Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync Release with Main #1068

Merged
merged 2 commits into from
Nov 1, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
chore(release): v0.7.2 ✨
narekhovhannisyan committed Nov 1, 2024
commit 3e49888ff390eb3ccea00aeff4f4ff7e031510bc
3 changes: 0 additions & 3 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -4,11 +4,8 @@
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.


### A description of the changes proposed in the Pull Request
2 changes: 1 addition & 1 deletion .github/workflows/release-commit.yml
Original file line number Diff line number Diff line change
@@ -94,4 +94,4 @@ jobs:
- name: Create pull request
run: gh pr create -B ${{ vars.RELEASE_BRANCH_NAME }} -H $PR_BRANCH_NAME --title "Release ${{github.event.release.tag_name}}" --body "${{github.event.release.body}}"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
2 changes: 1 addition & 1 deletion .github/workflows/release-publish.yml
Original file line number Diff line number Diff line change
@@ -70,4 +70,4 @@ jobs:
run: npm publish
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
3 changes: 0 additions & 3 deletions .gitmodules

This file was deleted.

150 changes: 68 additions & 82 deletions Refactor-migration-guide.md
Original file line number Diff line number Diff line change
@@ -105,45 +105,24 @@ There have also been some changes to the structure of manifest files. Some of th
method: SciEmbodied
```

- **Global config**
We have introduced the concept of global config to the plugins. This is truly global configuration data that should be kept constant regardless of where the plugin is invoked across the manifest file.
- **Config**
We have introduced the concept of config to the plugins. This is truly configuration data that should be kept constant regardless of where the plugin is invoked across the manifest file.

A good example is the `interpolation` method to use in the Teads curve plugin - this is not expected to vary from component to component and can therefore be defined in global config. The plugin code itself must expect the global config. Then, the config can be passed in the `Initialize` block, for example:
A good example is the `interpolation` method to use in the Teads curve plugin - this is not expected to vary from component to component and can therefore be defined in config. The plugin code itself must expect the config. Then, the config can be passed in the `Initialize` block, for example:

```yaml
initialize:
plugins:
'time-sync':
method: TimeSync
path: 'builtin'
global-config:
config:
start-time: '2023-12-12T00:00:00.000Z'
end-time: '2023-12-12T00:01:00.000Z'
interval: 5
allow-padding: true
```

- **Node level config**

We have also introduced the concept of node-level config. This is designed for pluin configuration that might vary between components in the tree. For example, for each child in the tree you might wish to use the `regroup` feature to group the outputs according to a different set of keys.

```yaml
tree:
children:
child-1:
pipeline:
compute:
- teads-curve
- sci-e
- sci-embodied
- sci-o
- time-sync
- sci
regroup:
- region
- cloud/instance-type
```

- **Defaults**

We have also introduced the concept of `defaults`. This is a section in each component's definition that can be used to provide fallbacks for missing input data. For example, perhaps you have a value arriving from an external API that should be present in every observation in your inputs array, but for soem reason the API fails to deliver a value for some timestamps. In this case, IF would fallback to the value provided for that metric in the `defaults` section of the manifest for that component.
@@ -178,15 +157,15 @@ There have also been some changes to the structure of manifest files. Some of th

Technically time-sync is not a new feature as it was present in IF before the refactor, but there are some tweaks to how the plugin is configured that are worth explaining here. Time sync snaps all input arrays across an entire graph to a common time grid.

This means you have to define a global start time, end time and interval to use everywhere. There is also a boolean to toggle whether you should allow the time sync model to pad the start and end of your time series with zeroes. You should default to `true` unless you have a specific reason not to. In the refactored IF we expect this information to be provided in global config, as follows:
This means you have to define a start time, end time and interval to use everywhere. There is also a boolean to toggle whether you should allow the time sync model to pad the start and end of your time series with zeroes. You should default to `true` unless you have a specific reason not to. In the refactored IF we expect this information to be provided in config, as follows:

```yaml
initialize:
plugins:
'time-sync':
method: TimeSync
path: 'builtin'
global-config:
config:
start-time: '2023-12-12T00:00:00.000Z'
end-time: '2023-12-12T00:01:00.000Z'
interval: 5
@@ -220,68 +199,75 @@ Details tbc...

## Plugins

The plugins themselves require some changes to keep them compatible with the refactored IF.
Plugins require some modifications to remain compatible with the refactored IF interface.

Instead of the old class-based model, plugins are now functions. They conform to the following interface:
Each plugin follows the `PluginFactory` interface, which is a higher-order function that accepts a `params` object of type `PluginFactoryParams`. This function returns another function (the inner function), which handles the plugin’s `config`, `parametersMetadata`, and `mapping`.

```ts
export type PluginInterface = {
execute: (
inputs: PluginParams[],
config?: Record<string, any>
) => PluginParams[];
metadata: {
kind: string;
};
[key: string]: any;
};
export const PluginFactory =
(params: PluginFactoryParams) =>
(
config: ConfigParams = {},
parametersMetadata: PluginParametersMetadata,
mapping: MappingParams
) => ({
metadata: {
inputs: {...params.metadata.inputs, ...parametersMetadata?.inputs},
outputs: parametersMetadata?.outputs || params.metadata.outputs,
},
execute: async (inputs: PluginParams[]) => {
// Generic plugin functionality goes here
// E.g., mapping, arithmetic operations, validation
// Process inputs and mapping logic
});
})
```

The plugin still requires an execute function. This is where you implement the plugin logic.
Inner Function Parameters:

- `config`: This is of type `ConfigParams` and has a default value of an empty object ({}). This might hold configuration settings for the plugin.
- `parametersMetadata`: A `PluginParametersMetadata` object that describes the metadata for the plugin’s parameters.
- `mapping`: A `MappingParams` object, describing parameters are mapped.

Implementation Function:

Here's a minimal example for a plugin that sums some inputs defined in global config - see inline comments for some important notes:
The plugin requires an `implementation` function, where the actual plugin logic is defined.
Here’s a minimal example of a plugin that sums inputs as defined in the config. See the inline comments for further clarification.

```ts
// Here's the function definition - notice that global config is passed in here!
export const Sum = (globalConfig: SumConfig): PluginInterface => {
const inputParameters = globalConfig['input-parameters'] || [];
const outputParameter = globalConfig['output-parameter'];

// we also return metadata now too - you can add more or just use this default
const metadata = {
kind: 'execute',
};

/**
* Calculate the sum of the input metrics for each timestamp.
*/
const execute = async (inputs: PluginParams[]): Promise<PluginParams[]> => {
inputs.map(input => {
return calculateSum(input, inputParameters, outputParameter);
// Here's the function definition!
export const Sum = PluginFactory({
configValidation: z.object({
'input-parameters': z.array(z.string()),
'output-parameter': z.string().min(1),
}),
inputValidation: (input: PluginParams, config: ConfigParams) => {
return validate(validationSchema, inputData);
},
implementation: async (inputs: PluginParams[], config: ConfigParams) => {
const {
'input-parameters': inputParameters,
'output-parameter': outputParameter,
} = config;

return inputs.map(input => {
const calculatedResult = calculateSum(input, inputParameters);

return {
...input,
[outputParameter]: calculatedResult,
};
});
return inputs;
};

/**
* Calculates the sum of the energy components.
*/
const calculateSum = (
input: PluginParams,
inputParameters: string[],
outputParameter: string
) => {
input[outputParameter] = inputParameters.reduce(
(accumulator, metricToSum) => {
return accumulator + input[metricToSum];
},
0
);
};

// return the metadata and the execute function
return {
metadata,
execute,
};
};
},
allowArithmeticExpressions: [],
});

/**
* Calculates the sum of the energy components.
*/
const calculateSum = (input: PluginParams, inputParameters: string[]) =>
inputParameters.reduce(
(accumulator, metricToSum) => accumulator + input[metricToSum],
0
);
```
12 changes: 3 additions & 9 deletions github-processes.md
Original file line number Diff line number Diff line change
@@ -3,24 +3,18 @@

- [`if`](https://github.com/Green-Software-Foundation/if)
- source code for the IF
- [`if-core`](https://github.com/Green-Software-Foundation/if-core)
- helper types, interfaces and utilities for IF and plugin development
- [`if-plugins`](https://github.com/Green-Software-Foundation/if-plugins) **DEPRECATED**
- source code for standard library of plugins
- IF core team commit to maintaining these plugins
- [`if-unofficial-plugins`](https://github.com/Green-Software-Foundation/if-unofficial-plugins)
- [`if-unofficial-plugins`](https://github.com/Green-Software-Foundation/if-unofficial-plugins) **DEPRECATED**
- source code for plugins relying on third-party data/APIs
- intended to be deprecated and removed in mid-term future
- plugins in this repo should be handed over to relevant organizations to maintain
- [`if-plugin-template`](https://github.com/Green-Software-Foundation/if-plugin-template)
- template for new plugins
- intended for builders to bootstrap IF-compatible plugin development
- [`if-standards`](https://github.com/Green-Software-Foundation/if-standards)
- not currently used, but intended to be the home of params.ts
- will have a dedicated discussion board and governance to set IF standards
- [`if-exhaust plugins`](https://github.com/Green-Software-Foundation/if-exhaust-plugins)
- not currently used
- intended to become a separate repo just for exhaust plugins
- requires strict rules from if


## Branch names and purposes

71 changes: 0 additions & 71 deletions grafana/IF_GRAFANA_SETUP.md

This file was deleted.

370 changes: 0 additions & 370 deletions grafana/if_grafana_config.json

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
name: coefficient-demo
description: failure with ivalid `global-config.input-parameter`
description: failure with ivalid `config.input-parameter`
tags:
initialize:
plugins:
coefficient:
method: Coefficient
path: "builtin"
global-config:
config:
input-parameter: 4
coefficient: 3
output-parameter: "carbon-product"
@@ -16,8 +16,6 @@ tree:
pipeline:
compute:
- coefficient
config:
sum:
inputs:
- timestamp: 2023-08-06T00:00
duration: 3600
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
coefficient:
method: Coefficient
path: "builtin"
global-config:
config:
input-parameter: "carbon"
coefficient: 3
output-parameter:
@@ -16,8 +16,6 @@ tree:
pipeline:
compute:
- coefficient
config:
sum:
inputs:
- timestamp: 2023-08-06T00:00
duration: 3600
4 changes: 1 addition & 3 deletions manifests/examples/builtins/coefficient/success.yml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
coefficient:
method: Coefficient
path: "builtin"
global-config:
config:
input-parameter: "carbon"
coefficient: 3
output-parameter: "carbon-product"
@@ -16,8 +16,6 @@ tree:
pipeline:
compute:
- coefficient
config:
sum:
inputs:
- timestamp: 2023-08-06T00:00
duration: 3600
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
cloud-metadata:
path: builtin
method: CSVLookup
global-config:
config:
filepath: >-
https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
query:
@@ -18,7 +18,6 @@ tree:
pipeline:
compute:
- cloud-metadata
config:
inputs:
- timestamp: 2023-07-06T00:00 # [KEYWORD] [NO-SUBFIELDS] time when measurement occurred
cloud/vendor: aws
Original file line number Diff line number Diff line change
@@ -6,19 +6,18 @@ initialize:
cloud-metadata:
path: builtin
method: CSVLookup
global-config:
config:
filepath: >-
https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
query:
instance-class: cloud/instance-type
output: ["cpu-cores-utilized", "vcpus-allocated"]
instance-class: cloud/vendor
output: ["cpu-cores-utilized", "cloud/instance-type"]
tree:
children:
child:
pipeline:
compute:
- cloud-metadata
config:
inputs:
- timestamp: 2023-07-06T00:00 # [KEYWORD] [NO-SUBFIELDS] time when measurement occurred
cloud/vendor: gcp
Original file line number Diff line number Diff line change
@@ -6,19 +6,18 @@ initialize:
cloud-metadata:
path: builtin
method: CSVLookup
global-config:
config:
filepath: >-
https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
query:
instance-class: cloud/instance-type
instance-class: cloud/vendor
output: ["cpu-cores-utilized", "vcpus-allocated"]
tree:
children:
child:
pipeline:
compute:
- cloud-metadata
config:
inputs:
- timestamp: 2023-07-06T00:00 # [KEYWORD] [NO-SUBFIELDS] time when measurement occurred
#cloud/vendor: aws
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
cloud-metadata:
path: builtin
method: CSVLookup
global-config:
config:
filepath: >-
https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
query:
@@ -18,7 +18,6 @@ tree:
pipeline:
compute:
- cloud-metadata
config:
inputs:
- timestamp: 2023-07-06T00:00 # [KEYWORD] [NO-SUBFIELDS] time when measurement occurred
cloud/vendor: aws
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
cloud-metadata:
method: CSVLookup
path: "builtin"
global-config:
config:
filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/region-metadata.csv
query:
cloud-provider: "nonexistant"
Original file line number Diff line number Diff line change
@@ -6,12 +6,12 @@ initialize:
cloud-metadata:
method: CSVLookup
path: "builtin"
global-config:
config:
filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/region-metadata.csv
query:
cloud-provider: "cloud/provider"
cloud-region: "cloud/region"
output: "*"
output:
tree:
children:
child:
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
cloud-metadata:
method: CSVLookup
path: "builtin"
global-config:
config:
filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/region-metadata.csv
query:
cloud-provider: "cloud/provider"
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
cloud-metadata:
method: CSVLookup
path: "builtin"
global-config:
config:
filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/region-metadata.csv
query:
cloud-provider: "cloud/provider"
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
tdp-finder:
method: CSVLookup
path: "builtin"
global-config:
config:
filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/tdp-data-1.csv
query:
name: physical-processor
@@ -17,7 +17,6 @@ tree:
pipeline:
compute:
- tdp-finder
config:
inputs:
- timestamp: 2023-07-06T00:00
duration: 300
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
tdp-finder:
method: CSVLookup
path: "builtin"
global-config:
config:
filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/tdp-data-1.csv
query:
name: physical-processor
@@ -17,7 +17,6 @@ tree:
pipeline:
compute:
- tdp-finder
config:
inputs:
- timestamp: 2023-07-06T00:00
duration: 300
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
tdp-finder:
method: CSVLookup
path: "builtin"
global-config:
config:
filepath: https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/tdp-data-1.csv
query:
name: physical-processor
@@ -17,7 +17,6 @@ tree:
pipeline:
compute:
- tdp-finder
config:
inputs:
- timestamp: 2023-07-06T00:00
duration: 300

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
name: divide
description: failure when `global-config.denominator` is string
description: failure when `config.denominator` is string
tags:
initialize:
plugins:
cloud-metadata:
path: builtin
method: CSVLookup
global-config:
config:
filepath: >-
https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
query:
@@ -15,7 +15,7 @@ initialize:
divide:
method: Divide
path: "builtin"
global-config:
config:
numerator: vcpus-allocated
denominator: "vcpus"
output: cpu/number-cores
@@ -26,8 +26,6 @@ tree:
compute:
- cloud-metadata
- divide
config:
divide:
defaults:
cloud/vendor: aws
cloud/instance-type: m5n.large
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
cloud-metadata:
path: builtin
method: CSVLookup
global-config:
config:
filepath: >-
https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
query:
@@ -15,7 +15,7 @@ initialize:
divide:
method: Divide
path: "builtin"
global-config:
config:
#numerator: vcpus-allocated
denominator: 2
output: cpu/number-cores
@@ -26,8 +26,6 @@ tree:
compute:
- cloud-metadata
- divide
config:
divide:
defaults:
cloud/vendor: aws
cloud/instance-type: m5n.large
4 changes: 2 additions & 2 deletions manifests/examples/builtins/divide/success.yml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
cloud-metadata:
path: builtin
method: CSVLookup
global-config:
config:
filepath: >-
https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-aws-instances.csv
query:
@@ -15,7 +15,7 @@ initialize:
divide:
method: Divide
path: "builtin"
global-config:
config:
numerator: vcpus-allocated
denominator: 2
output: cpu/number-cores
2 changes: 1 addition & 1 deletion manifests/examples/builtins/exponent/success.yml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
exponent:
method: Exponent
path: "builtin"
global-config:
config:
input-parameter: "cpu/energy"
exponent: 2
output-parameter: "energy"
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
interpolation:
method: Interpolation
path: "builtin"
global-config:
config:
method: linear
x: [0, 10, 50, 100]
y: [0.12, 0.32, 0.75, 1.02]
@@ -21,4 +21,4 @@ tree:
inputs:
- timestamp: 2023-07-06T00:00
duration: 3600
cpu/utilization: 45
cpu/utilization: off
2 changes: 1 addition & 1 deletion manifests/examples/builtins/interpolation/success.yml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
interpolation:
method: Interpolation
path: "builtin"
global-config:
config:
method: linear
x: [0, 10, 50, 100]
y: [0.12, 0.32, 0.75, 1.02]
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
name: mock-observation-demo
description: failure with `global-config->generators->randint->cpu/utilization->min` is greater than `max`
description: failure with `config->generators->randint->cpu/utilization->min` is greater than `max`
tags:
initialize:
plugins:
mock-observations:
kind: plugin
method: MockObservations
path: "builtin"
global-config:
config:
timestamp-from: 2023-07-06T00:00
timestamp-to: 2023-07-06T00:10
duration: 60
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ initialize:
kind: plugin
method: MockObservations
path: "builtin"
global-config:
config:
timestamp-from: 2023-07-06T00:00
timestamp-to: 2023-07-06T00:10
duration: 60
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ initialize:
kind: plugin
method: MockObservations
path: "builtin"
global-config:
config:
#timestamp-from: 2023-07-06T00:00
timestamp-to: 2023-07-06T00:10
duration: 60
2 changes: 1 addition & 1 deletion manifests/examples/builtins/mock-observations/success.yml
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ initialize:
kind: plugin
method: MockObservations
path: "builtin"
global-config:
config:
timestamp-from: 2023-07-06T00:00
timestamp-to: 2023-07-06T00:10
duration: 60
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
multiply:
method: Multiply
path: "builtin"
global-config:
config:
input-parameters: ["cpu/energy", "network/energy"]
output-parameter: "energy-product"
tree:
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
multiply:
method: Multiply
path: "builtin"
global-config:
config:
input-parameters: ["cpu/energy", "network/energy"]
output-parameter: "energy-product"
tree:
2 changes: 1 addition & 1 deletion manifests/examples/builtins/multiply/success.yml
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ initialize:
multiply:
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu/energy", "network/energy"]
output-parameter: "energy-product"
tree:
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
regex:
method: Regex
path: "builtin"
global-config:
config:
parameter: physical-processor
match: ^(.*),
output: cpu/name
Original file line number Diff line number Diff line change
@@ -6,9 +6,9 @@ initialize:
regex:
method: Regex
path: "builtin"
global-config:
config:
parameter: physical-processor
match: ^
match: ^$
output: cpu/name
tree:
children:
2 changes: 1 addition & 1 deletion manifests/examples/builtins/regex/success.yml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
regex:
method: Regex
path: "builtin"
global-config:
config:
parameter: physical-processor
match: ^(.*),
output: cpu/name

This file was deleted.

This file was deleted.

27 changes: 27 additions & 0 deletions manifests/examples/builtins/sci-embodied/scenario-1.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: embodied-carbon demo
description:
tags:
aggregation:
metrics:
- embodied-carbon
type: "both"
initialize:
plugins:
embodied-carbon:
method: SciEmbodied
path: builtin
config:
output-parameter: "embodied-carbon"
tree:
children:
child:
pipeline:
compute:
- embodied-carbon
inputs:
- timestamp: 2023-08-06T00:00
duration: 3600
hdd: 2
- timestamp: 2023-08-06T10:00
duration: 3600
hdd: 2
37 changes: 37 additions & 0 deletions manifests/examples/builtins/sci-embodied/scenario-2.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: embodied-carbon demo
description:
tags:
initialize:
plugins:
embodied-carbon:
method: SciEmbodied
path: builtin
config:
baseline-vcpus: 1
baseline-memory: 16
lifespan: 157680000
baseline-emissions: 2000000
vcpu-emissions-constant: 100000
memory-emissions-constant: 1172
ssd-emissions-constant: 50000
hdd-emissions-constant: 100000
gpu-emissions-constant: 150000
output-parameter: "embodied-carbon"
tree:
children:
child:
pipeline:
compute:
- embodied-carbon
defaults:
vCPUs: 4
memory: 32
ssd: 1
hdd: 1
gpu: 1
total-vcpus: 16
inputs:
- timestamp: 2023-08-06T00:00
duration: 3600
- timestamp: 2023-08-06T10:00
duration: 3600
13 changes: 13 additions & 0 deletions manifests/examples/builtins/sci-embodied/success.yml
Original file line number Diff line number Diff line change
@@ -3,6 +3,15 @@ description: successful path
tags:
initialize:
plugins:
"csv-lookup":
path: builtin
method: CSVLookup
config:
filepath: >-
https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-azure-instances.csv
query:
instance-class: cloud/instance-type
output: ["cpu-cores-utilized", "vcpus-allocated"]
"sci-embodied": # a model that calculates m from te, tir, el, rr and rtor
method: SciEmbodied
path: "builtin"
@@ -11,6 +20,7 @@ tree:
child:
pipeline:
compute:
- csv-lookup
- sci-embodied # duration & config -> embodied
defaults:
device/emissions-embodied: 1533.120 # gCO2eq
@@ -21,3 +31,6 @@ tree:
inputs:
- timestamp: 2023-07-06T00:00
duration: 3600
cloud/vendor: intel
cloud/instance-type: Standard_A1_v2
cpu/utilization: 10
Original file line number Diff line number Diff line change
@@ -7,17 +7,16 @@ initialize:
kind: plugin
method: Sci
path: "builtin"
# global-config:
# config:
# functional-unit: 1 minute
tree:
children:
child:
pipeline:
compute:
- sci
config:
sci:
functional-unit: 999 # factor to convert per time to per f.unit
defaults:
functional-unit: 999 # factor to convert per time to per f.unit
inputs:
- timestamp: 2023-07-06T00:00
duration: 3600
Original file line number Diff line number Diff line change
@@ -8,7 +8,7 @@ initialize:
kind: plugin
method: Sci
path: "builtin"
global-config:
config:
functional-unit: requests
tree:
children:
2 changes: 1 addition & 1 deletion manifests/examples/builtins/sci/success.yml
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ initialize:
kind: plugin
method: Sci
path: "builtin"
global-config:
config:
functional-unit: requests
tree:
children:
4 changes: 2 additions & 2 deletions manifests/examples/builtins/shell/failure-invalid-command.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
name: shell
description: falure with `global-config.command` being number instead od string
description: falure with `config.command` being number instead od string
tags:
initialize:
plugins:
shell:
method: Shell
path: "builtin"
global-config:
config:
command: 1000
tree:
children:
4 changes: 2 additions & 2 deletions manifests/examples/builtins/shell/success.yml
Original file line number Diff line number Diff line change
@@ -5,8 +5,8 @@ initialize:
plugins:
shell:
method: Shell
path: 'builtin'
global-config:
path: "builtin"
config:
command: python3 /usr/local/bin/sampler
tree:
children:
2 changes: 1 addition & 1 deletion manifests/examples/builtins/subtract/success.yml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
subtract:
method: Subtract
path: "builtin"
global-config:
config:
input-parameters: ["cpu/energy", "network/energy"]
output-parameter: "energy/diff"
tree:
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
name: sum
description: failure with `inputs[0]` misses one of `global-config.input-parameters`
description: failure with `inputs[0]` misses one of `config.input-parameters`
tags:
initialize:
plugins:
sum:
method: Sum
path: "builtin"
global-config:
config:
input-parameters: ["cpu/energy", "network/energy"]
output-parameter: "energy"
tree:
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
name: sum
description: missing `output-parameter` in global-config
description: missing `output-parameter` in config
tags:
initialize:
plugins:
sum:
method: Sum
path: "builtin"
global-config:
config:
input-parameters: ["cpu/energy", "network/energy"]
# output-parameter: "energy"
tree:
2 changes: 1 addition & 1 deletion manifests/examples/builtins/sum/success.yml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
sum:
method: Sum
path: "builtin"
global-config:
config:
input-parameters: ["cpu/energy", "network/energy"]
output-parameter: "energy"
tree:
6 changes: 3 additions & 3 deletions manifests/examples/builtins/time-converter/success.yaml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
time-converter:
method: TimeConverter
path: builtin
global-config:
config:
input-parameter: "energy-per-year"
original-time-unit: "year"
new-time-unit: "duration"
@@ -15,8 +15,8 @@ tree:
children:
child:
pipeline:
- time-converter
config:
compute:
- time-converter
defaults:
energy-per-year: 10000
inputs:
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
name: time-sync
description: failure with `global-config.start-time` being later than `global-config.end-time`
description: failure with `config.start-time` being later than `config.end-time`
tags:
initialize:
output:
- yaml
plugins:
'time-sync':
"time-sync":
method: TimeSync
path: "builtin"
global-config:
start-time: '2023-12-12T00:01:00.000Z'
end-time: '2023-12-12T00:00:00.000Z'
config:
start-time: "2023-12-12T00:01:00.000Z"
end-time: "2023-12-12T00:00:00.000Z"
interval: 5
allow-padding: true
tree:
@@ -20,15 +18,15 @@ tree:
compute:
- time-sync
inputs:
- timestamp: '2023-12-12T00:00:00.000Z'
- timestamp: "2023-12-12T00:00:00.000Z"
duration: 1
energy-cpu: 0.001
- timestamp: '2023-12-12T00:00:01.000Z'
- timestamp: "2023-12-12T00:00:01.000Z"
duration: 5
energy-cpu: 0.001
- timestamp: '2023-12-12T00:00:06.000Z'
- timestamp: "2023-12-12T00:00:06.000Z"
duration: 7
energy-cpu: 0.001
- timestamp: '2023-12-12T00:00:13.000Z'
- timestamp: "2023-12-12T00:00:13.000Z"
duration: 30
energy-cpu: 0.001
energy-cpu: 0.001
Original file line number Diff line number Diff line change
@@ -1,18 +1,13 @@
name: time-sync
description: missing global config
description: missing config
tags:
initialize:
output:
- yaml
plugins:
'time-sync':
method: TimeSync
path: "builtin"
global-config:
# start-time: '2023-12-12T00:00:00.000Z'
# end-time: '2023-12-12T00:01:00.000Z'
# interval: 5
# allow-padding: true
path: 'builtin'
tree:
children:
child:
@@ -31,4 +26,4 @@ tree:
energy-cpu: 0.001
- timestamp: '2023-12-12T00:00:13.000Z'
duration: 30
energy-cpu: 0.001
energy-cpu: 0.001
4 changes: 2 additions & 2 deletions manifests/examples/builtins/time-sync/success.yml
Original file line number Diff line number Diff line change
@@ -7,8 +7,8 @@ initialize:
plugins:
'time-sync':
method: TimeSync
path: "builtin"
global-config:
path: 'builtin'
config:
start-time: '2023-12-12T00:00:00.000Z'
end-time: '2023-12-12T00:01:00.000Z'
interval: 5
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: regroup
description:
initialize:
plugins:
plugins: {}
tree:
children:
my-app:
2 changes: 1 addition & 1 deletion manifests/examples/features/regroup/success.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: regroup
description: successful path
initialize:
plugins:
plugins: {}
tree:
children:
my-app:
16 changes: 8 additions & 8 deletions manifests/examples/pipelines/generics.yml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
"interpolate":
method: Interpolation
path: "builtin"
global-config:
config:
method: linear
x: [0, 10, 50, 100]
y: [0.12, 0.32, 0.75, 1.02]
@@ -15,47 +15,47 @@ initialize:
"cpu-factor-to-wattage":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-factor", "cpu/thermal-design-power"]
output-parameter: "cpu-wattage"
"wattage-times-duration":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-wattage", "duration"]
output-parameter: "cpu-wattage-times-duration"
"wattage-to-energy-kwh":
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-wattage-times-duration
denominator: 3600000
output: cpu-energy-raw
"calculate-vcpu-ratio":
method: Divide
path: "builtin"
global-config:
config:
numerator: vcpus-total
denominator: vcpus-allocated
output: vcpu-ratio
"correct-cpu-energy-for-vcpu-ratio":
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-energy-raw
denominator: vcpu-ratio
output: cpu-energy-kwh
"coefficient":
path: "builtin"
method: Coefficient
global-config:
config:
input-parameter: cpu-energy-kwh
coefficient: 2
output-parameter: energy-doubled
"multiply":
path: "builtin"
method: Multiply
global-config:
config:
input-parameters: ["cpu/utilization", "duration"]
output-parameter: "cpu-times-duration"
tree:
4 changes: 2 additions & 2 deletions manifests/examples/pipelines/instance-metadata.yml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
cloud-instance-metadata:
path: builtin
method: CSVLookup
global-config:
config:
filepath: >-
https://raw.githubusercontent.com/Green-Software-Foundation/if-data/main/cloud-metdata-azure-instances.csv
query:
@@ -15,7 +15,7 @@ initialize:
extract-processor-name:
path: builtin
method: Regex
global-config:
config:
parameter: cpu-model-name
match: /^([^,])+/g
output: cpu/name
108 changes: 75 additions & 33 deletions manifests/examples/pipelines/nesting.yml
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@ initialize:
"interpolate":
method: Interpolation
path: "builtin"
global-config:
config:
method: linear
x: [0, 10, 50, 100]
y: [0.12, 0.32, 0.75, 1.02]
@@ -24,43 +24,53 @@ initialize:
cpu/utilization:
unit: percentage
description: refers to CPU utilization.
aggregation-method: avg
aggregation-method:
time: avg
component: sum
outputs:
cpu-factor:
unit: kWh
description: result of interpolate
aggregation-method: avg
aggregation-method:
time: avg
component: avg
"cpu-factor-to-wattage":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-factor", "cpu/thermal-design-power"]
output-parameter: "cpu-wattage"
parameter-metadata:
inputs:
cpu-factor:
unit: kWh
description: result of interpolate
aggregation-method: avg
aggregation-method:
time: avg
component: avg
cpu/thermal-design-power:
unit: kWh
description: thermal design power for a processor
aggregation-method: avg
aggregation-method:
time: avg
component: avg
outputs:
cpu-wattage:
unit: kWh
description: the energy used by the CPU
aggregation-method: sum
aggregation-method:
time: sum
component: sum
"wattage-times-duration":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-wattage", "duration"]
output-parameter: "cpu-wattage-times-duration"
"wattage-to-energy-kwh":
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-wattage-times-duration
denominator: 3600000
output: cpu-energy-raw
@@ -69,16 +79,20 @@ initialize:
cpu-wattage-times-duration:
unit: kWh
description: CPU wattage multiplied by duration
aggregation-method: sum
aggregation-method:
time: sum
component: sum
outputs:
cpu-energy-raw:
unit: kWh
description: Raw energy used by CPU in kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
"calculate-vcpu-ratio":
method: Divide
path: "builtin"
global-config:
config:
numerator: vcpus-total
denominator: vcpus-allocated
output: vcpu-ratio
@@ -87,11 +101,13 @@ initialize:
vcpu-ratio:
unit: none
description: Ratio of vCPUs
aggregation-method: none
aggregation-method:
time: copy
component: copy
"correct-cpu-energy-for-vcpu-ratio":
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-energy-raw
denominator: vcpu-ratio
output: cpu-energy-kwh
@@ -101,62 +117,76 @@ initialize:
"operational-carbon":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-energy-kwh", "grid/carbon-intensity"]
output-parameter: "carbon-operational"
parameter-metadata:
inputs:
cpu-energy-kwh:
unit: kWh
description: Corrected CPU energy in kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
grid/carbon-intensity:
unit: gCO2eq/kWh
description: Carbon intensity for the grid
aggregation-method: avg
aggregation-method:
time: avg
component: avg
outputs:
carbon-operational:
unit: gCO2eq
description: Operational carbon footprint
aggregation-method: sum
aggregation-method:
time: sum
component: sum
sci:
path: "builtin"
method: Sci
global-config:
config:
functional-unit: "requests"
parameter-metadata:
inputs:
requests:
unit: none
description: expressed the final SCI value
aggregation-method: sum
aggregation-method:
time: sum
component: sum
"sum-carbon":
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- carbon-operational
- carbon-embodied
- embodied-carbon
output-parameter: carbon
parameter-metadata:
inputs:
carbon-operational:
description: Operational carbon footprint
unit: gCO2eq
aggregation-method: sum
carbon-embodied:
aggregation-method:
time: sum
component: sum
embodied-carbon:
description: Embodied carbon footprint
unit: gCO2eq
aggregation-method: sum
aggregation-method:
time: sum
component: sum
outputs:
carbon:
description: Total carbon footprint
unit: gCO2eq
aggregation-method: sum
aggregation-method:
time: sum
component: sum
time-sync:
method: TimeSync
path: "builtin"
global-config:
config:
start-time: "2023-12-12T00:00:00.000Z"
end-time: "2023-12-12T00:01:00.000Z"
interval: 5
@@ -166,27 +196,39 @@ initialize:
timestamp:
unit: RFC3339
description: refers to the time of occurrence of the input
aggregation-method: none
aggregation-method:
time: none
component: none
duration:
unit: seconds
description: refers to the duration of the input
aggregation-method: sum
aggregation-method:
time: sum
component: sum
cloud/instance-type:
unit: none
description: type of Cloud Instance name used in the cloud provider APIs
aggregation-method: none
aggregation-method:
time: copy
component: copy
cloud/region:
unit: none
description: region cloud instance
aggregation-method: none
aggregation-method:
time: copy
component: copy
time-reserved:
unit: seconds
description: time reserved for a component
aggregation-method: avg
aggregation-method:
time: avg
component: avg
network/energy:
description: "Energy consumed by the Network of the component"
unit: "kWh"
aggregation-method: "sum"
aggregation-method:
time: sum
component: sum
tree:
children:
child-0:
288 changes: 166 additions & 122 deletions manifests/examples/pipelines/outputs-if-diff/pipeline-with-aggregate.yaml

Large diffs are not rendered by default.

200 changes: 130 additions & 70 deletions manifests/examples/pipelines/outputs-if-diff/pipeline-with-mocks.yaml

Large diffs are not rendered by default.

30 changes: 15 additions & 15 deletions manifests/examples/pipelines/pipeline-teads-sci.yml
Original file line number Diff line number Diff line change
@@ -5,43 +5,43 @@ initialize:
plugins:
"interpolate":
method: Interpolation
path: 'builtin'
global-config:
path: "builtin"
config:
method: linear
x: [0, 10, 50, 100]
y: [0.12, 0.32, 0.75, 1.02]
input-parameter: 'cpu/utilization'
output-parameter: 'cpu-factor'
input-parameter: "cpu/utilization"
output-parameter: "cpu-factor"
"cpu-factor-to-wattage":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-factor", "cpu/thermal-design-power"]
output-parameter: "cpu-wattage"
"wattage-times-duration":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-wattage", "duration"]
output-parameter: "cpu-wattage-times-duration"
"wattage-to-energy-kwh":
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-wattage-times-duration
denominator: 3600000
output: cpu-energy-raw
"calculate-vcpu-ratio":
method: Divide
path: "builtin"
global-config:
config:
numerator: vcpus-total
denominator: vcpus-allocated
output: vcpu-ratio
"correct-cpu-energy-for-vcpu-ratio":
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-energy-raw
denominator: vcpu-ratio
output: cpu-energy-kwh
@@ -51,26 +51,26 @@ initialize:
"operational-carbon":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-energy-kwh", "grid/carbon-intensity"]
output-parameter: "carbon-operational"
"sci":
path: "builtin"
method: Sci
global-config:
config:
functional-unit: "component"
"sum-carbon":
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- carbon-operational
- carbon-embodied
- embodied-carbon
output-parameter: carbon
"time-sync":
method: TimeSync
path: "builtin"
global-config:
config:
start-time: "2023-12-12T00:00:00.000Z"
end-time: "2023-12-12T00:01:00.000Z"
interval: 5
@@ -123,4 +123,4 @@ tree:
cloud/instance-type: A1
cloud/region: uk-west
cpu/utilization: 15
network/energy: 0.000001
network/energy: 0.000001
84 changes: 57 additions & 27 deletions manifests/examples/pipelines/pipeline-with-aggregate.yml
Original file line number Diff line number Diff line change
@@ -10,7 +10,7 @@ initialize:
"interpolate":
method: Interpolation
path: "builtin"
global-config:
config:
method: linear
x: [0, 10, 50, 100]
y: [0.12, 0.32, 0.75, 1.02]
@@ -21,43 +21,53 @@ initialize:
cpu/utilization:
unit: percentage
description: refers to CPU utilization.
aggregation-method: avg
aggregation-method:
time: avg
component: avg
outputs:
cpu-factor:
unit: kWh
description: result of interpolate
aggregation-method: avg
aggregation-method:
time: avg
component: avg
"cpu-factor-to-wattage":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-factor", "cpu/thermal-design-power"]
output-parameter: "cpu-wattage"
parameter-metadata:
inputs:
cpu-factor:
unit: kWh
description: result of interpolate
aggregation-method: avg
aggregation-method:
time: avg
component: avg
cpu/thermal-design-power:
unit: kWh
description: thermal design power for a processor
aggregation-method: avg
aggregation-method:
time: avg
component: avg
outputs:
cpu-wattage:
unit: kWh
description: the energy used by the CPU
aggregation-method: sum
aggregation-method:
time: sum
component: sum
"wattage-times-duration":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-wattage", "duration"]
output-parameter: "cpu-wattage-times-duration"
"wattage-to-energy-kwh":
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-wattage-times-duration
denominator: 3600000
output: cpu-energy-raw
@@ -66,16 +76,20 @@ initialize:
cpu-wattage-times-duration:
unit: kWh
description: CPU wattage multiplied by duration
aggregation-method: sum
aggregation-method:
time: sum
component: sum
outputs:
cpu-energy-raw:
unit: kWh
description: Raw energy used by CPU in kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
"calculate-vcpu-ratio":
method: Divide
path: "builtin"
global-config:
config:
numerator: vcpus-total
denominator: vcpus-allocated
output: vcpu-ratio
@@ -84,20 +98,26 @@ initialize:
vcpus-total:
unit: count
description: total number of vcpus available on a particular resource
aggregation-method: none
aggregation-method:
time: copy
component: copy
vcpus-allocated:
unit: count
description: number of vcpus allocated to particular resource
aggregation-method: none
aggregation-method:
time: copy
component: copy
outputs:
vcpu-ratio:
unit: none
description: Ratio of vCPUs
aggregation-method: none
aggregation-method:
time: copy
component: copy
"correct-cpu-energy-for-vcpu-ratio":
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-energy-raw
denominator: vcpu-ratio
output: cpu-energy-kwh
@@ -107,53 +127,63 @@ initialize:
"operational-carbon":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-energy-kwh", "grid/carbon-intensity"]
output-parameter: "carbon-operational"
parameter-metadata:
inputs:
cpu-energy-kwh:
unit: kWh
description: Corrected CPU energy in kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
grid/carbon-intensity:
unit: gCO2eq/kWh
description: Carbon intensity for the grid
aggregation-method: avg
aggregation-method:
time: avg
component: avg
outputs:
carbon-operational:
unit: gCO2eq
description: Operational carbon footprint
aggregation-method: sum
aggregation-method:
time: sum
component: sum
"sci":
path: "builtin"
method: Sci
global-config:
config:
functional-unit: requests # factor to convert per time to per f.unit
parameter-metadata:
inputs:
requests:
unit: none
description: expressed the final SCI value
aggregation-method: sum
aggregation-method:
time: sum
component: sum
"sum-carbon":
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- carbon-operational
- carbon-embodied
- embodied-carbon
output-parameter: carbon
parameter-metadata:
outputs:
carbon:
unit: gCO2eq
description: product of carbon
aggregation-method: sum
aggregation-method:
time: sum
component: sum
"time-sync":
method: TimeSync
path: "builtin"
global-config:
config:
start-time: "2023-12-12T00:00:00.000Z"
end-time: "2023-12-12T00:01:00.000Z"
interval: 5
@@ -259,4 +289,4 @@ tree:
cpu/utilization: 33
cloud/instance-type: A1
cloud/region: uk-west
requests: 180
requests: 180
140 changes: 99 additions & 41 deletions manifests/examples/pipelines/pipeline-with-mocks.yml
Original file line number Diff line number Diff line change
@@ -11,7 +11,7 @@ initialize:
kind: plugin
method: MockObservations
path: "builtin"
global-config:
config:
timestamp-from: "2023-12-12T00:00:00.000Z"
timestamp-to: "2023-12-12T00:00:13.000Z"
duration: 30
@@ -29,23 +29,31 @@ initialize:
timestamp:
description: refers to the time of occurrence of the input
unit: RFC3339
aggregation-method: none
aggregation-method:
time: none
component: none
duration:
description: refers to the duration of the input
unit: seconds
aggregation-method: sum
aggregation-method:
time: sum
component: sum
cloud/instance-type:
description: type of Cloud Instance name used in the cloud provider APIs
unit: none
aggregation-method: none
aggregation-method:
time: none
component: none
cloud/region:
description: region cloud instance
unit: none
aggregation-method: none
aggregation-method:
time: none
component: none
interpolate:
method: Interpolation
path: "builtin"
global-config:
config:
method: linear
x: [0, 10, 50, 100]
y: [0.12, 0.32, 0.75, 1.02]
@@ -56,58 +64,74 @@ initialize:
cpu/utilization:
description: refers to CPU utilization.
unit: percentage
aggregation-method: avg
aggregation-method:
time: avg
component: avg
outputs:
cpu-factor:
description: result of interpolate
unit: kWh
aggregation-method: avg
aggregation-method:
time: avg
component: avg
cpu-factor-to-wattage:
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-factor", "cpu/thermal-design-power"]
output-parameter: "cpu-wattage"
parameter-metadata:
inputs:
cpu-factor:
description: result of interpolate
unit: kWh
aggregation-method: avg
aggregation-method:
time: avg
component: avg
cpu/thermal-design-power:
description: thermal design power for a processor
unit: kWh
aggregation-method: avg
aggregation-method:
time: avg
component: avg
outputs:
cpu-wattage:
description: the energy used by the CPU
unit: kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
wattage-times-duration:
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-wattage", "duration"]
output-parameter: "cpu-wattage-times-duration"
parameter-metadata:
inputs:
cpu-wattage:
description: Energy used by the CPU
unit: kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
duration:
description: Duration of the observation
unit: seconds
aggregation-method: sum
aggregation-method:
time: sum
component: sum
outputs:
cpu-wattage-times-duration:
description: CPU wattage multiplied by duration
unit: kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
wattage-to-energy-kwh:
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-wattage-times-duration
denominator: 3600000
output: cpu-energy-raw
@@ -116,16 +140,20 @@ initialize:
cpu-wattage-times-duration:
description: CPU wattage multiplied by duration
unit: kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
outputs:
cpu-energy-raw:
description: Raw energy used by CPU in kWh
unit: kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
calculate-vcpu-ratio:
method: Divide
path: "builtin"
global-config:
config:
numerator: vcpus-total
denominator: vcpus-allocated
output: vcpu-ratio
@@ -134,20 +162,26 @@ initialize:
vcpus-total:
description: total number of vcpus available on a particular resource
unit: count
aggregation-method: none
aggregation-method:
time: none
component: none
vcpus-allocated:
description: number of vcpus allocated to particular resource
unit: count
aggregation-method: none
aggregation-method:
time: none
component: none
outputs:
vcpu-ratio:
description: Ratio of vCPUs
unit: none
aggregation-method: none
aggregation-method:
time: none
component: none
correct-cpu-energy-for-vcpu-ratio:
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-energy-raw
denominator: vcpu-ratio
output: cpu-energy-kwh
@@ -156,83 +190,105 @@ initialize:
cpu-energy-raw:
description: Raw energy used by CPU in kWh
unit: kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
vcpu-ratio:
description: Ratio of vCPUs
unit: none
aggregation-method: none
aggregation-method:
time: none
component: none
outputs:
cpu-energy-kwh:
description: Corrected CPU energy in kWh
unit: kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
sci-embodied:
path: "builtin"
method: SciEmbodied
operational-carbon:
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-energy-kwh", "grid/carbon-intensity"]
output-parameter: "carbon-operational"
parameter-metadata:
inputs:
cpu-energy-kwh:
description: Corrected CPU energy in kWh
unit: kWh
aggregation-method: sum
aggregation-method:
time: sum
component: sum
grid/carbon-intensity:
description: Carbon intensity for the grid
unit: gCO2eq/kWh
aggregation-method: avg
aggregation-method:
time: avg
component: avg
outputs:
carbon-operational:
description: Operational carbon footprint
unit: gCO2eq
aggregation-method: sum
aggregation-method:
time: sum
component: sum
sum-carbon:
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- carbon-operational
- carbon-embodied
- embodied-carbon
output-parameter: carbon
parameter-metadata:
inputs:
carbon-operational:
description: Operational carbon footprint
unit: gCO2eq
aggregation-method: sum
carbon-embodied:
aggregation-method:
time: sum
component: sum
embodied-carbon:
description: Embodied carbon footprint
unit: gCO2eq
aggregation-method: sum
aggregation-method:
time: sum
component: sum
outputs:
carbon:
description: Total carbon footprint
unit: gCO2eq
aggregation-method: sum
aggregation-method:
time: sum
component: sum
sci:
path: "builtin"
method: Sci
global-config:
config:
functional-unit: "requests"
parameter-metadata:
inputs:
requests:
description: expressed the final SCI value
unit: none
aggregation-method: sum
aggregation-method:
time: sum
component: sum
outputs:
sci:
description: Scientific Carbon Intensity
unit: none
aggregation-method: none
aggregation-method:
time: none
component: none
time-sync:
method: TimeSync
path: "builtin"
global-config:
config:
start-time: "2023-12-12T00:00:00.000Z"
end-time: "2023-12-12T00:01:00.000Z"
interval: 5
@@ -266,6 +322,7 @@ tree:
device/expected-lifespan: 94608000 # 3 years in seconds
vcpus-total: 8
vcpus-allocated: 1
requests: 50
inputs:
child-2:
pipeline:
@@ -294,4 +351,5 @@ tree:
device/expected-lifespan: 94608000 # 3 years in seconds
vcpus-total: 8
vcpus-allocated: 1
requests: 50
inputs:
34 changes: 0 additions & 34 deletions manifests/examples/pipelines/scenario-1.yml

This file was deleted.

52 changes: 0 additions & 52 deletions manifests/examples/pipelines/scenario-2.yml

This file was deleted.

2 changes: 1 addition & 1 deletion manifests/examples/pipelines/scenario-3.yml
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@ initialize:
"sum":
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- cpu/energy
- network/energy
10 changes: 5 additions & 5 deletions manifests/examples/pipelines/scenario-4.yml
Original file line number Diff line number Diff line change
@@ -1,35 +1,35 @@
name: demo
description:
description:
tags:
initialize:
plugins:
"sum":
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- cpu/energy
- network/energy
output-parameter: energy-sum
"coefficient":
path: "builtin"
method: Coefficient
global-config:
config:
input-parameter: energy
coefficient: 2
output-parameter: energy-doubled
"multiply":
path: "builtin"
method: Multiply
global-config:
config:
input-parameters: ["cpu/utilization", "duration"]
output-parameter: "cpu-times-duration"
tree:
children:
child-1:
pipeline:
observe:
compute:
compute:
- sum
- coefficient
- multiply
6 changes: 3 additions & 3 deletions manifests/examples/pipelines/scenario-5.yml
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
name: demo
description:
description:
tags:
initialize:
plugins:
mock-observations:
kind: plugin
method: MockObservations
path: "builtin"
global-config:
config:
timestamp-from: 2023-07-06T00:00
timestamp-to: 2023-07-06T00:01
duration: 60
@@ -28,7 +28,7 @@ initialize:
sum:
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- cpu/utilization
- memory/utilization
24 changes: 12 additions & 12 deletions manifests/examples/pipelines/sci.yml
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ initialize:
interpolate:
method: Interpolation
path: "builtin"
global-config:
config:
method: linear
x: [0, 10, 50, 100]
y: [0.12, 0.32, 0.75, 1.02]
@@ -15,40 +15,40 @@ initialize:
cpu-factor-to-wattage:
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-factor", "cpu/thermal-design-power"]
output-parameter: "cpu-wattage"
wattage-times-duration:
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-wattage", "duration"]
output-parameter: "cpu-wattage-times-duration"
wattage-to-energy-kwh:
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-wattage-times-duration
denominator: 3600000
output: cpu-energy-raw
calculate-vcpu-ratio:
method: Divide
path: "builtin"
global-config:
config:
numerator: vcpus-total
denominator: vcpus-allocated
output: vcpu-ratio
correct-cpu-energy-for-vcpu-ratio:
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-energy-raw
denominator: vcpu-ratio
output: cpu/energy
sum-energy-components:
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- cpu/energy
- network/energy
@@ -59,21 +59,21 @@ initialize:
"operational-carbon":
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["energy", "grid/carbon-intensity"]
output-parameter: "carbon-operational"
"sum-carbon":
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- carbon-operational
- carbon-embodied
- embodied-carbon
output-parameter: carbon
"sci":
path: "builtin"
method: Sci
global-config:
config:
functional-unit: "component"
tree:
children:
@@ -126,4 +126,4 @@ tree:
cloud/instance-type: A1
cloud/region: uk-west
cpu/utilization: 15
network/energy: 0.000001
network/energy: 0.000001
20 changes: 10 additions & 10 deletions manifests/examples/pipelines/teads-curve.yml
Original file line number Diff line number Diff line change
@@ -1,47 +1,47 @@
name: carbon-intensity plugin demo
description:
description:
tags:
initialize:
plugins:
interpolate:
method: Interpolation
path: 'builtin'
global-config:
path: "builtin"
config:
method: linear
x: [0, 10, 50, 100]
y: [0.12, 0.32, 0.75, 1.02]
input-parameter: 'cpu/utilization'
output-parameter: 'cpu-factor'
input-parameter: "cpu/utilization"
output-parameter: "cpu-factor"
cpu-factor-to-wattage:
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-factor", "thermal-design-power"]
output-parameter: "cpu-wattage"
wattage-times-duration:
method: Multiply
path: builtin
global-config:
config:
input-parameters: ["cpu-wattage", "duration"]
output-parameter: "cpu-wattage-times-duration"
wattage-to-energy-kwh:
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-wattage-times-duration
denominator: 3600000
output: cpu-energy-raw
calculate-vcpu-ratio:
method: Divide
path: "builtin"
global-config:
config:
numerator: vcpus-total
denominator: vcpus-allocated
output: vcpu-ratio
correct-cpu-energy-for-vcpu-ratio:
method: Divide
path: "builtin"
global-config:
config:
numerator: cpu-energy-raw
denominator: vcpu-ratio
output: cpu-energy-kwh
46 changes: 23 additions & 23 deletions manifests/examples/pipelines/zeros.yml
Original file line number Diff line number Diff line change
@@ -6,101 +6,101 @@ initialize:
"sum-zero-and-one":
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- some-value
- zero-value
output-parameter: one-plus-zero
"sum-zero-and-zero":
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- zero-value
- zero-value
output-parameter: zero-plus-zero
"subtract-one-and-zero":
path: "builtin"
method: Subtract
global-config:
config:
input-parameters:
- some-value
- zero-value
output-parameter: one-minus-zero
"subtract-zero-and-zero":
path: "builtin"
method: Sum
global-config:
config:
input-parameters:
- zero-value
- zero-value
output-parameter: zero-minus-zero
"subtract-zero-and-one":
path: "builtin"
method: Subtract
global-config:
config:
input-parameters:
- zero-value
- some-value
output-parameter: zero-minus-one
"coefficient-one-times-zero":
path: "builtin"
method: Coefficient
global-config:
config:
input-parameter: zero-value
coefficient: 1
output-parameter: zero-times-one-coefficient
"coefficient-zero-times-one":
path: "builtin"
method: Coefficient
global-config:
config:
input-parameter: some-value
coefficient: 0
output-parameter: one-times-zero-coefficient
"coefficient-zero-times-zero":
path: "builtin"
method: Coefficient
global-config:
config:
input-parameter: zero-value
coefficient: 0
output-parameter: zero-times-zero-coefficient
"multiply-one-times-zero":
path: "builtin"
method: Multiply
global-config:
config:
input-parameters: ["some-value", "zero-value"]
output-parameter: "one-times-zero"
"multiply-zero-times-one":
path: "builtin"
method: Multiply
global-config:
config:
input-parameters: ["zero-value", "zero-value"]
output-parameter: "zero-times-one"
exponent-one-to-zero:
method: Exponent
path: 'builtin'
global-config:
input-parameter: 'some-value'
path: "builtin"
config:
input-parameter: "some-value"
exponent: 0
output-parameter: 'one-raised-to-zero-power'
output-parameter: "one-raised-to-zero-power"
exponent-zero-to-zero:
method: Exponent
path: 'builtin'
global-config:
input-parameter: 'zero-value'
path: "builtin"
config:
input-parameter: "zero-value"
exponent: 0
output-parameter: 'zero-raised-to-zero-power'
output-parameter: "zero-raised-to-zero-power"
exponent-zero-to-one:
method: Exponent
path: 'builtin'
global-config:
input-parameter: 'zero-value'
path: "builtin"
config:
input-parameter: "zero-value"
exponent: 1
output-parameter: 'zero-raised-to-first-power'
output-parameter: "zero-raised-to-first-power"
"sci":
path: "builtin"
method: Sci
global-config:
config:
functional-unit: "zero-value"
tree:
children:
80 changes: 40 additions & 40 deletions manifests/outputs/bugs/aggregation-error-wrong-metric.yaml
Original file line number Diff line number Diff line change
@@ -12,14 +12,14 @@ initialize:
interpolate:
method: Interpolation
path: builtin
global-config:
config:
method: linear
x:
- 0
- 10
- 50
- 100
"y":
'y':
- 0.12
- 0.32
- 0.75
@@ -29,37 +29,37 @@ initialize:
cpu-factor-to-wattage:
method: Multiply
path: builtin
global-config:
config:
input-parameters:
- cpu-factor
- cpu/thermal-design-power
output-parameter: cpu-wattage
wattage-times-duration:
method: Multiply
path: builtin
global-config:
config:
input-parameters:
- cpu-wattage
- duration
output-parameter: cpu-wattage-times-duration
wattage-to-energy-kwh:
method: Divide
path: builtin
global-config:
config:
numerator: cpu-wattage-times-duration
denominator: 3600000
output: cpu-energy-raw
calculate-vcpu-ratio:
method: Divide
path: builtin
global-config:
config:
numerator: vcpus-total
denominator: vcpus-allocated
output: vcpu-ratio
correct-cpu-energy-for-vcpu-ratio:
method: Divide
path: builtin
global-config:
config:
numerator: cpu-energy-raw
denominator: vcpu-ratio
output: cpu-energy-kwh
@@ -69,48 +69,47 @@ initialize:
operational-carbon:
method: Multiply
path: builtin
global-config:
config:
input-parameters:
- cpu-energy-kwh
- grid/carbon-intensity
output-parameter: carbon
sci:
path: builtin
method: Sci
global-config:
config:
functional-unit: requests
time-sync:
method: TimeSync
path: builtin
global-config:
start-time: "2023-12-12T00:00:00.000Z"
end-time: "2023-12-12T00:01:00.000Z"
config:
start-time: '2023-12-12T00:00:00.000Z'
end-time: '2023-12-12T00:01:00.000Z'
interval: 5
allow-padding: true
execution:
status: fail
command: >-
/Users/mariamkhalatova/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
/Users/mariamkhalatova/Projects/UK/if/src/index.ts -m
manifests/outputs/bugs/aggregation-error-wrong-metric.yml -o
manifests/outputs/bugs/aggregation-error-wrong-metric
/Users/manushak/.npm/_npx/1bf7c3c15bf47d04/node_modules/.bin/ts-node
/Users/manushak/Documents/Projects/Green-Software/if/src/if-run/index.ts -m
manifests/outputs/bugs/aggregation-error-wrong-metric.yaml
environment:
if-version: 0.4.0
if-version: 0.6.0
os: macOS
os-version: "13.2"
node-version: 18.14.2
date-time: 2024-07-01T19:25:34.759Z (UTC)
os-version: 14.6.1
node-version: 18.20.4
date-time: 2024-10-04T08:38:25.343Z (UTC)
dependencies:
- "@babel/core@7.22.10"
- "@babel/preset-typescript@7.23.3"
- "@commitlint/cli@18.6.0"
- "@commitlint/config-conventional@18.6.0"
- "@grnsft/if-core@0.0.10"
- "@jest/globals@29.7.0"
- "@types/jest@29.5.8"
- "@types/js-yaml@4.0.9"
- "@types/luxon@3.4.2"
- "@types/node@20.9.0"
- '@babel/core@7.22.10'
- '@babel/preset-typescript@7.23.3'
- '@commitlint/cli@18.6.0'
- '@commitlint/config-conventional@18.6.0'
- '@grnsft/if-core@0.0.25'
- '@jest/globals@29.7.0'
- '@types/jest@29.5.8'
- '@types/js-yaml@4.0.9'
- '@types/luxon@3.4.2'
- '@types/node@20.9.0'
- axios-mock-adapter@1.22.0
- axios@1.7.2
- cross-env@7.0.3
@@ -130,9 +129,10 @@ execution:
- typescript-cubic-spline@1.0.1
- typescript@5.2.2
- winston@3.11.0
- zod@3.22.4
- zod@3.23.8
error: >-
MissingInputDataError: `functional-unit` value is missing from input data or it is not a positive integer
MissingAggregationParamError: Aggregation metric dummy-param is not found in
inputs[0].
tree:
children:
child-1:
@@ -160,25 +160,25 @@ tree:
vcpus-allocated: 1
vcpus-total: 8
inputs:
- timestamp: "2023-12-12T00:00:00.000Z"
- timestamp: '2023-12-12T00:00:00.000Z'
cloud/instance-type: A1
cloud/region: uk-west
duration: 1
cpu/utilization: 10
requests: 100
- timestamp: "2023-12-12T00:00:01.000Z"
- timestamp: '2023-12-12T00:00:01.000Z'
duration: 5
cpu/utilization: 20
cloud/instance-type: A1
cloud/region: uk-west
requests: 100
- timestamp: "2023-12-12T00:00:06.000Z"
- timestamp: '2023-12-12T00:00:06.000Z'
duration: 7
cpu/utilization: 15
cloud/instance-type: A1
cloud/region: uk-west
requests: 100
- timestamp: "2023-12-12T00:00:13.000Z"
- timestamp: '2023-12-12T00:00:13.000Z'
duration: 30
cloud/instance-type: A1
cloud/region: uk-west
@@ -209,25 +209,25 @@ tree:
vcpus-allocated: 1
vcpus-total: 8
inputs:
- timestamp: "2023-12-12T00:00:00.000Z"
- timestamp: '2023-12-12T00:00:00.000Z'
duration: 1
cpu/utilization: 30
cloud/instance-type: A1
cloud/region: uk-west
requests: 100
- timestamp: "2023-12-12T00:00:01.000Z"
- timestamp: '2023-12-12T00:00:01.000Z'
duration: 5
cpu/utilization: 28
cloud/instance-type: A1
cloud/region: uk-west
requests: 100
- timestamp: "2023-12-12T00:00:06.000Z"
- timestamp: '2023-12-12T00:00:06.000Z'
duration: 7
cpu/utilization: 40
cloud/instance-type: A1
cloud/region: uk-west
requests: 100
- timestamp: "2023-12-12T00:00:13.000Z"
- timestamp: '2023-12-12T00:00:13.000Z'
duration: 30
cpu/utilization: 33
cloud/instance-type: A1
2 changes: 1 addition & 1 deletion manifests/outputs/bugs/input-error-missing-duration.yaml
Original file line number Diff line number Diff line change
@@ -8,7 +8,7 @@ initialize:
interpolate:
method: Interpolation
path: builtin
global-config:
config:
method: linear
x:
- 0
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ initialize:
kind: plugin
method: MockObservations
path: builtin
global-config:
config:
timestamp-from: 2023-07-06T00:00
timestamp-to: 2023-07-06T00:10
duration: 0
@@ -69,7 +69,7 @@ execution:
- typescript@5.2.2
- winston@3.11.0
- zod@3.22.4
error: "InputValidationError: \"duration\" parameter is number must be greater than 0. Error code: too_small."
error: 'InputValidationError: "duration" parameter is number must be greater than 0. Error code: too_small.'
tree:
children:
child:
2 changes: 1 addition & 1 deletion manifests/outputs/bugs/pipeline-error-naming-mismatch.yaml
Original file line number Diff line number Diff line change
@@ -8,7 +8,7 @@ initialize:
interpolate:
method: Interpolation
path: builtin
global-config:
config:
method: linear
x:
- 0
Original file line number Diff line number Diff line change
@@ -8,7 +8,7 @@ initialize:
interpolate:
method: Interpolation
path: builtin
global-config:
config:
method: linear
x:
- 0
12 changes: 6 additions & 6 deletions manifests/outputs/bugs/pipeline-ordering-error.yaml
Original file line number Diff line number Diff line change
@@ -8,7 +8,7 @@ initialize:
interpolate:
method: Interpolation
path: builtin
global-config:
config:
method: linear
x:
- 0
@@ -25,37 +25,37 @@ initialize:
cpu-factor-to-wattage:
method: Multiply
path: builtin
global-config:
config:
input-parameters:
- cpu-factor
- cpu/thermal-design-power
output-parameter: cpu-wattage
wattage-times-duration:
method: Multiply
path: builtin
global-config:
config:
input-parameters:
- cpu-wattage
- duration
output-parameter: cpu-wattage-times-duration
wattage-to-energy-kwh:
method: Divide
path: builtin
global-config:
config:
numerator: cpu-wattage-times-duration
denominator: 3600000
output: cpu-energy-raw
calculate-vcpu-ratio:
method: Divide
path: builtin
global-config:
config:
numerator: vcpus-total
denominator: vcpus-allocated
output: vcpu-ratio
correct-cpu-energy-for-vcpu-ratio:
method: Divide
path: builtin
global-config:
config:
numerator: cpu-energy-raw
denominator: vcpu-ratio
output: cpu-energy-kwh
72 changes: 0 additions & 72 deletions manifests/outputs/bugs/sci-embodied-missing-resources-total.yaml

This file was deleted.

Loading