Proposed specification for a Carbon Model interface.

[jawache]

Background

• Carbon models are models that given a certain set of inputs return carbon emissions data (and energy etc...)
• There are many models out there, some are APIs, some are just collections of CSV files with instructions on how to use them, some are buried in papers.
• Different models need different inputs and might provide different outputs.
• We need a common way to call all these models and a common way to interpret and process the outputs.

Scope

• This task is NOT going to create new models, this project will create a common interface to existing models so they can be more easily consumed, swapped out, compared, upgraded.
• Through creating a common interface and hopefully eventually library of carbon models that can be easily consumed by developers we probably will identify gaps, models that are needed and don't exist.
• By just being able to swap out the model and keeping the rest of the code the same we'll be able to more easily see the difference between models and when one is more suitable than another.
• As organizations are increasingly going to create their own private, NDA models, as long as they adhere to the interface we describe below they will bne able to use the models in any application/framework that supports CQL model interfaces.

Requirements

• Carbon models have different inputs, some use billing data, some use utilization - the interface needs to be flexible to accommodate all the existing types of inputs that we know of and all the future types of inputs.
• Carbon models should be muti-language, there should not be the Python ones the Node.js ones, one core code base (rust/c/c++/c-linkable) with bindings to a set of commonly supported languages and not all of those languages will be OO.
• Using a carbon module should be very quick, from running the install script to consuming the model should take no longer than a few mins (installing a docker image and launching a local service is not an option here).
• Every carbon model implementation should have exactly the same function signature, so swapping them out might just require one line of code change and every use of that model is the same across your application.
• Carbon models should have a verbose mode, we should be able to get a detailed dump of HOW this model has calculated a number for verification/debugging purposes.
• Carbon models might need different inputs, they will need to provide very detailed instructions to the developer for what data they need to function. E.g. you might swap out one model for another similar one but the new one requires one more config param, the error should be clear and instruct the developer on what's missing.
• They will probably need to be stateful, e.g. load a set of coefficients from a file/network on the creation and then in future calls just use the in-mem coeffs.

Proposed Interface

sequenceDiagram
    User->>Model: create(StaticParams: Dict): Model
    Model->>Model: Instantiate

    User->>Model: snapshot(Telemetry: Dict): EmissionsData
    Model->>User: Returns single emissions snapshot

    User->>Model: historical([Telemetry: Dict]): [EmissionsData]
    Model->>User: Returns array of emissions data

    User->>Model: realtime(TelemetrySteam: Stream): EmissionsStream
    Model->>User: Returns array of emissions data

    User->>Model: dispose()
    Model->>Model: Clean up

Loading

Create

create(StaticParams: Dict): Model

• Pass in static params as a dictionary/hash/object, there will be such variety of potential inputs only a dictionary will guarentee to work.
• Return a stateful model object which we can use to call functions with/on.

Returns errors

• Missing Params: Some static param is not avaialble.
• Incorrent Params: A param has been input but the value is weid, incorrect to just doesn't work with this model.
• External Error: Some other problem with perhaps an external service, netowrking, disk etc...

Snapshot

snapshot(Telemetry: Dict): EmissionsData

• Pass in a single snapshot of some telemetry data and returns a single snapshot of emissions data.

Returns errors

• Missing Telemetry: Some telemetry it needs to function isn't passed in.
• Incorrent Telemetry: A param has been input but the value is weid, incorrect to just doesn't work with this model.
• External Error: Some other problem with perhaps an external service, netowrking, disk etc...

Historical

historical([Telemetry: Dict]): [EmissionsData]

• Pass in a an array of historcal telemetry (e.g. every min for last hour) and returns an array of emissions data for the same frequency of inputs.

Returns errors

• Missing Telemetry: Some telemetry it needs to function isn't passed in.
• Incorrent Telemetry: A param has been input but the value is weid, incorrect to just doesn't work with this model.
• External Error: Some other problem with perhaps an external service, netowrking, disk etc...

Real-Time

realtime(TelemetrySteam: Stream): EmissionsStream

• Pass in a stream which contains future telemetry, as that telemetry is fed it in passes the results back in an emissions stream.
• This usecase would be for monitoring applications.

Returns errors

• Missing Telemetry: Some telemetry it needs to function isn't passed in.
• Incorrent Telemetry: A param has been input but the value is weid, incorrect to just doesn't work with this model.
• External Error: Some other problem with perhaps an external service, netowrking, disk etc...

Dispose

dispose()

• Let the model know it can remove data, clean things up.

Returns errors

• External Error: Some other problem with perhaps an external service, netowrking, disk etc...

[jawache]

@srini1978 @Oleg-Zhymolokhov proposed this initial spec just for an interface to a carbon model.

[jawache]

NOTE: @Oleg-Zhymolokhov I can't tag naveen so I think he's not been added to the repo as an lead, can you please make sure to add him (settings > team/roles etc...)

[Oleg-Zhymolokhov]

@navveenb Tagging you here

[srini1978]

Created a set of sample code files developing to the above specification https://github.com/srini1978/carbonQL. The code is not yet uploaded to this repository as it needs a lot of polishing.

@jawache @navveenb @Oleg-Zhymolokhov

[jawache]

The content in this issue is now all inside the PR #46, once that PR is merged we can close this issue

[jawache]

Closing