An API specification for exposing the environmental footprint for server-side software applications (SSAs)
This repository contains (1) the methodology for calculating the environmental footprint (including carbon emissions) of server-side applications and (2) the API specification on how to expose the measured footprint as an API within IT infrastructure software. The specification is developed and governed by the Digital Carbon Footprint Steering Group (SG) of the SDIA.
Read more about our approach on our blog.
The repository include methodology, specifications, use cases and requirements, best practices and guidelines, primers and notes. The SG's Work Items are linked from the SG homepage.
This repository contains of two parts:
- An methodology for measuring & assembling the environmental footprint
- A REST API specification to expose the footprint within IT infrastructure (e.g. within a container or virtualization platform)
When a server-side application is running, it is consuming resources - such as CPU time, memory and storage capacity. These resources are produced by physical IT infrastructure - physical servers sitting inside data centers or server rooms. The resources are often managed by virtualization or container orchestration platforms, or both. Each virtual resource unit is produced by a physical server, which comes with an embedded (e.g. from manufacturing or transport) and operational footprint (e.g. from the electricity needed to produce it). The server itself, also requires physical infrastructure, such as a building, electricity and cooling, in order to be able to produce resources.
Therefore, each resource (CPU time, memory, storage) that a server-side application is consuming, comes with a measurable environmental & social costs which this project aims to expose to the application through an API within the IT infrastructure.
The challenge now is to bring together a virtual machine, or more bleeding-edge, a Kubernetes pod, with the actual power consumption of the server. By design, one is not supposed to access the physical server from within a pod, or a virtual machine. It’s the abstraction provided by the system.
Therefore we have chosen (for now) to build an architecture around the Observer pattern, whereby each part of the system reports its relevant metrics to an Agent (the observer), which, using the collected information, in turn provides each part with a the information on the actual power consumption.
The observer (The Environmental Footprint Data Agent - EDA) thus records three types of time-series through observation:
- A time-series of power consumption (kW per second)
- A time-series of resource utilization (CPU, memory, storage & network capacity) of a pod or virtual machine
- A time-series of the schedule on which virtual machines and pods where running (e.g. recording when they started and stopped)
Combining these three time series with a solver that attributes the different resource types to the power consumption (e.g. 60% to CPU, 20% to memory, 10% to storage, 10% to the network) then leads us to be able to determine how much power a pod or virtual machine was consuming, based on it’s resource use, while it was running. Even better: It allows us to measure the opposite as well - how much power was wasted on idling/non-use of available resources.
An implementation of the EDA is currently developed by Helio.
- The solver is described further in this document
- Our approach & fallback to power consumption measurement is here
The API specification in this repository defines the APIs necessary within the EDA Agent:
- Environmental Footprint Query API - to query the environmental footprint of a VM or a Pod
Of course, given the virtualized nature of today’s IT infrastructure, the EDA itself can be running within the IT infrastructure that is being monitored itself. Meaning it can be deployed within the same Kubernetes cluster that it observes.
This also means that the EDA is being observed itself - by itself and can determine its power consumption - which should be considered overhead and to work towards minimizing this overhead.
The role of the EDA can also be assumed by existing systems, such as the virtualization layer (e.g. OpenStack or VMWare) or it can be assumed by the physical infrastructure (e.g. the Data Center Infrastructure Management System - DCIM). As long as the physical server, the pods or virtual machines are able to reach the agent, it’s actual location is not important.
This specification focusses on assessing the environmental footprint of both physical IT infrastructure (servers, racks, etc.) and the supporting infrastructure (cooling, power conversion, backup power generation and storage, etc.) both from an operational and embedded perspective. It is aimed creating a realistic and true assessment of the carbon footprint of server-side applications.
In the first version, we have decided to focus on the server itself, while using constants/pre-defined values for most of the infrastructure. Therefore we focus on measuring two values only at the moment:
- The physical power consumption of the server
- The embedded carbon of the server from transport & manufacturing
The open data hub, operated as a public data repository by the SDIA is used by the EDA to enrich the collected information from the physical infrastructure. As an example, the ODH contains a database of server hardware and its embedded emissions. It also provides real-time APIs for the CO2-concentration of electricity for different locations.
- Boavitza API for embodied carbon of server hardware
In order to ensure that the carbon footprints are calculated using the same methodology across different environments, we are working on an open standard which can be continously improved as we learn more about the footprint of software. The standard is developed by the members of the SG as part of a monthly meeting. Anyone can contribute, please see participation below.
For the time being, we have decided to focus on the environmental footprint, specifically energy-use and embodied carbon of servers.
As the precision of data available to convert electrical power into carbon emissions (e.g. based on values reported by the grid) is very low, we find it more useful to simply measure the power consumption itself, instead of using carbon emissions as a potentially misleading proxy.
All substantive contributors must be members of the SDIA SG. It’s easy to join the SG if you’d like to contribute.
Anyone can open issues and contribute to this repository.