Tracking CO2 Emissions

[x-tabdeveloping]

Since we are trying to put emphasis on benchmark efficiency and would like to be transparent about the compute and environmental impact of our benchmark, it might be a good idea to track CO2 emissions for the runs we are going to conduct.
This is also something reviewers might give us comments about, as ACL is usually quite sensitive to ethical considerations.

I think adding emissions tracking with CodeCarbon and adding total emissions per task per model to the results file could be a great idea. This would also give us an idea about which models are most and least polluting.

What do you think? (@isaac-chung, @KennethEnevoldsen, @imenelydiaker, @orionw, @Muennighoff )

[imenelydiaker]

I love this idea! Thanks for bringing this up 🙂
Some reflexions and questions here:

1. I don't expect us to run more than 5-10 models for the paper. How should we select them?
2. Emissions depend on the models we run, bigger the model, bigger the carbon footprint as more hardware is needed. It also depends on the task, e.g. retrieval and reranking tasks are the most consuming in terms of hardware as very large corpuses are embedded.
3. This could be a section in the paper: "Ethical considerations", we could report the carbon footprint of our evaluations for the paper. Btw, making the Clustering task faster helps reducing the carbon footprint.
4. Long term view, indeed it would be nice to have an estimation of the CO2 emission per task, it could be added to the results file along with mean_eval_time and scores.

[x-tabdeveloping]

Yeah I think estimating how much we managed to reduce CO2 emissions with some of the changes in the clustering tasks could also be a great point.

[x-tabdeveloping]

Regarding selecting the models:
I think we should probably choose popular open source models that are diverse enough in size and performance.
How many models and how are we planning to run? Do you have any info or insights on this @Muennighoff ?

[KennethEnevoldsen]

We can add it as a context manager around our tasks. Then each task can get a carbon emission similar to evaluation time.

[KennethEnevoldsen]

Task emission is then sum across task runs and the model emission sum across model.

[imenelydiaker]

@KennethEnevoldsen a suggested implementation here: #712

[isaac-chung]

At the model running stage, instead of reporting the emissions per task, would we want to present the net decrease in emissions per task / overall as well? That might require running tasks before the speedups with emissions enabled.

[mrshu]

@imenelydiaker the branch at main...mrshu:mteb:mrshu/port-carbon-emissions-estimation now has results for

• average_word_embeddings_komninos
• sentence-t5-xxl
• intfloat/multilingual-e5-large

Would you find that enough comparison-wise? What do you think would be the best way and/or format of reporting carbon footprint based on the data we have?

[imenelydiaker]

Hey @mrshu , thank you for this! Yes it should be enough for the comparison.

For the respresentaiton, we will probably sum values accross tasks for each model. And do the same with the lite version. Finally, we'll have a barplot that should look like this:

Although, we may want to include other modalities such as the performance or the time it took to run for better insights. Welcoming suggestions 🙂

[mrshu]

Ok, that's certainly doable.

I will need to run the MTEB-lite too for that to be doable. Would you happen to have any thoughts on what would be the best way of doing so now?

Thanks!

[KennethEnevoldsen]

Running MTEB-lite, seems to require #836, #835 (and #837). However, we can start running it on all tasks except for classification and retrieval

[mrshu]

Thanks a bunch @KennethEnevoldsen. I just wanted to quickly circle back to ask whether there might be a better way of running the MTEB-lite now.

Thanks!