Excessive memory usage in rule build_line_rating

[koen-vg]

Checklist

• I am using the current master branch or the latest release. Please indicate.
• I am running on an up-to-date pypsa-eur environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

As pointed out on the mailing list, the build_line_rating rule seems to use tens of GBs of memory, and takes a really long time (I didn't get it to complete). Dynamic line rating is turned off by default, which I'm sure is why this wasn't caught before.

In the snakefile the rule is given 1GB per thread (with a default of 4 threads), but actual memory usage seems to be at least an order of magnitude about that.

When I was testing the rule out, memory usage seemed to be increasing gradually, suggesting some kind of memory leak leading to issues over time? I unfortunately haven't had time to look into it properly. The last major thing that happened to the line rating rule is the switch-up of when clustering happens in the workflow. Maybe it has something to do with that??