Keep track of historical load shedding #466
Comments
|
What do you think of the following for the Historical Data Format (following a similar pattern to where historical: It is a boolean field which will be |
|
I'm actually wondering if there are any disadvantages to keeping the format identical, so future changes look the same as historical changes: historical_changes:
- stage: 4
start: 2023-08-31T14:00:00
finish: 2023-09-02T05:00:00
source: https://twitter.com/Eskom_SA/status/1697210092179935262
exclude: coct
- stage: 2
start: 2023-09-02T05:00:00
finish: 2023-09-02T16:00:00
source: https://twitter.com/Eskom_SA/status/1697210092179935262
exclude: coctVery happy to hear feedback/opinions on this, but my reasoning is:
Here's a link to the
|
|
Agreed. I don't see a disadvantage in keeping the format the same. To explain my thought process for dealing with two files that have overlapping times or conflicting stages like this: and a newer file: where the differences are stage 3 in the first entry moving to stage 5 for a portion of the overlapping time and a change from stage 5 to 6 for a whole time frame. Then the output in the historical data would be where the rules for these changes can be summarised as "new entries replace older entries for overlapping times". Essentially, we delete the incorrect entry from the file completely... |
|
Yes that looks correct to me. Although attempting to read these is making me remember why I tried to make a schedule visualiser a while back (it's trickier than it might seem at first glance). If I get a chance later on today, I'll write up some test cases (probably formatted as a multi-document yaml file) so that we can get the computer verifying these things for us. I'll write out some high-level test examples below: It'll be useful to have a little custom syntax:
With the above, we can define some mini-test examples like: If there are conflicts in the future, the newer file should take precedence: If there are conflicts in the past, the newer file should still take If the If an old file says "stage 6 for the rest of time" but a newer file updates to The above example also shows that there must be an option to specify "unknown Finally, here's one big example, just to stress test things a bit I'll try write these up as YAML files tonight, but this should give you a good |
|
I will have a look at this and get back to you but an issue (maybe something I missed?) is the different formatting of the manually_specified.yaml file. I can get you the commit hash if necessary. Easy enough to skip the files that are misbehaving and mark a period between two successful reads as unknown. For example: as compared to: |
|
@beyarkay have you had any time to generate some test yaml files? (I am also working on some). I have started some rudimentary test cases for a data aggregation file I wrote here. I have generated the historical data using this file but I am afraid it will (may) be riddled with errors until some proper testing is done. |
|
Hey, sorry for the delay. Yes you're correct, the misbehaving file should be omitted (the one formatted like: ---
- stage: 3
start: 2023-08-27T16:00:00
finsh: 2023-08-28T05:00:00
source: https://twitter.com/Eskom_SA/status/1695790083796828468
exclude: coct
- stage: 1
start: 2023-08-28T05:00:00
...I'm not sure what happened there). The correctly formatted file should have two keys: Busy working on the test files now, should have them uploaded to your PR in a bit. |
Keeping track of historical loadshedding is technically feasible at the moment, but it isn't easy to accomplish.
Basically all the information is in the git log for the file
manually_specified.yaml, but extracting and compiling it would be a pain.Probably the easiest way to make historical loadshedding data available would be to have a CI/CD script that runs every time the calendars get built. This script should calculate the historical loadshedding (either by updating the previously calculated data or by recalculating everything from scratch) and emit a file containing that information.
For parsing, it would be easiest if that file were formatted in the same way as
manually_specified.yaml:Keeping the format the same would mean the main codebase is equally able to calculate historical loadshedding and future loadshedding. However, it shouldn't be too much work to parse some different format, if that format provided some benefits.
Note that YAML is a superset of JSON, so the below snippet is valid YAML, while requiring fewer characters:
Keeping the format the same is not a hard requirement, but alternatives should be properly motivated.
Here's a high level checklist:
eskom-calendar-devrepo. This is a private mirror ofeskom-calendar, used to test CI/CD things. You can also set up your own, but getting the private GitHub keys setup (which allow GH actions to run faster) can be a pain.calendars/directory, as that's the only directory that gets uploaded to GitHub releases.manually_specified.yamland asserting that the updated changes get properly integrated.If the above are done, then all should be good! @beyarkay will check things over and merge.
The text was updated successfully, but these errors were encountered: