-
Notifications
You must be signed in to change notification settings - Fork 0
/
TODO
140 lines (129 loc) · 6.85 KB
/
TODO
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
TODO
-----------------------------
* move filter_kruskal.rb into graph.rb
* also integrate k_means_pp.rb et al into graph.rb
* How do we decide whether to enlarge a generator or build a new one?
* there is currently no method to this madness
* Physarum polycephalum is the bacteria that can develop optimal transportation networks
* Add DB caching via memcache
* Plot a choropleth of the US as a background
* `Grid` only works with a ConnectedGraph. Is there a way to make it work with
a DisjointGraph?
* producing DG connected subgraphs is slow
* No longer have the algorithm to build new generators. Where do we build new sources if
there are already generators on the map?
* How do we decide which generators to grow?
* introduce non-load nodes to theoretical grid
* Should I change drakos to do A_gen * A^2 instead of A^3?
* we don't care about ALL walks/paths, only those walks/paths originating
from a generator
* but we DO call about all paths, since that is how electricity flows
* but there should be a bonus for simply adding a generator to a spot
* weird bug. checking percentiles 4..8 should include everything that's in
5..8, but that doesn't appear to be happening
* improve resilience
* increase connectivity of lowest-degree nodes
* hard
* reduce # of paths that go through node with highest sigma
* easier but still hard
* increase # of overall paths without affecting node with highest sigma
* now we're talking
* congestion reduction for scale-free graphs
* find long stretch of high-flow
* find a way to shorten it
* pick two nodes on either end
* remove the subgraph of top-sigma node + 3 steps outward
* connect subgraphs
* uhoh. Only 5% transmission loss
* https://www.eia.gov/totalenergy/data/flow-graphs/electricity.php
* need to find a new raison d'etre
* Maybe this is the end of the project
* gotta pivot to resilience
* memory profiling
* https://www.toptal.com/ruby/hunting-ruby-memory-issues
* accept range as CLI input (as opposed to only using 10x10 grid)
* duplicate edges appear in reduce_congestion
TASKS
-------------------------
* import "miso_north_tx_lines.geosjon"
DONE
----------------------------
* Fix k_means where it doesn't converge sometimes
* i cheated, though: i fixed it by using a different k-means implementation
* edges need to have transmission loss. there needs to be a penalty for longer transmission lines
* Easy way to get the total load of a node
* given node and path and *where it is on its journey*, calculate how
much load is passing through that part of the trip
* Instead of looking for high-flow edges to connect to, look for nodes that are
drawing power from far-away generators. the distances are calculated during the
flow calculations, and we just have to beat that number.
* When constructing an edge between two nodes, see if part of the edge can be
shortened by going through already-created edges that pass nearby it
* How do we add edges between nodes?
* How do we do it smartly?
* **BUILD A DB FOR TRACKING NODES AND TRANSMISSION LINES**
* at the very least, just for nodes, since those come from disparate resources
* but yeah, lines too
* make edge/node more specific to line/load
* in the DB, it'll be line/load, but when doing graph operations, it'll be node/edge
* Identifying the shortest cycle isn't working the way it should... but once we can
identify the shortest cycles, we can delete an edge from them and restore balance to
the graph. Only looking at cycles <= 4
* PARALLELIZE THE SHORTEST CYCLE ALGORITHM
* easy. parallelize the cycle from each node.
* Okay, we have all of the data we need. HIFLD already join lines at substations.
* and if you look at the plotting, you'll see that 99% of new england is already
connected!
* so how do we turn those lines into simplified edges?
* rebuild the line simplification algorithms
* Split out the grid.rb interface from interacting with the DB and
having all relevant classes be available
* map all loads to OSM points so we can ensure that they connect to the grid
* done for New England, at least. the data is so shitty that it doesn't even matter
* parallelize calculate flows
* each thread does its own subset of nodes and then the partial tallies are summed
* no, that wouldn't work, because they have interdependence on when a generator
can support more load
* maybe i can copy parallelization of the MST algorithm? does such a parallelization
exist?
* that wouldn't even work, because it's just fundamentally different and i don't think
MST can be parallelized
* fixed by rewriting it entirely
* reduce congestion:
* need to take the two biggest CGs that contain the sources, not simply the two biggest CGs
* flow detection
* instead of having a load be entirely powered along the path to the closest
generator, have only a *proportion* be powered by that generator
* a true real-world sim would have some fraction of the voltage travel along
ALL possible paths to each generator, but this is close enough
* get rid of the concept of reach
* estrada index is standard for graph resilience
* randic index? maybe? somehow?
* Randić index could be useful?
* https://www.sciencedirect.com/science/article/pii/S0972860017301408
* reduce_congestion - wow this bad boy uses a lot of RAM
* gonna have to parallelize #reduce_congestion as well
* verify #reduce_congestion works on NEW_ENGLAND
* well... it works on a simplified version
* **ADD PARALLELIZATION SERIALIZATION METHOD TO EASE PARALLELIZATION**
* for when dealing with the serialization of objects (mapping the copies
to their originals based on IDs/returning IDs instead of modifying
objects) across parallelization boundaries
* Grid#simplify seems to fundamentally change the graph when it simplifies them
* peak congestion is different
* thus can't trust any congestion reduction
* turns out it was because CG#path was doing # of edges instead of total path weight
* change ConnectedGraph#path_to to be about edge weight instead of # of edges
* add the ability to load a graph and data from a marshal file
* kinda done
* maybe try `jemalloc`
BUGGY RUNS
---------------------------------
* freq calculation seems... off
DATA
-------------------------------
https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::transmission-lines/explore?location=44.098241%2C-72.184173%2C8.55
https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::electric-planning-areas/explore?location=35.996286%2C-95.848305%2C5.17
https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::independent-system-operators-1/explore?location=36.658645%2C-95.679925%2C5.18
https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::ferc-regions/explore?location=42.550218%2C-122.156318%2C4.20
https://hifld-geoplatform.opendata.arcgis.com/datasets/geoplatform::power-plants-2/explore?location=44.159264%2C-120.288643%2C4.25