-
Notifications
You must be signed in to change notification settings - Fork 14
Race-condition triggered when reading events takes too long #13
Comments
Well that's neat. I'd have to look into that code again, but I bet there was a reason I made it based on time. If not, or if that's ultimately not workable, I suspect that the solution would be to send some kind of barrier (or whatever) at the end of each polling cycle prune events that are x cycles old. I think this might cause an issue if there were multiple independent sources of data that were sending to the data aggregation layer. If you can attach a fine containing a 2000+ resource output of You can also increase the polling time, |
As far as I understand the code (which may be wrong...), it seems that prune() was supposed to remove entries that were not updated within the last 3 poll cycles - based on the poll interval time. I changed pruning so that it is triggered once after each poll cycle by the collector (a different collector, e.g. an event-based one, may still instead trigger it once per received event), and the timestamp for pruning is based on the earliest event timestamp from the current poll cycle. It also makes parsing noticeably faster, because it makes the prune operation per poll-cycle approximately an O(n) operation instead of the previous O(e * n), where e is the number of events received and n is up to the number of resources (depending on whether or not, and when it prunes entries, and how many of them, it's complicated...). It's not perfect, but it seems that it fixes most issues and does not add any new breakage, so I guess it might just be good enough for now. |
Problem description
When reading events from drbdsetup takes too long, e.g. when there are lots of resources and therefore lots of event lines to parse, then the ResourceCollection.prune() method in pkg/update/update.go will remove resources that have just been added in the same events collection cycle, because the time stamp is already too old.
The result is that resources are either never displayed, or, depending on the exact timing, the display may flap from showing the resources to not showing them, or possibly showing only some resources.
Steps to reproduce
Run with lots of resources configured (seen with 2000 resources on our pina/colada test systems). The problem does not seem to affect resources that are not started - those are always listed.
Possible solution
It may make sense to do a single prune() run before starting a new events poll cycle, so that no matter how long each cycle takes, it would always show at least the resources that were seen in the last cycle.
The text was updated successfully, but these errors were encountered: