Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent enter_at from using enter_at #469

Merged
merged 1 commit into from
Mar 21, 2024

Conversation

frankmcsherry
Copy link
Member

Collection::enter_at used timely's enter_at trait and method, but .. it is bad. Its implementation is roughly enter().delay(), where delay() does us the favor of queueing all input data until the delayed-to timestamp has passed. In a nested timestamp context, this means until the outer timestamp has fully passed, because it is not until then that we might not see (time, 0) in the loop.

So that means that users of enter_at introduce a stalling operator that prevents updates from passing through linear operators (map, filter) and from landing in a consolidating merge batcher where they will be compactly maintained.

The PR just deletes the use of enter_at, replacing it with enter. This seems harmless, as the (_, time, _) field of the triple is still updated, and it is only the introduced capability that is not delayed, but .. nothing good will happen until the outer timestamp clears anyhow, and the update's time needs to be consulted by the receiving operator in any case.

The PR also updates the accumulate.rs example to use a nested scope and enter_at, which without the fix grows without bound, and with the fix stays at about 2MB or so.

@frankmcsherry frankmcsherry merged commit dfc0062 into TimelyDataflow:master Mar 21, 2024
7 checks passed
@frankmcsherry frankmcsherry deleted the unblock_enter_at branch March 21, 2024 23:14
This was referenced Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant