Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make live point generation deterministic #105

Merged
merged 17 commits into from
Jan 9, 2024

Conversation

AdamOrmondroyd
Copy link
Contributor

@AdamOrmondroyd AdamOrmondroyd commented Oct 24, 2023

Currently, live point generation (and therefore the entire algorithm) is non-deterministic, even if the seed is fixed.

Old approach:

Root process waits as nprocs-1 other processes generate live points at random positions, calculate logL and pass to the root, and keep going until told to stop when the root has nprior samples.
The issue here is that the processes will run at different speeds, leading to the random number generations getting all mixed up.

New approach:

Root process generates nprocs random live points, and scatters these to all processes (including itself). Each process calculates logL and the results are gathered. The root then adds the points >log0, and repeats the above until it has nprior (and sends arrays full of -1 to tell the other workers to stop).
This way, only the root is generating random numbers, so they will appear in the same order for a given seed.

Possible issues:

Since the root waits for all workers to complete their calculation, if one of them gets stuck, the whole algorithm will get stuck.

Second pass:

A simpler approach is to just offload all the random number generation to the root node, and pass this along when requesting points from the workers. The key is that PolyChord waits for all active workers to finish generating prior samples before it moves on.

This avoids the issue from the first pass where all workers have to wait for a slow evaluation.

@AdamOrmondroyd AdamOrmondroyd marked this pull request as ready for review October 25, 2023 14:16
@AdamOrmondroyd
Copy link
Contributor Author

AdamOrmondroyd commented Nov 24, 2023

While this method will always generate the same set of live points, it will not put them in the same order every time. This is important as generateseed() will always choose the $n^\text{th}$ prior sample, which won't always be the same.

My plan is to record the order in which live points were requested, and sort them by this

@AdamOrmondroyd
Copy link
Contributor Author

Think this is now ready to review

Copy link
Member

@williamjameshandley williamjameshandley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of name changes -- otherwise this is a very clever bit of code. Well done @AdamOrmondroyd

src/polychord/mpi_utils.F90 Outdated Show resolved Hide resolved
src/polychord/mpi_utils.F90 Outdated Show resolved Hide resolved
src/polychord/mpi_utils.F90 Outdated Show resolved Hide resolved
@AdamOrmondroyd AdamOrmondroyd merged commit 8107fd2 into PolyChord:master Jan 9, 2024
7 checks passed
@AdamOrmondroyd AdamOrmondroyd deleted the mpistartup branch January 9, 2024 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants