Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Termination detection fault with dtd #634

Open
abouteiller opened this issue Feb 7, 2024 · 2 comments
Open

Termination detection fault with dtd #634

abouteiller opened this issue Feb 7, 2024 · 2 comments
Assignees
@therault
Labels
bug Something isn't working
Milestone
v4.0

Comments

@abouteiller
Copy link
Contributor

Describe the bug

Seen only once on #321, need to see if it also happens on master

To Reproduce

29302 Command: "/apps/spacks/2023-08-14/opt/spack/linux-rocky9-x86_64/gcc-11.3.1/openmpi-4.1.5-2rgaqk2wseegpmbdbbygvwrljccjaqsk/bin/mpiexec" "-n" "4" "dsl/did/dtd_test_task_insertion"
...
 29329 dtd_test_task_insertion: /home/bouteill/parsec/dplasma/parsec/parsec/mca/termdet/local/termdet_local_module.c:114: parsec_termdet_local_termination_dete
       cted: Assertion `tp->tdm.monitor == PARSEC_TERMDET_LOCAL_TERMINATED' failed.
 29330 [leconte:4113702] *** Process received signal ***
 29340 [leconte:4113702] [ 7] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(+0xb042f)[0x7fa93d0fc42f]
 29341 [leconte:4113702] [ 8] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(parsec_release_dtd_task_to_mempool+0x32)[0x7fa93d0ce596]
 29342 [leconte:4113702] [ 9] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(__parsec_complete_execution+0xc6)[0x7fa93d0b7f70]
 29343 [leconte:4113702] [10] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(__parsec_task_progress+0x12e)[0x7fa93d0b80ca]
 29344 [leconte:4113702] [11] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(__parsec_context_wait+0x2ee)[0x7fa93d0b8c0a]
 29345 [leconte:4113702] [12] /home/bouteill/parsec/dplasma/build.cuda/parsec/parsec/libparsec.so.4(+0x49343)[0x7fa93d095343]
@abouteiller abouteiller added the bug Something isn't working label Feb 7, 2024
@abouteiller
Copy link
Contributor Author

this is somewhat rare but can be reproduced for sure with ctest --repeat until-fail:100

Another variant

dtd_test_task_insertion: /home/bouteill/parsec/dplasma/parsec/parsec/mca/termdet/local/termdet_local_module.c:106: parsec_termdet_local_taskpool_state: Assertion `0' failed.

@abouteiller
Copy link
Contributor Author

I have it on master

@abouteiller abouteiller added this to the v4.0 milestone Nov 13, 2024
therault added a commit to therault/parsec that referenced this issue Nov 13, 2024
@therault
Fix a race condition in DTD for the local termdet
8dea7af 
DTD uses termination_detected() to destroy the taskpool,
and on_leave_wait() to reset the taskpool and reinitialize
the termination detector.

The 'local' termination detector module had a race condition:
it would mark the taskpool as 'terminated', then call the
termination_detected() callback.

As any thread can detect the termination, a worker thread could
enter taskpool destruction in parallel with the main thread
calling on_leave_wait() because taskpool_state() has returned
TERMINATED already.

Thus, the taskpool could be destroyed *after* its termination
detector is reset, leading to asserts.

Proposed solution consists of adding an intermediary state in
the termination detector, 'TERMINATING'. TERMINATING is shown
as 'BUSY' to the scheduler, so the main thread does not return
from the main loop and does not call on_leave_wait() while the
thread that detected termination is still executing
termination_detected(). TERMINATING is still a necessary
intermediary step in order to ensure that only one thread calls
termination_detected().

Solves issue ICLDisco#634
@therault therault mentioned this issue Nov 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@therault therault

Labels
bug Something isn't working
Projects
None yet
Milestone
v4.0
Development

No branches or pull requests
2 participants
@therault @abouteiller