-
Notifications
You must be signed in to change notification settings - Fork 93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ucc_context_create
seems to take a constant time
#881
Comments
ucc_context create time depends on multiple factors, e.g. what UCC TLs you are using and whether it's global context or not. It's expected if you run with TL UCP only, since we don't connect endpoints in advance and instead connection between peers is established only if it's really needed.
Again it depends on TLs being used. Context create for TL UCP includes initializing UCP context and UCP worker. |
Hi, I am trying to see the overhead of creating a ucc communication context in a 4 node cluster with 64 slots/node.
![AVERAGE of create ctx and AVERAGE of destroy ctx](https://private-user-images.githubusercontent.com/5582709/283900463-26e17af9-f7fa-4bbd-8341-31c3ae8701b4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU4MTc3MTAsIm5iZiI6MTcyNTgxNzQxMCwicGF0aCI6Ii81NTgyNzA5LzI4MzkwMDQ2My0yNmUxN2FmOS1mN2ZhLTRiYmQtODM0MS0zMWMzYWU4NzAxYjQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDkwOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA5MDhUMTc0MzMwWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZjViYjcwMTdhYTNjNGRjNDMzZDg0NGQyZDI0NTI2Yzk4YmFiYTgzNWY0M2QwNTZkZGFiZDdlNDJmYWMxZmNjMyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.obh8o6HGXLrKOiC-plhVhsh8vDWrpbWyveGGrOdXt28)
This is a graph of calling
ucc_context_create
anducc_context_destroy
for 50 iterations and the average time spent on each operation. I was surprised to see that context creation takes constant time across the 4 nodes. Is this the expected behavior?I measured the timings for the oob allgather operation (time between when the request was created and when the request completes) and it doesn't come close to this 3s mark. Can anyone shed some light on this?
The text was updated successfully, but these errors were encountered: