-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
why context parallel do not support alibi? #1046
Comments
Hi @Monekyzoon this is because after we split sequence into multiple chunks, we cannot tell flash attn kernel what's the sequence offset of each chunk, so flash attn kernel cannot know the correct relative positions of tokens, and the ALiBi bias generated inside the kernel is wrong for each sequence chunk. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/attention.py#L3237
assert ( alibi_slopes is None ), "Alibi slope bias addition is not supported with context parallelism."
Why not support alibi?
The text was updated successfully, but these errors were encountered: