Skip to content

Commit

Permalink
add assert in get_assigned_chunk to check if tensor.shape[dim] is div…
Browse files Browse the repository at this point in the history
…isible by world_size
  • Loading branch information
chengzeyi committed Dec 19, 2024
1 parent fbfd4ed commit 8280d22
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions src/para_attn/primitives.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,8 @@ def get_assigned_chunk(
if idx is None:
idx = get_rank(group)
world_size = get_world_size(group)
total_size = tensor.shape[dim]
assert total_size % world_size == 0, f"tensor.shape[{dim}]={total_size} is not divisible by world_size={world_size}"
return tensor.chunk(world_size, dim=dim)[idx]


Expand Down

0 comments on commit 8280d22

Please sign in to comment.