Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large offset support #499

Open
llaniewski opened this issue Jan 17, 2024 · 1 comment
Open

Large offset support #499

llaniewski opened this issue Jan 17, 2024 · 1 comment
Assignees
Labels
enhancement New features/updates watch out! Things to watch out for (bug/non-bug weird behavior)

Comments

@llaniewski
Copy link
Member

TCLB currently does not support cases in which overall offset overflows (32bit) integer for:

  • load_ functions (dynamic and static access of fields) anywhere where nx*ny*nz*fields is larger than 2^31
  • pop_ functions (loading fields through densities) anywhere where
    nx*ny*nz is larger than 2^31

This can be fixed by appropriate casting in offset calculations in LatticeAccess, but care has to be taken to not slow down the performance by making unnecessary int64_t operations.

Originally posted by @shkodm in #496 (comment)
[...] Some things still don't work as expected (also the same on master branch). I run on 2 V100 on Bunya, each with 80GB GPUs, my case is large, so I split between 2.
I get:
Cumulative allocation of 63.GB)
and then
an illegal memory access was encountered in Lattice.hpp at line 279

The error is the same even if try I split between 3 GPUs (40GB each, so plenty of space even if there is some unaccounted memory)

@llaniewski llaniewski self-assigned this Jan 17, 2024
@llaniewski llaniewski added enhancement New features/updates watch out! Things to watch out for (bug/non-bug weird behavior) labels Jan 17, 2024
@llaniewski
Copy link
Member Author

@shkodm @TravisMitchell After some investigation, we should really think if we want to calculate 64bit indexes, as 64bit multiplication on CUDA is around 20x cost of 32bit multiplication.

@shkodm can you check the sizes in your case? nx,ny,nz, but also number of fields?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New features/updates watch out! Things to watch out for (bug/non-bug weird behavior)
Projects
None yet
Development

No branches or pull requests

1 participant