[Bug] CUDA error: an illegal memory access was encountered on v.0.0.18

[NanoCode012]

Hello, I have encountered this new CUDA error on the latest tag. I initially transitioned from v0.0.12 (which worked) to v0.0.18(which got this error). v0.0.17 from source worked.

My env is a GCP L4 VM, cuda 11.8. I've made sure to use torch==2.2.0 and clear torch extensions cache. I also tried changing VM gpus, updating dependencies, installing from wheels vs source.

Full stack:

CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Traceback (most recent call last):
    ...
    res = self.generator.stream_ex()
  File "/home/{user}/workspace/exllamav2/exllamav2/generator/streaming.py", line 346, in stream_ex
    chunk, eos, chunk_token_ids, probs, ptokens, pprobs, logits = self._stream()
  File "/home/{user}/workspace/exllamav2/exllamav2/generator/streaming.py", line 432, in _stream
    next_token, next_ptokens, next_pprobs, next_prob, eos, next_logits = self._gen_single_token(self.settings)
  File "/home/{user}/workspace/exllamav2/exllamav2/generator/streaming.py", line 695, in _gen_single_token
    logits = self.model.forward(self.sequence_ids[:, -1:],
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/{user}/workspace/exllamav2/exllamav2/model.py", line 694, in forward
    r, ls = self._forward(input_ids = input_ids[:, chunk_begin : chunk_end],
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/{user}/workspace/exllamav2/exllamav2/model.py", line 775, in _forward
    x = safe_move_tensor(x, device)
  File "/home/{user}/workspace/exllamav2/exllamav2/compat.py", line 59, in safe_move_tensor
    return tensor.to(device)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I do not see this function safe_move_tensor being changed itself. I tried the streaming.py example, and it usually runs, but it also sometimes throw this error.

The code that generates this issue can be trimmed to:

ids = self.tokenizer.encode(
                prompt, add_bos=state["add_bos_token"], encode_special_tokens=True
            )
self.generator.warmup() # sometimes throws error here
self.generator.begin_stream(
            ids, settings, input_mask=None, position_offsets=None, loras=None
        )
for i in range(max_new_tokens):
  res = self.generator.stream_ex() # error

[turboderp]

I'll try to see if I can get access to an L4 VM and maybe reproduce this, but I am a little swamped. Also a little confused if 0.0.17 works, since there shouldn't really be any changes to inference for 0.0.18.

[NanoCode012]

Thanks for the reply. I tested it briefly on a 3090 as well receiving a similar error.

I've kept to 0.0.17 for now, while I'm trying to debug why CommandR isn't working (the output just stalls). Edit: It was due to a badly written code.

[NanoCode012]

Unclear why, but reinstalling today worked.. Very weird. Couldn't find why.