-
Notifications
You must be signed in to change notification settings - Fork 112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NotImplementedError: Cannot copy out of meta tensor; no data! #50
Comments
This is a bug in DeepSpeed. Can you report it there? |
Thanks for your reply. When I changged torch down to 1.12.1 and brought cuda up to the suitable version (10.2.89), the previous error indeed disappeared, but a new one came, as shown below.
My conda env ls (Python 3.10.9):
The
Can you help me to solve it? Thanks. |
I am not really sure. Haven't seen this before but seems like CUDA is not able to compile some kernels in DeepSpeed. However, you will need to modify it a bit for the standalone script. |
Actually, I can only use cuda with version 10.2, as I am using other versions of cuda that report the following error:
The
The
The
It feels like a version issue, but I've tried to make sure the version is the same as your docker file. So, have you encountered this problem? Thank you again. |
I think your environment is configured with CUDA 11.1 and torch is compiled using 10.2. |
Hi @mayank31398, I ran into a similar issue when employing the deepspeed framework to speed up the inference of BLOOM 7.1B. Could you please take a look? Many thanks The cmd is shown below: The log is listed as follows:
Here, I use the docker image generated by the Dockerfile from https://github.com/huggingface/transformers-bloom-inference/blob/main/Dockerfile. The
The
The
|
The dockerfile works out of the box. |
Many thanks for your prompt response @mayank31398 The dockerfile is as follows:
I simply commend the last 5 lines and do them in the docker manually (to avoid repeated git clone the repo when I docker exec the created instance with another terminal)
Then in the docker, make bloom-176b, launch the benchmark, and hit the NotImplementedError: Cannot copy out of meta tensor; no data!
To supplement: I could successfully run the benchmark of bloom3b and get the perf data. First add the following lines in Makefile
Then
|
Not sure why 176b is not working. I will try to look into it :) |
Hi,
I now employ the deepspeed framework to speed up the inference of BLOOM 7.1B, as shown below:
deepspeed --num_gpus 4 bloom-inference-scripts/bloom-ds-inference.py --name bigscience/bloom-7b1
But instead I got the following bugs:
My main conda environment is:
My nvidia-smi info is:
Can you help me to solve this bug? Thank you very much!
The text was updated successfully, but these errors were encountered: