-
Notifications
You must be signed in to change notification settings - Fork 154
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory Error while running the training code #50
Comments
Hi @Rufaida94, try reducing the batch_size? |
I've reduced the batch size and removed any writing into memory except for the model and it worked fine. Thanks |
@wuyifan18 Is there any way to make the code run faster (especially the training code) for a very large dataset? Currently, it is taking about 1 hour for each epoch with either a GPU or a CPU. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi @wuyifan18 , thank you for the great tool. It works perfectly with a very small dataset, but whenever I try running it with a larger dataset I get this error during the training phase:
[enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 278528000 bytes. Error code 12 (Cannot allocate memory)
My machine has large RAM but I am not sure why is this happening or how can I resolve this issue? Or how can we edit the code so that we use less memory space with each epoch? now each epoch takes approximately 1 GB of memory.
Any suggestion is highly appreciated.
Thanks
The text was updated successfully, but these errors were encountered: