Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wavenet not running fully on Mac #11

Open
zannads opened this issue Feb 15, 2024 · 1 comment
Open

Wavenet not running fully on Mac #11

zannads opened this issue Feb 15, 2024 · 1 comment
Assignees
Labels
bug Something isn't working
Milestone

Comments

@zannads
Copy link
Contributor

zannads commented Feb 15, 2024

When training the Wavenet model on the Mac with Metal Acceleration (device 'mps:0'), the training works but at the end when computing the final tensor it fails.
It works on CPU and on CUDA however.

This is an important Issue because it allows a 5x 6x acceleration.

@zannads zannads added the bug Something isn't working label Feb 15, 2024
@zannads zannads added this to the Internal milestone Feb 15, 2024
@zannads zannads self-assigned this Feb 15, 2024
@zannads
Copy link
Contributor Author

zannads commented Feb 16, 2024

Wavenet is not running on the Mac GPU.
It runs the training and it finishes, however, when adjusting the bias at the end of the iteration the MultiOutputWrapper function throws an error.
In particular, in file multioutput.py, function _get_args_kwargs_by_output, at row 110, inside apply_to_collection(args, Tensor,.. the error is

return tuple(function(x, *args, **kwargs) for x in data)
IndexError: Dimension specified as -1 but tensor has no dimensions

Probably something that has to do on some internal definition of the metrics of the multioutput wrapper (?).

No problem appear when running on the cpu device!

zannads added a commit that referenced this issue Feb 16, 2024
I tried to reinstall the libraries but it didn't work.

**Solution**:
since the tensors are small and on Mac there is a unified memory for GPU and CPU, the solution is to send them back to 'cpu' and then back to 'mps:0'
Dirty solution but better then nothing :)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

1 participant