Error when training using single-animal pipeline #743
-
|
Hello! I was trying to track a single fly, and after labeling a set of suggested frames, If I tried to train a model using the single animal pipeline, I got a popup saying there was an error, and to check the command line for more information. Attached are screenshots of the message and the most recent info on the command line terminal. I have used SLEAP successfully to track multiple flies before. Also, when I tried to view training image inputs for the single fly, nothing would show up. The second screenshot shows the command line terminal when I clicked view training image inputs.
What could be going wrong? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Hi @akshitasax, This issue is related to GPU memory. You can tell from the console error that starts with For single instance models, SLEAP uses the entire frame as input to the network and this can be quite large as in your case (1280 x 720), which will fill up your GPU memory if you don't have one with >>8GB of GPU RAM. For your data, our recommendation would be to try using a top-down model. Even though it's meant for multiple animals, you can filter it down to a single animal later by running tracking on it with a max number of animals set to 1. It's useful in this case since your animal is much smaller than the frame size, so the top-down model will first detect the centroid of the fly using a lower resolution version of the frame, and then crop a smaller (but full resolution) box around it to detect the individual body parts. Give it a go and let us know if you run into any issues! Talmo |
Beta Was this translation helpful? Give feedback.
Hi @akshitasax,
This issue is related to GPU memory. You can tell from the console error that starts with
ResourceExhaustedError: OOM when allocating tensor with shape....For single instance models, SLEAP uses the entire frame as input to the network and this can be quite large as in your case (1280 x 720), which will fill up your GPU memory if you don't have one with >>8GB of GPU RAM.
For your data, our recommendation would be to try using a top-down model. Even though it's meant for multiple animals, you can filter it down to a single animal later by running tracking on it with a max number of animals set to 1. It's useful in this case since your animal is much smaller than the frame s…