is training on smaller dataset possible?

[philippbb]

Would it be possible to receive some advice how to train/predict the gesticulator on a smaller data set?

I just tried with one bvh/audio/json around 15s, but mostly prediction step via the prediction model from demo folder doesnt give me anything I expected it to be.

For example if I use the same audio and text I used for training only the first 2s or so the skeleton is doing something with the arms, then stands still.

Thank you very much in advance.

[Svito-zar]

Hi @philippbb ,
Thank you for bringing this up.
I have developed Gesticulator to be a very simple model (there are no convolutions, recurrences, etc.) , because there is not that much data to begin with. It should work on smaller datasets, even though I did not experiment with it much yet.
One aspect that's important to keep in mind is that in the default setting Gesticulator is using quite a lot of speech context: 0.5s of the past context and 1s of future context. That's why predictions on very short sequences don't work that well.
I am not sure if I understood your question correctly, @philippbb . Maybe you can clarify it.

[philippbb]

Hi @Svito-zar

Thank you very much for your answer.

My dataset is basically a 12s long. During this time (around 2s to 5s) the actor specifically says top and reaches with his hand from an almost A pose to point with his right hand to his chest to point out something on her cloth.

Training this dataset and then use the same file with the gesture predictor via the demo folder, I was expecting at some point the skeleton would do a similar kind of pointing gesture, but all I get is the movement below. And this is one of the better results I got.

temp.mp4

The dataset is in japanese and I used Japanese Bert model 'cl-tohoku/bert-base-japanese' . As the initial version of Gesticulator was in japanese, I thought this woulnd't be the source of the problem.

Moreover: In the original bvh the left hand constantly is holding a microphone pointing to the head.

[Svito-zar]

actually, Gesticulator was developed using the English dataset only.

I used a Japanese dataset in my previous system which is different.

[Svito-zar]

Hmm ... training on 12s of data is quite extreme.

Normally a model should be able to overfit to a small dataset, but 12s is not small, it is tiny :)

Did you manage to reproduce the results with the original dataset: Trinity-College Speech-Gesture dataset?
I wonder if the model is working in normal settings for you ...

[philippbb]

I applied for the trinity dataset, but answer is still pending.

Thats why I wanted to try it out with the dataset I could come up with.

How much would you "unofficially" suggest for the bare minimum as a dataset? :)

[Svito-zar]

another tricky question :)
I would say 30 minutes is the bare minimum and at least 1h is probably needed for decent results

P.S. : you might want to remind authors of the Trinity Dataset if you are waiting for more than 2 weeks