I can't replicate the results

[mathigatti]

Hi! Thanks for working on such a cool project. I'm testing the shared trained hhar model checking if the 120 frames long samples align with their expected activity labels after running inference using these models:

BERT_PATH = 'saved/pretrain_base_motion_20_120/motion.pt'
CLASSIFIER_PATH = 'saved/classifier_base_gru_motion_20_120/motion.pt'

But I'm not getting good results, it almost always fails (19% accuracy). Do you know why it might be?

True label: downstairs
Predicted label: walk

More details about the prediction:
[('bike', 0.076339796),
 ('sit', 0.1319973),
 ('downstairs', 0.037129793),
 ('upstairs', 0.018817274),
 ('stand', 0.019472582),
 ('walk', 0.71624327)]

My code is here. You can run it just by going to "Runtime" -> "Run all".

[dapowan]

Hi, Thanks for your interest. It seems that you load the wrong models. If you want to classify the data in the hhar dataset, please load the models trained in the same dataset, i.e., saved/pretrain_base_hhar_20_120/hhar.pt and saved/classifier_base_gru_hhar_20_120/hhar.pt in your case.

[mathigatti]

oh right thanks! I just updated the notebook with the right models. It's still having pretty much the same accuracy anyway :'( (18% now)

[dapowan]

A possible reason is that the classifier is trained on data with a sequence length of 20 instead of 120. You can check the Section 3.6 in our paper for more details. You may transform the shape of embedding generated by LIMU-BERT from 12072 to 620*72, and then input the transformed embedding into the classifier for inference.

[DLuminary]

Hi @mathigatti , did you manage to sort it out?
I'm facing the same issue. @dapowan I guess you meant 120 * 72 -> 6 * 20 * 72, or am I missing something? (I guess you do and it's GitHub that breaks it)
I tried to do so by:

reshaped_embeddings = embeddings.reshape(6, 20, embeddings.size(-1)) # Shape: (6, 20, 72),
output = classifier_model(reshaped_embeddings)

but with no luck.

[dapowan]

Sorry for the late reply. Yes, @DLuminary you are right. Maybe you two can build the model based on 'classifier.py' or 'classifier_bert.py'. And please note the index of activity labels on hhar dataset should be 2 instead of 0 as indicated in dataset/data_config.json, which means please set -l 2 when you want to run 'classifier.py' on hhar dataset.