Performance List Reproduction 性能榜单复现

[zRzRzRzRzRzRzR]

Any questions about the Benchmark in paper can be raised here.

任何对榜单有疑问的内容都可以在这里进行提出

[neuwangmq]

是否可以提供验证脚本能够复现一下在VQAV2 testVQA这些数据集上的指标？谢谢。

我使用cogagent-4bit的openai-api测试上面VQAV2的时候出现了上面的问题，生成的文本与答案匹配度比较低。但是在网页端这种情况就比较少？请问可能是什么原因？

[runzeer]

Thanks for your excellent work. But I have some questiosn for the VQA benchmark experiments. Could you help me out?
In the paper VQA-v2 experiments, the model are finetuned with LoRA or full parameters? And which pretrained model weights are loaded? It is the base model or the chat model? Thanks a lot.

[JohnTang93]

无法复现指标，使用原始prompt DocVQA是73.4% anls，你们用的是什么prompt，能不能提供一下测试脚本，我们还想在我们的paper中对比一下其他benchmark的指标

[dmndxld]

感谢您的工作，请问能否提供有关aitw数据集的测试脚本？

[zhiyuanhubj]

Hi I tried to reimplement the results in AITW based on Cogagent. While, I only can get around 34% in overall performance when I follow the repo readme instruction and the discussion in #259.

Would you like to share more details for the implementation? For example, whether you incorporate summary of historical actions and how cogagent can do this summary? Also, would you like to share the script for this dataset?

[Originlightwkp]

My test results are as follows, is this considered good or bad?

[2024-07-19 09:38:21,589] [INFO] [RANK 0] validation loss at the end of training for test data | loss: 0.000000E+00 | PPL: 1.000000E+00 acc 5.319865E-02 | acc_w/o_case 5.319865E-02 |