Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How many iterations have the pretrained models been trained for? #20

Open
mayank-git-hub opened this issue Apr 17, 2024 · 2 comments
Open

Comments

@mayank-git-hub
Copy link

Hello Authors, thank you for the wonderful work!

While trying to reproduce the results for the pretrained models, I found a mismatch between the metrics mentioned in the paper.
One of the small bugs that i found in the evaluation script is that instead of multiplying the FAD by 1e4, it has been multiplied by 1e3.

Even ignoring that, the results do not match and I believe it could be because of the difference in the number of iterations that the model has been trained for compared to the one mentioned in the paper.

Can you provide details about the pretrained models that you have shared?

Thank you so much!

@kaiw7
Copy link

kaiw7 commented May 3, 2024

Hello Authors, thank you for the wonderful work!

While trying to reproduce the results for the pretrained models, I found a mismatch between the metrics mentioned in the paper. One of the small bugs that i found in the evaluation script is that instead of multiplying the FAD by 1e4, it has been multiplied by 1e3.

Even ignoring that, the results do not match and I believe it could be because of the difference in the number of iterations that the model has been trained for compared to the one mentioned in the paper.

Can you provide details about the pretrained models that you have shared?

Thank you so much!

Hi, I also met this issue. Could I know where you found the 1e4 or 1e3? And how many samples did you use for the FVD evaluation?

@mayank-git-hub
Copy link
Author

Could I know where you found the 1e4 or 1e3?

https://github.com/researchmm/MM-Diffusion/blob/1d2d5ad9b47f57e7d300e087af8eb93181da094d/mm_diffusion/evaluator.py#L170C5-L170C43

And how many samples did you use for the FVD evaluation?

2048 (I used the default settings of the repository)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants