Memory leak?

[SparkJiao]

I think there may be memory leakage.

The memory usage is increasing with the training process and has cost more than 100GB before the first epoch ends.

Did anyone encounter similar problem?

[Jingkang50]

Have you solved it yet?

[SparkJiao]

Yes, I use del command to free the memory used by inputs after each forward. At least it works for me.

[Jingkang50]

Cool, that pretty makes sense.

[cxzhang93]

Yes, I use del command to free the memory used by inputs after each forward. At least it works for me.

Hi @SparkJiao , I was encountering same problem as you. could you elaborate how did you resolve this issue like what command you added to which location? Many thanks!