Fix memory leak for pipelined optimizer swapper

[mauryaavinash95]

We identified a memory leak when training with NVMe offloaded optimizer states. The issue occurs when pipeline_write=true, where the tensors that have swapped out and written to NVMe are not deallocated, leading to a memory leak.

This PR resolves the issue by deallocating the unused tensors which have swapped out to NVMe.

[tjruwase]

@mauryaavinash95, thanks for fixing this leak. As this is not a commonly used feature, I am curious if you are seeing any benefits from the pipelined swapper?

[mauryaavinash95]

@tjruwase: We see a 20-30% improvement in the optimizer update phase when using the pipelined swapper-- most likely because of the full-duplex PCIe through which the NVMes are connected. We are running some extensive experiments on this and can share some results in a few weeks if you're interested.

[tjruwase]

@tjruwase: We see a 20-30% improvement in the optimizer update phase when using the pipelined swapper-- most likely because of the full-duplex PCIe through which the NVMes are connected. We are running some extensive experiments on this and can share some results in a few weeks if you're interested.

@mauryaavinash95, wow! It is great to learn that you are seeing speedups from pipelined swapper. Thanks for sharing.

The back story is that we actually never got a chance to properly evaluate the perf benefits, but released the code anyways in hope that someday somebody might find it useful :). And so, I am very curious about your work and results in this line, if you don't mind sharing.

[mauryaavinash95]

Sorry, I accidentally made some other commits to my master branch which caused this PR to get closed and I had to open it again.