Make full use of GPU memory.

[toanhuynhnguyen]

I have 32GB GPU memory. How to make full use of GPU memory to run Batched Faster Whisper?
Such as: multithreading, or any way?
Thank you for your help.

[MahmoudAshraf97]

using the whole memory will speed anything up, there is a limit to the speed you get from batching which is bounded by memory bandwidth, not memory size

[toanhuynhnguyen]

@MahmoudAshraf97
I mean:
I have a server with 32GB GPU memory. I need to transcribe 10,000 audio files, each audio file is in around 10 minutes.
When I run batched faster-whisper with one process/thread, it takes around 7GB GPU memory for each audio file.
I want to use all the memory to speed up the transcription, how can I do it? Can I use multi-threading or other ways to make full use of GPU memory?
Thank you so much, bro!

[MahmoudAshraf97]

running multiple instances will not yield the speedup you are looking for, you should use TRT-LLM instead as it is at least 3 times faster if you implement it correctly

[toanhuynhnguyen]

Can you provide me some documents of this TRT-LLM for Batched Faster Whisper.
Thank you so much.