WhisperX missing audio part while original whisper and fast whisper working fine

[polyglotSE]

I am trying to transcribe audio using WhisperX. It's giving the best transcription, but the only problem I am having is that it's missing some audio parts. I don't think it's a VAD issue because I have used the same hyperparameters while using VAD standalone. I have traced the code and printed the vad_segments after merge_chunks, and it's giving the correct vad_segments.

File: asr.py

def transcribe(
        self, audio: Union[str, np.ndarray], batch_size=None, num_workers=0, language=None, task=None, chunk_size=30, print_progress = False, combined_progress=False
    ) -> TranscriptionResult:
        if isinstance(audio, str):
            audio = load_audio(audio)

        def data(audio, segments):
            for seg in segments:
                f1 = int(seg['start'] * SAMPLE_RATE)
                f2 = int(seg['end'] * SAMPLE_RATE)
                # print(f2-f1)
                yield {'inputs': audio[f1:f2]}

        vad_segments = self.vad_model({"waveform": torch.from_numpy(audio).unsqueeze(0), "sample_rate": SAMPLE_RATE})
        vad_segments = merge_chunks(
            vad_segments,
            chunk_size,
            onset=self._vad_params["vad_onset"],
            offset=self._vad_params["vad_offset"],
        )

Steps to Reproduce:

1. Use the provided audio file with WhisperX.
2. Compare the transcription result with the original Whisper and Fast Whisper transcriptions.
3. Notice that certain audio parts are missing in the WhisperX transcription.

Expected Behavior:

WhisperX should transcribe the entire audio without missing any parts, similar to the original Whisper and Fast Whisper.

Actual Behavior:

WhisperX is missing some parts of the audio during transcription.

Additional Information:

• I think that the issue is not related to VAD by using the same hyperparameters with VAD standalone.
• The vad_segments after merge_chunks are correct, as printed during debugging.

Any guidance or ideas on how to troubleshoot this issue and ensure WhisperX captures the entire audio content would be greatly appreciated.

[BenMaheu]

Same problem here !

[reasv]

It seems similar to my issue #844

[ncuxzy]

Did you solve this problem? If I denoise the voice first, this problem will not exist. I used demucs to remove noise, but the denoising effect was unstable, so it was not completely solved

[pneuly]

It also seems to be the same issue as #574. I cannot find any solutions, but the workaround is to set chunk_size to less than 30.