"beamsearch" and "beamsearch_ngram" flag on script: "eval_beamsearch_ngram.py" not working

[asierhv]

Hello, I have an issue with the script eval_beamsearch_ngram.py.

When changing the flag to indicate the decoding_mode, there is no difference between choosing beamsearch or beamsearch_ngram. The documentation states that this option indicates whether beamsearch is done using the language model or not, but it seems that no matter which one I choose, it always uses the language model, and it even requires it, as it doesn't run without it.

The results obtained are the same, even different from when beam_beta is 0. It's as if the script is hard-coded to always use beamsearch_ngram.

Is there a solution to this?

[titu1994]

@karpnv could you take a look ?

[karpnv]

@AsierH can you specify whole command? Especially interested the value of decoding_strategy parameter

[asierhv]

Sure, this is the command I use:

python eval_beamsearch_ngram.py nemo_model_file=mymodel.nemo
input_manifest=test.json
beam_width=[32]
beam_alpha=[4]
beam_beta=[4]
acoustic_batch_size=32
preds_output_folder=./predictions
decoding_mode=beamsearch
decoding_strategy=beam
use_amp=true

[karpnv]

@AsierH which Python version do you use? Is it from docker container or not?

[asierhv]

I use recomended python version 3.10.12 installed on a conda environment, then used pip install nemo_toolit[all] for installing nemo on the environment.

[asierhv]

Any news with this? @karpnv @titu1994