(Possible) SpecAug-related Error on modified ASR tutorial #10 #2496

feddybear · 2021-07-16T14:29:30Z

feddybear
Jul 16, 2021

I am currently trying the asr tutorial, particularly, 10_ASR_CTC_Language_Finetuning.
The first part of the tutorial is a case in Japanese, using a small set from Common Voice.

I decided to try the standard, larger (500 hrs) CSJ corpus. Replaced the data manifests carefully.
I didn't freeze the encoder as it's a large corpus, also didn't do the dakuten thing.

In the first training cell, it ran for about 10 minutes before showing me this error that I have no idea how to troubleshoot.
I can see it has something to do with specAugment but, what should I check to ensure it won't happen again?
If anyone has an idea, kindly advise. Thank you!

Update: I used the commented additional values for the specAug configuration:

with open_dict(char_model.cfg.spec_augment):
    char_model.cfg.spec_augment.freq_masks = 2
    char_model.cfg.spec_augment.freq_width = 25
    char_model.cfg.spec_augment.time_masks = 2
    char_model.cfg.spec_augment.time_width = 0.05

and this changed the error to:
ValueError: empty range for randrange() (0, -39, -39)

The original log is below:

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
[NeMo I 2021-07-16 13:56:33 modelPT:751] Optimizer config = Novograd (
    Parameter Group 0
        amsgrad: False
        betas: [0.95, 0.5]
        eps: 1e-08
        grad_averaging: False
        lr: 0.01
        weight_decay: 0.001
    )
[NeMo I 2021-07-16 13:56:33 lr_scheduler:621] Scheduler "<nemo.core.optim.lr_scheduler.CosineAnnealing object at 0x7fc4f0454520>" 
    will be used during training (effective maximum steps = 11008000) - 
    Parameters : 
    (warmup_steps: null
    warmup_ratio: 0.05
    min_lr: 1.0e-05
    last_epoch: -1
    max_steps: 11008000
    )

  | Name              | Type                              | Params
------------------------------------------------------------------------
0 | preprocessor      | AudioToMelSpectrogramPreprocessor | 0     
1 | encoder           | ConvASREncoder                    | 18.9 M
2 | spec_augmentation | SpectrogramAugmentation           | 0     
3 | _wer              | WER                               | 0     
4 | decoder           | ConvASRDecoder                    | 3.3 M 
5 | loss              | CTCLoss                           | 0     
------------------------------------------------------------------------
22.2 M    Trainable params
0         Non-trainable params
22.2 M    Total params
88.969    Total estimated model params size (MB)
Validation sanity check: 0it [00:00, ?it/s]
[NeMo I 2021-07-16 13:56:36 wer:225] 
    
[NeMo I 2021-07-16 13:56:36 wer:226] reference:えーこれは従来の規則手法
[NeMo I 2021-07-16 13:56:36 wer:227] predicted:錨嘘醜錨普錨ぷ瀋貯固錨山火ｙ正ほ嘘紗内錨荻判錨滴嘘湘鍛十既十既加猿虐娘ピ既糾既荻割固湿試立篆躯５油鯨浮醜株玉ｚ環既奢既瓜既詳十魚鰹事械内聞煙践勇貯丼幇顫貯普固初善殷挿庵個紗煙萎
[NeMo I 2021-07-16 13:56:37 wer:225] 

(OUTPUT TRUNCATED)

[NeMo I 2021-07-16 14:04:51 wer:226] reference:一対比較法というのは
[NeMo I 2021-07-16 14:04:51 wer:227] predicted:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<timed eval> in <module>

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py in fit(self, model, train_dataloader, val_dataloaders, datamodule)
    458         )
    459 
--> 460         self._run(model)
    461 
    462         assert self.state.stopped

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py in _run(self, model)
    756 
    757         # dispatch `start_training` or `start_evaluating` or `start_predicting`
--> 758         self.dispatch()
    759 
    760         # plugin will finalized fitting (e.g. ddp_spawn will load trained model)

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py in dispatch(self)
    797             self.accelerator.start_predicting(self)
    798         else:
--> 799             self.accelerator.start_training(self)
    800 
    801     def run_stage(self):

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py in start_training(self, trainer)
     94 
     95     def start_training(self, trainer: 'pl.Trainer') -> None:
---> 96         self.training_type_plugin.start_training(trainer)
     97 
     98     def start_evaluating(self, trainer: 'pl.Trainer') -> None:

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py in start_training(self, trainer)
    142     def start_training(self, trainer: 'pl.Trainer') -> None:
    143         # double dispatch to initiate the training loop
--> 144         self._results = trainer.run_stage()
    145 
    146     def start_evaluating(self, trainer: 'pl.Trainer') -> None:

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py in run_stage(self)
    807         if self.predicting:
    808             return self.run_predict()
--> 809         return self.run_train()
    810 
    811     def _pre_training_routine(self):

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py in run_train(self)
    869                 with self.profiler.profile("run_training_epoch"):
    870                     # run train epoch
--> 871                     self.train_loop.run_training_epoch()
    872 
    873                 if self.max_steps and self.max_steps <= self.global_step:

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py in run_training_epoch(self)
    497             # ------------------------------------
    498             with self.trainer.profiler.profile("run_training_batch"):
--> 499                 batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
    500 
    501             # when returning -1 from train_step, we end epoch early

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py in run_training_batch(self, batch, batch_idx, dataloader_idx)
    736 
    737                         # optimizer step
--> 738                         self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
    739                         if len(self.trainer.optimizers) > 1:
    740                             # revert back to previous state

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py in optimizer_step(self, optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
    432 
    433         # model hook
--> 434         model_ref.optimizer_step(
    435             self.trainer.current_epoch,
    436             batch_idx,

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py in optimizer_step(self, epoch, batch_idx, optimizer, optimizer_idx, optimizer_closure, on_tpu, using_native_amp, using_lbfgs)
   1401 
   1402         """
-> 1403         optimizer.step(closure=optimizer_closure)
   1404 
   1405     def optimizer_zero_grad(self, epoch: int, batch_idx: int, optimizer: Optimizer, optimizer_idx: int):

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py in step(self, closure, *args, **kwargs)
    212             profiler_name = f"optimizer_step_and_closure_{self._optimizer_idx}"
    213 
--> 214         self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
    215         self._total_optimizer_step_calls += 1
    216 

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py in __optimizer_step(self, closure, profiler_name, **kwargs)
    132 
    133         with trainer.profiler.profile(profiler_name):
--> 134             trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
    135 
    136     def step(self, *args, closure: Optional[Callable] = None, **kwargs):

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py in optimizer_step(self, optimizer, opt_idx, lambda_closure, **kwargs)
    327         )
    328         if make_optimizer_step:
--> 329             self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs)
    330         self.precision_plugin.post_optimizer_step(optimizer, opt_idx)
    331         self.training_type_plugin.post_optimizer_step(optimizer, opt_idx, **kwargs)

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py in run_optimizer_step(self, optimizer, optimizer_idx, lambda_closure, **kwargs)
    334         self, optimizer: Optimizer, optimizer_idx: int, lambda_closure: Callable, **kwargs: Any
    335     ) -> None:
--> 336         self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs)
    337 
    338     def optimizer_zero_grad(self, current_epoch: int, batch_idx: int, optimizer: Optimizer, opt_idx: int) -> None:

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py in optimizer_step(self, optimizer, lambda_closure, **kwargs)
    191 
    192     def optimizer_step(self, optimizer: torch.optim.Optimizer, lambda_closure: Callable, **kwargs):
--> 193         optimizer.step(closure=lambda_closure, **kwargs)
    194 
    195     @property

/opt/conda/lib/python3.8/site-packages/torch/optim/lr_scheduler.py in wrapper(*args, **kwargs)
     63                 instance._step_count += 1
     64                 wrapped = func.__get__(instance, cls)
---> 65                 return wrapped(*args, **kwargs)
     66 
     67             # Note that the returned function here is no longer a bound method,

/opt/conda/lib/python3.8/site-packages/torch/optim/optimizer.py in wrapper(*args, **kwargs)
     86                 profile_name = "Optimizer.step#{}.step".format(obj.__class__.__name__)
     87                 with torch.autograd.profiler.record_function(profile_name):
---> 88                     return func(*args, **kwargs)
     89             return wrapper
     90 

/opt/conda/lib/python3.8/site-packages/nemo/core/optim/novograd.py in step(self, closure)
     81         loss = None
     82         if closure is not None:
---> 83             loss = closure()
     84 
     85         for group in self.param_groups:

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py in train_step_and_backward_closure()
    730 
    731                         def train_step_and_backward_closure():
--> 732                             result = self.training_step_and_backward(
    733                                 split_batch, batch_idx, opt_idx, optimizer, self.trainer.hiddens
    734                             )

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py in training_step_and_backward(self, split_batch, batch_idx, opt_idx, optimizer, hiddens)
    821         with self.trainer.profiler.profile("training_step_and_backward"):
    822             # lightning module hook
--> 823             result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
    824             self._curr_step_result = result
    825 

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py in training_step(self, split_batch, batch_idx, opt_idx, hiddens)
    288             model_ref._results = Result()
    289             with self.trainer.profiler.profile("training_step"):
--> 290                 training_step_output = self.trainer.accelerator.training_step(args)
    291                 self.trainer.accelerator.post_training_step()
    292 

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py in training_step(self, args)
    202 
    203         with self.precision_plugin.train_step_context(), self.training_type_plugin.train_step_context():
--> 204             return self.training_type_plugin.training_step(*args)
    205 
    206     def post_training_step(self) -> None:

/opt/conda/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py in training_step(self, *args, **kwargs)
    153 
    154     def training_step(self, *args, **kwargs):
--> 155         return self.lightning_module.training_step(*args, **kwargs)
    156 
    157     def post_training_step(self):

/opt/conda/lib/python3.8/site-packages/nemo/utils/model_utils.py in wrap_training_step(wrapped, instance, args, kwargs)
    337 @wrapt.decorator
    338 def wrap_training_step(wrapped, instance: pl.LightningModule, args, kwargs):
--> 339     output_dict = wrapped(*args, **kwargs)
    340 
    341     if isinstance(output_dict, dict) and output_dict is not None and 'log' in output_dict:

/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/ctc_models.py in training_step(self, batch, batch_nb)
    557             )
    558         else:
--> 559             log_probs, encoded_len, predictions = self.forward(input_signal=signal, input_signal_length=signal_len)
    560 
    561         loss_value = self.loss(

/opt/conda/lib/python3.8/site-packages/nemo/core/classes/common.py in __call__(self, wrapped, instance, args, kwargs)
    790 
    791         # Call the method - this can be forward, or any other callable method
--> 792         outputs = wrapped(*args, **kwargs)
    793 
    794         instance._attach_and_validate_output_types(

/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/ctc_models.py in forward(self, input_signal, input_signal_length, processed_signal, processed_signal_length)
    541 
    542         if self.spec_augmentation is not None and self.training:
--> 543             processed_signal = self.spec_augmentation(input_spec=processed_signal, length=processed_signal_length)
    544 
    545         encoded, encoded_len = self.encoder(audio_signal=processed_signal, length=processed_signal_length)

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1013         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1014                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1015             return forward_call(*input, **kwargs)
   1016         # Do not call functions when jit is used
   1017         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.8/site-packages/nemo/core/classes/common.py in __call__(self, wrapped, instance, args, kwargs)
    790 
    791         # Call the method - this can be forward, or any other callable method
--> 792         outputs = wrapped(*args, **kwargs)
    793 
    794         instance._attach_and_validate_output_types(

/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/modules/audio_preprocessing.py in forward(self, input_spec, length)
    492     @typecheck()
    493     def forward(self, input_spec, length):
--> 494         augmented_spec = self.spec_cutout(input_spec=input_spec)
    495 
    496         # To run the Numba kernel, correct numba version is required as well as

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1013         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1014                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1015             return forward_call(*input, **kwargs)
   1016         # Do not call functions when jit is used
   1017         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.8/site-packages/nemo/core/classes/common.py in __call__(self, wrapped, instance, args, kwargs)
    790 
    791         # Call the method - this can be forward, or any other callable method
--> 792         outputs = wrapped(*args, **kwargs)
    793 
    794         instance._attach_and_validate_output_types(

/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     25         def decorate_context(*args, **kwargs):
     26             with self.__class__():
---> 27                 return func(*args, **kwargs)
     28         return cast(F, decorate_context)
     29 

/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/parts/submodules/spectr_augment.py in forward(self, input_spec)
    146             for i in range(self.rect_masks):
    147                 rect_x = self._rng.randint(0, sh[1] - self.rect_freq)
--> 148                 rect_y = self._rng.randint(0, sh[2] - self.rect_time)
    149 
    150                 w_x = self._rng.randint(0, self.rect_freq)

/opt/conda/lib/python3.8/random.py in randint(self, a, b)
    246         """
    247 
--> 248         return self.randrange(a, b+1)
    249 
    250     def _randbelow_with_getrandbits(self, n):

/opt/conda/lib/python3.8/random.py in randrange(self, start, stop, step, _int)
    224             return istart + self._randbelow(width)
    225         if step == 1:
--> 226             raise ValueError("empty range for randrange() (%d, %d, %d)" % (istart, istop, width))
    227 
    228         # Non-unit step argument supplied.

ValueError: empty range for randrange() (0, -7, -7)

Answered by feddybear

Jul 16, 2021

Update 2: The training seems to be running smoothly now.
I just removed all rectangular masks and went with the time-frequency-based ones.

View full answer

feddybear · 2021-07-16T15:34:29Z

feddybear
Jul 16, 2021
Author

Update 2: The training seems to be running smoothly now.
I just removed all rectangular masks and went with the time-frequency-based ones.

2 replies

titu1994 Jul 16, 2021
Maintainer

Which Nemo version was this ? 1.1.0? Or main? I patched this in main recently

feddybear Jul 16, 2021
Author

Yes it's 1.1.0. I was working inside a docker container and didn't have time to check out the update. Thanks for the info!

raman-r-4978 · 2023-04-06T03:47:08Z

raman-r-4978
Apr 6, 2023

I have question related Conformer SpecAugment

I see we use

'spec_augment': {
  '_target_': 'nemo.collections.asr.modules.SpectrogramAugmentation',
  'freq_masks': 2,
  'time_masks': 10,
  'freq_width': 27,
  'time_width': 0.05
},

Does this actually work?

Do you apply SpecAug to all the training datapoints? Or Are there any trick to this while applying?

So far I haven't got success with SpecAug. Any tips can be helpful

5 replies

VahidooX Apr 6, 2023
Collaborator

Yes, that is the setting we use to train all our ASR models. It is applied to all the samples during training.
What do you mean by no success with SpecAug?

raman-r-4978 Apr 6, 2023

The training does not seem to converge when SpecAug is applied. I realize that the Training/Validation loss will be significantly higher than in standard training, but training metrics (CER and WER on development sets) are also deteriorating.

Yellow line - Without SpecAug
Purple line - SpecAug applied intermittently between steps

As illustrated, the WER increases dramatically upon applying SpecAug and returns to normal when its application is stopped. And using SpecAug on all training batches is not effective.

Please note - The aforementioned observations are based on our in-house ASR tools. We do not utilize NeMo for our development.

Is it necessary to train for an extended number of epochs? Whats the trick here?

VahidooX Apr 6, 2023
Collaborator

Actually your plots shows that SpecAug is working and helping. Take a look at the models when SpecAug is disabled, the purple model is giving better accuracy on train comparing to the yellow one. When SpecAug is enabled, the train WER should be high, even as large as 20% or more. That is how it is supposed to work. The important thing is the validation WER. SpecAug should help the validation metrics. When you use SpecAug, a significant portion of the input is masked and that is why train loss and WER is higher than validation sets.
SpecAug is helpful significantly specially if you have smaller datasets which is the case for most users.

titu1994 Apr 6, 2023
Maintainer

You're monitoring training wer - that is completely dependent on whether spec augment is applied or not. ASR is an alignment task - if audio frames are 0 it's wer on that blank region will spike. When applying spec Aug only monitor val wer because spec Aug is not applied to it.

And yes, spec Aug should be applied through training.

raman-r-4978 Apr 10, 2023

I get the point now. Thanks to you both for the clarification. 🙂

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(Possible) SpecAug-related Error on modified ASR tutorial #10 #2496

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

(Possible) SpecAug-related Error on modified ASR tutorial #10 #2496

feddybear Jul 16, 2021

Replies: 2 comments · 7 replies

feddybear Jul 16, 2021 Author

titu1994 Jul 16, 2021 Maintainer

feddybear Jul 16, 2021 Author

raman-r-4978 Apr 6, 2023

VahidooX Apr 6, 2023 Collaborator

raman-r-4978 Apr 6, 2023

VahidooX Apr 6, 2023 Collaborator

titu1994 Apr 6, 2023 Maintainer

raman-r-4978 Apr 10, 2023

feddybear
Jul 16, 2021

Replies: 2 comments 7 replies

feddybear
Jul 16, 2021
Author

titu1994 Jul 16, 2021
Maintainer

feddybear Jul 16, 2021
Author

raman-r-4978
Apr 6, 2023

VahidooX Apr 6, 2023
Collaborator

VahidooX Apr 6, 2023
Collaborator

titu1994 Apr 6, 2023
Maintainer