How to start training again from last checkpoint

[FatimaArshad-DS]

Hi,

My training got interrupted and Im trying to restart training from last checkpoint. However, training starts from the beginning. How do I make it start from last checkpoint?

[titu1994]

If you're using exp manager, it has two flags for resuming training v

Without it, you can try loading your checkpoint using load_from_checkpoinr() and then calling trainer.train() again but we haven't tested that

[yunbin]

What is involved to restart training using speech_to_text_finetune.py for model type EncDecHybridRNNTCTCBPEModel? The script states: "Currently for simplicity of single script for all model types, we only support init_from_nemo_model and init_from_pretrained_model", but resume from a ckpt file is essential for any model building tool.

[FatimaArshad-DS]

Are you talking about this one? create_checkpoint_callback. It is set to True @titu1994

[itzsimpl]

@FatimaArshad-DS see

NeMo/nemo/utils/exp_manager.py

Lines 197 to 200 in 8a172df

	- resume_if_exists (bool): Whether this experiment is resuming from a previous run. If True, it sets
	trainer._checkpoint_connector.resume_from_checkpoint_fit_path so that the trainer should auto-resume. exp_manager will move files
	under log_dir to log_dir/run_{int}. Defaults to False. From v1.0.0, when resume_if_exists is True,
	we would not create version folders to make it easier to find the log folder for next runs.

and

NeMo/nemo/utils/exp_manager.py

Lines 204 to 206 in 8a172df

	- resume_ignore_no_checkpoint (bool): exp_manager errors out if resume_if_exists is True and no checkpoint
	could be found. This behaviour can be disabled, in which case exp_manager will print a message and
	continue without restoring, by setting resume_ignore_no_checkpoint to True. Defaults to False.

[itzsimpl]

Note that you will need to fix eithr

NeMo/nemo/utils/exp_manager.py

Lines 188 to 189 in 8a172df

	- explicit_log_dir (str, Path): Can be used to override exp_dir/name/version folder creation. Defaults to
	None, which will use exp_dir, name, and version to construct the logging directory.

or set

NeMo/nemo/utils/exp_manager.py

Lines 194 to 195 in 8a172df

	- version (str): The version of the experiment. Defaults to None which uses either a datetime string or
	lightning's TensorboardLogger system of using version_{int}.

to point to your specific run, that you wish to resume.

[ericharper]

We also some documentation here: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/core/exp_manager.html

[XuesongYang]

You could continue training from the latest checkpoint following the steps:

1. go to your checkpoint directory, and rename the latest a.ckpt file to a-last.ckpt because https://github.com/NVIDIA/NeMo/blob/main/nemo/utils/exp_manager.py#L420 ;
2. override the default YAML config with ++exp_manager.resume_if_exists=True and run the same training recipe again.

[FatimaArshad-DS]

This worked!!!!
However, what happens if training is interrupted before the epoch ends? What that cause any issue? As far as I have observed it trains on already trained data

[XuesongYang]

if training is interrupted within an epoch, then no new checkpoint (a+1)-last.ckpt would be saved. So you have to load the previous a-last.ckpt again and train.

[FatimaArshad-DS]

Training on a-last.ckpt starts training on already seen data samples. How do I avoid it?

[XuesongYang]

it did not happen to me. Maybe change another random seed? @titu1994 are you aware of this observation?

[titu1994]

If you have not turned on shuffle, or have not used shuffle_n in train_ds, then data loader will not shuffle the data (if using tarred dataset).

[XuesongYang]

@FatimaArshad-DS Do you have any other questions about this?