You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As stated by other discussions here, I used resume_if_exists to resume my previous training, however it resulted in a sudden spike of val_wer as you can see in the image.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
As stated by other discussions here, I used
resume_if_exists
to resume my previous training, however it resulted in a sudden spike ofval_wer
as you can see in the image.For context here is the full command that i used:
/home/sabat/NeMo/examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py --config-path=/home/sabat/NeMo/examples/asr/conf/contextnet_rnnt --config-name=config_rnnt_bpe model.train_ds.manifest_filepath=/home/sabat/NeMo/data/train_clean_100.json model.validation_ds.manifest_filepath=/home/sabat/NeMo/data/dev_clean.json model.test_ds.manifest_filepath=/home/sabat/NeMo/data/test_clean.json model.tokenizer.dir=/home/sabat/NeMo/tokenizers/tokenizer_spe_bpe_v70 model.tokenizer.type=bpe trainer.devices=-1 trainer.accelerator=gpu trainer.max_epochs=100 trainer.precision=16 exp_manager.create_wandb_logger=True exp_manager.wandb_logger_kwargs.name=rnnt_bpe_v70_clean100 exp_manager.wandb_logger_kwargs.project=Cleft2Speech ++exp_manager.resume_if_exists=True +exp_manager.explicit_log_dir=/home/sabat/NeMo/nemo_experiments/ConvRNNTBPE5x1/2025-03-18_07-06-32/ ++exp_manager.create_early_stopping_callback=True ++exp_manager.disable_validation_on_resume=False
Beta Was this translation helpful? Give feedback.
All reactions