Commit Graph

76 Commits

Author SHA1 Message Date
Edresson 0bdfd3cb50 Add the ValueError in the restore checkpoint exception to avoid problems with the optimizer restauration when new keys are addition 2021-12-20 11:54:09 +00:00
Edresson f996afedb0 Implement multilingual dataloader support 2021-12-20 11:54:09 +00:00
Eren Gölge 512ada7548 Fix callbacks against multi-gpu training 2021-12-01 10:32:14 +00:00
Eren Gölge 70e4d0e524 Fix grad_norm handling 2021-10-21 16:29:06 +00:00
Eren Gölge 588da1a24e Simplify grad_norm handling in trainer 2021-10-19 16:33:04 +00:00
Eren Gölge b4b890df03 Update trainer's initialization 2021-10-18 08:53:19 +00:00
Eren Gölge 073a2d2eb0 Refactor VITS multi-speaker initialization 2021-10-15 10:20:00 +00:00
Eren Gölge e15bc157d8 Fix #873 2021-10-14 14:39:45 +00:00
Eren Gölge 37959ad0c7 Make linter 2021-09-30 23:02:16 +00:00
Eren Gölge 55d9209221 Remote STT tokenizer 2021-09-30 14:58:26 +00:00
Eren Gölge 8ada870a57 Refactor `trainer.py` for v2 2021-09-30 14:16:34 +00:00
Eren Gölge a97dc8d09f Fix trainer malformatted print 2021-09-13 08:32:02 +00:00
Eren Gölge 66732025e1 Add `base_model` field to `forward_tts` configs 2021-09-10 17:23:48 +00:00
Eren Gölge 2dfc5bdd11 Fix best_model_path init if no best_mode 2021-09-09 09:01:52 +00:00
Eren Gölge abf5e48177 Fix logging current learning rate in trainer 2021-09-09 09:01:04 +00:00
Eren Gölge 6c4c1065b0 Fix trainer's scheduler restoring 2021-09-09 09:00:27 +00:00
Eren Gölge e20ea57c87 Update comment and add a warning 2021-09-07 12:23:32 +00:00
Eren Gölge 91a70e80b2 Refactor TTSDataset
Return a dict by `collate`
Refactor batch handling in `collate`
A couple of bug fixes
2021-09-06 15:16:58 +00:00
Eren Gölge 8584f2b82d Update docstring format 2021-09-06 15:16:58 +00:00
Eren Gölge 9af42f7886 Restore `last_epoch` of the scheduler 2021-09-06 15:16:58 +00:00
Eren Gölge 738eee0cf9 Fix style 2021-08-30 13:12:13 +00:00
Eren Gölge c560114324 Fix #750 2021-08-30 13:06:50 +00:00
Eren Gölge 18da8f5dbd Update pylint 2.10.2 and fix lint issues 2021-08-30 08:10:35 +00:00
Eren Gölge f186856e5d Add option to sort input sequnce by audio len 2021-08-30 08:10:35 +00:00
Eren Gölge 2620f62ea8 Move duration_loss inside VitsGeneratorLoss 2021-08-27 07:07:07 +00:00
Eren Gölge 5911eec3b1 Small trainer refactoring
1. Use a single Gradscaler for all the optimizers
2. Save terminal logs to a file. In DDP mode, each worker creates `trainer_N_log.txt`.
3. Fixes to allow only the main worker (rank==0) writing to Tensorboard
4. Pass parameters owned by the target optimizer to the grad_clip_norm
2021-08-26 17:08:58 +00:00
Eren Gölge c5d1dd9d1b Fix restoring best_loss
Keep the default value if model checkpoint has no `model_loss`
2021-08-17 12:12:36 +00:00
Eren Gölge c8bbcdfd07 Fix `test_run` for DDP 2021-08-13 19:39:02 +00:00
Eren Gölge 7c0d564965 Syncronize DDP processes 2021-08-13 10:40:50 +00:00
Eren Gölge ecf5f17dca Fix distribute.py and ddp training 2021-08-12 22:22:32 +00:00
Eren Gölge 39004484b9 Fix 🐛
Fix synthesizer multi-speaker init
Fix #712
2021-08-10 12:56:32 +00:00
Ayush Chaurasia e685ddfca7 Update trainer.py 2021-08-09 18:37:46 +00:00
Ayush Chaurasia 8a246cbb66 Update trainer.py 2021-08-09 18:35:08 +00:00
Ayush Chaurasia f3e9d61330 Refactor logging initialization 2021-08-09 18:35:08 +00:00
Ayush Chaurasia 79b74a989d Update: add_text 2021-08-09 18:34:38 +00:00
Ayush Chaurasia 936a47504d Update Logger API, recipes 2021-08-09 18:34:00 +00:00
Ayush Chaurasia f63cf46c55 Unified logger API 2021-08-09 18:34:00 +00:00
Ayush Chaurasia f4434da5a3 Update disabled structure 2021-08-09 18:31:16 +00:00
Ayush Chaurasia f606741dc4 Add artifacts logging , wandb args 2021-08-09 18:31:16 +00:00
Ayush Chaurasia f5e50ad502 WandbLogger 2021-08-09 18:27:06 +00:00
Eren Gölge bf562cf437 Update `trainer.py`
Fix multi-speaker initialization of models. Add changes for end2end`tts`
models.
2021-08-09 18:02:36 +00:00
Agrin Hilmkil ced4cfdbbf Allow saving / loading checkpoints from cloud paths (#683)
* Allow saving / loading checkpoints from cloud paths

Allows saving and loading checkpoints directly from cloud paths like
Amazon S3 (s3://) and Google Cloud Storage (gs://) by using fsspec.

Note: The user will have to install the relevant dependency for each
protocol. Otherwise fsspec will fail and specify which dependency is
missing.

* Append suffix _fsspec to save/load function names

* Add a lower bound to the fsspec dependency

Skips the 0 major version.

* Add missing changes from refactor

* Use fsspec for remaining artifacts

* Add test case with path requiring fsspec

* Avoid writing logs to file unless output_path is local

* Document the possibility of using paths supported by fsspec

* Fix style and lint

* Add missing lint fixes

* Add type annotations to new functions

* Use Coqpit method for converting config to dict

* Fix type annotation in semi-new function

* Add return type for load_fsspec

* Fix bug where fs not always created

* Restore the experiment removal functionality
2021-08-09 18:02:36 +00:00
Eren Gölge 05c75aa9d5 Fix linter issues 2021-07-16 13:37:38 +02:00
Eren Gölge 58cc414477 Fix WaveGrad `test_run` 2021-07-16 13:02:25 +02:00
WeberJulian 25832eb97b Changes for review 2021-07-15 11:38:45 +02:00
WeberJulian c79a82ed07 refix linter 2021-07-13 23:12:18 +02:00
WeberJulian 7d92b30946 Fix tests 2021-07-13 23:00:34 +02:00
WeberJulian 32974dd6a9 Fix test sentences synthesis 2021-07-13 16:07:13 +02:00
Eren Gölge 2e1a428b83 Update glowtts docstrings and docs 2021-06-30 14:30:55 +02:00
Eren Gölge 5723eb4738 Fix config init in `process_args` 2021-06-29 16:41:08 +02:00