Commit Graph

68 Commits

Author SHA1 Message Date
Eren Gölge 37959ad0c7 Make linter 2021-09-30 23:02:16 +00:00
Eren Gölge 55d9209221 Remote STT tokenizer 2021-09-30 14:58:26 +00:00
Eren Gölge 8ada870a57 Refactor `trainer.py` for v2 2021-09-30 14:16:34 +00:00
Eren Gölge a97dc8d09f Fix trainer malformatted print 2021-09-13 08:32:02 +00:00
Eren Gölge 66732025e1 Add `base_model` field to `forward_tts` configs 2021-09-10 17:23:48 +00:00
Eren Gölge 2dfc5bdd11 Fix best_model_path init if no best_mode 2021-09-09 09:01:52 +00:00
Eren Gölge abf5e48177 Fix logging current learning rate in trainer 2021-09-09 09:01:04 +00:00
Eren Gölge 6c4c1065b0 Fix trainer's scheduler restoring 2021-09-09 09:00:27 +00:00
Eren Gölge e20ea57c87 Update comment and add a warning 2021-09-07 12:23:32 +00:00
Eren Gölge 91a70e80b2 Refactor TTSDataset
Return a dict by `collate`
Refactor batch handling in `collate`
A couple of bug fixes
2021-09-06 15:16:58 +00:00
Eren Gölge 8584f2b82d Update docstring format 2021-09-06 15:16:58 +00:00
Eren Gölge 9af42f7886 Restore `last_epoch` of the scheduler 2021-09-06 15:16:58 +00:00
Eren Gölge 738eee0cf9 Fix style 2021-08-30 13:12:13 +00:00
Eren Gölge c560114324 Fix #750 2021-08-30 13:06:50 +00:00
Eren Gölge 18da8f5dbd Update pylint 2.10.2 and fix lint issues 2021-08-30 08:10:35 +00:00
Eren Gölge f186856e5d Add option to sort input sequnce by audio len 2021-08-30 08:10:35 +00:00
Eren Gölge 2620f62ea8 Move duration_loss inside VitsGeneratorLoss 2021-08-27 07:07:07 +00:00
Eren Gölge 5911eec3b1 Small trainer refactoring
1. Use a single Gradscaler for all the optimizers
2. Save terminal logs to a file. In DDP mode, each worker creates `trainer_N_log.txt`.
3. Fixes to allow only the main worker (rank==0) writing to Tensorboard
4. Pass parameters owned by the target optimizer to the grad_clip_norm
2021-08-26 17:08:58 +00:00
Eren Gölge c5d1dd9d1b Fix restoring best_loss
Keep the default value if model checkpoint has no `model_loss`
2021-08-17 12:12:36 +00:00
Eren Gölge c8bbcdfd07 Fix `test_run` for DDP 2021-08-13 19:39:02 +00:00
Eren Gölge 7c0d564965 Syncronize DDP processes 2021-08-13 10:40:50 +00:00
Eren Gölge ecf5f17dca Fix distribute.py and ddp training 2021-08-12 22:22:32 +00:00
Eren Gölge 39004484b9 Fix 🐛
Fix synthesizer multi-speaker init
Fix #712
2021-08-10 12:56:32 +00:00
Ayush Chaurasia e685ddfca7 Update trainer.py 2021-08-09 18:37:46 +00:00
Ayush Chaurasia 8a246cbb66 Update trainer.py 2021-08-09 18:35:08 +00:00
Ayush Chaurasia f3e9d61330 Refactor logging initialization 2021-08-09 18:35:08 +00:00
Ayush Chaurasia 79b74a989d Update: add_text 2021-08-09 18:34:38 +00:00
Ayush Chaurasia 936a47504d Update Logger API, recipes 2021-08-09 18:34:00 +00:00
Ayush Chaurasia f63cf46c55 Unified logger API 2021-08-09 18:34:00 +00:00
Ayush Chaurasia f4434da5a3 Update disabled structure 2021-08-09 18:31:16 +00:00
Ayush Chaurasia f606741dc4 Add artifacts logging , wandb args 2021-08-09 18:31:16 +00:00
Ayush Chaurasia f5e50ad502 WandbLogger 2021-08-09 18:27:06 +00:00
Eren Gölge bf562cf437 Update `trainer.py`
Fix multi-speaker initialization of models. Add changes for end2end`tts`
models.
2021-08-09 18:02:36 +00:00
Agrin Hilmkil ced4cfdbbf Allow saving / loading checkpoints from cloud paths (#683)
* Allow saving / loading checkpoints from cloud paths

Allows saving and loading checkpoints directly from cloud paths like
Amazon S3 (s3://) and Google Cloud Storage (gs://) by using fsspec.

Note: The user will have to install the relevant dependency for each
protocol. Otherwise fsspec will fail and specify which dependency is
missing.

* Append suffix _fsspec to save/load function names

* Add a lower bound to the fsspec dependency

Skips the 0 major version.

* Add missing changes from refactor

* Use fsspec for remaining artifacts

* Add test case with path requiring fsspec

* Avoid writing logs to file unless output_path is local

* Document the possibility of using paths supported by fsspec

* Fix style and lint

* Add missing lint fixes

* Add type annotations to new functions

* Use Coqpit method for converting config to dict

* Fix type annotation in semi-new function

* Add return type for load_fsspec

* Fix bug where fs not always created

* Restore the experiment removal functionality
2021-08-09 18:02:36 +00:00
Eren Gölge 05c75aa9d5 Fix linter issues 2021-07-16 13:37:38 +02:00
Eren Gölge 58cc414477 Fix WaveGrad `test_run` 2021-07-16 13:02:25 +02:00
WeberJulian 25832eb97b Changes for review 2021-07-15 11:38:45 +02:00
WeberJulian c79a82ed07 refix linter 2021-07-13 23:12:18 +02:00
WeberJulian 7d92b30946 Fix tests 2021-07-13 23:00:34 +02:00
WeberJulian 32974dd6a9 Fix test sentences synthesis 2021-07-13 16:07:13 +02:00
Eren Gölge 2e1a428b83 Update glowtts docstrings and docs 2021-06-30 14:30:55 +02:00
Eren Gölge 5723eb4738 Fix config init in `process_args` 2021-06-29 16:41:08 +02:00
Eren Gölge 4b5421b42f Remove FAQ link from README.md 2021-06-29 13:20:40 +02:00
Eren Gölge ae6405bb76 Docstrings for `Trainer` 2021-06-28 17:03:47 +02:00
Eren Gölge ab563ce7cd Start training by config.json using `register_config` 2021-06-28 17:03:47 +02:00
Eren Gölge fbba37e01e Fix loading the `amp` scaler from a checkpoint 🛠️ 2021-06-28 17:03:47 +02:00
Eren Gölge cfa5041db7 Fix `eval_log` for `gan.py` 🛠️ 2021-06-28 17:03:47 +02:00
Eren Gölge 5b89cb4fec Fixup `trainer.py` 🛠️ 2021-06-28 17:03:47 +02:00
Eren Gölge c7aad884cd Implement unified trainer 2021-06-28 17:03:19 +02:00
Eren Gölge c754a0e17d `TrainerAbstract` and related updates for `TrainerTTS` 2021-06-28 17:03:19 +02:00