v1.0 the total recall edition

Everything has changed! And yet, nothing has. Some defaults may. No, will - be different. It's hard to know which ones.

For those who can do so, it's recommended to use configure.py to reconfigure your environment on this new release.

It should go without saying, but for those in the middle of a training run, do not upgrade to this release until you finish.

Refactoring and Enhancements:

1. Refactor train.py into a Trainer Class:

   • The core logic of train.py has been restructured into a Trainer class, improving modularity and maintainability.
   • Exposes an SDK for reuse elsewhere.

2. Model Family Unification:

   • References to specific model types (--sd3, --flux, etc.) have been replaced with a unified --model_family argument, streamlining model specification and reducing clutter in configurations.

3. Configuration System Overhaul:

   • Switched from .env configuration files to JSON (config.json), with multiple backends supporting JSON configuration loading. This allows more flexible and readable configuration management.
   • Updated the configuration loader to auto-detect the best backend when launching.

4. Enhanced Argument Handling:

   • Deprecated old argument references and moved argument parsing to helpers/configuration/cmd_args.py for better organization.
   • Introduced support for new arguments such as --model_card_safe_for_work, --flux_schedule_shift, and --disable_bucket_pruning.

5. Improved Hugging Face Integration:

   • Modified configure.py to avoid asking for Hugging Face model name details unless required.
   • Added the ability to pass the SFW (safe-for-work) argument into the training script.

6. Optimizations and Bug Fixes:

   • Fixed several references to learning rate (lr) initialization and corrected --optimizer usage.
   • Addressed issues with attention masks swapping and fixed the persistence of text encoders in RAM after refactoring.

7. Training and Validation Enhancements:

   • Added better dataset examples with support for multiple resolutions and mixed configurations.
   • Configured training scripts to disable gradient accumulation steps by default and provided better control over training options via the updated config files.

8. Enhanced Logging and Monitoring:

   • Improved the handling of Weights & Biases (wandb) logs and updated tracker argument references.

9. Documentation Updates:

   • Revised documentation to reflect changes in model family handling, argument updates, and configuration management.
   • Added guidance on setting up the new configuration files and examples for multi-resolution datasets.

10. Miscellaneous Improvements:

    • Enabled support for NSFW tags in model cards enabled by default.
    • Updated train.sh to minimal requirements, reducing complexity and streamlining the training process.

More detailed change log

• lycoris model card updates by @bghira in #820
• Generate and store attention masks for T5 for flux by @AmericanPresidentJimmyCarter in #821
• Fix validation by @AmericanPresidentJimmyCarter in #822
• backwards-compatible flux embedding cache masks by @bghira in #823
• merge by @bghira in #824
• parquet add width and height columns by @frankchieng in #825
• quanto: remove warnings about int8/fp8 confusion as it happened so long ago now; add warning about int4 by @bghira in #826
• remove clip warning by @bghira in #827
• update lycoris to dev branch, 3.0.1dev3 by @bghira in #828
• Fix caption_with_blip3.py on CUDA by @anhi in #833
• fix quanto resuming by @bghira in #834
• lycoris: resume should use less vram now by @bghira in #835
• (#644) temporarily block training on multi-gpu setup with quanto + PEFT, inform user to go with lycoris instead by @bghira in #837
• quanto + deepspeed minor fixes for multigpu training by @bghira in #839
• deepspeed sharding by @bghira in #840
• fix: only run save full model on main process by @ErwannMillon in #838
• merge by @bghira in #841
• clean-up by @bghira in #842
• follow-up fixes for quanto limitation on multigpu by @bghira in #846
• merge by @bghira in #850
• (#851) remove shard merge code on load hook by @bghira in #853
• csv backend updates by @williamzhuk in #645
• csv fixes by @bghira in #856
• add schedulefree optim w/ kahan summation by @bghira in #857
• merge by @bghira in #858
• merge by @bghira in #861
• schedulefree: return to previous stable settings and add a new preset for aggressive training by @bghira in #862
• fix validation image filename only using resolution from first img, and, unreadable/untypeable parenthesis by @bghira in #863
• (#519) add side by side comparison with base model by @bghira in #865
• merge fixes by @bghira in #870
• (#864) add flux final export for full tune by @bghira in #871
• wandb gallery mode by @bghira in #872
• sdxl: dtype inference followup fix by @bghira in #873
• merge by @bghira in #878
• combine the vae cache clear logic with bucket rebuild logic by @bghira in #879
• flux: mobius-style training via augmented guidance scale by @bghira in #880
• track flux cfg via wandb by @bghira in #881
• multigpu VAE cache rebuild fixes; random crop auto-rebuild; mobius flux; json backend now renamed to discovery ; wandb guidance tracking by @bghira in #888
• fixing typo in flux document for preserve_data_backend_cache key by @riffmaster-2001 in #882
• reintroduce timestep dependent shift as an option during flux training for dev and schnell, disabled by default by @bghira in #892
• adding SD3 timestep-dependent shift for Flux training by @bghira in #894
• fix: set optimizer details to empty dict w/ deepspeed by @ErwannMillon in #895
• fix: make sure wandb_logs is always defined by @ErwannMillon in #896
• merge by @bghira in #900
• Dataloader Docs - Correct caption strategy for instance prompt by @barakyo in #902
• refactor train.py into Trainer class by @bghira in #899
• Update TRAINER_EXTRA_ARGS for model_family by @barakyo in #903
• Fix text encoder nuking regression by @mhirki in #906
• added lokr lycoris init_lora by @flotos in #907
• Fix Flux schedule shift and add resolution-dependent schedule shift by @mhirki in #905
• Swap the attention mask location, because Flux swapped text and image… by @AmericanPresidentJimmyCarter in #908
• support toml, json, env config backend, and multiple config environments by @bghira in #909
• Add "none" to --report_to argument by @twri in #911
• Add support for tiny PEFT-based Flux LoRA based on TheLastBen's post on Reddit by @mhirki in #912
• Update lycoris_config.json.example with working defaults by @mhirki in #918
• fix constant_with_warmup not being so constant or warming up by @bghira in #919
• follow-up fix for setting last_epoch by @bghira in #920
• fix multigpu schedule issue with LR on resume by @bghira in #921
• multiply the resume state step by the number of GPUs in an attempt to overcome accelerate v0.33 issue by @bghira in #922
• default to json/toml before the env file in case multigpu is configured by @bghira in #923
• fix json/toml configs str bool values by @bghira in #924
• bypass some "helpful" diffusers logic that makes random decisions to run on CPU by @bghira in #925
• v1.0 merge by @bghira in #910
• doc updates by @bghira in #927
• merge by @bghira in #928

New Contributors

Christ, that's a lot of new people. Welcome!

• @anhi made their first contribution in #833
• @ErwannMillon made their first contribution in #838
• @williamzhuk made their first contribution in #645
• @riffmaster-2001 made their first contribution in #882
• @barakyo made their first contribution in #902
• @flotos made their first contribution in #907
• @twri made their first contribution in #911

Full Changelog: v0.9.8.3.2...v1.0