"Expectations" section in the Quick Start Guide?

[OnePawProductions]

Hello!

I've been doing SD for a few months now, and come from a background in the arts, with only a passing familiarity for the technical aspects of SD. I'm learning as fast as I can, but often find myself not knowing if the problems I'm having are from things I've done wrong, incorrect expectations, or some other reason.

Would it be possible to have a "Reasonable Expectations" section in the guide, to help people differentiate troubleshooting?

Here's what I mean:

-An average training session should take between X minutes up to X hours

• images should look like X around 10 steps, 20 steps, 30 steps, etc
• if images look like X, you probably set it up wrong, or have an error in the console

Specifically, I'm getting output like this:

And I can't quite figure out if I'm sampling too early, if I've done something wrong, or if it's because there's an error in Terminal.

I'm training a LORA for SDXL using about 100 images that have captioned .txt files with hand-written prompts

Usually, I see that sort of noise when there's been an egregious error. I've also done embeddings, though, and the first few seem to be quite messy, up until about step 500 or so. But I have no frame of reference for LORA, other than my SD and Embed experince

[OnePawProductions]

Been poking around, trying to figure it out. FIgured I'd post it here? If it's better posted in a different place, let me know.
Images like this:

For me, were associated with a mismatch in Resolution. As I was training SDXL, setting the Sample resolution to 512x512 confused the model, and gave noise, instead of image output.

Now, I'm getting images. However, the images are unrelated to the dataset, even several epoch's into training. I'm not sure what I'm doing wrong, but if its ok, I figured this might be a good place to document my struggles. Once I figure it out, perhaps it could help others!

I will have images in the dataset like this:

But receive output from the sampler like this:

Madness..

Config is as follows:
"training_method": "LORA", "model_type": "STABLE_DIFFUSION_XL_10_BASE", "debug_mode": false, "debug_dir": "debug", "workspace_dir": "workspace7/run", "cache_dir": "workspace7-cache/run", "tensorboard": false, "continue_last_backup": false, "base_model_name": "C:/Auto1111.v3/webui/models/Stable-diffusion/Worm Color Nv_SDXL_base_1.0.safetensors", "weight_dtype": "FLOAT_16", "output_dtype": "FLOAT_32", "output_model_format": "SAFETENSORS", "output_model_destination": "C:/Auto1111.v3/webui/models/Lora/Custom/THvt7", "gradient_checkpointing": true, "concept_file_name": "training_concepts/Taylor Hebert.json", "circular_mask_generation": false, "random_rotate_and_crop": false, "aspect_ratio_bucketing": false, "latent_caching": true, "latent_caching_epochs": 1, "clear_cache_before_training": true, "learning_rate_scheduler": "CONSTANT", "learning_rate": 0.0003, "learning_rate_warmup_steps": 200, "learning_rate_cycles": 1, "epochs": 100, "batch_size": 4, "gradient_accumulation_steps": 1, "ema": "OFF", "ema_decay": 0.999, "ema_update_step_interval": 5, "train_device": "cuda", "temp_device": "cpu", "train_dtype": "FLOAT_16", "only_cache": false, "resolution": "1024", "attention_mechanism": "DEFAULT", "align_prop": false, "align_prop_probability": 0.1, "align_prop_loss": "AESTHETIC", "align_prop_weight": 0.01, "align_prop_steps": 20, "align_prop_truncate_steps": 0.5, "align_prop_cfg_scale": 7.0, "train_unet": true, "train_unet_epochs": 100000, "unet_learning_rate": null, "offset_noise_weight": 0.0, "perturbation_noise_weight": 0.0, "rescale_noise_scheduler_to_zero_terminal_snr": false, "force_v_prediction": false, "force_epsilon_prediction": false, "max_noising_strength": 1.0, "unet_weight_dtype": "NONE", "train_prior": true, "train_prior_epochs": 10000, "prior_learning_rate": null, "prior_weight_dtype": "NONE", "train_text_encoder": false, "train_text_encoder_epochs": 30, "text_encoder_learning_rate": null, "text_encoder_layer_skip": 0, "text_encoder_weight_dtype": "NONE", "train_text_encoder_2": false, "train_text_encoder_2_epochs": 30, "text_encoder_2_learning_rate": null, "text_encoder_2_layer_skip": 0, "text_encoder_2_weight_dtype": "NONE", "vae_weight_dtype": "FLOAT_32", "effnet_encoder_model_name": "", "effnet_encoder_weight_dtype": "NONE", "decoder_model_name": "", "decoder_weight_dtype": "NONE", "decoder_text_encoder_weight_dtype": "NONE", "decoder_vqgan_weight_dtype": "NONE", "masked_training": false, "unmasked_probability": 0.1, "unmasked_weight": 0.1, "normalize_masked_area_loss": false, "embedding_model_names": [], "token_count": 1, "initial_embedding_text": "*", "embedding_weight_dtype": "FLOAT_32", "lora_model_name": "", "lora_rank": 16, "lora_alpha": 1.0, "lora_weight_dtype": "FLOAT_32", "optimizer": "ADAMW", "optimizer_adam_w_mode": false, "optimizer_alpha": null, "optimizer_amsgrad": false, "optimizer_beta1": 0.9, "optimizer_beta2": 0.999, "optimizer_beta3": null, "optimizer_bias_correction": false, "optimizer_block_wise": false, "optimizer_capturable": false, "optimizer_centered": false, "optimizer_clip_threshold": null, "optimizer_d0": null, "optimizer_d_coef": null, "optimizer_dampening": null, "optimizer_decay_rate": null, "optimizer_decouple": false, "optimizer_differentiable": false, "optimizer_eps": 1e-08, "optimizer_eps2": null, "optimizer_foreach": false, "optimizer_fsdp_in_use": false, "optimizer_fused": true, "optimizer_growth_rate": null, "optimizer_initial_accumulator_value": null, "optimizer_is_paged": false, "optimizer_log_every": null, "optimizer_lr_decay": null, "optimizer_max_unorm": null, "optimizer_maximize": false, "optimizer_min_8bit_size": null, "optimizer_momentum": null, "optimizer_nesterov": false, "optimizer_no_prox": false, "optimizer_optim_bits": null, "optimizer_percentile_clipping": null, "optimizer_relative_step": false, "optimizer_safeguard_warmup": false, "optimizer_scale_parameter": false, "optimizer_use_bias_correction": false, "optimizer_use_triton": false, "optimizer_warmup_init": false, "optimizer_weight_decay": 0.01, "sample_definition_file_name": "training_samples/sampleTHvt1.json", "sample_after": 1, "sample_after_unit": "EPOCH", "sample_image_format": "JPG", "backup_after": 1, "backup_after_unit": "EPOCH", "rolling_backup": false, "rolling_backup_count": 3, "backup_before_save": true, "save_after": 0, "save_after_unit": "NEVER"

[sangoi-exe]

Hello, try training only the UNet, without the text encoder (this suggestion is specifically for the SDXL).

[OnePawProductions]

No commandline arguments. Not sure if commandline arguments are a thing with OneTrainer

[OnePawProductions]

I'm ecstatic to be talking to someone who might know more than I do, but I gotta go get my dinner, so I'll be afk off and on, but I'm very happy to be talking to you, so I'll be checking in as I can

[sangoi-exe]

Okay, first thing, set the batch size to 1, SDXL is really heavy, I also have a 3060 12GB.

Second thing, change the optimizer to dadapt lion and set the main learning rate to 1, and leave the learning rate for the unet blank.

Check if the base model is really float 16 (if it's 2GB, it's very likely that it's f16), and set the output to f16 as well.

You can try turning off gradient checkpointing too, but check the VRAM usage in the task manager when you start training, if it starts using shared memory, reactivate the gradient checkpoint.

[OnePawProductions]

That's all really great info, I'll try it! Thanks!

I'll let you know if that works!

[OnePawProductions]

I don't want to call it too soon, but the data quality has definitely shifted alot.

She's still covered in fish, for some reason. But at least at Epoch 2 now, there's a recognizable person with dark curly hair and a serious look on her face!

[OnePawProductions]

I realized I had also incorrectly checked where the prompts were located. It's now set to read them from each image file, ie, "from image file name"

That was apparently also not it, as I'm still getting unrelated output:

[OnePawProductions]

Does that mean I need to train the text encoders if I added prompt captions to every image, ie, each image labeled with something like:
Taylor Hebert, wearing a brown shirt, long dark curly hair

Also, with this iteration, instead of selecting "From Image File Name" under concepts, I used A1111 Dataset Tag Editor to generate a matching .txt file from those captions, and set it to "From Text File Per Sample"

So I"m not sure if it's your advice, that change, or a combination of the two

[OnePawProductions]

YESS!!!! DevArqSangoi, you are WONDERFUL! Thank you!

It Is working now!!!!! Each sampler prompt is putting out variations on the concepts being trained ("Taylor Hebert", "Young Taylor", "Adult Taylor", "Taylor Hebert, perfect"). Even at Epoch 3, I can see the image variance between the concepts!

Fantastic!

I'm gonna post here my understanding of what you said that seemed to make the difference. Please correct me if I'm wrong, that's the only way to learn, after all!

Okay, first thing, set the batch size to 1, SDXL is really heavy, I also have a 3060 12GB.

this helps speed it up....by limiting the amount of (produced? sampled?) images per (step? sampler output?)

Second thing, change the optimizer to dadapt lion and set the main learning rate to 1, and leave the learning rate for the unet blank.

Optimizer changes how...something important, I assume. I can probably google that. Main learning rate is tied to optimizer, I'm guessing. More Googling.

Check if the base model is really float 16 (if it's 2GB, it's very likely that it's f16), and set the output to f16 as well.

Helps with speed, in exchange for a small drop in quality

---

Ah, just in the time i wrote this, another epoch complete! Thank you, sir!

If what I wrote is wrong, please correct me. It you wanna wax poetic about what all these things do in detail, I would love to listen. If not, I can add it to my stack of Things I Must Google. Thanks for pointing me in the right direction!

[sangoi-exe]

That's great that you managed to do it!

The text encoder is the 'translator' of the prompts for the Unet; it learns the relationship between the images and the training weights. When you create a prompt to draw something, it's the text encoder that 'explains' to the Unet what should be generated.

this helps speed it up....by limiting the amount of (produced? sampled?) images per (step? sampler output?)

When a neural network is trained, the batch size is the number of images it 'sees' in each step to calculate the gradient and update the weights. So, if the batch size is 4, it sees 4 images and calculates the average gradient of these 4.

But everything involving neural network training consumes VRAM, so a batch size of 1 uses less than a batch size of 4. Since VRAM is limited, in our case to 12GB, to have an effectively fast training, the trainer should not consume more than this during training, because if it exceeds the total VRAM of the GPU, it will start using RAM (NVIDIA recently released a driver that makes it possible to disable this shared memory, which I prefer, so if the training consumes more VRAM than is available, it will cause an error), which is much slower compared to VRAM. Just to give you an idea, a reasonable time for each step is about 2 seconds.

Optimizer changes how...something important, I assume. I can probably google that. Main learning rate is tied to optimizer, I'm guessing. More Googling.

The optimizer is the function that performs the calculation of the gradient based on the loss function, and there are various ways to do this calculation.

In the case of AdamW, you set a fixed learning rate and it does the rest internally. In the case of Dadapt_Lion, you set the learning rate to 1 and it takes care of dynamically adjusting the learning rate as necessary (according to the loss function).

Helps with speed, in exchange for a small drop in quality
F16 or f32 refers to the floating-point precision of the values, F32 has more precision than F16.

To put it simply, let's say F32 can store 20 digits after the decimal point of pi, and F16 stores 10. In summary, more precision means more fidelity in the training.

Good luck with the training! The tedious part has just started, lol. 😂

[OnePawProductions]

Alrighty, so it goes:
{Images--(batch size)-->Optimizer via text encoder: "ok, so what this means is..."-->UNET} which is "gradient accumulation" ?

Holy Hannah, DevArqSangoi, IT WORKED~! I was telling myself "ok, it'll probably be hot mess garbage, that's ok, it's your first try, lets see what work there is to do next, but..." It is viable on the first try!! Dude, you are my hero! I've been trying to figure out how to do an SDXL LORA for 6 months. I lack the technical knowledge to parse most tutorials, and those that I could figure out, are for SD1.5. (I think I should write a guide when I'm done for people who wouldn't know a CSS sheet from a .yaml file, let alone how to use either one. It's where I started from, so maybe I can help people avoid my pitfalls.)

I see that there's alot of work to do. I can now use this LORA to start accumulating correlative effects without contaminating the prompt! (for example: Taylor in a hoodie will always wear glasses. But trying to pair "Glasses" with "Hoodie" in an SDXL prompt invariably leads to prompt leak of the "youtuber" concept, especially "east asian cute girl youtuber". But with the LORA, her face is baked in, so I should be able to either generate new sets without the leak, or use the existing images with retouch img2img//ControlNet IP Adapt txt2img to generate newer, better, v5 dataset.

But look! I tried getting an image of a Secret Service AU Taylor Hebert for days, but with the promping so highly weighted towards her face, relatively little attention was paid to the rest of the scene. I like to run gens of fanart while I'm reading a fanfic that I love. But hot damn, if this doesn't look like the scene in Recoil, where she's standing on the stage at Winslow High, facing down Maquis! PERFECT

I have so much wonderful, wonderful work to do! I love the tedious process of refinement, on all different projects. It's like sculpting, but in a digital medium. Meditative, and fun. What a puzzle! Delightful

THANK YOU! <3

[sangoi-exe]

It's funny how things are.

Until a week ago, I used Kohya to train LoRAs, I didn't even know about OneTrainer. Then, I decided to create a little tutorial on Reddit to help people with the issue of configuring parameters and VRAM overflow, which make training ridiculously slow, and people didn't know why.

Someone mentioned OneTrainer on that thread, and I came to check it out. I really liked it because the UI is made by the same person who created the scripts, creating better synergy, plus there's the option to create backups and pause/restart training whenever I want, among many other advantages, which made me switch to it.

So, after all this, I ended up here in the discussions to suggest a feature and saw your post, and I realized it was a simple error for me, but for those without experience, it's a struggle, haha.

I'm from IT, so everything is easier for me in this area, and I've absorbed a tremendous amount of knowledge about neural networks since I started dealing with SD, and it's not even been a year since I started.

Make good use of the information, I'm always happy to help (without making those long, shit YouTube videos) people who have similar problems to what I had at the beginning.

I know how satisfying it is when we manage to generate an image that we really want, especially as a result of our own effort training a model. I feel this almost daily with my tests and everything else. 😂

[OnePawProductions]

I hear ya. I find the OneTrainer UI to be just fantastic. Cards, Tabs, and Accordians/Dropdowns--that's my jam!

Especially the Concepts cards. I love having the graphical representation of each folder put on display, and being able to mess with saturation automatically is really important to me (I render cover art in color, but chapter art in grayscale sketch style).

I am definitely a written tutorial kind of person. How people can learn anything from a talking head on YouTube is completely beyond me.

I would absolutely love to hit you up from time to time. I've been offering the same to beginners on Reddit. People ask questions because they just downloaded it, and are stuck, because they lack the skills to understand how to understand how to make it work. I may not know it all, but I definitely know about some of the basics!

For sure, one person's week-long struggle could be another person's 2 minute explanation! That's why it's always good to ask around :D

[hyppyhyppo]

Hi,
I'm also a written tutorial person. And it's a great idea to contribute on OneTrainer documentation, but know that one exist already in the Wiki this wiki is open for suggestions and contribution. I've initiated it but it's lacking some information, VRAM optimization, optimizer correct usage, when text and unet are required and what for..., whatever seams usefull.
It's also open for edition. I'd suggest you create a page in it and put your tutorial there so the documentation is centralized somewhere and we could point people to it.
There is also a discussion on OneTrainer Discord Wiki-Discussion where we could talk on its organization and enhacement.
Any contribution and good ideas are welcome.

[sangoi-exe]

@OnePawProductions nice! When you need something, just tag me on the post and I will try to help. 😁

[hyppyhyppo]

Thanks,
But I'd suggest you to wrap up your information in a FAQ or tIps and tricks style as a beguinning, best would be join the Discord and talk on the Wiki-discussion. The community is there and they also could benefit of it or give valuable feedbacks.
Q: what are text encoder and unet, when to use them?
Tip: VRAM optimization
Tip: Captioning for LoRA (like being an apple or not an apple)

[OnePawProductions]

Those are great ideas! That's a good way to go I think!

How professional do the FAQ and tips have to be? On the one hand, I'll be taking my own notes, and I'd love to share those with other people, but on the other, rambling musings on the nature of settings might not be very easy to read. Maybe I'm overthinking it? I'd like to contribute, I'm just not sure how yet

But the Discord thing sounds like a good idea! And if/when I have something more formal, contributing to the Wiki.

Can...can you contribute small bits to the Wiki? Like...a paragraph of the one thing you understand really well, or going in and adding a bit at a time?

[hyppyhyppo]

Yes, it can done pieces by pieces. The FAQ and the lessons learnt sessions are made this way by collecting the frequent questions and tips shared by users.
The others sections are intended to stand more for a general documentation.
Thanks for your future contribution !

[Aamir3d]

This is a great discussion, thank you @DevArqSangoi and @OnePawProductions for contributing to this.

OneTrainer is great, and looks straightforward, but there are certain differences when you're used to Kohya/Dreambooth.

[elen07zz]

Hi guys, im also having some troubles to make this work, could you share your .json preset?

These were my results, terrible and meaningless:
https://imgur.com/a/sTHqJCK

This was my preset:
14_5GB_Best_Settings.json

[OnePawProductions]

Looking over your uploads now. My guess, from looking at the output, is that the ai is confused as heck. Do you have any more output from further along in the process? Like, early sampling (less that 10 epoch), mid (greater than 10 but less that 50), late (greater than 50)?

For example, in my Mascot Muse LORA, which worked very well and I'm now on v3, I have a sampling prompt of "Muse Jumping". Early output has...the rock band Muse...jumping. Lol. By mid, I began to get cohesion: Muse....but they're also cats!. By late, I get what I really want: Mascot Muse, the pink calico cartoon cat, jumping in a cute lil' pose. In this way, I can see that the Ai is confused at first, but then sees my data over the epochs, and learns

However! If the Ai is confused and not seeing data, you would not see that happy progression. If the metrics are off, you'll not see much progression at all, just noise. Such as you had before.

Here's what I see from your JSON. I'll explain the why, if I know it, but alot of it I don't yet fully understand:

{
"training_method": "FINE_TUNE", ((here you are fine-tuning an entire model, not creating a LORA))
"model_type": "STABLE_DIFFUSION_XL_10_BASE",
"debug_mode": false,
"debug_dir": "debug",
"workspace_dir": "G:/OneTrainer/workspace",
"cache_dir": "G:/OneTrainer/cache",
"tensorboard": false,
"tensorboard_expose": false,
"continue_last_backup": false,
"base_model_name": "F:/0 models/sd_xl_base_1.0.safetensors",
"weight_dtype": "BFLOAT_16", ((try using float_16 instead of Bfloat. I don't know why, =P))
"output_dtype": "BFLOAT_16",
"output_model_format": "SAFETENSORS",
"output_model_destination": "G:/automatic1111/stable-diffusion-webui/models/Stable-diffusion/one_train.safetensors",
"gradient_checkpointing": true,
"concept_file_name": "training_concepts/concepts.json",
"circular_mask_generation": false,
"random_rotate_and_crop": false,
"aspect_ratio_bucketing": false,
"latent_caching": true,
"clear_cache_before_training": true,
"learning_rate_scheduler": "CONSTANT",
"learning_rate": 1e-05, ((I did a learning rate of 1, with the Optimizer DADAPT_LION))
"learning_rate_warmup_steps": 200,
"learning_rate_cycles": 1,
"epochs": 500, ((wowie, alot of epochs. Why so high? You might be getting overbaked. Try 100))
"batch_size": 1,
"gradient_accumulation_steps": 1,
"ema": "OFF",
"ema_decay": 0.999,
"ema_update_step_interval": 1, ((I did 5. I don't know why, but it worked. =P))
"train_device": "cuda",
"temp_device": "cpu",
"train_dtype": "BFLOAT_16", ((try float_16. I don't know why =P))
"only_cache": false,
"resolution": "1024",
"attention_mechanism": "DEFAULT", ((I did XFORMERS. I don't actually use XFORMERS in A111. but it worked alright. Don't know this, either ))
"align_prop": false,
"align_prop_probability": 0.1,
"align_prop_loss": "AESTHETIC",
"align_prop_weight": 0.01,
"align_prop_steps": 20,
"align_prop_truncate_steps": 0.5,
"align_prop_cfg_scale": 7.0,
"mse_strength": 1.0, ((I don't even have these in my JSON. I wonder what they do? Try turning them off?))
"mae_strength": 0.0,
"loss_scaler": "NONE",
"learning_rate_scaler": "NONE",
"train_unet": true,
"train_unet_epochs": 100000,
"unet_learning_rate": 1e-05, ((I set mine to null. Don't remember why))
"offset_noise_weight": 0.0,
"perturbation_noise_weight": 0.0,
"rescale_noise_scheduler_to_zero_terminal_snr": false,
"force_v_prediction": false,
"force_epsilon_prediction": false,
"max_noising_strength": 1.0,
"unet_weight_dtype": "BFLOAT_16", ((I set this to None))
"train_prior": true,
"train_prior_epochs": 10000,
"prior_learning_rate": null,
"prior_weight_dtype": "NONE",
"train_text_encoder": true, (( I did not train the text encoder))
"train_text_encoder_epochs": 500,
"text_encoder_learning_rate": 3e-06,
"text_encoder_layer_skip": 0,
"text_encoder_weight_dtype": "BFLOAT_16",
"train_text_encoder_2": false,
"train_text_encoder_2_epochs": 30,
"text_encoder_2_learning_rate": null,
"text_encoder_2_layer_skip": 0,
"text_encoder_2_weight_dtype": "BFLOAT_16",
"vae_weight_dtype": "FLOAT_32",
"effnet_encoder_model_name": "",
"effnet_encoder_weight_dtype": "NONE",
"decoder_model_name": "",
"decoder_weight_dtype": "NONE",
"decoder_text_encoder_weight_dtype": "NONE",
"decoder_vqgan_weight_dtype": "NONE",
"masked_training": false,
"unmasked_probability": 0.1,
"unmasked_weight": 0.1,
"normalize_masked_area_loss": false,
"embedding_model_names": [],
"token_count": 1,
"initial_embedding_text": "*",
"embedding_weight_dtype": "FLOAT_32",
"lora_model_name": "",
"lora_rank": 16,
"lora_alpha": 1.0,
"lora_weight_dtype": "FLOAT_32",
"optimizer": "ADAFACTOR", ((Ah, here's the optimizer. Try DADAPT_LION. Better for SDXL, I think))
"optimizer_adam_w_mode": false,
"optimizer_alpha": null,
"optimizer_amsgrad": false,
"optimizer_beta1": null,
"optimizer_beta2": 0.999,
"optimizer_beta3": null,
"optimizer_bias_correction": false,
"optimizer_block_wise": false,
"optimizer_capturable": false,
"optimizer_centered": false,
"optimizer_clip_threshold": 1.0, ((I set this to null))
"optimizer_d0": null, (( I set this to 1e-06))
"optimizer_d_coef": null,
"optimizer_dampening": null,
"optimizer_decay_rate": -0.8, ((null))
"optimizer_decouple": false,
"optimizer_differentiable": false,
"optimizer_eps": 1e-30, ((1e-08))
"optimizer_eps2": 0.001, ((null))
"optimizer_foreach": false,
"optimizer_fsdp_in_use": false,
"optimizer_fused": true,
"optimizer_growth_rate": null,
"optimizer_initial_accumulator_value": null,
"optimizer_is_paged": false,
"optimizer_log_every": null,
"optimizer_lr_decay": null,
"optimizer_max_unorm": null,
"optimizer_maximize": false,
"optimizer_min_8bit_size": null,
"optimizer_momentum": null,
"optimizer_nesterov": false,
"optimizer_no_prox": false,
"optimizer_optim_bits": null,
"optimizer_percentile_clipping": null,
"optimizer_relative_step": false,
"optimizer_safeguard_warmup": false,
"optimizer_scale_parameter": false,
"optimizer_use_bias_correction": false,
"optimizer_use_triton": false,
"optimizer_warmup_init": false,
"optimizer_weight_decay": 0.01, ((0.0, not the ascii emoji, just 0.0, lol))
"sample_definition_file_name": "training_samples/samples.json",
"sample_after": 5,
"sample_after_unit": "EPOCH",
"sample_image_format": "PNG", ((setting to JPEG will be faster, but shouldn't affect results))
"samples_to_tensorboard": true,
"non_ema_sampling": true,
"backup_after": 50,
"backup_after_unit": "NEVER",
"rolling_backup": false,
"rolling_backup_count": 3,
"backup_before_save": false,
"save_after": 30,
"save_after_unit": "EPOCH"
}

[elen07zz]

thank you, im going to try that, unfortunately I didn't early samplings so

[OnePawProductions]

You're welcome! I'll be poking around all day, so feel free to hit me up and I'll try and help

[OnePawProductions]

I find that setting the sampling to once per epoch, in a 100 epoch run, gives a good idea of convergence ratios.

[hyppyhyppo]

Btw, thanks OnePawProduction to your contribution to the Wiki. It's always valuable :)