Replies: 5 comments 13 replies
-
|
try setting default to true for your text embed cache. it should actually use the sole text embed backend as the default, if none is explicitly defined. so there may be a bug, but as far as i've seen, i remember mine overriding it and warning me that it will do so. the default text embed backend is required, as it is where the validation prompts are cached and stored - and that is where this process is failing |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
ah, yeah, yours warns too:
but it's interesting that this still failed later. hmm |
Beta Was this translation helpful? Give feedback.
-
[
{
"id": "bloodimage-local",
"type": "local",
"instance_data_dir": "/store/CIA/scfc3/diffusion/SimpleTuner/TEST_DATA",
"crop": false,
"resolution": 768,
"resolution_type": "pixel",
"prepend_instance_prompt": true,
"instance_prompt": "a microscope image of a ",
"only_instance_prompt": false,
"caption_strategy": "filename",
"cache_dir_vae": "vae_cache",
"vae_cache_clear_each_epoch": true,
"probability": 1.0,
"repeats": 1,
"text_embeds": "alt-embed-cache"
},
{
"id": "alt-embed-cache",
"dataset_type": "text_embeds",
"default": true,
"type": "local",
"cache_dir": "text_embed_cache"
}
]try this, it has the one line corrected. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Thank you! However, including the suggested lines gives another even longer error message. |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
see in |
Beta Was this translation helpful? Give feedback.
-
|
to be honest, i'm not sure how Spack is involved here. and i never use miniconda, simply poetry and venvs. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you!! I had to change both |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, I want to use a folder structure like this:
and I want the text input to the model to be: and I only get cached results under the Full error message (when I have images that are under subfolders) |
Beta Was this translation helpful? Give feedback.
9 replies
-
|
Also, the lr wandb end the tqdm bar says I have do not correspond to the lr I try to use: But the info says: and I use However, I don't know if this was the case before the new push or not, but I assume it was because my loss was around 1 then as well. |
Beta Was this translation helpful? Give feedback.
-
|
i've popped a check in to the main branch for the subfolders element length. i had a bunch of other changes in another branch for the subfolder name prepending to the caption, and it is indeed as tricky to implement as i thought. i'm not sure that one will happen |
Beta Was this translation helpful? Give feedback.
-
|
which LR scheduler are you using? |
Beta Was this translation helpful? Give feedback.
-
|
ah, nvm. i see 'constant' with 3 warmup steps. not sure why the LR is so high there 🤔 |
Beta Was this translation helpful? Give feedback.
-
|
i assume you use Adafactor optimizer, which is pretty complicated and can be set up in multiple ways. SimpleTuner only has one means of it implemented, which is using relative_step and auto-optimizing the learning rate. there's another which is to use it as a drop-in for AdamW but that didn't seem to work very well in testing. i'm not sure why the loss value is so high, but maybe the learning rate can be retrieved more accurately. |
Beta Was this translation helpful? Give feedback.
-
|
well since it seems the original issue is resolved and i'm not clear on the settings or remaining problems, please open an issue report with some new logs and your complete |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Thanks for an excellent repo!
I have been struggling to get training to work and would greatly appreciate guidance.
I try to train the XL model but run into the following problem:
Initially, I just want to use the file name as the input text to see if I can get things to work. (However, later I want to use the folder name instead of the file name.)
I've put 4 images in a folder called
TEST_DATAand I use the followingmultidatabackend.json.I appreciate any guidance you can provide to help get my training started. Thank you!
Beta Was this translation helpful? Give feedback.
All reactions