Parallel Multi-GPU training?

[JimikoSK]

Figured out how to get int8-quanto working with mutliple GPUs thanks to old questions, but only one GPU is really being used, so kinda underutilized.

I get only one GPU used, even bumping the batch size up to 2, still just one. Is there any way to run them in parallel?

My settings are below:

Accelerate

compute_environment: LOCAL_MACHINE
debug: false
distributed_type: MULTI_GPU
#distributed_type: NO
downcast_bf16: 'no'
enable_cpu_affinity: true
#gpu_ids: "2"
gpu_ids: 1,2
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

config.json

{
    "--resume_from_checkpoint": "latest",
    "--data_backend_config": "config/multidatabackend.json",
    "--aspect_bucket_rounding": 2,
    "--seed": 42,
    "--minimum_image_size": 0,
    "--disable_benchmark": false,
    "--output_dir": "/mnt/output/SimpleTuner",
    "--lora_type": "lycoris",
    "--lycoris_config": "config/lycoris_config.json",
    "--num_train_epochs": 5,
    "--checkpointing_steps": 250,
    "--checkpoints_total_limit": 7,
    "--model_type": "lora",
    "--pretrained_model_name_or_path": "black-forest-labs/FLUX.1-dev",
    "--model_family": "flux",
    "--train_batch_size": 1,
    "--gradient_checkpointing": "true",
    "--caption_dropout_probability": 0.1,
    "--resolution_type": "pixel_area",
    "--resolution": "512",
    "--validation_seed": 42,
    "--validation_steps": 250,
    "--validation_resolution": "512x512",
    "--validation_guidance": 3.0,
    "--validation_guidance_rescale": "0.0",
    "--validation_num_inference_steps": "20",
    "--validation_prompt": "A cat with watermelon.",
    "--disable_tf32": "true",
    "--mixed_precision": "bf16",
    "--optimizer": "adamw_bf16",
    "--learning_rate": "1e-4",
    "--lr_scheduler": "polynomial",
    "--lr_warmup_steps": 100,
    "--base_model_precision": "int8-quanto",
    "--validation_torch_compile": "false"
}

mutlidatabackend.json

[
    {
        "id": "LORA_NAME",
        "type": "local",
        "instance_data_dir": "/mnt/dataset",
        "crop": false,
        "crop_style": "random",
        "minimum_image_size": 128,
        "resolution": 512,
        "resolution_type": "pixel_area",
        "repeats": 10,
        "metadata_backend": "discovery",
        "caption_strategy": "filename",
        "cache_dir_vae": "cache//vae-512"
    },
    {
        "id": "text-embed-cache",
        "dataset_type": "text_embeds",
        "default": true,
        "type": "local",
        "cache_dir": "cache//text"
    }
]

+-----------------------------------------+----------------------+----------------------+
|   1  Tesla P40                      On  | 00000000:21:00.0 Off |                  Off |
| N/A   24C    P8               9W / 150W |    150MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   2  Tesla P40                      On  | 00000000:2D:00.0 Off |                  Off |
| N/A   41C    P0              88W / 150W |  14022MiB / 24576MiB |    100%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    1   N/A  N/A     11412      C   ...iconda3/envs/simpletuner/bin/python      148MiB |
|    2   N/A  N/A     11444      C   ...iconda3/envs/simpletuner/bin/python    14020MiB |
+---------------------------------------------------------------------------------------+