关于 llamafactory-cli train 和 torchrun 训练耗费时间以及效果均不同的疑惑

[Maydaytyh]

Reminder

• I have read the README and searched the existing issues.

System Info

• llamafactory version: 0.9.1.dev0
• Platform: Linux-5.15.0-105-generic-x86_64-with-glibc2.17
• Python version: 3.8.19
• PyTorch version: 2.3.0+cu121 (GPU)
• Transformers version: 4.44.2
• Datasets version: 2.21.0
• Accelerate version: 0.34.2
• PEFT version: 0.12.0
• TRL version: 0.9.6
• GPU type: NVIDIA RTX A6000
• vLLM version: 0.4.3

Reproduction

llamafactory-cli train

使用 llamafactory-cli train 时的配置文件如下

### model
model_name_or_path: meta-llama/Llama-2-7b-hf

### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: q_proj,v_proj

### dataset
template: llama2
dataset: **
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/**
logging_steps: 10
save_steps: 1000
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 5.0e-5
num_train_epochs: 100.0
lr_scheduler_type: cosine
warmup_ratio: 0.0
ddp_timeout: 180000000
fp16: true

训练时间所耗费时间为

torchrun

使用 torchrun 参数设置为

CUDA_VISIBLE_DEVICES=3,4,5,6 torchrun  --standalone --nnodes=1 --nproc-per-node=4 new_LLMs/LLaMA/src/train_bash.py --stage sft --model_name_or_path meta-llama/Llama-2-7b-hf --do_train --dataset_dir LLMs/data --dataset ** --template llama2 --finetuning_type lora --lora_target q_proj,v_proj --output_dir **/checkpoint --overwrite_cache --per_device_train_batch_size 4 --gradient_accumulation_steps 4 --lr_scheduler_type cosine --logging_steps 10 --save_steps 1000 --learning_rate 5e-5 --num_train_epochs 100.0 --plot_loss --fp16

运行时间为

Expected behavior

为何两者运行时间和效果都不同呢，是哪里的设置有问题吗？
感谢！

Others

No response