Skip to content

Commit

Permalink
add qlora
Browse files Browse the repository at this point in the history
  • Loading branch information
lwaekfjlk committed Oct 11, 2023
1 parent 26a64e5 commit fad73a7
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 0 deletions.
Empty file added llm_ft/data/dummyfile
Empty file.
25 changes: 25 additions & 0 deletions llm_ft/dummy_qlora.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
deepspeed fastchat/train/train_lora.py \
--model_name_or_path ./vicuna-7b-1.5 \
--lora_r 8 \
--lora_alpha 16 \
--lora_dropout 0.05 \
--data_path ./data/dummy_conversation.json \
--bf16 True \
--output_dir ./checkpoints \
--num_train_epochs 3 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 1200 \
--save_total_limit 100 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--tf32 True \
--model_max_length 2048 \
--q_lora True \
--deepspeed playground/deepspeed_config_s2.json

0 comments on commit fad73a7

Please sign in to comment.