Skip to content

Issues: intelligent-machine-learning/dlrover

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

57 Open 179 Closed
57 Open 179 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

DLRover - Flyte integration collaboration
#1275 opened Sep 19, 2024 by davidmirror-ops
2
missing elastic_training_pb2
#1266 opened Sep 12, 2024 by NiushanDong
1
Flash checkpoint does not support safetensors
#1263 opened Sep 10, 2024 by Alex-Ruan
Erros in dlrover, after pip installed the dlrover package
#1260 opened Sep 9, 2024 by Desperadoze
2
Can you create a dlrover arm64 image for Ascend NPU?
#1248 opened Aug 22, 2024 by xmarker
1
Question: How DLRover integrate with Llama Factory?
#1244 opened Aug 21, 2024 by hetingyou
1
What is the relationship with DLRover and Megatron? Can I integrate DLRover with Megatron with fault-tolerance and monitoring capabilities. How DLRover can recover from GPU offline problems with TP and PP needing to be reorganized?
#1243 opened Aug 19, 2024 by dotsonliu
1
while using megatron distributed flash-checkpoint to recovery, error ocurs when load_checkpoint
#1233 opened Aug 13, 2024 by deepcoldfish
Why model_optim_rng.pt is saved in a seperate directory?
#1225 opened Aug 2, 2024 by zhaoyang-star
7
easydl/elasticjob-controller:master image pull error
#1222 opened Aug 1, 2024 by xywangbuaa
1
scale down allreduct pytorch job won't complete and report error
#1215 opened Jul 29, 2024 by cocodee
1
When performing multi-node, multi-GPU training with Megatron-LM, if the 'rank' is only input in the startup script and not set in the environment variables, an exception may occur (stroagetype is disk)
#1208 opened Jul 23, 2024 by lkq51
2
[Error] When using deepspeed to start a megatron training task, only rank 0 of the flash checkpoint saves the model
#1199 opened Jul 17, 2024 by liangxuZhang
1
Error encountered while using flash attention in TensorFlow
#1180 opened Jul 8, 2024 by monatis
Worker pod stuck in Pending state causing TimeoutError and incorrect handling by master
#1175 opened Jul 2, 2024 by TheAriaYang
1
How to use the elasticity and fault tolerance in a Volcano job.
#1172 opened Jun 30, 2024 by workingloong
1
xpu timer python package
#1159 opened Jun 17, 2024 by zxyyzx
3
megatron-lm flash-ckpt can not save ckpt to disk when use pipeline parallel
#1146 opened May 29, 2024 by Lzhang-hub
6
Error encountered when using falsh checkpoint
#1144 opened May 29, 2024 by chencjcj
1
example failed: examples/tensorflow/criteo_deeprec/manual_job.yaml
#1136 opened May 21, 2024 by jason-i-vv
1
Incomplete save of ckpt files
#1135 opened May 21, 2024 by husky23333
3
[observability] OTEL Trace/Event for training rendezvous, gpu check, flash checkpoint, etc. enhancement New feature or request
#1132 opened May 17, 2024 by liyzcj
make deploy IMG=easydl/elasticjob-controller:master
#1115 opened May 11, 2024 by yangzhipeng1108
1
Use Gang Scheduling in ElasticJob of DLRover.
#1075 opened Apr 14, 2024 by workingloong Backlog
The job stops restarting workers and exits if the traceback is a code bug. enhancement New feature or request question Further information is requested
#1068 opened Apr 8, 2024 by workingloong Backlog
@BalaBalaYi
2
ProTip! Updated in the last three days: updated:>2024-09-27.