Best practice for training LLaMA models in Megatron-LM
-
Updated
Jan 2, 2024 - Python
#
megatron-lm
Here are 11 public repositories matching this topic...
Annotations of the interesting ML papers I read
nlp machine-learning deep-learning transformers gpt research-paper bert gpt-2 xlnet annotated-paper megatron-lm papers-annotations
-
Updated
Nov 10, 2024
Super-Efficient RLHF Training of LLMs with Parameter Reallocation
distributed-systems reinforcement-learning distributed-computing transformers large-scale-machine-learning deepspeed megatron-lm large-language-models llm reinforcement-learning-from-human-feedback llm-training llm-framework
-
Updated
Dec 15, 2024 - Python
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
transformers moe data-parallelism distributed-optimizers model-parallelism megatron mixture-of-experts pipeline-parallelism huggingface-transformers megatron-lm tensor-parallelism large-scale-language-modeling 3d-parallelism zero-1 sequence-parallelism
-
Updated
Dec 14, 2023 - Python
Odysseus: Playground of LLM Sequence Parallelism
-
Updated
Jun 17, 2024 - Python
A LLaMA1/LLaMA12 Megatron implement.
-
Updated
Dec 13, 2023 - Python
Training NVIDIA NeMo Megatron Large Language Model (LLM) using NeMo Framework on Google Kubernetes Engine
-
Updated
Nov 19, 2024 - HCL
Megatron-LM/GPT-NeoX compatible Text Encoder with 🤗Transformers AutoTokenizer.
-
Updated
Nov 16, 2023 - Python
Running Large Language Model easily.
-
Updated
Dec 19, 2024 - Jupyter Notebook
Minimal yet high performant code for pretraining llms. Attempts to implement some SOTA features. Implements training through: Deepspeed, Megatron-LM, and FSDP. WIP
-
Updated
Feb 6, 2024 - Python
Improve this page
Add a description, image, and links to the megatron-lm topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the megatron-lm topic, visit your repo's landing page and select "manage topics."