Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding the new feature of FPDT #441

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Adding the new feature of FPDT #441

wants to merge 8 commits into from

Conversation

YJHMITWEB
Copy link

@YJHMITWEB YJHMITWEB commented Aug 29, 2024 •
edited by samadejacobs
Loading

FPDT can only be work with this version of DeepSpeed.

@YJHMITWEB YJHMITWEB mentioned this pull request Aug 29, 2024
Open
@delock
Copy link

delock commented Aug 30, 2024

Hi @YJHMITWEB , is FPDT referring to this paper? https://ui.adsabs.harvard.edu/abs/2023JARS...17b6510H/abstract

@tohtana
Copy link

tohtana commented Aug 30, 2024 •
edited
Loading

@YJHMITWEB Do we need changes in gpt2-merge.txt / gpt2-vocab.json? I'm not sure if we should check them in.

tohtana
tohtana reviewed Aug 30, 2024
View reviewed changes
megatron/initialize.py Outdated
@@ -349,9 +349,12 @@ def _warmup_jit_function():
dtype = torch.float32

# Warmup fused bias+gelu
seq_length = args.seq_length
if args.ds_sequence_parallel_fpdt:
seq_length = 8192
Copy link

@tohtana tohtana Aug 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you define this as another variable like "FPDT_SEQ_LEN" and give a description in a comment why we have this setting?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fixed by setting it to be ds_sequence_parallel_fpdt_chunk_size if FPDT is enabled.

inkcherry
inkcherry reviewed Aug 30, 2024
View reviewed changes
megatron/model/rotary_pos_embedding.py
@@ -32,7 +35,9 @@ def forward(self, max_seq_len, offset=0):
emb = torch.cat((freqs, freqs), dim=-1)
# emb [seq_length, .., dim]
from einops import rearrange
return rearrange(emb, 'n d -> n 1 1 d')
base = rearrange(emb, 'n d -> n 1 1 d')
Copy link

@inkcherry inkcherry Aug 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this change the output when use --use-rotary-position-embeddings, in llama style model?
FYI https://github.com/microsoft/Megatron-DeepSpeed/blob/main/examples_deepspeed/pretrain_llama2_distributed.sh

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have tested both GPT and Llama models, this works well with both.

@samadejacobs
Copy link

Hi @YJHMITWEB , is FPDT referring to this paper? https://ui.adsabs.harvard.edu/abs/2023JARS...17b6510H/abstract

@delock, no, FPDT refers to this paper, aka Ulysses-Offload

@YJHMITWEB
Copy link
Author

Hi @YJHMITWEB , is FPDT referring to this paper? https://ui.adsabs.harvard.edu/abs/2023JARS...17b6510H/abstract

@delock, no, FPDT refers to this paper, aka Ulysses-Offload

Thanks @samadejacobs for pointing.

@YJHMITWEB
Copy link
Author

@microsoft-github-policy-service agree

@YJHMITWEB
Copy link
Author

@microsoft-github-policy-service agree

github-merge-queue bot pushed a commit to microsoft/DeepSpeed that referenced this pull request Nov 26, 2024
@YJHMITWEB @samadejacobs
@loadams @tjruwase @tohtana
Adding the new feature of FPDT (#6462)
404074e 
[FPDT](https://arxiv.org/abs/2408.16978) can only be used with [this
version](microsoft/Megatron-DeepSpeed#441) of
Megatron-DeepSpeed.

---------

Co-authored-by: Jinghan Yao <yjhmitweb@ascend-rw02.ten.osc.edu>
Co-authored-by: Sam Ade Jacobs <samjacobs@microsoft.com>
Co-authored-by: Jinghan Yao <yjhmitweb@ascend-rw01.ten.osc.edu>
Co-authored-by: Logan Adams <loadams@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Jinghan Yao <yjhmitweb@cardinal-rw02.ten.osc.edu>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <mtanaka@microsoft.com>
github-merge-queue bot pushed a commit to microsoft/DeepSpeed that referenced this pull request Nov 26, 2024
@YJHMITWEB @samadejacobs
@loadams @tjruwase @tohtana
Adding the new feature of FPDT (#6462)
13f4943 
[FPDT](https://arxiv.org/abs/2408.16978) can only be used with [this
version](microsoft/Megatron-DeepSpeed#441) of
Megatron-DeepSpeed.

---------

Co-authored-by: Jinghan Yao <yjhmitweb@ascend-rw02.ten.osc.edu>
Co-authored-by: Sam Ade Jacobs <samjacobs@microsoft.com>
Co-authored-by: Jinghan Yao <yjhmitweb@ascend-rw01.ten.osc.edu>
Co-authored-by: Logan Adams <loadams@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Jinghan Yao <yjhmitweb@cardinal-rw02.ten.osc.edu>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <mtanaka@microsoft.com>
github-merge-queue bot pushed a commit to microsoft/DeepSpeed that referenced this pull request Nov 26, 2024
@YJHMITWEB @samadejacobs
@loadams @tjruwase @tohtana
Adding the new feature of FPDT (#6462)
648fbb4 
[FPDT](https://arxiv.org/abs/2408.16978) can only be used with [this
version](microsoft/Megatron-DeepSpeed#441) of
Megatron-DeepSpeed.

---------

Co-authored-by: Jinghan Yao <yjhmitweb@ascend-rw02.ten.osc.edu>
Co-authored-by: Sam Ade Jacobs <samjacobs@microsoft.com>
Co-authored-by: Jinghan Yao <yjhmitweb@ascend-rw01.ten.osc.edu>
Co-authored-by: Logan Adams <loadams@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Jinghan Yao <yjhmitweb@cardinal-rw02.ten.osc.edu>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <mtanaka@microsoft.com>
github-merge-queue bot pushed a commit to microsoft/DeepSpeed that referenced this pull request Nov 26, 2024
@YJHMITWEB @samadejacobs
@loadams @tjruwase @tohtana
Adding the new feature of FPDT (#6462)
464796d 
[FPDT](https://arxiv.org/abs/2408.16978) can only be used with [this
version](microsoft/Megatron-DeepSpeed#441) of
Megatron-DeepSpeed.

---------

Co-authored-by: Jinghan Yao <yjhmitweb@ascend-rw02.ten.osc.edu>
Co-authored-by: Sam Ade Jacobs <samjacobs@microsoft.com>
Co-authored-by: Jinghan Yao <yjhmitweb@ascend-rw01.ten.osc.edu>
Co-authored-by: Logan Adams <loadams@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Jinghan Yao <yjhmitweb@cardinal-rw02.ten.osc.edu>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <mtanaka@microsoft.com>
github-merge-queue bot pushed a commit to microsoft/DeepSpeed that referenced this pull request Dec 2, 2024
@YJHMITWEB @samadejacobs
@loadams @tjruwase @tohtana
Adding the new feature of FPDT (#6462)
3076b72 
[FPDT](https://arxiv.org/abs/2408.16978) can only be used with [this
version](microsoft/Megatron-DeepSpeed#441) of
Megatron-DeepSpeed.

---------

Co-authored-by: Jinghan Yao <yjhmitweb@ascend-rw02.ten.osc.edu>
Co-authored-by: Sam Ade Jacobs <samjacobs@microsoft.com>
Co-authored-by: Jinghan Yao <yjhmitweb@ascend-rw01.ten.osc.edu>
Co-authored-by: Logan Adams <loadams@microsoft.com>
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Jinghan Yao <yjhmitweb@cardinal-rw02.ten.osc.edu>
Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <mtanaka@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tohtana tohtana tohtana left review comments

@inkcherry inkcherry inkcherry left review comments

@tjruwase tjruwase Awaiting requested review from tjruwase tjruwase is a code owner

@awan-10 awan-10 Awaiting requested review from awan-10 awan-10 is a code owner

@eltonzheng eltonzheng Awaiting requested review from eltonzheng

@duli2012 duli2012 Awaiting requested review from duli2012

@arashb arashb Awaiting requested review from arashb

@GuanhuaWang GuanhuaWang Awaiting requested review from GuanhuaWang GuanhuaWang is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@YJHMITWEB @delock @tohtana @samadejacobs @inkcherry