Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hierarchical CP implementation (Ulysses + Ring) #1209

Open
wants to merge 24 commits into
base: main
Choose a base branch
from
Open

Hierarchical CP implementation (Ulysses + Ring) #1209

wants to merge 24 commits into from

Commits on Sep 19, 2024

  1. change API for hierarchical CP 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    2ee1389 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into xren/cp_a2a_p2p

    @xrennvidia
    xrennvidia committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    3743c5f View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. move fp8 code before qkv reshape 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    0e826e7 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. try to insert A2A for hierarchical CP 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    44db43b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into xren/cp_a2a_p2p

    @xrennvidia
    xrennvidia committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    e717b81 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. make fwd work 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    705da7e View commit details
    Browse the repository at this point in the history

  2. remove a redundant sync 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    99ace46 View commit details
    Browse the repository at this point in the history

  3. make bwd of hierarchical CP work 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    f73b2fb View commit details
    Browse the repository at this point in the history

  4. fix dout a2a in bwd 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    0be5dd7 View commit details
    Browse the repository at this point in the history

  5. fix q_f16 with fp8 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    a6833c7 View commit details
    Browse the repository at this point in the history

  6. assert hierarchical CP implementation does not support THD format 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    caf3746 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. bug fix 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    03806c4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into xren/cp_a2a_p2p

    @xrennvidia
    xrennvidia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    d3d336a View commit details
    Browse the repository at this point in the history

  3. assert hierarchical CP does not support attn bias 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    ea1e4a3 View commit details
    Browse the repository at this point in the history

  4. add unit test for hierarchical CP 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    3cd21ab View commit details
    Browse the repository at this point in the history

  5. fix cp_comm_type in unit test 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    90c0bb8 View commit details
    Browse the repository at this point in the history

  6. bug fix and code cleaning 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    8c6139d View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into xren/cp_a2a_p2p

    @xrennvidia
    xrennvidia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    52c7ae3 View commit details
    Browse the repository at this point in the history

  8. minor change 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    1db4e8b View commit details
    Browse the repository at this point in the history

  9. an assert info change 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    1655edc View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. dout shape fix 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    edbe898 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    b096051 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into xren/cp_a2a_p2p

    @xrennvidia
    xrennvidia committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    6063bec View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. move function definitions to the front of the first call 

    Signed-off-by: Xiaowei Ren <xren@nvidia.com>
    @xrennvidia
    xrennvidia committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    67c590d View commit details
    Browse the repository at this point in the history