Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add partial RMSnorm "pRMSNorm" variation #153

Merged
merged 4 commits into from
Apr 19, 2024

Commits on Apr 18, 2024

  1. Add partial RMSnorm "pRMSNorm" variation

    This was described in the RMSNorm paper as being able to accomplish the
    same task as RMSNorm usually with only performing calculations on the
    first 6% of entries.
    
    This is because the average of the RMSNorm changes more slowly the more
    items are added, and the RMSNorm authors noted that the tokens they
    measured had around the same value.
    gkielian committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    fa56821 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3ea7279 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    9c8a4df View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2024

  1. Add kRMSNorm variation

    Add variation of pRMSNorm that simply sets a max number of tokens to
    utilize for the RMSNorm normalization.
    
    This should be much easier to implement in hardware.
    gkielian committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    0c5a510 View commit details
    Browse the repository at this point in the history