Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate Marlin Kernels for Int4 GPTQ inference #2497

Merged
merged 52 commits into from
Mar 1, 2024
Merged

Integrate Marlin Kernels for Int4 GPTQ inference #2497

merged 52 commits into from
Mar 1, 2024

Commits on Jan 17, 2024

  1. marlin

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    f30ed49 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'marlin' of github.com:neuralmagic/vllm into marlin

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    afd6547 View commit details
    Browse the repository at this point in the history

  3. added marlin

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    837d344 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2024

  1. trying to load packed weights turning out to be tricky

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    7a43b29 View commit details
    Browse the repository at this point in the history

  2. trying to load packed weights turning out to be tricky due to qkv

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    e034640 View commit details
    Browse the repository at this point in the history

  3. integrated marlin for single gpu

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    15e8f9c View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2024

  1. Update llama.py

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Jan 19, 2024
    Configuration menu
    Copy the full SHA
    d8286fb View commit details
    Browse the repository at this point in the history

  2. Fixes to Marlin quantization to allow execution via CUDA graphs captu… 

    …re (eager_force=False)
    @alexm-neuralmagic
    alexm-neuralmagic committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    8bc625f View commit details
    Browse the repository at this point in the history

  3. Integrate @efrantar's changes for CUDA graphs

    @alexm-neuralmagic
    alexm-neuralmagic committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    2691e89 View commit details
    Browse the repository at this point in the history

  4. review comments based on zhyncs

    @alexm-neuralmagic
    alexm-neuralmagic committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    92f7290 View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2024

  1. (1) Integrate the latest changes from Elias that improve large batch … 

    …size by running multiple parallel problems of size 64. (2) Refactor the workspace to be dynamic per layer
    @alexm-neuralmagic
    alexm-neuralmagic committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    bc10e4b View commit details
    Browse the repository at this point in the history

  2. add bug fix

    @alexm-neuralmagic
    alexm-neuralmagic committed Jan 30, 2024
    Configuration menu
    Copy the full SHA
    47987da View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2024

  1. refactored some of alex's work to be consistent with the gptq config

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    43aa818 View commit details
    Browse the repository at this point in the history

  2. updated to load model based on hf_config from AutoGPTQ

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    5906a60 View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2024

  1. Reduce Marlin's kernel limitation of thread_n from 256 to 64 (to avoi… 

    …d issues with tensor parallel runs)
    @alexm-neuralmagic
    alexm-neuralmagic committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    8dfeaa2 View commit details
    Browse the repository at this point in the history

  2. Update checks related to MarlinConfig

    @alexm-neuralmagic
    alexm-neuralmagic committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    c7fb928 View commit details
    Browse the repository at this point in the history

  3. formatting

    @alexm-neuralmagic
    alexm-neuralmagic committed Feb 2, 2024
    Configuration menu
    Copy the full SHA
    1ea85f3 View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2024

  1. Merge branch 'main' into marlin

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Feb 7, 2024
    Configuration menu
    Copy the full SHA
    c876b79 View commit details
    Browse the repository at this point in the history

  2. Update pybind.cpp

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Feb 7, 2024
    Configuration menu
    Copy the full SHA
    a435c97 View commit details
    Browse the repository at this point in the history

  3. Update ops.h 

    cleanup to undo autoformatting
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Feb 7, 2024
    Configuration menu
    Copy the full SHA
    90e8b8f View commit details
    Browse the repository at this point in the history

  4. Update ops.h 

    cleanup formatting
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Feb 7, 2024
    Configuration menu
    Copy the full SHA
    b03af7d View commit details
    Browse the repository at this point in the history

  5. readded marlin

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    9192287 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2024

  1. Bug fix for determination of the scales size in marlin layer

    @alexm-neuralmagic
    alexm-neuralmagic committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    ce50dd4 View commit details
    Browse the repository at this point in the history

  2. Ensure marlin only compiles for GPU compute capability >= 8.0

    @alexm-neuralmagic
    alexm-neuralmagic committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    5a305d3 View commit details
    Browse the repository at this point in the history

  3. fix marlin compilation again

    @alexm-neuralmagic
    alexm-neuralmagic committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    b1773aa View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2024

  1. Merge branch 'vllm-project:main' into marlin

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Feb 9, 2024
    Configuration menu
    Copy the full SHA
    036e0ca View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2024

  1. added marlin test

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    d63627e View commit details
    Browse the repository at this point in the history

  2. added marlin test

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    18981b1 View commit details
    Browse the repository at this point in the history

  3. updated skipping logic

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    828c621 View commit details
    Browse the repository at this point in the history

  4. updated skipping logic

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    4f1759b View commit details
    Browse the repository at this point in the history

  5. added memory profiling

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    f1714e9 View commit details
    Browse the repository at this point in the history

  6. added memory profiling

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    e3a4706 View commit details
    Browse the repository at this point in the history

  7. test wout memory utilization

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    efd886c View commit details
    Browse the repository at this point in the history

  8. updating memory profiling

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    70f5850 View commit details
    Browse the repository at this point in the history

  9. adding more profiling

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    567fe38 View commit details
    Browse the repository at this point in the history

  10. updating memory profiling

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    01f5e40 View commit details
    Browse the repository at this point in the history

  11. removed memory profiling

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    fc5310c View commit details
    Browse the repository at this point in the history

  12. cleaned up

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    99ab19d View commit details
    Browse the repository at this point in the history

  13. added newline

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    eabeea6 View commit details
    Browse the repository at this point in the history

  14. ran ./format.sh

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    d064595 View commit details
    Browse the repository at this point in the history

  15. Merge branch 'upstream-main' into marlin

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    721351e View commit details
    Browse the repository at this point in the history

  16. merged into upstream main

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    9b1bc5f View commit details
    Browse the repository at this point in the history

  17. Update test_marlin.py

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Feb 18, 2024
    Configuration menu
    Copy the full SHA
    013f10f View commit details
    Browse the repository at this point in the history

  18. Update test_marlin.py

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Feb 18, 2024
    Configuration menu
    Copy the full SHA
    7f2165e View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2024

  1. Merge branch 'main' into marlin

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic authored Feb 19, 2024
    Configuration menu
    Copy the full SHA
    79081ff View commit details
    Browse the repository at this point in the history

  2. updated retry testing to use pytest-flaky rather than implementing th… 

    …is logic myself.
    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    7a9b828 View commit details
    Browse the repository at this point in the history

  3. missed newline

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    c23902f View commit details
    Browse the repository at this point in the history

  4. formatting

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    e7aba66 View commit details
    Browse the repository at this point in the history

  5. removed silly print

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    2403f7d View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. added license

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    aabaed2 View commit details
    Browse the repository at this point in the history

  2. format

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    a67dc8d View commit details
    Browse the repository at this point in the history

  3. minor change for ruff

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    8ff42c0 View commit details
    Browse the repository at this point in the history