[TKW] Add CDNA2 + CDNA3 Int8 intrinsics and refactor intrinsic enums #279

raikonenfnu · 2024-11-19T22:56:01Z

Added CDNA2 int8 intrinsic layouts
Modified iree_ref to handle int gemms
Modified certain e2e test to require certain GPU arch to be available
Modified enum for easy handling in the future
Get default architecture function
Borrowed device_randint from Ivan
Turn on CDNA2 runner for TK-CI

Manually tested that the generated iree_ref for int gemms are working as expected!

iree/turbine/kernel/wave/iree_utils.py

iree/turbine/kernel/wave/utils.py

iree/turbine/kernel/wave/iree_utils.py

Hardcode84

Couple more comments, but otherwise LGTM

iree/turbine/kernel/wave/utils.py

tests/kernel/wave/wave_gemm_test.py

harsh-nod

This looks great and I am very surprised that you were able to reuse all the other intrinsics. One final ask - can you add a lit test showing the IR form of the mfmas?

iree/turbine/kernel/wave/constraints.py

tests/kernel/wave/wave_gemm_test.py

saienduri · 2024-11-20T00:37:08Z

.github/workflows/perf.yaml

@@ -61,3 +61,10 @@ jobs:
        export WAVE_RUN_E2E_TESTS=1
        export TEST_PARAMS_PATH="tests/kernel/wave/test_param.json"
        pytest -n 1 --capture=tee-sys -vv ./tests/kernel/wave/
+
+    - name: Run e2e tests on MI250


You can remove this step completely, and just keep the one before it. Just hange its name from Run e2e tests on MI300 to Run e2e tests on AMD GPU or something and remove the if: "contains(matrix.os, 'mi300') && !cancelled()" line

Can you further explain? If I understand correctly, you want to only have one test? but we need to run both MI250 and MI300 though

actually I see what you mean! that's a great idea, thanks!

Done! thanks you again for the tip! :)

Hardcode84 · 2024-11-20T00:41:27Z

For correctness check probably should add to ci-tk.yaml instead. perf.yaml was meant for performance measurement, but this work was never finished.

raikonenfnu · 2024-11-20T00:49:10Z

For correctness check probably should add to ci-tk.yaml instead. perf.yaml was meant for performance measurement, but this work was never finished.

Makes sense! thanks!

Giuseppe5 · 2024-11-20T14:11:26Z

tests/kernel/wave/wave_gemm_test.py

+@pytest.mark.parametrize(
+    "mfma_variant",
+    [
+        MMAType.F32_16x16x32_F8,


Shouldn't this be I8?

ahh, good catch! it was passing because the generated instructions between them are technically in the mlir vector level are the same haha, will fix that. :)

Giuseppe5 · 2024-11-20T14:57:07Z

tests/kernel/wave/wave_gemm_test.py

+        dynamic_symbols_map=dynamic_symbols_map,
+    ):
+        randint_hi = 4
+        a = device_randint(randint_hi, (shape[0], shape[2]), dtype=torch.int16)


Would it be possible to pass here directly a torch.int8?

yeap, done!

- Added CDNA2 int8 intrinsic layouts - Modified iree_ref to handle int gemms - Modified certain e2e test to require certain GPU arch to be available - Modified enum for easy handling in the future - Get default architecture function - Borrowed device_randint from Ivan Signed-off-by: Stanley Winata <stanley.winata@amd.com> Co-authored-by: Ivan Butygin <ivan.butygin@gmail.com> Signed-off-by: Stanley Winata <stanley.winata@amd.com>

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

harsh-nod

thanks! this looks great!

harsh-nod · 2024-11-20T01:48:25Z

.github/workflows/ci-tk.yaml

@@ -21,7 +21,7 @@ jobs:
      fail-fast: false
      matrix:
        version: [3.11]
-        os: [ubuntu-latest, nodai-amdgpu-mi300-x86-64]
+        os: [ubuntu-latest, nodai-amdgpu-mi300-x86-64, nodai-amdgpu-mi250-x86-64]


very nice :)

raikonenfnu requested review from harsh-nod and Hardcode84 November 19, 2024 22:56

raikonenfnu force-pushed the intGemms branch from 996c40f to 1a1c8fc Compare November 19, 2024 23:08

Hardcode84 reviewed Nov 19, 2024

View reviewed changes

iree/turbine/kernel/wave/iree_utils.py Outdated Show resolved Hide resolved

Hardcode84 reviewed Nov 19, 2024

View reviewed changes

iree/turbine/kernel/wave/utils.py Outdated Show resolved Hide resolved

raikonenfnu changed the title ~~[TKW] Add CDNA2 Int8 intrinsics and refactor intrinsic enums~~ [TKW] Add CDNA2 + CDNA3 Int8 intrinsics and refactor intrinsic enums Nov 19, 2024

raikonenfnu commented Nov 19, 2024

View reviewed changes

iree/turbine/kernel/wave/iree_utils.py Show resolved Hide resolved

Hardcode84 approved these changes Nov 20, 2024

View reviewed changes

iree/turbine/kernel/wave/utils.py Show resolved Hide resolved

tests/kernel/wave/wave_gemm_test.py Outdated Show resolved Hide resolved

harsh-nod requested changes Nov 20, 2024

View reviewed changes

iree/turbine/kernel/wave/constraints.py Show resolved Hide resolved

iree/turbine/kernel/wave/constraints.py Show resolved Hide resolved

tests/kernel/wave/wave_gemm_test.py Show resolved Hide resolved

raikonenfnu force-pushed the intGemms branch from 3660dd8 to fa61434 Compare November 20, 2024 00:33

saienduri reviewed Nov 20, 2024

View reviewed changes

raikonenfnu force-pushed the intGemms branch from a92a16a to 9b7d04d Compare November 20, 2024 01:09

raikonenfnu requested review from harsh-nod and saienduri November 20, 2024 01:11

raikonenfnu force-pushed the intGemms branch from 9b7d04d to 5a5c76f Compare November 20, 2024 01:14

Giuseppe5 reviewed Nov 20, 2024

View reviewed changes

raikonenfnu and others added 10 commits November 20, 2024 08:44

Fix NIT and add CDNA3 int gemms

7e60f37

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

Fix NIT

93bc238

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

fix some more nits

9b08f6c

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

naming nit

48fa66d

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

Add Mi250 runner

1385eb4

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

Clean up runner yaml

968c2e3

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

Integrate back perf yamls

e2f7983

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

Make get_default_run_config use default device + move test to CI-TK.yaml

54a6345

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

turn off igemm for cdna2 temporarily + fix yaml

4b2d452

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

raikonenfnu force-pushed the intGemms branch from 5a5c76f to 3a3f9f3 Compare November 20, 2024 17:12

rebase, nit, and lit

3a3f9f3

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

harsh-nod approved these changes Nov 20, 2024

View reviewed changes

raikonenfnu merged commit f8e0cbb into iree-org:main Nov 20, 2024
8 checks passed

raikonenfnu mentioned this pull request Nov 20, 2024

[Wave] Fun projects for beginnners #278

Open

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TKW] Add CDNA2 + CDNA3 Int8 intrinsics and refactor intrinsic enums #279

[TKW] Add CDNA2 + CDNA3 Int8 intrinsics and refactor intrinsic enums #279

raikonenfnu commented Nov 19, 2024 •

edited

Loading

Hardcode84 left a comment

harsh-nod left a comment

saienduri Nov 20, 2024

raikonenfnu Nov 20, 2024

raikonenfnu Nov 20, 2024 •

edited

Loading

raikonenfnu Nov 20, 2024

Hardcode84 commented Nov 20, 2024 •

edited

Loading

raikonenfnu commented Nov 20, 2024

Giuseppe5 Nov 20, 2024

raikonenfnu Nov 20, 2024

raikonenfnu Nov 20, 2024

Giuseppe5 Nov 20, 2024

raikonenfnu Nov 20, 2024 •

edited

Loading

harsh-nod left a comment

harsh-nod Nov 20, 2024

[TKW] Add CDNA2 + CDNA3 Int8 intrinsics and refactor intrinsic enums #279

[TKW] Add CDNA2 + CDNA3 Int8 intrinsics and refactor intrinsic enums #279

Conversation

raikonenfnu commented Nov 19, 2024 • edited Loading

Hardcode84 left a comment

Choose a reason for hiding this comment

harsh-nod left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raikonenfnu Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hardcode84 commented Nov 20, 2024 • edited Loading

raikonenfnu commented Nov 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raikonenfnu Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

harsh-nod left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raikonenfnu commented Nov 19, 2024 •

edited

Loading

raikonenfnu Nov 20, 2024 •

edited

Loading

Hardcode84 commented Nov 20, 2024 •

edited

Loading

raikonenfnu Nov 20, 2024 •

edited

Loading