[SYCL] Add support for JIT-ing in AMD and NVIDIA backends #14280

jchlanda · 2024-06-25T10:00:32Z

This patch provides the following:

support for JIT compilation of Nvidia and AMD kernels
This is guarded by SYCL_JIT_KERNELS environment variable. Target CPU and features can be provided through environment variables (SYCL_JIT_TARGET_CPU and SYCL_JIT_TARGET_FEATURES respectively), otherwise default, per-backend, values will be chosen.
caching of JIT-compiled kernels
The runtime maintains a map of available JIT-ed kernels, accessible through a key, which is constructed from kernel's name and values of specialization constant (if provided).
materialization of specialization
Materialization is done through a SYCLSpecConstMaterializer pass that receives the values of all specialization constants used by a kernel (from SYCLSpecConstDataInserter) and then walks all the uses of implicit kernel argument (_arg__specialization_constants_buffer), representing emulated specialization constants, with concrete values, turning them to de-facto compile time constants.

This PR extends the work done for kernel fusion and in a similar fashion it requires embedding of IR (-fsycl-embed-ir) during the initial compilation.

Also allow resetting of SYCL_CACHE_IN_MEM.

steffenlarsen · 2024-07-24T12:12:46Z

Would it be possible to write some tests for this? Even maybe unittests?

I just remembered that I asked about tests. What is the plan for E2E or SYCL-level unittests?

jchlanda · 2024-07-24T12:17:18Z

Would it be possible to write some tests for this? Even maybe unittests?

I just remembered that I asked about tests. What is the plan for E2E or SYCL-level unittests?

Testing of this is a bit tricky, but I though we could use the debug output of the specialization constant materializer pass as a proof of the JIT taking place (as the pass is only pass of the JIT pipeline). I have it almost ready, exercising both a regular kernel and kernel bundle. Going to add it to e2e.

Will ping you once it's push to this PR.

jchlanda · 2024-07-24T12:47:25Z

Would it be possible to write some tests for this? Even maybe unittests?

I just remembered that I asked about tests. What is the plan for E2E or SYCL-level unittests?

Testing of this is a bit tricky, but I though we could use the debug output of the specialization constant materializer pass as a proof of the JIT taking place (as the pass is only pass of the JIT pipeline). I have it almost ready, exercising both a regular kernel and kernel bundle. Going to add it to e2e.

Will ping you once it's push to this PR.

OK, @steffenlarsen let me know what you thing about this test, thank you!

steffenlarsen

Thanks for adding the test! Looks like a reasonable way to cover it.

steffenlarsen · 2024-07-24T13:09:54Z

sycl/test-e2e/AmdNvidiaJIT/kernel_and_bundle.cpp

+// RUN: %{build} -fsycl-embed-ir -o %t.out
+// RUN: env SYCL_JIT_AMDGCN_PTX_KERNELS=1 env SYCL_JIT_COMPILER_DEBUG="sycl-spec-const-materializer" %{run} %t.out &> %t.txt ; FileCheck %s --input-file %t.txt
+
+#include <sycl/sycl.hpp>


sycl/sycl.hpp is not permitted in E2E tests. You want sycl/detail/core.hpp and then probably some other headers.

jchlanda · 2024-07-25T09:39:24Z

@intel/llvm-gatekeepers I think we might be good to go on this one.

steffenlarsen · 2024-07-25T11:28:11Z

@jchlanda - There are some post-commit failures caused by this PR. I am hoping #14770 will address them all.

jchlanda · 2024-07-25T11:30:14Z

@jchlanda - There are some post-commit failures caused by this PR. I am hoping #14770 will address them all.

Yeap, just saw the PR, thank you for taking the time to address it.

steffenlarsen · 2024-07-25T14:31:22Z

@jchlanda - Sadly there seems to be more failures related to it. Could you please take a look?

sarnex · 2024-07-25T15:06:23Z

@jchlanda Do you mind confirming if you're working on the problem? If it's outside of working hours for you we might want to revert. Thanks

Naghasan · 2024-07-25T15:11:41Z

@jchlanda Do you mind confirming if you're working on the problem? If it's outside of working hours for you we might want to revert. Thanks

I'm looking into this, I think Jakub is out of office now.

sarnex · 2024-07-25T16:20:18Z

@Naghasan Thanks!

Naghasan · 2024-07-25T16:36:06Z

Seems it was the NDEBUG macro causing issue, this should fix #14777

sarnex · 2024-07-25T16:38:06Z

@Naghasan There are also unused param warnings:

/Users/runner/work/llvm/llvm/src/sycl-fusion/jit-compiler/lib/translation/KernelTranslation.cpp:232:24: error: unused parameter 'TargetCPU' [-Werror,-Wunused-parameter]
    const std::string &TargetCPU, const std::string &TargetFeatures) {
                       ^
/Users/runner/work/llvm/llvm/src/sycl-fusion/jit-compiler/lib/translation/KernelTranslation.cpp:232:54: error: unused parameter 'TargetFeatures' [-Werror,-Wunused-parameter]
    const std::string &TargetCPU, const std::string &TargetFeatures) {
                                                     ^
/Users/runner/work/llvm/llvm/src/sycl-fusion/jit-compiler/lib/translation/KernelTranslation.cpp:313:24: error: unused parameter 'TargetCPU' [-Werror,-Wunused-parameter]
    const std::string &TargetCPU, const std::string &TargetFeatures) {
                       ^
/Users/runner/work/llvm/llvm/src/sycl-fusion/jit-compiler/lib/translation/KernelTranslation.cpp:313:54: error: unused parameter 'TargetFeatures' [-Werror,-Wunused-parameter]
    const std::string &TargetCPU, const std::string &TargetFeatures) {
                                                     ^

https://github.com/intel/llvm/actions/runs/10095439923/job/27915595323

This patch provides the following: * support for JIT compilation of Nvidia and AMD kernels This is guarded by `SYCL_JIT_KERNELS` environment variable. Target CPU and features can be provided through environment variables (`SYCL_JIT_TARGET_CPU` and `SYCL_JIT_TARGET_FEATURES` respectively), otherwise default, per-backend, values will be chosen. * caching of JIT-compiled kernels The runtime maintains a map of available JIT-ed kernels, accessible through a key, which is constructed from kernel's name and values of specialization constant (if provided). * materialization of specialization Materialization is done through a `SYCLSpecConstMaterializer` pass that receives the values of all specialization constants used by a kernel (from `SYCLSpecConstDataInserter`) and then walks all the uses of implicit kernel argument (`_arg__specialization_constants_buffer`), representing emulated specialization constants, with concrete values, turning them to de-facto compile time constants. This PR extends the work done for kernel fusion and in a similar fashion it requires embedding of IR (`-fsycl-embed-ir`) during the initial compilation.

jchlanda requested a review from Naghasan June 25, 2024 10:00

jchlanda had a problem deploying to WindowsCILock June 25, 2024 10:00 — with GitHub Actions Error

jchlanda force-pushed the jakub/jit_spec_const branch from a5928b4 to 432ea9b Compare June 25, 2024 10:15

jchlanda had a problem deploying to WindowsCILock June 25, 2024 10:17 — with GitHub Actions Error

jchlanda force-pushed the jakub/jit_spec_const branch from 432ea9b to ac5a346 Compare June 25, 2024 10:25

jchlanda had a problem deploying to WindowsCILock June 25, 2024 10:27 — with GitHub Actions Error

jchlanda force-pushed the jakub/jit_spec_const branch from ac5a346 to 21e38f2 Compare June 25, 2024 10:30

jchlanda had a problem deploying to WindowsCILock June 25, 2024 10:31 — with GitHub Actions Error

jchlanda force-pushed the jakub/jit_spec_const branch from 21e38f2 to adfbf76 Compare June 25, 2024 10:46

jchlanda temporarily deployed to WindowsCILock June 25, 2024 10:53 — with GitHub Actions Inactive

jchlanda temporarily deployed to WindowsCILock June 25, 2024 11:42 — with GitHub Actions Inactive

jchlanda force-pushed the jakub/jit_spec_const branch 3 times, most recently from 1408f59 to ef2a74a Compare June 26, 2024 06:07

jchlanda temporarily deployed to WindowsCILock June 26, 2024 06:08 — with GitHub Actions Inactive

jchlanda temporarily deployed to WindowsCILock June 26, 2024 06:39 — with GitHub Actions Inactive

jchlanda changed the title ~~[WIP] Add support for jitting in AMD and NVIDIA backends~~ [WIP] Add support for JIT-ing in AMD and NVIDIA backends Jun 27, 2024

jchlanda temporarily deployed to WindowsCILock June 27, 2024 09:12 — with GitHub Actions Inactive

jchlanda temporarily deployed to WindowsCILock June 27, 2024 09:43 — with GitHub Actions Inactive

jchlanda force-pushed the jakub/jit_spec_const branch from 59078a8 to 04281f5 Compare June 27, 2024 12:07

jchlanda temporarily deployed to WindowsCILock June 27, 2024 12:08 — with GitHub Actions Inactive

jchlanda temporarily deployed to WindowsCILock June 27, 2024 12:31 — with GitHub Actions Inactive

[SYCL] Introduce SYCL_JIT_KERNELS env var

4158578

Also allow resetting of SYCL_CACHE_IN_MEM.

jchlanda force-pushed the jakub/jit_spec_const branch from 04281f5 to 434bc9f Compare July 1, 2024 12:32

jchlanda added 5 commits July 1, 2024 12:34

[SYCL] Extend kernel fusion with JIT-ing

9bf827f

[SYCL] Define JIT pipeline and introduce materializer pass

0bc1146

[SYCL] Add functionality to create/cache/retrieve materialized kernels

1aaf3e5

[SYCL] Introduce SYCL_JIT_TARGET_{CPU,FEATURES} env variables

8221f0f

[SYCL] Document SYCL_JIT_{KERNELS,TARGET_CPU,TARGET_FEATURES} env vars

434bc9f

jchlanda temporarily deployed to WindowsCILock July 1, 2024 12:44 — with GitHub Actions Inactive

jchlanda had a problem deploying to WindowsCILock July 24, 2024 12:06 — with GitHub Actions Error

JIT e2e test

509e3e6

jchlanda had a problem deploying to WindowsCILock July 24, 2024 12:44 — with GitHub Actions Error

clang format the test

d8c6499

jchlanda temporarily deployed to WindowsCILock July 24, 2024 12:48 — with GitHub Actions Inactive

steffenlarsen approved these changes Jul 24, 2024

View reviewed changes

jchlanda had a problem deploying to WindowsCILock July 24, 2024 13:24 — with GitHub Actions Error

include fix in the test

66bd110

jchlanda temporarily deployed to WindowsCILock July 24, 2024 13:41 — with GitHub Actions Inactive

jchlanda temporarily deployed to WindowsCILock July 24, 2024 14:44 — with GitHub Actions Inactive

jchlanda temporarily deployed to WindowsCILock July 25, 2024 05:42 — with GitHub Actions Inactive

Merge remote-tracking branch 'upstream/sycl' into jakub/jit_spec_const

a403149

jchlanda temporarily deployed to WindowsCILock July 25, 2024 06:13 — with GitHub Actions Inactive

martygrant merged commit 63c61d8 into intel:sycl Jul 25, 2024
16 checks passed

jchlanda mentioned this pull request Jul 25, 2024

[SYCL] Rename sycl-fusion to sycl-jit #14762

Merged

KornevNikita mentioned this pull request Jul 25, 2024

[CI][SYCL-CTS] Run nightly SYCL-CTS on L0 gen12 #14325

Merged

frasercrmck mentioned this pull request Jul 29, 2024

[SYCLLowerIR] Remove !amdgcn.annotations metadata #14713

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SYCL] Add support for JIT-ing in AMD and NVIDIA backends #14280

[SYCL] Add support for JIT-ing in AMD and NVIDIA backends #14280

jchlanda commented Jun 25, 2024 •

edited

Loading

steffenlarsen commented Jul 24, 2024

jchlanda commented Jul 24, 2024

jchlanda commented Jul 24, 2024

steffenlarsen left a comment

steffenlarsen Jul 24, 2024

jchlanda Jul 24, 2024

jchlanda commented Jul 25, 2024

steffenlarsen commented Jul 25, 2024

jchlanda commented Jul 25, 2024

steffenlarsen commented Jul 25, 2024

sarnex commented Jul 25, 2024

Naghasan commented Jul 25, 2024

sarnex commented Jul 25, 2024

Naghasan commented Jul 25, 2024

sarnex commented Jul 25, 2024 •

edited

Loading

[SYCL] Add support for JIT-ing in AMD and NVIDIA backends #14280

[SYCL] Add support for JIT-ing in AMD and NVIDIA backends #14280

Conversation

jchlanda commented Jun 25, 2024 • edited Loading

steffenlarsen commented Jul 24, 2024

jchlanda commented Jul 24, 2024

jchlanda commented Jul 24, 2024

steffenlarsen left a comment

Choose a reason for hiding this comment

steffenlarsen Jul 24, 2024

Choose a reason for hiding this comment

jchlanda Jul 24, 2024

Choose a reason for hiding this comment

jchlanda commented Jul 25, 2024

steffenlarsen commented Jul 25, 2024

jchlanda commented Jul 25, 2024

steffenlarsen commented Jul 25, 2024

sarnex commented Jul 25, 2024

Naghasan commented Jul 25, 2024

sarnex commented Jul 25, 2024

Naghasan commented Jul 25, 2024

sarnex commented Jul 25, 2024 • edited Loading

jchlanda commented Jun 25, 2024 •

edited

Loading

sarnex commented Jul 25, 2024 •

edited

Loading