ENH: dpnp interop for sklearnex #1374

samir-nasibli · 2023-07-24T16:40:26Z

Description

interop for DPNP ndarrays
API support for dpnp ndarrays
added dpnp installation into env
added examples with dpnp, dpctl usage.
- updated runner for this

TODO

update docs -> on separate PR
testing for different dataframe libs on sklearnex own tests -> on separate PR
interop for SPMD interfaces

api support for dpnp ndarrays

update runner for added examples codestyle with black and isort check

ahuber21 · 2023-07-25T11:03:46Z

Here's what I see in the CI summary

======================================================================
ERROR: test_knn_bf_classification_dpnp_batch (test_examples_sklearnex.TestsklearnexExamples.test_knn_bf_classification_dpnp_batch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vsts/work/1/s/tests/test_examples_sklearnex.py", line 43, in testit
    process = subprocess.run(
              ^^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/CB/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/home/vsts/work/1/s/examples/sklearnex/knn_bf_classification_dpnp_batch.py']' returned non-zero exit status 1.

======================================================================
ERROR: test_random_forest_classifier_dpctl_batch (test_examples_sklearnex.TestsklearnexExamples.test_random_forest_classifier_dpctl_batch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vsts/work/1/s/tests/test_examples_sklearnex.py", line 43, in testit
    process = subprocess.run(
              ^^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/CB/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/home/vsts/work/1/s/examples/sklearnex/random_forest_classifier_dpctl_batch.py']' returned non-zero exit status 1.

======================================================================
ERROR: test_random_forest_regressor_dpnp_batch (test_examples_sklearnex.TestsklearnexExamples.test_random_forest_regressor_dpnp_batch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vsts/work/1/s/tests/test_examples_sklearnex.py", line 43, in testit
    process = subprocess.run(
              ^^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/CB/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/home/vsts/work/1/s/examples/sklearnex/random_forest_regressor_dpnp_batch.py']' returned non-zero exit status 1.

Do these checks work locally for you?

samir-nasibli · 2023-07-25T11:26:13Z

Here's what I see in the CI summary

======================================================================
ERROR: test_knn_bf_classification_dpnp_batch (test_examples_sklearnex.TestsklearnexExamples.test_knn_bf_classification_dpnp_batch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vsts/work/1/s/tests/test_examples_sklearnex.py", line 43, in testit
    process = subprocess.run(
              ^^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/CB/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/home/vsts/work/1/s/examples/sklearnex/knn_bf_classification_dpnp_batch.py']' returned non-zero exit status 1.

======================================================================
ERROR: test_random_forest_classifier_dpctl_batch (test_examples_sklearnex.TestsklearnexExamples.test_random_forest_classifier_dpctl_batch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vsts/work/1/s/tests/test_examples_sklearnex.py", line 43, in testit
    process = subprocess.run(
              ^^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/CB/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/home/vsts/work/1/s/examples/sklearnex/random_forest_classifier_dpctl_batch.py']' returned non-zero exit status 1.

======================================================================
ERROR: test_random_forest_regressor_dpnp_batch (test_examples_sklearnex.TestsklearnexExamples.test_random_forest_regressor_dpnp_batch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vsts/work/1/s/tests/test_examples_sklearnex.py", line 43, in testit
    process = subprocess.run(
              ^^^^^^^^^^^^^^^
  File "/usr/share/miniconda/envs/CB/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '/home/vsts/work/1/s/examples/sklearnex/random_forest_regressor_dpnp_batch.py']' returned non-zero exit status 1.

Do these checks work locally for you?

Yeah I am working on it, trying understand public CI env, will reproduce it. Locally it works

samir-nasibli · 2023-07-25T22:35:08Z

sklearnex/_device_offload.py

+            result = _copy_to_usm(usm_iface["syclobj"], result)
+            if dpnp_available and isinstance(data[0], dpnp.ndarray):
+                result = dpnp.array(result, copy=False)


This and related dpnp import are only functional changes in this file, the rest is formatting.

samir-nasibli · 2023-07-25T23:54:25Z

/intelci: run

samir-nasibli · 2023-07-26T01:31:31Z

/intelci: run

samir-nasibli · 2023-07-26T12:00:00Z

Job: http://intel-ci.intel.com/ee2baa7b-5115-f127-aa37-a4bf010d0e2e. Specifically for sklearnex examples GPU.

samir-nasibli · 2023-07-26T12:50:21Z

Job: http://intel-ci.intel.com/ee2baa7b-5115-f127-aa37-a4bf010d0e2e. Specifically for sklearnex examples GPU.

examples/sklearnex/random_forest_classifier_dpctl_batch.py
	PASSED	random_forest_classifier_dpctl_batch.py

examples/sklearnex/random_forest_regressor_dpnp_batch.py
	PASSED	random_forest_regressor_dpnp_batch.py

examples/sklearnex/knn_bf_classification_dpnp_batch.py
	PASSED	knn_bf_classification_dpnp_batch.py

samir-nasibli · 2023-07-26T13:13:10Z

/intelci: run

examples/sklearnex/knn_bf_classification_dpnp_batch.py

KulikovNikita · 2023-07-26T13:05:59Z

examples/sklearnex/knn_bf_classification_dpnp_batch.py

+dpnp_X_test = dpnp.asarray(X_test, usm_type="device", sycl_queue=q)
+
+knn_mdl = KNeighborsClassifier(
+    algorithm="brute", n_neighbors=20, weights="uniform", p=2, metric="minkowski"


I suppose weights="uniform", p=2, metri="minkowski" - correspond to the Euclidean metric and normal mode of classification which are default.

should be patched for both spmd and batch examples. Let's do it on separate PR

ethanglaser · 2023-07-26T14:08:13Z

examples/sklearnex/knn_bf_classification_dpnp_batch.py

+# ===============================================================================
+
+# sklearnex kNN example for GPU offloading with DPNP ndarray:
+#    python ./knn_bf_classification_dpnp_batch.py.py


Suggested change

# python ./knn_bf_classification_dpnp_batch.py.py

# python ./knn_bf_classification_dpnp_batch.py

Good catch!

.ci/pipeline/build-and-test-lnx.yml

ENH: dpnp interop in sklearnex

8a712d1

api support for dpnp ndarrays

samir-nasibli requested review from Alexsandruss, KulikovNikita and napetrov as code owners July 24, 2023 16:40

samir-nasibli marked this pull request as draft July 24, 2023 16:42

samir-nasibli added 3 commits July 24, 2023 12:25

fix: make dpnp as a testing dependency

10ee686

fix isort check

edd94d9

add examples for using dpnp ndarrays

7c485e1

update runner for added examples codestyle with black and isort check

samir-nasibli changed the title ~~ENH: dpnp interop in sklearnex~~ ENH: dpnp interop for sklearnex Jul 24, 2023

samir-nasibli mentioned this pull request Jul 25, 2023

MAINT: align daal4py, onedal and sklearnex modules with isort and black #1375

Merged

samir-nasibli added 3 commits July 25, 2023 14:29

Merge branch 'intel:master' into enh/dpnp_api_support

9cd0085

fix sklearnex examples runner condition

e05b895

formating tests/test_examples_sklearnex.py

f68f989

samir-nasibli commented Jul 25, 2023

View reviewed changes

minor fix in condition

5ea882a

samir-nasibli requested review from ahuber21 and razdoburdin July 26, 2023 00:35

fix runner for sklearnex examples

dff97fa

samir-nasibli marked this pull request as ready for review July 26, 2023 08:58

Merge branch 'master' into enh/dpnp_api_support

a471aed

samir-nasibli requested a review from ethanglaser July 26, 2023 12:59

ethanglaser reviewed Jul 26, 2023

View reviewed changes

examples/sklearnex/knn_bf_classification_dpnp_batch.py Show resolved Hide resolved

KulikovNikita approved these changes Jul 26, 2023

View reviewed changes

ethanglaser reviewed Jul 26, 2023

View reviewed changes

.ci/pipeline/build-and-test-lnx.yml Show resolved Hide resolved

ethanglaser approved these changes Jul 26, 2023

View reviewed changes

samir-nasibli merged commit 0ccbbfa into intel:master Jul 26, 2023
20 checks passed

samir-nasibli deleted the enh/dpnp_api_support branch July 26, 2023 16:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: dpnp interop for sklearnex #1374

ENH: dpnp interop for sklearnex #1374

samir-nasibli commented Jul 24, 2023 •

edited

Loading

ahuber21 commented Jul 25, 2023

samir-nasibli commented Jul 25, 2023

samir-nasibli Jul 25, 2023 •

edited

Loading

samir-nasibli commented Jul 25, 2023

samir-nasibli commented Jul 26, 2023

samir-nasibli commented Jul 26, 2023

samir-nasibli commented Jul 26, 2023 •

edited

Loading

samir-nasibli commented Jul 26, 2023

KulikovNikita Jul 26, 2023

samir-nasibli Jul 26, 2023

ethanglaser Jul 26, 2023

samir-nasibli Jul 26, 2023

	# python ./knn_bf_classification_dpnp_batch.py.py
	# python ./knn_bf_classification_dpnp_batch.py

ENH: dpnp interop for sklearnex #1374

ENH: dpnp interop for sklearnex #1374

Conversation

samir-nasibli commented Jul 24, 2023 • edited Loading

Description

TODO

ahuber21 commented Jul 25, 2023

samir-nasibli commented Jul 25, 2023

samir-nasibli Jul 25, 2023 • edited Loading

Choose a reason for hiding this comment

samir-nasibli commented Jul 25, 2023

samir-nasibli commented Jul 26, 2023

samir-nasibli commented Jul 26, 2023

samir-nasibli commented Jul 26, 2023 • edited Loading

samir-nasibli commented Jul 26, 2023

KulikovNikita Jul 26, 2023

Choose a reason for hiding this comment

samir-nasibli Jul 26, 2023

Choose a reason for hiding this comment

ethanglaser Jul 26, 2023

Choose a reason for hiding this comment

samir-nasibli Jul 26, 2023

Choose a reason for hiding this comment

samir-nasibli commented Jul 24, 2023 •

edited

Loading

samir-nasibli Jul 25, 2023 •

edited

Loading

samir-nasibli commented Jul 26, 2023 •

edited

Loading