Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: oneMKL migration #2865

Conversation

Alexandr-Solovev
Copy link
Contributor

@Alexandr-Solovev Alexandr-Solovev commented Aug 1, 2024

Description:

Feature: Migration from FPKs to oneMKL

Summary:

This PR updates the oneDAL codebase to ensure compatibility with oneMKL. It includes various fixes and modifications across multiple files to address issues related to the integration with the new libraries.

Key Changes:

  1. Updating Public CI Environment:

    • Since MKL FPKs are no longer used, publicly available oneMKL packages have been added to the CI environment.
  2. Removing macOS Public CI Step:

    • As oneMKL does not support macOS, the macOS step has been removed from the CI pipeline.
  3. Bazel Changes:

    • Instead of using two different Bazel libraries (mklcpu and mklgpu), a unified MKL library is now utilized.
  4. Service Functions:

    • The first set of FPK service functions has been replaced with public oneMKL functions, and the remaining ones have been replaced with standard C++ library functions.
  5. Naming Updates:

    • References to FPK have been removed or replaced with MKL.
  6. Macros Usage and CPU Instructions Updates:

    • As oneMKL does not allow switching between CPU instruction sets, all DAAL_MKL_CALL macros have been replaced with fname_fargs macros to call the equivalent functions in oneMKL.
  7. Sparse API Updates:

    • The deprecated spBLAS stack from MKL has been replaced with the new oneMKL Inspector-Executor API.
  8. PCA Algorithm Refactoring:

    • With the removal of the micromkl library, the PCA algorithm has been migrated to use GPU functions (syevd, gesvd).
  9. Makefile Updates:

    • Library names and paths have been updated to support oneMKL.

Known Issues/Looking for Help:

  • 1. Syevd and Gesvd Tests:

    • Assistance needed with these tests.
  • 2. Public Build for Windows (WindowsMakeVC Step):

    • Guidance required to obtain a public build for this step.
  • 3. Removing Unnecessary CPU Instructions and Macros:

    • Help needed with this cleanup.
  • 4. Fixing Path Substitution for Windows:

    • Need to correct the substitution of \, /, and $ for proper Windows path handling.

@icfaust
Copy link
Contributor

icfaust commented Aug 1, 2024

@Alexandr-Solovev You'll need to work on that public CI still uses mircomkl (so far seeing no change there), and please include an informative description (or is this still supposed to be in draft?)

@Alexandr-Solovev
Copy link
Contributor Author

@icfaust Hi! Yes, sorry its a draft, but thanks for the comment about WORKSPACE

@Alexandr-Solovev
Copy link
Contributor Author

/azp run CI

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@samir-nasibli
Copy link
Contributor

Question: do we need to update scikit-learn-intelex deps in case of these migration?

Copy link
Contributor

@samir-nasibli samir-nasibli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huge work done. Thank you @Alexandr-Solovev !
I am sure we should have follow up tickets for all TODOs.
Provided just minor comments.

For merging, I recommend getting approval from other reviewers as well.

.ci/env/apt.sh Show resolved Hide resolved
@@ -87,16 +72,23 @@ mkl_repo(
name = "mkl",
root_env_var = "MKLROOT",
urls = [
# TODO: when the issue with binutils will be solved, replace 2023.0 to 2024.2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please cover this with a ticket

WORKSPACE Show resolved Hide resolved
memcpy(dest, src, srcSize);
return 0;
// TODO: safe funtion
// return memcpy_s(dest, destSize, src, srcSize);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add some ticket for this kind of refactoring


static float serv_string_to_float(const char * nptr, char ** endptr) { return __FPK_string_to_float(nptr, endptr); }
// TODO: not a safe function - no control for the input buffer end
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ticket

__DAAL_VSLFN_CALL(fpk_vsl_sub_kernel, vsliSSEditTask, (task, __DAAL_VSL_SS_ED_CP_STORAGE, &cpStorage), errcode);
__DAAL_VSLFN_CALL(fpk_vsl_sub_kernel, vsldSSEditTask, (task, __DAAL_VSL_SS_ED_ACCUM_WEIGHT, weight), errcode);

ThreadingFuncs threading = { _daal_mkl_threader_for, _daal_mkl_threader_for_ordered, _daal_mkl_threader_sections, _daal_mkl_threader_ordered,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this far somehow discussed, make sense for reply comment with the answer before marking it resolved

namespace mkl
{
//It's a placeholder, the real function calls exact in xfunctions.
//TODO: add correct threading control
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ticket

@samir-nasibli
Copy link
Contributor

intel/scikit-learn-intelex#2066 is merged.
@Alexandr-Solovev you can launch the CI for the scikit-learn-intelex main.

Copy link
Contributor

@Vika-F Vika-F left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! The changes look good to me.

@Alexandr-Solovev Alexandr-Solovev merged commit ce4a8f8 into oneapi-src:main Sep 27, 2024
17 checks passed
Alexandr-Solovev added a commit to Alexandr-Solovev/oneDAL that referenced this pull request Sep 27, 2024
napetrov pushed a commit that referenced this pull request Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file infra
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants