Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vectorscan: 5.4.10.1 -> 5.4.11 #303207

Merged
merged 2 commits into from
May 18, 2024
Merged

Conversation

tnias
Copy link
Contributor

@tnias tnias commented Apr 10, 2024

Description of changes

Release: https://github.com/VectorCamp/vectorscan/releases/tag/vectorscan%2F5.4.11
Changelog: https://github.com/VectorCamp/vectorscan/blob/develop/CHANGELOG-vectorscan.md#5411-2023-11-19

tldr: This PR bumps the version and explicitly enables all available CPU extensions the FAT_RUNTIME binaries.

With the version bump the FAT_RUNTIME won't build without AVX2 support explicitly enabled.
And upsteam now enabled all extensions for fat binaries (not released yet).
And Debian has almost all extensions enabled. (I assume they did not enable BUILD_AVX512VBMI, because they want it to be buildable by an older toolchain?)

This let me to belive it would be sensible enable all FAT_RUNTIME extensions.

FAT_RUNTIME bundles optimized implementations for different CPU extensions and uses CPUID to transparently select the fastest for the current hardware.
This feature is only available on linux for x86, x86_64, and aarch64.

If FAT_RUNTIME is not available, we fall back to building for a single extension based on stdenv.hostPlatform.
For generic builds (e.g. x86_64) this can mean using an implementation not optimized for the potentially available more modern hardware extensions (e.g. x86_64 with AVX512).

Effectively this adds avx512 and avx512vbmi support for linux x86_64 binaries and follows upstreams direction.

"nm -D ..." showing what symbols are available from what library builds linux x86_64:
# before
$ nm -D result/lib/libhs.so | grep -E "_hs_scan_stream"
00000000005bdbb0 T avx2_hs_scan_stream
00000000003b7140 T core2_hs_scan_stream
00000000004bb1b0 T corei7_hs_scan_stream

# after
$ nm -D result-/lib/libhs.so | grep -E "_hs_scan_stream"
00000000005c0bf0 T avx2_hs_scan_stream
00000000006d72a0 T avx512_hs_scan_stream
00000000007ff920 T avx512vbmi_hs_scan_stream
00000000003ba180 T core2_hs_scan_stream
00000000004be1f0 T corei7_hs_scan_stream

linux aarch64:

#before
$ nm -D result-old/lib/libhs.so | grep -E "_hs_scan_stream"
000000000033a7c4 T neon_hs_scan_stream
0000000000505994 T sve2_hs_scan_stream
0000000000421034 T sve_hs_scan_stream

#after
$ nm -D result-new/lib/libhs.so | grep -E "_hs_scan_stream"
000000000033a7c4 T neon_hs_scan_stream
0000000000506744 T sve2_hs_scan_stream
0000000000421034 T sve_hs_scan_stream

Result of nixpkgs-review pr 303207 run on x86_64-linux 1

5 packages built:
  • python311Packages.pyperscan
  • python311Packages.pyperscan.dist
  • python312Packages.pyperscan
  • python312Packages.pyperscan.dist
  • vectorscan

Result of nixpkgs-review pr 303207 run on aarch64-linux 1

5 packages built:
  • python311Packages.pyperscan
  • python311Packages.pyperscan.dist
  • python312Packages.pyperscan
  • python312Packages.pyperscan.dist
  • vectorscan

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin (ofborg did)
    • aarch64-darwin (ofborg did)
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@ofborg ofborg bot requested a review from vlaci April 10, 2024 21:39
@tnias tnias force-pushed the update/vectorscan-5.4.11 branch 3 times, most recently from 2b14e8c to 24af5ca Compare April 12, 2024 23:53
@tnias tnias marked this pull request as ready for review April 13, 2024 12:25
@tnias
Copy link
Contributor Author

tnias commented Apr 13, 2024

Hey @vlaci, could you check this PR? It uses a more recent boost version directly here in nixpkgs. I believe this allows you to drop some override stuff around libcxx/boost from your pyperscan repo.

Copy link
Member

@Atemu Atemu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some small things, otherwise this LGTM.

Result of nixpkgs-review pr 303207 run on x86_64-linux 1

5 packages built:
  • python311Packages.pyperscan
  • python311Packages.pyperscan.dist
  • python312Packages.pyperscan
  • python312Packages.pyperscan.dist
  • vectorscan

Comment on lines 63 to 69
checkPhase = ''
./bin/unit-hyperscan
'';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
checkPhase = ''
./bin/unit-hyperscan
'';
checkPhase = ''
runHook preCheck
./bin/unit-hyperscan
runHook postCheck
'';

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

applied

# potentially available more modern hardware extensions (e.g. x86_64 with AVX512).
cmakeFlags = [ (if enableShared then "-DBUILD_SHARED_LIBS=ON" else "BUILD_STATIC_LIBS=ON") ]
++
(if (stdenv.isLinux && (stdenv.isx86_32 || stdenv.isx86_64)) then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: We only have a limited set of hostPlatforms for which this applies; why not simply list them?

Suggested change
(if (stdenv.isLinux && (stdenv.isx86_32 || stdenv.isx86_64)) then
(if lib.elem stdenv.hostPlatform.system [ "x86_64-linux" "i686-linux" ] then

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

applied

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: You could do the same for the aarch64-linux check but I won't block on this.

@tnias tnias force-pushed the update/vectorscan-5.4.11 branch from 24af5ca to 111f1da Compare May 17, 2024 19:52
tnias added 2 commits May 17, 2024 21:57
The darwin (or libcxxStdenv) builds use C++17, which only seems to work
with boost >= 1.83.

The errors look like this:

include/boost/functional.hpp:454:62: error: no template named 'binary_function' in namespace 'boost::functional::detail'; did you mean '__binary_function'?
    class mem_fun1_ref_t : public boost::functional::detail::binary_function<T&, A, S>
                                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
include/boost/functional.hpp:46:24: note: '__binary_function' declared here
            using std::binary_function;
@tnias tnias force-pushed the update/vectorscan-5.4.11 branch from 111f1da to dbaa125 Compare May 17, 2024 20:01
Copy link
Member

@Atemu Atemu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Result of nixpkgs-review pr 303207 run on x86_64-linux 1

5 packages built:
  • python311Packages.pyperscan
  • python311Packages.pyperscan.dist
  • python312Packages.pyperscan
  • python312Packages.pyperscan.dist
  • vectorscan

@Atemu Atemu merged commit 1b765a2 into NixOS:master May 18, 2024
23 of 24 checks passed
@Atemu
Copy link
Member

Atemu commented May 18, 2024

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants