feat: shared headers, `rules_cc` support #221

sgammon · 2024-01-02T05:46:03Z

Summary

This changeset adds out_headers and additional_outputs, which, together, allow a developer to add additional outputs to a regular native_image build.

Because a header is generated per C entrypoint class, the developer needs to be able to declare as many as they want.

Additionally, a CcInfo provider needs to be returned from the native_image target to facilitate downsream support for rules_cc targets.

Tasklist

Related PRs and issues:

Other changes enclosed:

Initial support for rules_cc use downstream
File-based attributes now have multi-label alternatives
Debug builds now keep symbols and activate source-level debugging by default
New strict attribute applies latest strictness flags

Changelog

feat: add additional_outputs attribute
feat: add default_outputs attribute to opt-out of new behavior
feat: resolve and add appropriate CcInfo to shared library targets
feat: implement new native_image_shared_library rule
feat: implement output groups
feat: implement support for neverlink libs
feat: implement multi-label config attributes
feat: integrate graalvm build sandbox with bazel
feat: support for $(location ...) expansion
feat: support for make var expansion
fix: include data attribute in direct inputs
fix: provide GRAALVM_HOME to native-image build

graalvm/nativeimage/rules.bzl

fmeum · 2024-01-02T06:58:21Z

internal/native_image/common.bzl

+ # and `graal_isolate_dynamic.h` files are included. additional headers can be added by the `out_headers`
+ # attribute.
+ if ctx.attr.shared_library and ctx.attr.default_outputs:
+ additional_outputs.append(_GRAALVM_ISOLATE_HEADER)


Is there any case in which adding these headers would be problematic for a shared library build? If not, I think we should not add default_outputs for now. I am a bit worried of native_image gaining too many knobs that depend on each other.

these headers are produced automatically by the native-image build -- so if they aren't added here, they end up as undeclared outputs

Yeah, I'm in favor of adding them here. But maybe we could just get rid of default_outputs and have this behavior in place unconditionally?

I guess I'm concerned about multiple shared library targets within the same package, but that would be something the user has to deal with anyway?

In my downstream test codebase, I ended up needing to manage graal_*.h headers manually, because that codebase expected angle syntax for importing headers. So I needed to pull them into their own cc_library target with an include = path set. In that case, I flip default_outputs to False.

If the headers are considered outputs unconditionally by the rule, I worry there would be no easy way to prevent a file collision under this use case. Maybe I'm missing something.

The default functionality for the rules seems to align with including them as defaults, otherwise a user would have to declare them unconditionally, so I agree this isn't ideal and would love a better way.

FWIW I think I may merge additional_outputs and out_headers, but those may need to remain separate so that we can attach headers to providers. We could also sniff the filename, if that's preferable let me know. Open to your advice on all aspects above.

Could we separate the headers by having them emitted into a subdirectory? E.g., if we declare the headers for target foo with package relative path foo/graal_something.h, we can have multiple native images coexist in a package and also provide a way to disambiguate headers via the prefix.

I can check for hidden/expert native-image flags, but AFAIK those are calculated from the binary name, so they are already under the user's control via executable_name. I'm not sure about a subfolder with executable_name, I think we'd have to slice that and append it to -H:Path, because the binary name is passed separately from the output prefix (to native-image).

In any case, I agree with you, I'd like to find a cleaner way to do it.

(Lots of this is telling me that a dedicated shared_native_image rule might be a good idea, by the way, because the shared library function of native-image demands radically different Bazel behavior than an executable, as evidenced by executable_name being a little wonky in this case. Just food for later thought, your ideas are always welcome of course.)

Yes, I also like a dedicated rule better. For example, such a rule could be marked as non-executable and could also statically declare that it emits CcSharedLibraryInfo.

Sorry, to clarify, the regular header names are under the user's control, and follow the pattern of the executable_name (an executable name of libsample gets libsample.so and libsample.h, etc).

The standard GraalVM headers, i.e. graal_isolate.h and family, are generated unconditionally by native-image at those names. I don't see a way to override those.

Could we prepend foo/ to shared_library_name and then have a separate action symlink it to the location without the prefix? That would get us prefixed headers without the user having to refer to the shared library with the prefix.

internal/native_image/rules.bzl

This changeset adds `out_headers` and `additional_outputs`, which, together, allow a developer to add additional outputs to a regular `native_image` build. Because a header is generated per C entrypoint class, the developer needs to be able to declare as many as they want. Additionally, a `CcInfo` provider needs to be returned from the `native_image` target to facilitate downsream support for `rules_cc` targets. - feat: add `out_headers` and `additional_outputs` attributes - feat: add `default_outputs` attribute to opt-out of new behavior - feat: resolve and add `CcInfo` to shared library targets Signed-off-by: Sam Gammon <sam@elide.ventures>

Signed-off-by: Sam Gammon <sam@elide.ventures>

github-actions · 2024-01-13T07:06:13Z

Dependency Review

✅ No vulnerabilities or license issues found.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA 454dc5c.

Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

Scanned Manifest Files

Adds Makefile variable expansion support to the `extra_args` flags passed to `native-image`. Signed-off-by: Sam Gammon <sam@elide.ventures>

- feat: full implementation this time of make variable and location expansion - feat: managed build temp directory for `native-image` Relates-To: #244 Relates-To: #245 Signed-off-by: Sam Gammon <sam@elide.ventures>

- create base attributes, split into main/shared target attrs - create new macro and rules Signed-off-by: Sam Gammon <sam@elide.ventures>

- support `main_module` attribute - format with `main_module` attr if present Signed-off-by: Sam Gammon <sam@elide.ventures>

Signed-off-by: Sam Gammon <sam@elide.ventures>

- add transitive inputs for compile-time jars - mark applicable dependencies with `neverlink` Relates-To: #243 Signed-off-by: Sam Gammon <sam@elide.ventures>

Signed-off-by: Sam Gammon <sam@elide.ventures>

- add new `module_deps` attribute - add example of modular java use - wire in `main_module` attribute - process into separate module path (direct only) - append to `native-image` invocation as `--module-path` Signed-off-by: Sam Gammon <sam@elide.ventures>

Signed-off-by: Sam Gammon <sam@elide.ventures>

Relates-To: #220 Relates-To: bazelbuild/bazel#12714 Signed-off-by: Sam Gammon <sam@elide.ventures>

Signed-off-by: Sam Gammon <sam@elide.ventures>

- add rule and macro definition/entrypoint for test targets - rollup test targets dependencies in native image build - inject native test feature, runner as entrypoint - provide dependencies for junit5 Signed-off-by: Sam Gammon <sam@elide.ventures>

Signed-off-by: Sam Gammon <sam@elide.ventures>

fmeum · 2024-01-15T08:23:04Z

example/shared/BUILD.bazel

+ name = "mylib",
+ deps = [":java"],
+ strict = True,
+ lib_name = "libmylib", # becomes `libmylib.so`, `libmylib.dylib`, etc.


It would customarily be called mylib.dll on Windows though. How about letting the user specify just mylib? This matches the behavior of cc_* rules.

libmylib is literal, so users can specify mylib if they want to. Are you saying I should auto-add/manage the prefix? I'm happy to do so but I figured users would want it under their control.

fmeum · 2024-01-15T08:24:27Z

graalvm/artifacts/maven.bzl

 SDK = struct(
 artifact = "graal-sdk",
 group = "org.graalvm.sdk",
+ options = {"neverlink": True},
+ NATIVEIMAGE = struct(


Is this nesting intentional?

Yes, because it allows references like graalvm.catalog.SDK.NATIVEIMAGE in the artifact system

fmeum · 2024-01-15T08:25:28Z

graalvm/nativeimage/rules.bzl

- executable = True,
- fragments = [
+_TEST_NAME_CONDITION = select({
+ "@bazel_tools//src/conditions:windows": "%target%-test.exe",


These targets may go away soon, we should define our own ones.

Should I pull from @platforms instead? They ship default constraints, right?

fmeum · 2024-01-15T08:26:03Z

graalvm/nativeimage/rules.bzl

- fragments = [
+_TEST_NAME_CONDITION = select({
+ "@bazel_tools//src/conditions:windows": "%target%-test.exe",
+ "//conditions:default": "%target%-test",


Most test rule names will end with test anyway. Do we need the suffix?

No, I don't think we need it. Will drop

Signed-off-by: Sam Gammon <sam@elide.ventures>

sonarcloud · 2024-01-25T22:29:06Z

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

sgammon added feature Mainline feature work native-image Features and issues relating to the Native Image tool 🚧 WIP Work-in-progress, do not merge labels Jan 2, 2024

sgammon added this to the 1.0.0 milestone Jan 2, 2024

sgammon requested a review from fmeum January 2, 2024 05:46

sgammon self-assigned this Jan 2, 2024

fmeum requested changes Jan 2, 2024

View reviewed changes

sgammon force-pushed the feat/shared-headers branch 2 times, most recently from ad11d1e to 1a987f3 Compare January 2, 2024 23:16

sgammon linked an issue Jan 2, 2024 that may be closed by this pull request

Shared libraries with C/C++ targets #226

Open

sgammon added 8 commits January 5, 2024 18:45

fix: restore cpp toolchain symbol for classic rules

b6c47ef

Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: inclusion of native jvm headers in native-image builds

4e0f1bf

Signed-off-by: Sam Gammon <sam@elide.ventures>

chore: relock bazel mods

55af619

Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: provide GRAALVM_HOME to native-image builds

62d95e5

Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: gvm home parameter in classic rules

7b53b59

Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: off-by-one in graalvm home path

9d6d956

Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: include data attribute in direct inputs

223e0b8

Signed-off-by: Sam Gammon <sam@elide.ventures>

sgammon force-pushed the feat/shared-headers branch from 2fc91c1 to 0d24285 Compare January 13, 2024 07:05

feat: makevar expansion for extra_args

ed40131

Adds Makefile variable expansion support to the `extra_args` flags passed to `native-image`. Signed-off-by: Sam Gammon <sam@elide.ventures>

sgammon force-pushed the feat/shared-headers branch 3 times, most recently from 64eda9d to 687ff90 Compare January 13, 2024 23:58

feat: makevar expansion pt2, managed build tmpdir

fb6a8e2

- feat: full implementation this time of make variable and location expansion - feat: managed build temp directory for `native-image` Relates-To: #244 Relates-To: #245 Signed-off-by: Sam Gammon <sam@elide.ventures>

This was linked to issues Jan 13, 2024

Sandbox integration with GraalVM #244

Open

Support for Make variable expansion #245

Open

sgammon added 3 commits January 13, 2024 18:17

feat: initial work on shared native image rule

2a70e11

- create base attributes, split into main/shared target attrs - create new macro and rules Signed-off-by: Sam Gammon <sam@elide.ventures>

feat: modular native builds

3383a37

- support `main_module` attribute - format with `main_module` attr if present Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: override native image tmpdir for shared libs

e5ec856

Signed-off-by: Sam Gammon <sam@elide.ventures>

sgammon added 2 commits January 13, 2024 19:22

feat: support for neverlink

b61399c

- add transitive inputs for compile-time jars - mark applicable dependencies with `neverlink` Relates-To: #243 Signed-off-by: Sam Gammon <sam@elide.ventures>

chore: update blzmod lock

85cfed7

Signed-off-by: Sam Gammon <sam@elide.ventures>

sgammon linked an issue Jan 14, 2024 that may be closed by this pull request

Compatibility with neverlink #243

Open

sgammon force-pushed the feat/shared-headers branch from 687ff90 to 85cfed7 Compare January 14, 2024 03:24

sgammon added 12 commits January 13, 2024 19:50

fix: don't pull in jvm external from main rule entrypoint

f3a8030

Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: don't provide UnlockExperimentalOptions yet

e36ba06

Signed-off-by: Sam Gammon <sam@elide.ventures>

chore: update blzmod lock

167d4f6

Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: artifact options with structs

811d5fe

Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: proper print for debug message

ca03088

Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: main_class not available for shared libraries

2ed1f4f

Signed-off-by: Sam Gammon <sam@elide.ventures>

test: move to new native_image_shared_library rule

1e9c24b

Signed-off-by: Sam Gammon <sam@elide.ventures>

chore: build locally with strict java deps

442935d

Signed-off-by: Sam Gammon <sam@elide.ventures>

chore: add local example for shared library use

db9f530

Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: native example reflection config

b9ce2c0

Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: don't append -bin suffix to shared libs

1e79b60

Signed-off-by: Sam Gammon <sam@elide.ventures>

This was linked to issues Jan 14, 2024

Support for Output Groups #121

Open

Support for global javac options #246

Open

Modular Java builds #247

Open

sgammon added 8 commits January 13, 2024 21:27

chore: add third_party target for jacoco

a186a27

Relates-To: #220 Relates-To: bazelbuild/bazel#12714 Signed-off-by: Sam Gammon <sam@elide.ventures>

fix: sdk artifact coordinate

53686ca

Signed-off-by: Sam Gammon <sam@elide.ventures>

chore: update bzlmod locks

1aefe81

Signed-off-by: Sam Gammon <sam@elide.ventures>

chore: drop old disabled tests

1662194

Signed-off-by: Sam Gammon <sam@elide.ventures>

chore: add nativeimage artifact

2dcad3f

Signed-off-by: Sam Gammon <sam@elide.ventures>

chore: adopt new attr structure in macro entrypoints

8188160

Signed-off-by: Sam Gammon <sam@elide.ventures>

feat: native junit testing

a5b91cc

- add rule and macro definition/entrypoint for test targets - rollup test targets dependencies in native image build - inject native test feature, runner as entrypoint - provide dependencies for junit5 Signed-off-by: Sam Gammon <sam@elide.ventures>

chore: relock bzlmod for tests

454dc5c

Signed-off-by: Sam Gammon <sam@elide.ventures>

fmeum reviewed Jan 15, 2024

View reviewed changes

fix: expansion of profiles attr

72d7f6b

Signed-off-by: Sam Gammon <sam@elide.ventures>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: shared headers, `rules_cc` support #221

feat: shared headers, `rules_cc` support #221

sgammon commented Jan 2, 2024 •

edited

Loading

fmeum Jan 2, 2024

sgammon Jan 2, 2024

fmeum Jan 2, 2024

sgammon Jan 2, 2024 •

edited

Loading

fmeum Jan 5, 2024

sgammon Jan 5, 2024 •

edited

Loading

sgammon Jan 5, 2024 •

edited

Loading

fmeum Jan 5, 2024

sgammon Jan 5, 2024

fmeum Jan 5, 2024

github-actions bot commented Jan 13, 2024 •

edited

Loading

fmeum Jan 15, 2024

sgammon Jan 15, 2024

fmeum Jan 15, 2024

sgammon Jan 15, 2024

fmeum Jan 15, 2024

sgammon Jan 15, 2024

fmeum Jan 15, 2024

sgammon Jan 15, 2024

sonarcloud bot commented Jan 25, 2024

feat: shared headers, rules_cc support #221

Are you sure you want to change the base?

feat: shared headers, rules_cc support #221

Conversation

sgammon commented Jan 2, 2024 • edited Loading

Summary

Tasklist

Changelog

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgammon Jan 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgammon Jan 5, 2024 • edited Loading

Choose a reason for hiding this comment

sgammon Jan 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 13, 2024 • edited Loading

Dependency Review

Snapshot Warnings

Scanned Manifest Files

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Jan 25, 2024

Quality Gate passed

feat: shared headers, `rules_cc` support #221

feat: shared headers, `rules_cc` support #221

sgammon commented Jan 2, 2024 •

edited

Loading

sgammon Jan 2, 2024 •

edited

Loading

sgammon Jan 5, 2024 •

edited

Loading

sgammon Jan 5, 2024 •

edited

Loading

github-actions bot commented Jan 13, 2024 •

edited

Loading