Add code to cross compile x64 jdk8 on arm64 mac

[Haroon-Khel]

ref #3487, adoptium/infrastructure#2536 (comment), #3354 (comment)

Added the code/steps required to:

• Cross compile mac x64 jdk8 on arm64 mac
• Use Xcode11.7 for mac x64 jdk8 builds on both x64 and arm64 mac machines

Considering the successful cross compilation builds of jdk11 17 and 20 in adoptium/infrastructure#2536 (comment) I could add the --openjdk-target=x86_64-apple-darwin option to $CONFIGURE_ARGS_FOR_ANY_PLATFORM for any mac jdk cross compilation build not just jdk8 but thats up for discussion.

One of the consequences of this PR is that we can no longer build x64 JDK8 on 1014 systems as Xcode11.7 needs macOS 10.15.2+. Of the Macos10 machines we have, only test-macstadium-macos1015-x64-1 fits this

[Haroon-Khel]

The x64 mac jdk8 build github workflow already uses xcode11.7 to build jdk8, so we've been building it this whole time with xcode11.7 😄

Ive made the /Applications/Xcode-11.7.app link as that is how we have the path named on our machines. I've kept the /Applications/Xcode.app link but I do not think it is needed since the jdk11 build uses the command line tool's Xcode, which in the image is Xcode13, https://github.com/actions/runner-images/blob/main/images/macos/macos-11-Readme.md#tools

[sxa]

This looks a bit strange - is this because the github actions runners have it in a different place from the default? i.e. with a _ instead of an - and we have stuff hard coded to use the location with a -?

[Haroon-Khel]

Yes, the runner has it in /Applications/Xcode_11.7.app while the build expects it to be in /Applications/Xcode-11.7.app

[sxa]

Presumably one of those is the default location and the discrepancy isn't just because we've changed the default (in which case we'd be able to change it)?

[Haroon-Khel]

Running the changes by kicking off an x64 jdk8 mac build on https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-mac-x64-temurin/377/console

[Haroon-Khel]

Installed git via brew onto test-macstadium-macos11-arm64-1. Its executable is in /opt/homebrew/bin. Ive added this to the front of the PATH for arm64 mac machines building jdk8 in the latest changes. Im hoping the PATH stays this way for the whole build, including post build where git will be used

https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-mac-x64-temurin/392/console

[Haroon-Khel]

Thats strange, MACHINEARCITECTURE returned x86_64 on test-macstadium-macos11-arm64-1

13:17:28  MACHINEARCHITECTURE: x86_64
13:17:28  ARCHITECTURE: x64

[Haroon-Khel]

Ok ive got it to use the right git, /opt/homebrew/bin/git, which wont fail while xcode11.7 is being used. But the build is behaving unusually on test-macstadium-macos11-arm64-1. It wont detect that it is an arm64 system therefore it isnt cross compiling, it is building it as if it is an x64 system
https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-mac-x64-temurin/394/console

[sxa]

Thats strange, MACHINEARCITECTURE returned x86_64 on test-macstadium-macos11-arm64-1

13:17:28  MACHINEARCHITECTURE: x86_64
13:17:28  ARCHITECTURE: x64

Looking at the machine it looks like uname -m returns the correct thing by default but the JDK being used for the jenkins agent may be an x86-64 one. In that situation it's possible that it gets "stuck" in Rosetta which makes it return the "wrong" value when running through jenkins.

[sxa@fedora ~]$ ssh jenkins@199.7.163.51 uname -m
arm64
[sxa@fedora ~]$ ssh jenkins@199.7.163.51 file /Library/Java/JavaVirtualMachines/jdk-17/Contents/Home/bin/java
/Library/Java/JavaVirtualMachines/jdk-17/Contents/Home/bin/java: Mach-O 64-bit executable x86_64

Suggest comparing that with a machine that is behaving if you're seeing differences:

[Haroon-Khel]

Changed test-macstadium-macos11-arm64-1's agent to use an arm64 jdk17 binary. Kicked off a jdk8 build on test-macstadium-macos11-arm64-1 while Xcode11.7 is selected prebuild
https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-mac-x64-temurin/395/console
Ive added /opt/homebrew/bin/git to the machine's jenkins config.

Strange behaviour

16:14:46  The recommended git tool is: /opt/homebrew/bin/git
16:15:12  No credentials specified
16:15:12  Cloning the remote Git repository
16:15:12  Cloning repository https://github.com/adoptium/ci-jenkins-pipelines.git
16:15:12   > /opt/homebrew/bin/git init /Users/jenkins/workspace/build-scripts/jobs/jdk8u/jdk8u-mac-x64-temurin # timeout=10
16:15:13  Fetching upstream changes from https://github.com/adoptium/ci-jenkins-pipelines.git
16:15:13   > /opt/homebrew/bin/git --version # timeout=10
16:15:13   > git --version # 'git version 2.42.0'
16:15:13   > /opt/homebrew/bin/git fetch --tags --force --progress -- https://github.com/adoptium/ci-jenkins-pipelines.git +refs/heads/*:refs/remotes/origin/* # timeout=10
16:15:14   > /opt/homebrew/bin/git config remote.origin.url https://github.com/adoptium/ci-jenkins-pipelines.git # timeout=10
16:15:14   > /opt/homebrew/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
16:15:23  Avoid second fetch
16:15:23  Checking out Revision 16502bc800e90c614dd9aea46c4d221222b9161b (origin/master)
16:15:23   > /opt/homebrew/bin/git rev-parse origin/master^{commit} # timeout=10
16:15:24   > /opt/homebrew/bin/git config core.sparsecheckout # timeout=10
16:15:24   > /opt/homebrew/bin/git checkout -f 16502bc800e90c614dd9aea46c4d221222b9161b # timeout=10
16:15:28  Commit message: "Add JMOD signing retry and check curl rc (#823)"
[Pipeline] sh
16:15:30  + git clean -fdx
16:15:30  xcrun: error: unable to load libxcrun (dlopen(/Applications/Xcode-11.7.app/Xcode.app/Contents/Developer/usr/lib/libxcrun.dylib, 0x0005): could not use '/Applications/Xcode-11.7.app/Xcode.app/Contents/Developer/usr/lib/libxcrun.dylib' because it is not a compatible arch).

Uses /opt/homebrew/bin/git until git clean -fdx

[Haroon-Khel]

On test-macstadium-macos11-arm64-1

bash-3.2$ which git
/opt/homebrew/bin/git
bash-3.2$ git clean -fdx
bash-3.2$ /usr/bin/git clean -fdx
xcrun: error: unable to load libxcrun (dlopen(/Applications/Xcode-11.7.app/Xcode.app/Contents/Developer/usr/lib/libxcrun.dylib, 0x0005): could not use '/Applications/Xcode-11.7.app/Xcode.app/Contents/Developer/usr/lib/libxcrun.dylib' because it is not a compatible arch).

[Haroon-Khel]

Added /opt/homebrew/bin to the beginning of PATH in test-macstadium-macos11-arm64-1's jenkins config
https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-mac-x64-temurin/396/console
This looks good

[Haroon-Khel]

Build 396 looks good. But again I do not know why it didnt kick off the tests despite the parameters.

While test-macstadium-macos11-arm64-1 is still switched to xcode11.7 I want to see if it affects the test jobs. https://ci.adoptium.net/job/AQA_Test_Pipeline/191/console

[sxa]

16:15:24 > /opt/homebrew/bin/git checkout -f 16502bc800e90c614dd9aea46c4d221222b9161b # timeout=10
16:15:28 Commit message: "Add JMOD signing retry and check curl rc (#823)"
[Pipeline] sh
16:15:30 + git clean -fdx
16:15:30 xcrun: error: unable to load libxcrun (dlopen(/Applications/Xcode-11.7.app/Xcode.app/Contents/Developer/usr/lib/libxcrun.dylib, 0x0005): could not use '/Applications/Xcode-11.7.app/Xcode.app/Contents/Developer/usr/lib/libxcrun.dylib' because it is not a compatible arch).

Uses /opt/homebrew/bin/git until git clean -fdx

I would imagine that this is because jenkins (which does some of the checkouts etc. itself) has a definition on the machine under "Tool locations" that says git isexplicitly /opt/homebrew/bin/git so that's what it's invoking it as. Once it goes into running a shell script under [Pipeline] sh then it's just running git without a full path so it'll use the default on the machine

[Haroon-Khel]

ref #3492 (comment)
I think ive been looking at the screen for too long. The error wasnt with git but with xcrun whose error line is in the output in that comment. Nevermind that. Either way the xcrun error did not appear in the subsequent build https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-mac-x64-temurin/396/console. In between build 395 and 396 I added /opt/homebrew/bin to the PATH variable in test-macstadium-1's jenkins config.

But it has reared its head in the test jobs https://ci.adoptium.net/job/Test_openjdk8_hs_sanity.system_x86-64_mac/898/console

16:58:48  xcrun: error: unable to load libxcrun (dlopen(/Applications/Xcode-11.7.app/Xcode.app/Contents/Developer/usr/lib/libxcrun.dylib, 0x0005): could not use '/Applications/Xcode-11.7.app/Xcode.app/Contents/Developer/usr/lib/libxcrun.dylib' because it is not a compatible arch).

[Haroon-Khel]

xcrun is called in that test job by make
https://github.com/adoptium/aqa-tests/blob/74be47cf2956509483e1e77d1fb9929eb987df12/compile.sh#L43 and on arm64 mac it uses /usr/bin/make because there isnt a /opt/homebrew/bin/make

[Haroon-Khel]

Ive installed brew make onto test-macstadium-1 but brew installed it as gmake so i've linked it.

lrwxr-xr-x    1 administrator  admin     30  6 Oct 09:21 gmake -> ../Cellar/make/4.4.1/bin/gmake
lrwxr-xr-x    1 administrator  admin     23  6 Oct 09:29 make -> /opt/homebrew/bin/gmake

[Haroon-Khel]

Kicked off the test jobs again with xcode11.7 selected before the jobs. https://ci.adoptium.net/job/AQA_Test_Pipeline/192/console

https://ci.adoptium.net/job/Test_openjdk8_hs_sanity.functional_x86-64_mac/566/console looks good so far

[Haroon-Khel]

Ive installed make and git, and updated the jenkins config of the other 3 arm mac machines. Will add make and git to the playbooks

[Haroon-Khel]

Removed the debug bits. If https://ci.adoptium.net/job/AQA_Test_Pipeline/192/console completes without any errors related to this pr then this is pr can be merged (as far as I can see).

If this does get merged, then the jdk8 x64 mac build job will need to be updated to allow it to run on arm64 mac machines and not the 1014 x64 mac machines

[karianna]

Removed the debug bits. If https://ci.adoptium.net/job/AQA_Test_Pipeline/192/console completes without any errors related to this pr then this is pr can be merged (as far as I can see).

If this does get merged, then the jdk8 x64 mac build job will need to be updated to allow it to run on arm64 mac machines and not the 1014 x64 mac machines

https://ci.adoptium.net/job/AQA_Test_Pipeline/192/console failed I'm afraid

[Haroon-Khel]

linux jdk smoke test fails due to no space left on device

Attempt to copy /__w/temurin-build/temurin-build/aqa-tests/TKG/lib/jtreg_7_3_1_1.tar.gz to /__w/temurin-build/temurin-build/jvmtest/TKG/lib/jtreg_7_3_1_1.tar.gz using NIO Channels failed due to 'No space left on device'.

ping @andrew-m-leonard for review

[Haroon-Khel]

https://ci.adoptium.net/job/AQA_Test_Pipeline/192/console failed I'm afraid

The last few tests failed because the link to the binary expired. I am not bothered by the failures as we have already shown an x64 jdk8 binary built this way to pass the AQA test pipeline job. The 192 run was to see if test jobs could still run on an arm mac machine that has selected xcode11.7, and they can.