Enable reproducible comparing tests on jdk21+ macos #3844

sophia-guo · 2024-06-10T15:12:39Z

No description provided.

sophia-guo · 2024-06-10T19:47:47Z

I have tried both static mac and orka. Both no jq installed.

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10327/console
https://ci.adoptium.net/view/Test_grinder/job/Grinder/10328

Error: jq is not installed. Please install jq before running this script

Might be easy to install jq on static mac and orka?

sophia-guo · 2024-06-18T21:19:37Z

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10427/

17:21:04  Xcode 10.3
17:21:04  Build version 10G8
17:21:04  Xcode Build Success - Get Version
17:21:04  ERROR - No Xcode Installation Matching 15.2 Could Be Identified

Does this mean we can't do reproducing on this environment https://ci.adoptium.net/computer/test-macstadium-macos1014-x64-1/ due to the Xcode version not matching. @steelhead31 ?

steelhead31 · 2024-06-18T21:37:23Z

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10427/
17:21:04  Xcode 10.3
17:21:04  Build version 10G8
17:21:04  Xcode Build Success - Get Version
17:21:04  ERROR - No Xcode Installation Matching 15.2 Could Be Identified
Does this mean we can't do reproducing on this environment https://ci.adoptium.net/computer/test-macstadium-macos1014-x64-1/ due to the Xcode version not matching. @steelhead31 ?

Yup, that machines OS I think is too old for Xcode 15.2, and I think it has to be the same version of Xcode for reproducibility to work, @andrew-m-leonard can confirm this I think :)

andrew-m-leonard · 2024-06-19T08:22:02Z

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10427/
17:21:04  Xcode 10.3
17:21:04  Build version 10G8
17:21:04  Xcode Build Success - Get Version
17:21:04  ERROR - No Xcode Installation Matching 15.2 Could Be Identified
Does this mean we can't do reproducing on this environment https://ci.adoptium.net/computer/test-macstadium-macos1014-x64-1/ due to the Xcode version not matching. @steelhead31 ?
Yup, that machines OS I think is too old for Xcode 15.2, and I think it has to be the same version of Xcode for reproducibility to work, @andrew-m-leonard can confirm this I think :)

Yes, correct Xcode 15.2 requires MacOS 13.5 minimum : https://developer.apple.com/support/xcode/

sophia-guo · 2024-06-19T12:46:04Z

So we need extra label for macos to ensure the reproducing tests run on correct environment. Ether xcode15.2 or sw.os.osx.13_5. Two Mac static agents wont work for reproducing tests. https://ci.adoptium.net/label/ci.role.test&&hw.arch.x86&&%28sw.os.osx%7C%7Csw.os.mac%29/

Current orka ones are label as xcode15.0.1&&build&&mac&&x64 so can't work for reproducing either. Should we open an issue in infra to see if it's possible to update orka's os version?

sophia-guo · 2024-06-19T12:54:27Z

Interesting orka ones with label xcode15.0.1&&build&&mac&&x64 can do the reproducing build. https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-mac-x64-temurin/146/ https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-mac-x64-temurin/146/artifact/workspace/target/OpenJDK21U-sbom_x64_mac_hotspot_2024-05-31-22-18.json

andrew-m-leonard · 2024-06-19T13:42:36Z

Interesting orka ones with label xcode15.0.1&&build&&mac&&x64 can do the reproducing build. https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-mac-x64-temurin/146/ https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-mac-x64-temurin/146/artifact/workspace/target/OpenJDK21U-sbom_x64_mac_hotspot_2024-05-31-22-18.json

@sophia-guo so the Orka nodes are using Xcode 15.2, the 15.0.1 label has not been changed just because of the hassle involved! should have been called Xcode15 !

sophia-guo · 2024-06-26T20:13:09Z

orka updated with jq rerun adoptium/infrastructure#3593

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10470/console

sophia-guo · 2024-06-28T20:17:11Z

16:32:23  Make JDK Any Platform Argument List = 
16:32:23  --clean-git-repo --jdk-boot-dir /Users/admin/comp-jdk-build/jdk-20.0.2+9/Contents/Home --target-file-name OpenJDK21U-jdk_x64_mac_hotspot_21.0.4_5-ea.tar.gz --disable-shallow-git-clone -bjdk-21.0.4+5_adopt --create-jre-image --create-sbom --build-reproducible-date "2024-06-25 11:04:23"--user-openjdk-build-root-directory /Users/admin/comp-jdk-build/temurin-build/workspace/build/openjdkbuild/ --freetype-dir bundled --use-jep319-certs --create-debug-image  --configure-args "--disable-warnings-as-errors --openjdk-target=x86_64-apple-darwin --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk --with-extra-cxxflags='-mmacosx-version-min=10.9' --enable-dtrace --with-version-opt=ea"" --build-variant temurin jdk21u 
16:32:23  
16:32:23  Parameters Parsed Successfully
16:32:23  ---------------------------------------------
16:32:23  Building JDK...
16:32:23  sh: line 1: unexpected EOF while looking for matching `"'
16:32:23  sh: line 2: syntax error: unexpected end of file
16:32:23  -----------------------------------
16:32:23  Rebuild_Same_JDK_Reproducibility_Test_Mac_0_FAILED

parsing issue as SBOM --configure-args value format updated with double quotes. Parsing in mac script also need to update.

sophia-guo · 2024-08-16T21:04:31Z

New grinder with latest and label https://ci.adoptium.net/label/ci.role.test&&sw.os.osx&&orka&&hw.arch.x86/ to make sure running on orka

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10769/ - waiting

sophia-guo · 2024-08-20T15:42:57Z

Running with orka, issue message complains the Xcode can't be found.
https://ci.adoptium.net/view/Test_grinder/job/Grinder/10771/console.

Test is running on orka node , which should have Xcode 15.2 available #3844 (comment)
https://ci.adoptium.net/label/ci.role.test&&sw.os.osx&&orka&&hw.arch.x86/

11:27:51  Compiler Version Returned From SBOM: clang (clang/LLVM from Xcode 15.2)
11:27:51  Checking For Xcode Compilers In : /Applications/
11:27:51  Error - An Xcode Installaton Could Not Be Found In The Default Path : /Applications / Xcode 
11:27:51  -----------------------------------
11:27:51  Rebuild_Same_JDK_Reproducibility_Test_Mac_0_FAILED

sophia-guo · 2024-08-20T16:07:57Z

#3844 (comment)

@steelhead31 If it's related with how the Xcode installation is checked? #3844 (comment)

sophia-guo · 2024-08-20T16:10:45Z

rerun on build orka node https://ci.adoptium.net/label/xcode15.0.1&&build&&mac&&x64/ got different error from running on test orka node https://ci.adoptium.net/label/ci.role.test&&sw.os.osx&&orka&&hw.arch.x86/

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10775/console

12:01:08  ---------------------------------------------
12:01:08  Building JDK...
12:01:08  sh: line 1: unexpected EOF while looking for matching `"'
12:01:08  sh: line 2: syntax error: unexpected end of file
12:01:08  -----------------------------------
12:01:08  Rebuild_Same_JDK_Reproducibility_Test_Mac_0_FAILED
12:01:08  -----------------------------------

steelhead31 · 2024-08-21T09:20:09Z

@sophia-guo I've put some debugging in, and it does appear the script is checking correctly, but not finding an Xcode installation on that Orka node..

00:02:46.365 ---------------------------------------------
00:02:46.365 System Architecture: 64-bit (Intel)
00:02:46.365 SBOM & SYSTEM Architectures Match - All OK
00:02:46.365 ---------------------------------------------
00:02:46.365 Checking For Supported Compiler Versions...
00:02:46.365
00:02:46.365 Compiler Version Returned From SBOM: clang (clang/LLVM from Xcode 15.2)
00:02:46.366 Checking For Xcode Compilers In : /Applications/
00:02:46.366 Debug Xcode Find
00:02:46.366 total 0
00:02:46.366 drwxr-xr-x 3 root wheel 96 Oct 20 2023 Utilities
00:02:46.366 lrwxr-xr-x@ 1 root wheel 54 Oct 20 2023 Safari.app -> ../System/Cryptexes/App/System/Applications/Safari.app
00:02:46.366 drwxr-xr-x 3 root wheel 96 Jul 10 03:52 Packages.app
00:02:46.366 End Debug
00:02:46.366 Error - An Xcode Installaton Could Not Be Found In The Default Path : /Applications / Xcode

steelhead31 · 2024-08-21T09:33:53Z

@sophia-guo I've run a grinder here https://ci.adoptium.net/view/Test_grinder/job/Grinder/10782/console with these labels.. ( borrowed from a build job )

xcode15.0.1&&build&&mac&&x64&&!build-orka-macos14-arm64-fwc8f

So this issue is down to the orka test nodes not having Xcode installed... I'll take a look at the other issue now

steelhead31 · 2024-08-21T10:34:04Z

@sophia-guo I've fixed the issue with the repro script, and it now attempts to build, but its complaining about a missing SPEC.mk file..

Error: Cannot locate spec.gmk, given by SPEC=osx_x86-64.

@andrew-m-leonard may also be able to help in my absence.

See the grinder https://ci.adoptium.net/view/Test_grinder/job/Grinder/10791/console

sophia-guo · 2024-08-21T12:54:13Z

@steelhead31 so it does mean orka test nodes https://ci.adoptium.net/label/xcode15.0.1&&build&&mac&&x64/ are configured different from orka build nodes https://ci.adoptium.net/label/ci.role.test&&sw.os.osx&&orka&&hw.arch.x86/ ? So we will need to open an infra issue to have orka test nodes installed right xcode.

steelhead31 · 2024-08-21T13:10:26Z

@steelhead31 so it does mean orka test nodes https://ci.adoptium.net/label/xcode15.0.1&&build&&mac&&x64/ are configured different from orka build nodes https://ci.adoptium.net/label/ci.role.test&&sw.os.osx&&orka&&hw.arch.x86/ ? So we will need to open an infra issue to have orka test nodes installed right xcode.

Yes definitely, Im not sure how to do this though :)

sophia-guo · 2024-08-21T13:17:15Z

There are xcode related check error message even with build orka node. Are those the reasons of the failure build?

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10775/console

12:00:59  Compiler Version Returned From SBOM: clang (clang/LLVM from Xcode 15.2)
12:00:59  Checking For Xcode Compilers In : /Applications/
12:00:59  At Least One Xcode Installation Was Found ... Checking Versions
12:00:59  Current User Can Use Xcode Select... Continuing
12:00:59  Checking Path ... /Applications/Xcode.app
12:00:59  Xcode 15.2
12:00:59  Build version 15C500b
12:00:59  Xcode Build Success - Get Version
12:00:59  Checking Path ... /Applications/Xcode-11.7.app
12:00:59  xcrun: error: unable to load libxcrun (dlopen(/Applications/Xcode-11.7.app/Contents/Developer/usr/lib/libxcrun.dylib, 0x0005): tried: '/Applications/Xcode-11.7.app/Contents/Developer/usr/lib/libxcrun.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need '')), '/System/Volumes/Preboot/Cryptexes/OS/Applications/Xcode-11.7.app/Contents/Developer/usr/lib/libxcrun.dylib' (no such file), '/Applications/Xcode-11.7.app/Contents/Developer/usr/lib/libxcrun.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need ''))).
12:00:59  The Xcode Build Version For The Xcode Installation In /Applications/Xcode-11.7.app Could Not Be Obtained
12:00:59  Installed clang version: 15.0.0

steelhead31 · 2024-08-21T13:19:40Z

There are xcode related check error message even with build orka node. Are those the reasons of the failure build?

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10775/console

12:00:59  Compiler Version Returned From SBOM: clang (clang/LLVM from Xcode 15.2)
12:00:59  Checking For Xcode Compilers In : /Applications/
12:00:59  At Least One Xcode Installation Was Found ... Checking Versions
12:00:59  Current User Can Use Xcode Select... Continuing
12:00:59  Checking Path ... /Applications/Xcode.app
12:00:59  Xcode 15.2
12:00:59  Build version 15C500b
12:00:59  Xcode Build Success - Get Version
12:00:59  Checking Path ... /Applications/Xcode-11.7.app
12:00:59  xcrun: error: unable to load libxcrun (dlopen(/Applications/Xcode-11.7.app/Contents/Developer/usr/lib/libxcrun.dylib, 0x0005): tried: '/Applications/Xcode-11.7.app/Contents/Developer/usr/lib/libxcrun.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need '')), '/System/Volumes/Preboot/Cryptexes/OS/Applications/Xcode-11.7.app/Contents/Developer/usr/lib/libxcrun.dylib' (no such file), '/Applications/Xcode-11.7.app/Contents/Developer/usr/lib/libxcrun.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need ''))).
12:00:59  The Xcode Build Version For The Xcode Installation In /Applications/Xcode-11.7.app Could Not Be Obtained
12:00:59  Installed clang version: 15.0.0

Yes, they can be ignored, in addition to the xcode15.2 there is also an old Xcode11.7 installation, the script attempts to check the version and fails, but then switches to the correct 15.2 version, I'll see if I can tidy up the output, but its safe to ignore in the meantime.

sophia-guo · 2024-08-21T18:57:32Z

#3844 (comment)

Just realized it's rebuilding x64mac on aarch64mac if using build label https://ci.adoptium.net/label/xcode15.0.1&&build&&mac&&x64/ , not sure if that's the reason for themissing SPEC.mk file?

adoptium/infrastructure#3720 (comment)

sophia-guo · 2024-09-06T20:08:38Z

@sophia-guo I've fixed the issue with the repro script, and it now attempts to build, but its complaining about a missing SPEC.mk file..

@steelhead31 what's the fix? I did some force push and lost the changes you did, could you redo it so I can do a rebase?

@andrew-m-leonard any idea about the missing spec.mk file error? The original jdk is mac-x64 jdk( build on aarch64 ) the reproduced jdk is also mac-x64 ( build on aarch64). test mac doesn't have xcode installed adoptium/infrastructure#3720. I'm not sure if #3923 fixed this issue, as I lost the changes of @steelhead31 and can't rerun build to get this issue yet.

13:46:45  Error: Cannot locate spec.gmk, given by SPEC=osx_x86-64.
13:46:45  /Users/admin/comp-jdk-build/temurin-build/workspace/build/src/make/Init.gmk:124: *** Cannot continue.  Stop.
13:46:45  OpenJDK make failed, archiving make failed logs
13:46:45  Archiving and compressing with gzip
13:46:45  
13:46:45  real	0m0.003s
13:46:45  user	0m0.001s
13:46:45  sys	0m0.002s

https://ci.adoptium.net/view/Test_grinder/job/Grinder/10791/console

Another question @andrew-m-leonard would be for mac64 build from temurin ( on aarch64) reproduced comparing jdk should be built on x64 or aarch64? Or it doesn't matter there should be no difference. @sxa has the same question adoptium/infrastructure#3720 (comment)

andrew-m-leonard · 2024-09-09T09:40:34Z

@sophia-guo it's not clear what's wrong there, it's building on a mac arm64 node, so that is good. The "configure" has completed successfully, so it must be the "make" step that is failing, will need some extra debug there to work out why

andrew-m-leonard · 2024-09-09T15:16:59Z

@sophia-guo The problem I think is due to the TKG setting the "SPEC" make variable. I can see from the

temurin-build/test/system/reproducibleCompare/reproducible.mk

Line 20 in fdeec62

ifneq (,$(findstring win,$(SPEC)))

that it looks at a "SPEC" variable, within the make environment.
We need to ensure SPEC is "unset" before calling the mac repro compare build....

If I try this I get the same error:

anleonar@anleonar-mac macosx-aarch64-server-release % make images SPEC=asdfasf
Error: Cannot locate spec.gmk, given by SPEC=asdfasf.
/Users/anleonar/workspace/temurin-build/workspace/build/src/make/Init.gmk:124: *** Cannot continue.  Stop.

Signed-off-by: Sophia Guo <sophia.gwf@gmail.com>

sophia-guo marked this pull request as draft June 10, 2024 15:12

github-actions bot added the macos Issues that affect or relate to the MAC OS label Jun 10, 2024

sophia-guo force-pushed the mac branch 5 times, most recently from 5dd3e43 to 4e6a29f Compare June 10, 2024 19:30

sophia-guo mentioned this pull request Jun 11, 2024

update mac test agents with jq installation adoptium/infrastructure#3593

Closed

sophia-guo force-pushed the mac branch 2 times, most recently from a8cf660 to 524ffcb Compare August 16, 2024 20:49

sophia-guo self-assigned this Aug 16, 2024

sophia-guo force-pushed the mac branch 3 times, most recently from 0bc4005 to 3132741 Compare August 20, 2024 15:40

sophia-guo mentioned this pull request Aug 21, 2024

Xcode 15.2 on test orka nodes adoptium/infrastructure#3720

Closed

sophia-guo force-pushed the mac branch from 2633436 to 2c7926b Compare August 22, 2024 18:48

sophia-guo force-pushed the mac branch 2 times, most recently from 4bc0f37 to fdeec62 Compare September 9, 2024 14:33

sophia-guo force-pushed the mac branch 2 times, most recently from abc7b38 to b6c0af0 Compare September 9, 2024 15:30

sophia-guo added 5 commits September 9, 2024 13:27

Renable reproducible compare on mac

3efb001

Signed-off-by: Sophia Guo <sophia.gwf@gmail.com>

Permission issue?

2c1444b

Signed-off-by: Sophia Guo <sophia.gwf@gmail.com>

Update jdk tar path

03c0b0a

Signed-off-by: Sophia Guo <sophia.gwf@gmail.com>

rebase

ffddf0e

Signed-off-by: Sophia Guo <sophia.gwf@gmail.com>

Add execute permission to macos_repro_build_compare.sh

c5b1921

Signed-off-by: Sophia Guo <sophia.gwf@gmail.com>

sophia-guo force-pushed the mac branch from b6c0af0 to ef05815 Compare September 9, 2024 17:27

Using similar parsing method as linux

ec43d00

Signed-off-by: Sophia Guo <sophia.gwf@gmail.com>

sophia-guo force-pushed the mac branch from ef05815 to ec43d00 Compare September 9, 2024 17:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable reproducible comparing tests on jdk21+ macos #3844

Enable reproducible comparing tests on jdk21+ macos #3844

sophia-guo commented Jun 10, 2024

sophia-guo commented Jun 10, 2024

sophia-guo commented Jun 18, 2024 •

edited

Loading

steelhead31 commented Jun 18, 2024

andrew-m-leonard commented Jun 19, 2024

sophia-guo commented Jun 19, 2024

sophia-guo commented Jun 19, 2024

andrew-m-leonard commented Jun 19, 2024

sophia-guo commented Jun 26, 2024 •

edited

Loading

sophia-guo commented Jun 28, 2024

sophia-guo commented Aug 16, 2024 •

edited

Loading

sophia-guo commented Aug 20, 2024 •

edited

Loading

sophia-guo commented Aug 20, 2024 •

edited

Loading

sophia-guo commented Aug 20, 2024

steelhead31 commented Aug 21, 2024

steelhead31 commented Aug 21, 2024

steelhead31 commented Aug 21, 2024 •

edited by sophia-guo

Loading

sophia-guo commented Aug 21, 2024

steelhead31 commented Aug 21, 2024

sophia-guo commented Aug 21, 2024

steelhead31 commented Aug 21, 2024

sophia-guo commented Aug 21, 2024

sophia-guo commented Sep 6, 2024

andrew-m-leonard commented Sep 9, 2024

andrew-m-leonard commented Sep 9, 2024

Enable reproducible comparing tests on jdk21+ macos #3844

Are you sure you want to change the base?

Enable reproducible comparing tests on jdk21+ macos #3844

Conversation

sophia-guo commented Jun 10, 2024

sophia-guo commented Jun 10, 2024

sophia-guo commented Jun 18, 2024 • edited Loading

steelhead31 commented Jun 18, 2024

andrew-m-leonard commented Jun 19, 2024

sophia-guo commented Jun 19, 2024

sophia-guo commented Jun 19, 2024

andrew-m-leonard commented Jun 19, 2024

sophia-guo commented Jun 26, 2024 • edited Loading

sophia-guo commented Jun 28, 2024

sophia-guo commented Aug 16, 2024 • edited Loading

sophia-guo commented Aug 20, 2024 • edited Loading

sophia-guo commented Aug 20, 2024 • edited Loading

sophia-guo commented Aug 20, 2024

steelhead31 commented Aug 21, 2024

steelhead31 commented Aug 21, 2024

steelhead31 commented Aug 21, 2024 • edited by sophia-guo Loading

sophia-guo commented Aug 21, 2024

steelhead31 commented Aug 21, 2024

sophia-guo commented Aug 21, 2024

steelhead31 commented Aug 21, 2024

sophia-guo commented Aug 21, 2024

sophia-guo commented Sep 6, 2024

andrew-m-leonard commented Sep 9, 2024

andrew-m-leonard commented Sep 9, 2024

sophia-guo commented Jun 18, 2024 •

edited

Loading

sophia-guo commented Jun 26, 2024 •

edited

Loading

sophia-guo commented Aug 16, 2024 •

edited

Loading

sophia-guo commented Aug 20, 2024 •

edited

Loading

sophia-guo commented Aug 20, 2024 •

edited

Loading

steelhead31 commented Aug 21, 2024 •

edited by sophia-guo

Loading