jdk_tools_1 jpackage tests in Windows JDK 17 & 21 fail due to signing issue #4831

jiekang · 2023-10-19T20:56:19Z

From examining:
Failed run: https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_windows_testList_2/22/testReport/tools_jpackage_windows_WinConsoleTest/java/WinConsoleTest/
Successful run:
https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_windows_testList_2/18/testReport/tools_jpackage_windows_WinConsoleTest/java/WinConsoleTest/

Almost all of the failures mention something similar to:

[01:24:25.973] TRACE: exec: Execute tool provider [jpackage --input .\test.1ac8b3a6\input --dest .\test.1ac8b3a6\output --name WinConsoleTest --type app-image --main-jar hello.jar --main-class Hello --win-console --verbose](15)...
[2023/10/19 01:24:32.336, jpackage.dll (PID: 5868, TID: 10192), jpackage.cpp:51 (Java_jdk_jpackage_internal_ExecutableRebrander_lockResource)]
	ERROR: Exception with message 'ResourceEditor.cpp(37) at ResourceEditor::FileLock::FileLock(): BeginUpdateResource(C:\Jenkins\workspace\Test_openjdk21_hs_extended.openjdk_x86-64_windows_testList_2\aqa-tests\TKG\output_16976697789912\jdk_tools_1\work\scratch\0\.\test.1ac8b3a6\output\WinConsoleTest\WinConsoleTest.exe) failed. System error [1006](system error 1006 (The volume for a file has been externally altered so that the opened file is no longer valid))' caught
[...]
[01:24:32.337] java.lang.RuntimeException: Failed to lock: C:\Jenkins\workspace\Test_openjdk21_hs_extended.openjdk_x86-64_windows_testList_2\aqa-tests\TKG\output_16976697789912\jdk_tools_1\work\scratch\0\.\test.1ac8b3a6\output\WinConsoleTest\WinConsoleTest.exe

The notable output in the failure run that isn't in the successful run is:

[01:24:32.336] Warning: Windows Defender may prevent jpackage from functioning. If there is an issue, it can be addressed by either disabling realtime monitoring, or adding an exclusion for the directory "C:\Users\jenkins\AppData\Local\Temp\jdk.jpackage7026907677576338441".

Test Info
Test Name: jdk_tools_1
Test Duration: 18 min 1 sec
Machine: build-azure-win2012r2-x64-2
TRSS link for the test output: http://20.90.182.165/output/test?id=65301ea720748f006f8f3b66

Build Info
Build Name: Test_openjdk17_hs_extended.openjdk_x86-64_windows_testList_0
Jenkins Build start time: Oct 18 2023, 06:21 am
Jenkins Build URL: https://ci.adoptium.net/job/Test_openjdk17_hs_extended.openjdk_x86-64_windows_testList_0/117/
TRSS link for the build: http://20.90.182.165/allTestsInfo?buildId=65301def20748f006f8f3776

Java Version
openjdk version "17.0.9" 2023-10-17
OpenJDK Runtime Environment Temurin-17.0.9+9 (build 17.0.9+9)
OpenJDK 64-Bit Server VM Temurin-17.0.9+9 (build 17.0.9+9, mixed mode, sharing)

This test has been failed 3 times since Sep 24 2023, 09:31 am
Java Version when the issue first seen
openjdk version "17.0.9-beta" 2023-10-17
OpenJDK Runtime Environment Temurin-17.0.9+7-202309241213 (build 17.0.9-beta+7-202309241213)
OpenJDK 64-Bit Server VM Temurin-17.0.9+7-202309241213 (build 17.0.9-beta+7-202309241213, mixed mode, sharing)
Jenkins Build URL: https://ci.adoptium.net/job/Test_openjdk17_hs_extended.openjdk_x86-64_windows_testList_0/115/

The test failed on machine build-azure-win2012r2-x64-2 1 times
The test failed on machine test-azure-win2019-x64-1 1 times
The test failed on machine test-azure-win2016-x64-1 1 times

Rerun in Grinder

The text was updated successfully, but these errors were encountered:

sophia-guo · 2023-10-20T13:24:56Z

According to history tests only passed on test-azure-win2012r2-x64-1 and test-azure-win2012r2-x64-3.

test-azure-win2012r2-x64-1 is offline and rerun on test-azure-win2012r2-x64-3 got the same failure. https://ci.adoptium.net/job/Grinder/7783/ . Current tests failed on all machines.

sophia-guo · 2023-10-20T13:27:11Z

@sxa Could you take a look at this?

sxa · 2023-10-20T14:01:59Z

Brain is exploding today with too many things going on ... I've kicked off https://ci.adoptium.net/job/Grinder/7834/ on the previous GA level to verify whether this is somehow the result of a regression while I try and figure out the next step...

sxa · 2023-10-20T14:03:12Z

@sophia-guo @jiekang Can you try running it on one of the TC windows machines?

sxa · 2023-10-20T15:03:50Z

https://ci.adoptium.net/job/Grinder/7834/

OK that one with jdk-17.0.8 GA failed a couple of tzdata tests but passed the others that you were having trouble with - this is a direct comparison running on the same test-azure-win2012r2-x64-3 as your Grinder 7783, so this would seem to be something different in the new build rather than something having changed in the infrastructure.

Direct link to one particular test:

sophia-guo · 2023-10-23T14:47:01Z

I got same failure on TC windows machines temurin-compliance/job/Grinder/3828, temurin-compliance/job/Grinder/3827

sophia-guo · 2023-10-23T14:49:14Z

@judovana Could you please take a look at this? If it happens to red hat jdk?

sophia-guo · 2023-10-24T19:39:56Z

Running jdk21 with azul latest JDK21.0.1+12 on different machines passed

temurin-compliance/job/Grinder/3843 - win11-x64-4cpu-f5z0vu
temurin-compliance/job/Grinder/3846 - win2019-x64-4cpu-ohGh0x -one tests failure due to different issue `light.exe : error

00:26:34  Run c:/Users/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java -version
00:26:34  =JAVA VERSION OUTPUT BEGIN=
00:26:34  openjdk version "21.0.1" 2023-10-17 LTS
00:26:34  OpenJDK Runtime Environment Zulu21.30+15-CA (build 21.0.1+12-LTS)
00:26:34  OpenJDK 64-Bit Server VM Zulu21.30+15-CA (build 21.0.1+12-LTS, mixed mode, sharing)

Temurin build consistently hit the same issue on all machines.
temurin-compliance/job/Grinder/3845 -- failed.

Need deeper investigation
(
The latest successful one is on Sep 17 on test-azure-win2012r2-x64-1 ( offline)

Haroon-Khel · 2023-10-24T23:16:31Z

Warning: Windows Defender may prevent jpackage from functioning. If there is an issue, it can be addressed by either disabling realtime monitoring, or adding an exclusion for the directory "C:\Users\jenkins\AppData\Local\Temp\jdk.jpackage7026907677576338441".

I temporarily turned off windows defender on test-azure-win2022-x64-1 in light of the warning message above and kicked off jdk_tools jdk17

https://ci.adoptium.net/job/Grinder/7862/console still 57 failures

Karm · 2023-10-25T12:26:46Z

FWIW:

this is how I disable defender: Set-MpPreference -DisableRealtimeMonitoring $true
this is the tool I use to check who keeps a lock: https://learn.microsoft.com/en-us/sysinternals/downloads/handle

sxa · 2023-10-25T13:55:13Z

Noting that Sophia has confirmed in an edit to her previous comment that Azul and Temurin are showing different results when the test is run on the same machine.

judovana · 2023-10-25T14:13:00Z

@RadekCap do we have windows defender on on out vms? I really doubt

RadekCap · 2023-10-25T14:25:41Z

Windows Defender is disabled because it prevents running code from unknown locations. You can't create a folder in a disk root and run exe file from there. It's stopped by Defender.

sophia-guo · 2023-10-25T14:42:58Z

Noting that Sophia has confirmed in an edit to #4831 (comment) that Azul and Temurin are showing different results when the test is run on the same machine.

I think it's still worth disabling the defender on test-azure-win2012r2-x64-3 to see if tests behavior will be different as the tests result does suggest that might affect the tests. @Haroon-Khel

The latest successful run in on Sep 18 on test-azure-win2012r2-x64-1 .

sophia-guo · 2023-10-25T14:49:22Z

Disable the defender might help to find the real cause of the failure.

Haroon-Khel · 2023-10-25T15:30:27Z

Set-MpPreference -DisableRealtimeMonitoring $true

This method does not seem to work on test-azure-win2012r2-x64-3. Neither does the method I used to disable defender on test-azure-win2022-x64-1. From some google searches, i'm beginning to think that defender is not installed on test-azure-win2012r2-x64-3

Haroon-Khel · 2023-10-25T15:52:23Z

It is also worth noting that in the rerun on test-azure-win2022-x64-1 after disabling defender, https://ci.adoptium.net/job/Grinder/7862/, though the test tools/jpackage/windows/WinConsoleTest.java.WinConsoleTest failed again, the error log does not include the disable defender warning

Haroon-Khel · 2023-10-25T17:54:49Z

Some notes:
(JDK21)
Running jdk_tools with this release on all available machines:

test-ibmcloud-win2012r2-x64-2 https://ci.adoptium.net/job/Grinder/7876/
build-azure-win2012r2-x64-2 https://ci.adoptium.net/job/Grinder/7877/
test-azure-win2016-x64-1 https://ci.adoptium.net/job/Grinder/7878/
test-azure-win2022-x64-1 https://ci.adoptium.net/job/Grinder/7879/
test-azure-win2019-x64-1 https://ci.adoptium.net/job/Grinder/7880/
build-azure-win2022-x64-2 https://ci.adoptium.net/job/Grinder/7881/
build-ibmcloud-win2012r2-x64-1 https://ci.adoptium.net/job/Grinder/7882/
build-ibmcloud-win2022-x64-1 https://ci.adoptium.net/job/Grinder/7883/
build-azure-win2012r2-x64-1 https://ci.adoptium.net/job/Grinder/7884/

The main failing tests are the following 24

TEST: tools/jpackage/share/jdk/jpackage/tests/CookedRuntimeTest.java
TEST: tools/jpackage/share/jdk/jpackage/tests/DotInNameTest.java
TEST: tools/jpackage/share/jdk/jpackage/tests/JavaOptionsEqualsTest.java#id0
TEST: tools/jpackage/share/jdk/jpackage/tests/AppVersionTest.java
TEST: tools/jpackage/share/jdk/jpackage/tests/JavaOptionsEqualsTest.java#id1
TEST: tools/jpackage/share/jdk/jpackage/tests/JLinkOptionsTest.java
TEST: tools/jpackage/share/jdk/jpackage/tests/JavaOptionsTest.java
TEST: tools/jpackage/share/jdk/jpackage/tests/ModulePathTest.java
TEST: tools/jpackage/share/jdk/jpackage/tests/ModulePathTest2.java
TEST: tools/jpackage/share/jdk/jpackage/tests/MultipleJarAppTest.java
TEST: tools/jpackage/share/jdk/jpackage/tests/NoMPathRuntimeTest.java
TEST: tools/jpackage/share/jdk/jpackage/tests/UnicodeArgsTest.java
TEST: tools/jpackage/share/jdk/jpackage/tests/MainClassTest.java
TEST: tools/jpackage/share/AppLauncherEnvTest.java
TEST: tools/jpackage/share/ArgumentsTest.java
TEST: tools/jpackage/share/EmptyFolderTest.java
TEST: tools/jpackage/share/PerUserCfgTest.java
TEST: tools/jpackage/share/IconTest.java
TEST: tools/jpackage/windows/Win8282351Test.java
TEST: tools/jpackage/windows/WinConsoleTest.java
TEST: tools/jpackage/windows/WinLongVersionTest.java#id1
TEST: tools/jpackage/windows/WinRenameTest.java
TEST: tools/jpackage/windows/WinResourceTest.java
TEST: tools/jpackage/share/ServiceTest.java

For test jobs showing 48 failures, the TAP files consist of 24 duplicated tests
Similarly for test jobs showing 114 failures, 57 are non duplicates and within these there are the 24 from above. The remaining 33 pass in the test jobs which only have 48 failures. TLDR 24 actual failures (these do appear in the original failure test job)

I've rebuilt test-azure-win2019-x64-1's grinder after turning its defender off
https://ci.adoptium.net/job/Grinder/7886/console

jdk_tools using the last GA:
test-azure-win2012r2-x64-3 https://ci.adoptium.net/job/Grinder/7864/ None of the 24 failures above

sxa · 2023-10-25T21:18:25Z

Previous comment seems consistent with what I saw (which is good!).

sxa · 2023-10-25T23:01:17Z

The latest successful one is on Sep 17 on test-azure-win2012r2-x64-1 ( offline)

Some more Grinders to gether more data points ... All using the master branch of aqa-tests due to an issue with the AIX .1 respin on the 0.9.9-release branch

Build	Grinder	scm Ref	Bld ref
2023-09-08 (SBOM)	7901 ✅	a8a6879da05	fe5f881d34
2023-09-10 (SBOM)	7902 ✅	jdk-17.0.9+6_adopt-2-ga8a6879da05	f35f881d
2023-09-13 (SBOM)	7903 ✅	a8a6879da05	d42e421e83
2023-09-18 (SBOM)	7904 ✅	4c62c88cde0	c8b16c211
2023-09-20 (SBOM)	7905 ✅	4c62c88cde0	c8b16c211
09-13 rebuild? (SBOM)	7906 ❌	jdk-17.0.9+6_adopt-277-ga8a6879da05	c93b0ff62
2023-09-22 (SBOM)	7911 ✅	716c5db182e	5b7acc5f
2023-10-05 (SBOM)	7913 ❌	716c5db182e	467055a
2023-10-04 (SBOM)	7915 ❌	716c5db182e	751e5c84
2023-09-29 (SBOM)	7916 ✅	716c5db182e	716c5db182
2023-09-27 (SBOM)	7917 ✅
2023-09-25 (SBOM)	7919 ✅
2023-09-22 (SBOM)	7921
17.0.8 GA	7914 ✅

Note: 2023-09-15 had no win64 build
Once those are complete we should verify if there is any connection with

sxa · 2023-10-26T09:49:19Z

Based on the table above there was a difference in the windows signing process between the build that worked (29th September) and the first that didn't (4th October). FYI @andrew-m-leonard

andrew-m-leonard · 2023-10-26T10:00:39Z

Significance of jpackage is it is signed within the JMOD now:

// JDK 16 + jpackage needs to be signed as well
                                            "${base_path}/jdk/modules/jdk.jpackage/jdk/jpackage/internal/resources/jpackageapplauncher"

need to investigate

andrew-m-leonard · 2023-10-26T10:16:45Z

jmod jdk.jpackages file: classes\jdk\jpackage\internal\resources\jpackageapplauncher.exe
is 0 bytes long!
Wonder if EF signing failure went un-checked... leaving 0 length file

sxa · 2023-10-26T10:16:59Z

ARGH! That's in the pipelines which I wasn't looking it. On this basis I feel we need a ci-jenkins-pipelines reference in the SBOM as well which we don't currently have.

OK So adoptium/ci-jenkins-pipelines@9ff239d went in around the same time (PR 822)

sxa · 2023-10-26T10:17:31Z

jmod jdk.jpackages file: classes\jdk\jpackage\internal\resources\jpackageapplauncher.exe is 0 bytes long! Wonder if EF signing failure went un-checked... leaving 0 length file

Not sure but it's consistent with all the builds from a specific date based on the table above.
Should jpackageapplauncher.exe show up in https://ci.adoptium.net/job/build-scripts/job/release/job/sign_build/26953/console along with the other files?

andrew-m-leonard · 2023-10-26T10:33:24Z

So looks like it's not being signed, as it's not being stashed... I'll work on a fix

andrew-m-leonard · 2023-10-26T11:07:41Z

jmod jdk.jpackages file: classes\jdk\jpackage\internal\resources\jpackageapplauncher.exe is 0 bytes long! Wonder if EF signing failure went un-checked... leaving 0 length file

Not sure but it's consistent with all the builds from a specific date based on the table above. Should jpackageapplauncher.exe show up in https://ci.adoptium.net/job/build-scripts/job/release/job/sign_build/26953/console along with the other files?

No, jpackageapplauncher is unique as only being present in the JMOD, the signing job signs signs the visible jdk bin exe's the JMOD exe signing happens during the build.

gdams · 2023-10-26T11:11:17Z

jmod jdk.jpackages file: classes\jdk\jpackage\internal\resources\jpackageapplauncher.exe is 0 bytes long! Wonder if EF signing failure went un-checked... leaving 0 length file

Not sure but it's consistent with all the builds from a specific date based on the table above. Should jpackageapplauncher.exe show up in https://ci.adoptium.net/job/build-scripts/job/release/job/sign_build/26953/console along with the other files?

No, jpackageapplauncher is unique as only being present in the JMOD, the signing job signs signs the visible jdk bin exe's the JMOD exe signing happens during the build.

Yeah I remember having to specifically edge-case this with the macOS binaries when this change was first made to the JDK

gdams · 2023-10-26T11:16:13Z

@andrew-m-leonard see adoptium/ci-jenkins-pipelines#158

andrew-m-leonard · 2023-10-26T11:39:38Z

@andrew-m-leonard see adoptium/ci-jenkins-pipelines#158

Thanks @gdams yes, it was my misstake in adding the Windows signing of jpackagelauncher, not ensuring the correct windows exe's are signed!!

andrew-m-leonard · 2023-10-26T11:40:08Z

@sxa @Haroon-Khel fyi, we will also need to re-publish jdk17u Win32: http://20.90.182.165/output/test?id=65304a2220748f006f8fd465
as I suspect will have same issue.

andrew-m-leonard · 2023-10-26T15:17:47Z

Fixed by adoptium/ci-jenkins-pipelines#831

andrew-m-leonard · 2023-10-26T16:10:44Z

Re-builds in-progress for release:
jdk-17: win_64 https://ci.adoptium.net/job/build-scripts/job/release-openjdk17-pipeline/43/
jdk-17: win_32 https://ci.adoptium.net/job/build-scripts/job/release-openjdk17-pipeline/46/
jdk-21: win_64 https://ci.adoptium.net/job/build-scripts/job/release-openjdk21-pipeline/14/

sophia-guo mentioned this issue Oct 20, 2023

October 2023 release jdk17 Triage #4826

Closed

sophia-guo transferred this issue from adoptium/aqa-tests Oct 20, 2023

sophia-guo transferred this issue from adoptium/infrastructure Oct 23, 2023

This was referenced Oct 23, 2023

October 2023 Release Status per Platform, Version & Binary Type adoptium/temurin#6

Closed

October 2023 release jdk21 Triage #4827

Closed

KlemenDEV mentioned this issue Oct 25, 2023

Update JDK to Adoptium 17.0.9+9 MCreator/MCreator#4331

Merged

sxa assigned sxa and andrew-m-leonard Oct 26, 2023

andrew-m-leonard closed this as completed Oct 26, 2023

sxa changed the title ~~jdk_tools_1 jpackage tests in JDK 17 & 21 fail due to Windows Defender~~ jdk_tools_1 jpackage tests in Windows JDK 17 & 21 fail due to signing issue Oct 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jdk_tools_1 jpackage tests in Windows JDK 17 & 21 fail due to signing issue #4831

jdk_tools_1 jpackage tests in Windows JDK 17 & 21 fail due to signing issue #4831

jiekang commented Oct 19, 2023

sophia-guo commented Oct 20, 2023

sophia-guo commented Oct 20, 2023

sxa commented Oct 20, 2023 •

edited

Loading

sxa commented Oct 20, 2023

sxa commented Oct 20, 2023 •

edited

Loading

sophia-guo commented Oct 23, 2023

sophia-guo commented Oct 23, 2023

sophia-guo commented Oct 24, 2023 •

edited

Loading

Haroon-Khel commented Oct 24, 2023

Karm commented Oct 25, 2023

sxa commented Oct 25, 2023

judovana commented Oct 25, 2023

RadekCap commented Oct 25, 2023

sophia-guo commented Oct 25, 2023

sophia-guo commented Oct 25, 2023

Haroon-Khel commented Oct 25, 2023

Haroon-Khel commented Oct 25, 2023

Haroon-Khel commented Oct 25, 2023 •

edited

Loading

sxa commented Oct 25, 2023

sxa commented Oct 25, 2023 •

edited

Loading

sxa commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

sxa commented Oct 26, 2023 •

edited

Loading

sxa commented Oct 26, 2023 •

edited

Loading

andrew-m-leonard commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

gdams commented Oct 26, 2023 •

edited

Loading

gdams commented Oct 26, 2023 •

edited

Loading

andrew-m-leonard commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

jdk_tools_1 jpackage tests in Windows JDK 17 & 21 fail due to signing issue #4831

jdk_tools_1 jpackage tests in Windows JDK 17 & 21 fail due to signing issue #4831

Comments

jiekang commented Oct 19, 2023

sophia-guo commented Oct 20, 2023

sophia-guo commented Oct 20, 2023

sxa commented Oct 20, 2023 • edited Loading

sxa commented Oct 20, 2023

sxa commented Oct 20, 2023 • edited Loading

sophia-guo commented Oct 23, 2023

sophia-guo commented Oct 23, 2023

sophia-guo commented Oct 24, 2023 • edited Loading

Haroon-Khel commented Oct 24, 2023

Karm commented Oct 25, 2023

sxa commented Oct 25, 2023

judovana commented Oct 25, 2023

RadekCap commented Oct 25, 2023

sophia-guo commented Oct 25, 2023

sophia-guo commented Oct 25, 2023

Haroon-Khel commented Oct 25, 2023

Haroon-Khel commented Oct 25, 2023

Haroon-Khel commented Oct 25, 2023 • edited Loading

sxa commented Oct 25, 2023

sxa commented Oct 25, 2023 • edited Loading

sxa commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

sxa commented Oct 26, 2023 • edited Loading

sxa commented Oct 26, 2023 • edited Loading

andrew-m-leonard commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

gdams commented Oct 26, 2023 • edited Loading

gdams commented Oct 26, 2023 • edited Loading

andrew-m-leonard commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

andrew-m-leonard commented Oct 26, 2023

sxa commented Oct 20, 2023 •

edited

Loading

sxa commented Oct 20, 2023 •

edited

Loading

sophia-guo commented Oct 24, 2023 •

edited

Loading

Haroon-Khel commented Oct 25, 2023 •

edited

Loading

sxa commented Oct 25, 2023 •

edited

Loading

sxa commented Oct 26, 2023 •

edited

Loading

sxa commented Oct 26, 2023 •

edited

Loading

gdams commented Oct 26, 2023 •

edited

Loading

gdams commented Oct 26, 2023 •

edited

Loading