Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance Test result summary with testcases results #510

Closed
sophia-guo opened this issue Mar 8, 2024 · 3 comments · Fixed by #516
Closed

Enhance Test result summary with testcases results #510

sophia-guo opened this issue Mar 8, 2024 · 3 comments · Fixed by #516
Assignees

Comments

@sophia-guo
Copy link
Contributor

sophia-guo commented Mar 8, 2024

Currently test result summary is provided by Targets level, which doesn't provide an accurate test information per test cases. It might be helpful if test summary per test cases could be provided.

Both of JCK and JDK tests provide test summary per testcase (Test results: passed: 41; failed: 5; error:1; skipped: 1), which means this can be done for JCK and JDK tests.

Console output for each run could be something like:

TEST TARGETS SUMMARY
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++:
DISABLED test targets:
	jdk_awt_1
        ...
	jdk_sound_0

PASSED test targets:
        jdk_beans_2  - Test results: passed: 41; skipped: 1
        ...
  	jdk_other_2 - Test results: passed: 21

FAILED test targets:
        jdk_tools_2 - Test results: passed: 41; failed: 5; error:1; skipped: 1
        ...
        jdk_lang_1 -  Test results: passed: 541; failed: 5
TOTAL: 40   EXECUTED: 7   PASSED: 6   FAILED: 1   DISABLED: 11   SKIPPED: 22
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++:
 
TESTCASES RESULTS SUMMARY: passed: 214; failed: 1; error: 0; skipped: 0  
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Correspondingly tap file could also include those information for passed ones if needed ( failed one already include those information)

#SHA.txt content: 
...
# TEST TARGETS RESULTS SUMMARY: TOTAL: 2   EXECUTED: 2   PASSED: 1   FAILED: 1   DISABLED: 0   SKIPPED: 0
# TESTCASES RESULTS SUMMARY: passed: 214; failed: 1; error: 0; skipped: 0 
1..2
ok 1 - jdk_math_0
  ---
    output:
      |
        Test results: passed: 39
    duration_ms: 100014
  ...
not ok 2 - jdk_security1_0
  ---
    output:
      |
        Failed test cases: 
        TEST: java/security/cert/CertPathBuilder/akiExt/AKISerialNumber.java
        Test results: passed: 175; failed: 1
    duration_ms: 95974
  ...

Thoughts @smlambert @jiekang ?

@sophia-guo
Copy link
Contributor Author

Related adoptium/aqa-tests#5121

@smlambert
Copy link
Contributor

Big +1 to this suggestion, since once this info is put to the console, we can also grab it for inclusion to TRSS DB and relates to adoptium/aqa-test-tools#841

@jiekang
Copy link

jiekang commented Mar 11, 2024

The additional information sounds good to me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants