-
-
Notifications
You must be signed in to change notification settings - Fork 314
Graduated Testing & Test Numbers
Shelley Lambert edited this page Oct 5, 2022
·
27 revisions
Our goal is to employ graduated testing to optimize our use of the machine resources at our disposal. This is accomplished by tagging the tests by level, and choosing to run the 'sanity' level nightly, and the 'sanity' + 'extended' level against the release builds. Each test target may contain many tests, sometimes 1000's.
Some targets are rerun with variations of command-line options, in those cases, the number of tests are recounted, as the new input creates a unique test (a different code path is tested). In the case of system/load tests, some tests are run repeatedly (in some cases 100's to 10,000's of times).
The numbers vary slightly depending on the jdk version and platforms, but for the purpose of this wiki, we will document the average/common numbers and add notes for exceptional differences.
sanity = nightly (Triggered by nightly pipelines)
extended = weekly (Triggered via Jenkins schedule/cron to utilize machine farm during quiet periods on the weekend)
sanity + extended = per release (sanity triggered automatically, extended triggered manually at present, eventually to be triggered by release pipelines when we have sufficient machine resources)
| Test group | # of sanity targets | # of sanity test cases | # of extended targets | # of extended test cases | # of special targets | # of special test cases | Platforms | Notes |
|---|---|---|---|---|---|---|---|---|
| openjdk | 8 | ~1860 | 23 | ~2260 | all | jdk_custom and langtools_custom targets allow for running individual test classes | ||
| system | 89 | 4454 (bare number) / ~20,000 (with variants) | 27 | 482 (bare number) / ~5000 (with variants) | 366 | all | total: 4936 (bare number)/ 25,040 (with variants) / 31,727,361 (variants with repetitions), extended: 11 targets all jdk impls, 16 are openj9 specific | |
| external | 5 | ~25,000 | 12 | ~60,000 | 4 | ~40,000 | all docker archs x OSs we produce (with exception of windows) | sanity targets are microprofile tcks (which are actually being run nightly), extended are functional suites from applications, special are the functional and system test groups containerized |
| perf | 8 | 25 | 48 | all | microbenches are in special.perf | |||
| functional | 301 | ~25,000 | 364 | ~3250 | 371 | all | not enabled in automated runs at AdoptOpenJDK, used only in Grinders for now (running nightly at openj9) |
This wiki does not include the compliance test numbers in the listing (~70,000-120,000 tests per release, varies by jdk version).
For the most up-to-date view of these subgroups, please refer to the playlist. The openjdk group is divided into sub-targets, aligned with the jtreg test targets with which many people are already familiar (for example, sanity.openjdk == tier1).
| openjdk sanity targets | openjdk extended targets | Notes |
|---|---|---|
| jdk_math, jdk_math_jre, jdk_lang, jdk_io, jdk_net, jdk_nio, jdk_util, jdk_rmi, jdk_security1 | jdk_custom, langtools_custom, hotspot_all, hotspot_all_SE100, hotspot_jre, jdk_beans, jdk_other, jdk_security2, jdk_security3, jdk_security4, jdk_text, jdk_time, jdk_management, jdk_jmx, jdk_tools, jdk_jdi, jdk_jfr, jdk_awt (only enabled on linux platforms), jdk_swing (only enabled on linux platforms), jdk_sound (disabled) | extended: 2 targets only enabled on linux platforms (awt, swing), 1 target disabled (sound), 3 targets are hotspot specific, not used for openj9 testing (hotspot_all, hotspot_all_SE100, hotspot_jre) |
| system sanity targets | system extended targets | Notes |
|---|---|---|
| ClassLoadingTest -- runs 3 tests (3000 times) MauveSingleThreadLoadTest -- 4619 tests ( 500 times) MauveSingleInvocationLoadTest -- 4619 tests MauveMultiThreadLoadTest -- 4619 tests (* 1000 times) with at least 3 threads NioLoadTest - 11 tests (* 10 times), using at least 2 threads LambdaLoadTest - 14 tests (* 200 times), threads = 2 DaaLoadTest_daa1 - 19 tests (100 times) DaaLoadTest_daa2 - 28 tests(150 times) DaaLoadTest_daa3 - 1 test (300 times) TestJlmLocal - 1 TestJlmRemoteClassAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestJlmRemoteClassNoAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestJlmRemoteMemoryAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestJlmRemoteMemoryNoAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestJlmRemoteNotifierProxyAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestJlmRemoteThreadNoAuthTest - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestJlmRemoteThreadAuth- As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestIBMJlmLocal - 1 TestIBMJlmRemoteClassAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestIBMJlmRemoteClassNoAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestIBMJlmRemoteMemoryAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestIBMJlmRemoteMemoryNoAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestIBMJlmRemoteNotifierProxyAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestIBMJlmRemoteThreadNoAuth- As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times TestIBMJlmRemoteThreadAuth- As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times Modularity tests - 38 Test running in ConcurrentScavenge mode: ClassLoadingLoad test - runs 3 test (3000 times) LambdaLoadTest - 14 ( 200 times), threads = 2 mathLoadTest_autosimd - 39 (* 4000 times) MathLoadTest_bigdecimal = 63 (* 50 times) DaaLoadTest_daa1 - 19 (*100 times) DaaLoadTest_daa2 - 28 (150 times) DaaLoadTest_daa3 - 1 (300 times) MauveSingleThreadLoadTest -- 4619 ( 500 times) MauveSingleInvocationLoadTest -- 4619 MauveMultiThreadLoadTest -- 4619 ( 1000 times) with at least 3 threads |
ConcurrentLoadTest -- runs 28 tests (*20 tests) DirectByteBufferLoadTest -- runs 1 test (DirectMemoryTest) LangLoadTest -- runs 31 tests (*100 times) LockingLoadTest -- runs 2 tests MathLoadTest_all -- 3 tests (*50 times) + 63 tests (50 times) + 39 tests (4000 times) UtilLoadTest - 4 tests ( 500 times), using at elast 2 threads) DaaLoadTest_all - 48 tests(150 times) HCRLateAttachWorkload - run minimix load run (31 + 4 + 39 + 1) = 75 tests, with 3 threads JdiTest - 1 test HeapHogLoadTest - run minimix load run (31 + 4 + 39 + 1) = 75 tests, 150 times, with at least 2 threads ObjectTreeLoadTest - 2 tests ( 150 times) with 2 threads at least SharedClassesAPI - run minimix load run (31 + 4 + 39 + 1) = 75 tests, at least 3 threads SharedClassesWorkload - run minimix load run (31 + 4 + 39 + 1) = 75 tests, * 300 times, with at least 3 threads SharedClassesWorkloadTest_Softmx_Increase - 1 test SharedClassesWorkloadTest_Softmx_IncreaseDecrease - 1 test SharedClassesWorkloadTest_Softmx_Increase_JitAOT - 1 test SharedClasses.SCM01.SingleCL - 1 SharedClasses.SCM01.MultiCL - 1 SharedClasses.SCM01.MultiThread - 1 SharedClasses.SCM01.MultiThreadMultiCL - 1 SharedClasses.SCM23.MultiCL - 1 SharedClasses.SCM23.SingleCL - 1 SharedClasses.SCM23.MultiThreadMultiCL - 1 SharedClasses.SCM23.MultiThread - 1 Test running in ConcurrentScavenge mode: MathLoadTest_all -- 3 (*50 times) + 63 (*50 times) + 39 (*4000 times) DaaLoadTest_all = 48 (*150 times) |
| external sanity targets | external extended targets | Notes |
|---|---|---|
| openliberty_microprofile_tck thorntail_microprofile_tck: 395 testcases (metrics TCK FAT : 151, config TCK FAT : 53, fault-tolerance TCK FAT : 107, restclient TCK FAT : 56, openapi TCK FAT : 28, where FAT is Functional Acceptance Test) payara_microprofile_tck (derived from Glassfish): 683 testcases (Matrics TCK : 150, Config TCK : 125, Fault tolerance TCK : 186, OpenAPI TCK : 222) tomee_test quarkus_quickstarts_test |
camel_test jacoco_test jenkins_test kafka_test wildfly_test lucene_solr_nightly_smoketest elasticsearch_test netty-test scala_test tomcat_test derby_test_junit_all quarkus_java_test spring-test wildfly_test WycheproofTests |
| perf sanity targets | perf extended targets | Notes |
|---|---|---|
| dacapo-eclipse, dacapo-h2, dacapo-lusearch-fix, idlemicrobenchmark | remaining 8 dacapo benchmarks and 18 renaissance benchmarks |