diff --git a/content/blog/peeling-the-big-onion/bigo-peeled.jpg b/content/blog/peeling-the-big-onion/bigo-peeled.jpg new file mode 100644 index 0000000000..fcdf0807fa Binary files /dev/null and b/content/blog/peeling-the-big-onion/bigo-peeled.jpg differ diff --git a/content/blog/peeling-the-big-onion/bigo.jpg b/content/blog/peeling-the-big-onion/bigo.jpg new file mode 100644 index 0000000000..9a71ae57e7 Binary files /dev/null and b/content/blog/peeling-the-big-onion/bigo.jpg differ diff --git a/content/blog/peeling-the-big-onion/index.md b/content/blog/peeling-the-big-onion/index.md new file mode 100644 index 0000000000..f6f573122b --- /dev/null +++ b/content/blog/peeling-the-big-onion/index.md @@ -0,0 +1,45 @@ +--- +title: Peeling the Big Onion – Stripping out layers of indirection from test frameworks +date: "2023-06-14T11:00:00+00:00" +author: Mesbah-Alam +description: The article demonstrates the need and process of reducing indirection in the evolution of test frameworks. +tags: + - Adoptium + - Test + - Java + - Temurin +--- + +## Background + +Automation test frameworks should run test in the simplest way possible. However, we often over-engineer them by attempting to make one grand solution to support automation of a wide variety of tests. This results in layers of indirection, unnecessary technical debt, and typically— inefficiency. “Peeling the onion”, so to speak, thus becomes an imperative exercise to ensure a test framework's evolution in the right direction. + +## The Big onion + +Let’s use a real-world example. We run a large set of TCKs as part of our Java compatibility verification process of Semeru OpenJ9 and IBM JDKs. Historically, these TCKs used to be driven via an age-old cryptic Perl wrapper which was difficult to maintain and evolve to fit the needs of newer JDK versions. + +At one point we decided to replace this legacy TCK Perl wrapper with a new test framework that was being designed to drive stress tests— [STF](https://github.com/adoptium/STF). However, STF was simply too lofty for the needs of TCK execution. First, STF would need users to define test cases as a series of conceptual 'stages' using Java. It would then take those ‘stages’ and generate a set of Perl scripts to orchestrate the second layer of Java command-lines which would culminate in the eventual automated test job in the CI system. While this has benefits in terms of debug-ability of load/stress tests, it was not the best way to drive TCK, that itself comes with its own sophisticated test harness. + +
The Big Onion
+ +Result—the ‘Big Onion’. By adopting STF to run TCKs, not only did we introduce unnecessary layers of indirection in our automation story `(Java -> Perl -> Java)`, but we also ended up incurring additional compilation time in TCK builds in our CI system, since STF requires checking out and compiling of both its own Git repository, as well as one extra Git repository that of the System tests. + +## Peel the onion + +So, to run the TCK harness with less indirection and more efficiency, we decided to decouple it from the STF stress test framework altogether and replace it with [JavaTestRunner](https://github.com/adoptium/aqa-tests/blob/master/jck/jtrunner/JavaTestRunner.java)– a simple Java class that can perform the essential tasks of generating command files for TCK, building result summary, starting required services and the managing the execution TCK harness itself. + +
The Peeled Onion
+ +Unlike the STF framework, JavaTestRunner is housed within the same Adoptium aqa-tests repository, like the rest of the AQAvit test suites. This relieved us from having to checkout and compile two additional repositories— resulting in a dramatic reduction of compile time for TCK CI jobs (we ended up saving ~3285 hours of machine time per year). + +## Peel it more + +Stripping out layers of indirection from test frameworks typically happens in iterations. JavaTestRunner gave us the ability to run TCK jobs faster, with more debug-ability and less indirection. However, it still was an ‘onion’. Meaning, few more layers could potentially be peeled off still. + +One such layer was how JavaTestRunner drives the TCK harness itself. TCK harness is a Java application itself. JavaTestRunner would gather all the inputs from user via playlist, construct the command-line and run the TCK harness via ProcessBuilder. This layer of indirection `(JVM->JVM)` is not only inefficient, but it also results in hiding test command-lines— making the TCK builds harder to triage. + +The above issue was solved by stripping out JavaTestRunner and replacing it with [JavaTestUtil](https://github.com/adoptium/aqa-tests/blob/master/jck/jtrunner/JavatestUtil.java). JavaTestUtil performs the essential tasks of command file generation and result summary generation only, while the actual command-lines to start required services and running the TCK harness itself are placed in a makefile outside of the Java wrapper. This means one less layer of indirection and faster execution. Plus, users can now have more visibility to the actual command-lines that get executed for each TCK test target. + +## Summary + +Test frameworks evolve into a 'Big Onions'. Peeling those onions should be considered among software development best practices. It helps development teams to reduce technical debt while continuously improving QA automation efficiency.