Separate slow tests from fast tests #19
Comments
remkop
changed the title
|
Absolutely Remko!
I wrote something about the Tests in openVirus Slack #tech-driven.
I think we have to create a distrib with some minimal regression tests -
one or two per class.
On Wed, Apr 1, 2020 at 6:22 AM Remko Popma ***@***.***> wrote:
When running the maven build with mvn clean package, the tests to run for
several hours and at some point seem to get stuck.
Yes, this is a turn off. I think maybe @gilad Denneboom
<gilad.denneboom@gmail.com> encountered this.
I always build with
mvn install -Dmaven.test.skip=true
That's horrible but as a single author I didn't have time for anything
else. And when I had a
group in Cambridge we had good software discipline- a Jenkins server and
required clean builds.
Someone set up a CI for AMI but I am not sure it was deployed.
It would make sense to me to separate fast tests ("unit tests") from
slow-running tests ("integration", "acceptance" or "system" tests).
On the one hand, developers need fast feedback to make sure their changes
did not break anything, and on the other hand we need a set of thorough
tests that require a complete system setup.
There are several ways to accomplish this with Maven:
- naming convention, and include and exclude build configurations that
depend on these names
- separate module for slow-running tests
- JUnit Categories
Optionally, we can use the Maven failsafe plugin
<https://maven.apache.org/surefire/maven-failsafe-plugin/> for
integration tests. With the failsafe plugin enabled, we can run the
following:
1. mvn test (will run only the basic unit tests, and will stop the
build if any of them fails),
2. mvn integration-test (will run integration tests, and will not stop
the build if any of them fails), and
3. mvn verify (will stop the build if an integration test fails).
Resources:
- http://www.thinkcode.se/blog/2012/08/21/separating-tests-in-maven
-
https://semaphoreci.com/community/tutorials/how-to-split-junit-tests-in-a-continuous-integration-environment
The main problem will be going through the existing tests and filtering
the slow ones from the fast ones. Perhaps we can automate this from the
test log.
Great Idea.
I have done some naming - you will find many testFooIT() and these are
skipped but we need more.
There's always a tension between doing things properly and losing them, and
doing something quick.
AMI has HUGE technical debt!!
So I'll trust your judgment. I like the idea of triaging tests on time.
There are 447 hits in
public class *Test
so it's not trivial. It can probably only be triaged by me.
Idea?
How do we run just a single FooTest.class? or package? we could partition
the job if we get more volunteers.
P.
… —
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#19>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFTCSYMVMCYSZ2RPZJGV43RKLFSTANCNFSM4LYOG2BA>
.
--
Peter Murray-Rust
Founder ContentMine.org
and
Reader Emeritus in Molecular Informatics
Dept. Of Chemistry, University of Cambridge, CB2 1EW, UK
|
Separation: already done!I missed this, but looking at the POM, it turns out that both the Surefire and the Failsafe plugins are already configured. So, we can already use Maven to run the unit tests separately from the integration tests by using the naming conventions for these plugins:
I suspect that all that is needed is to rename some long-running tests from Triaging FindingsI found the following slow-running tests:
OverallAlso, there are quite a few test failures. One cause I can see is the test relying on data files existing in absolute paths like: |
|
Thanks looks good and what I expected
.am on phone will look later
…On Thu, 2 Apr 2020, 00:38 Remko Popma, ***@***.***> wrote:
Separation: already done!
I missed this, but looking at the POM, it turns out that both the Surefire
and the Failsafe plugins are already configured. So, we can already use
Maven to run the unit tests separately from the integration tests:
- mvn test or mvn package will only run the tests named Test*.java,
*Test.java and *TestCase.java
- mvn verify or mvn install will also run the tests named IT*.java,
*IT.java and *ITCase.java
I suspect that all that is needed is to rename some long-running tests
from ...Test.java to ...IT.java and that will help a lot.
Triaging Findings
I found the following slow-running tests:
-
JBrowserTest it took more than 10 minutes to run this test. I went
ahead and renamed this to JBrowserIT.
-
AMIDictionaryTest Tests run: 39, Failures: 3, Errors: 29, Skipped: 0, *Time
elapsed: 826.646 sec*. It gets stuck in
testCreateFromMediawikiTemplateListURL for 2 minutes, in
testCreateFromMediawikiTemplateURL for 6+ minutes, and again in
testCreateFromMediawikiTemplateListURLOld. I changed the test to run
those specific ones in a separate thread that times out after 10 seconds.
If these tests require more time perhaps they can be moved into an IT test?
-
org.contentmine.ami.tools.AMISectionToolTest Tests run: 26, Failures:
8, Errors: 4, Skipped: 0, *Time elapsed: 151.685 sec* (the bulk in
testAddAbstracts 48.087 sec and testAddBacks 14.658 sec) - no action taken
yet
-
org.contentmine.ami.tools.AMITransformToolTest Tests run: 4, Failures:
1, Errors: 0, Skipped: 0, *Time elapsed: 28.736 sec* - no action taken
yet
-
org.contentmine.ami.tools.AMIWordsToolTest Tests run: 4, Failures: 1,
Errors: 0, Skipped: 0, *Time elapsed: 23.569 sec* - no action taken yet
-
org.contentmine.graphics.svg.cache.SVGCacheTest Tests run: 5,
Failures: 0, Errors: 0, Skipped: 1, *Time elapsed: 145.624 sec* - no
action taken yet
-
org.contentmine.norma.sections.SectionTest Tests run: 21, Failures: 2,
Errors: 2, Skipped: 2, *Time elapsed: 202.66 sec* - no action taken yet
Overall
Also, there are quite a few test failures. One cause I can see is the test
relying on data files existing in absolute paths like:
"/Users/pm286/projects/openVirus/dictionaries/". Can these files be
committed to somewhere under /src/test/resources/? I can provide a
utility that converts a resource path to an absolute path that can be
passed as an argument to the tool that is being tested.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#19 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFTCS326SJNAAKKTAWQSWDRKPF6VANCNFSM4LYOG2BA>
.
|
When running the maven build with
mvn clean package, the tests to run for several hours and at some point seem to get stuck.It would make sense to me to separate fast tests ("unit tests") from slow-running tests ("integration", "acceptance" or "system" tests).
On the one hand, developers need fast feedback to make sure their changes did not break anything, and on the other hand we need a set of thorough tests that require a complete system setup.
There are several ways to accomplish this with Maven:
Optionally, we can use the Maven failsafe plugin for integration tests. With the failsafe plugin enabled, we can run the following:
mvn test(will run only the basic unit tests, and will stop the build if any of them fails),mvn integration-test(will run integration tests, and will not stop the build if any of them fails), andmvn verify(will stop the build if an integration test fails).Resources:
The main problem will be going through the existing tests and filtering the slow ones from the fast ones. Perhaps we can automate this from the test log.
The text was updated successfully, but these errors were encountered: