Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JUnit view issues with filtered repeated test results #1216

Closed
iloveeclipse opened this issue Feb 19, 2024 · 1 comment
Closed

JUnit view issues with filtered repeated test results #1216

iloveeclipse opened this issue Feb 19, 2024 · 1 comment
Assignees
Labels
bug Something isn't working regression Regression defect
Milestone

Comments

@iloveeclipse
Copy link
Member

Follow up on #946 and #1022.

There are (still) problems we see with the current state:

  1. The stack traces of "filtered" elements from templates are lost for the "Run History". The root cause is that the UI elements are serialized (and restored later) and if we filter children with stack traces in UI, we also don't serialize them and so on restore there is nothing... Stack for that (where getFailureTrace() returns null and there are no children):
Thread [main] (Suspended (breakpoint at line 268 in TestElement))	
	TestSuiteElement(TestElement).getFailureTrace() line: 268	
	TestRunSessionSerializer.addFailure(TestElement) line: 176	
	TestRunSessionSerializer.handleTestElement(ITestElement) line: 130	
	TestRunSessionSerializer.handleTestElement(ITestElement) line: 134	
	TestRunSessionSerializer.handleTestRun() line: 101	
	TestRunSessionSerializer.parse(InputSource) line: 73	
	TransformerImpl.transformIdentity(Source, SerializationHandler) line: 694	
	TransformerImpl.transform(Source, SerializationHandler, String) line: 781	
	TransformerImpl.transform(Source, Result) line: 395	
	JUnitModel.exportTestRunSession(TestRunSession, OutputStream) line: 528	
	JUnitModel.exportTestRunSession(TestRunSession, File) line: 499	
	TestRunSession.swapOut() line: 373	
  1. The "diff" (CompareResultDialog) dialog shown if the test fails comparing results is not shown for the "filtered out" template test results, because the code that asks for the diff checks org.eclipse.jdt.internal.junit.model.TestElement.isComparisonFailure() which is not true for the "parent" node that shows the fail after filtering out children.

The fixes in #946 and #1022 were unfortunately not complete.

One possibility to fix them would be replace parent TestSuiteElement element with one of the failed child elements instead of filtering children, or fully mimic "I'm not a suite, I'm a test" behavior by overriding some testsuite methods.

For the later, I will propose a PR in a moment, which will fix observed issues.

For 4.31 we can either take that or revert changes as proposed in #1203.

As before, the test case to play with is following:

import static org.junit.jupiter.api.Assertions.assertEquals;

import org.junit.Assume;
import org.junit.jupiter.api.MethodOrderer.MethodName;
import org.junit.jupiter.api.RepeatedTest;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(MethodName.class)
public class TestTemplatesJunit5 {

	@Test
	public void testOkNoRepeats() throws Exception {
		Thread.sleep(500);
	}
	
	@RepeatedTest(value = 1)
	public void testOkOneRepeat() throws Exception {
		Thread.sleep(500);
	}

	@RepeatedTest(value = 2)
	public void testOkTwoRepeats() throws Exception {
		Thread.sleep(500);
	}
	
	@Test
	public void testWithErrorNoRepeats() throws Exception {
		doStupidThings(null);
	}
	
	@RepeatedTest(value = 1)
	public void testErrorOneRepeat() throws Exception {
		doStupidThings(null);
	}
	
	@RepeatedTest(value = 2)
	public void testErrorTwoRepeats() throws Exception {
		doStupidThings(null);
	}

	@Test
	public void testFailNoRepeats() throws Exception {
		assertEquals("expected", "observed");
	}
	
	@RepeatedTest(value = 1)
	public void testFailOneRepeat() throws Exception {
		assertEquals("expected", "observed");
	}

	@RepeatedTest(value = 2)
	public void testFailTwoRepeats() throws Exception {
		assertEquals("expected", "observed");
	}


	@RepeatedTest(value = 0)
	public void testWrongRepeat() throws Exception {
		Thread.sleep(500);
	}
	
	@Test
	public void testAssume() throws Exception {
		Assume.assumeFalse(true);
	}
	
	@RepeatedTest(value = 1)
	public void testAssumeOneRepeat() throws Exception {
		Assume.assumeFalse(true);
	}
	
	@RepeatedTest(value = 2)
	public void testAssumeTwoRepeats() throws Exception {
		Assume.assumeFalse(true);
	}
	
	private void doStupidThings(String arg) {
		arg.toString();
	}
	
}
@iloveeclipse iloveeclipse added bug Something isn't working regression Regression defect labels Feb 19, 2024
@iloveeclipse iloveeclipse self-assigned this Feb 19, 2024
iloveeclipse added a commit to iloveeclipse/eclipse.jdt.ui that referenced this issue Feb 19, 2024
In case the testsuite contains a single dynamic test (like repeated
tests), provide the relevant test data if asked for that, not suite
data.

Fixes eclipse-jdt#1216
@iloveeclipse
Copy link
Member Author

Fixed by reverting new code via #1203

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working regression Regression defect
Projects
None yet
Development

No branches or pull requests

2 participants