Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Test suite didn't report a result" on macOS-arm64 #2689

Closed
mhsmith opened this issue Jul 2, 2024 · 3 comments · Fixed by #2730
Closed

"Test suite didn't report a result" on macOS-arm64 #2689

mhsmith opened this issue Jul 2, 2024 · 3 comments · Fixed by #2730
Labels
bug A crash or error in behavior. macOS The issue relates to Apple macOS support.

Comments

@mhsmith
Copy link
Member

mhsmith commented Jul 2, 2024

This has now happened in 2 out of the last 3 merges to main:

@mhsmith mhsmith added bug A crash or error in behavior. macOS The issue relates to Apple macOS support. labels Jul 2, 2024
@freakboy3742
Copy link
Member

This is proving very difficult to narrow down, because it's essentially not reproducible; re-running the same task almost always results in a successful test pass.

Looking at the failure mode: the error being reported is that the test suite didn't report a result - which means that Briefcase isn't seeing the >>>> EXIT ... test sentinel.

One observation is that the problem appears to coincide with log messages that read:

=== Messages dropped during live streaming (use `log show` to see what they were)

This run is a recent run that demonstrates this well. This suggests the issue might be that issue is that Briefcase is losing the log tail, and as a result, isn't seeing the sentinel value; but the process is exiting, so Briefcase is able to detect that the app has completed.

If this hypothesis is correct, there are three possible fixes:

  1. Use a mechanism other than the system log for tailing results. I'm not sure what that mechanism would be, though.
  2. Modify Briefcase so that, in the case where no sentinel is seen, an additional call to log show is made to get the last N lines of the system log. From what I'm reading, log show is guaranteed to return all log output, so
  3. Modify the test execution so that log streaming isn't needed. Log streaming is currently needed because the test suite is executed inside the app, so we are reliant on the app's output to evaluate test success. One of the approaches to Add testbed support for web backend #2662 is to modify the testbed so that the test suite runs locally, and communicates with the app via a remote control mechanism. This would mean we don't need to use the system log - we could use vanilla stdout handling for test reports.

@freakboy3742
Copy link
Member

freakboy3742 commented Jul 17, 2024

[Moved to #2732]

@freakboy3742
Copy link
Member

I've added a speculative workaround as part of #2666; it's difficult to tell if this will always work, but it at least passed its first CI run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A crash or error in behavior. macOS The issue relates to Apple macOS support.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants