Run testbed with Wayland #2670

rmartin16 · 2024-06-19T22:10:10Z

Changes

Run the testbed testing suite with Wayland
Closes Incorporate Wayland in to testing strategy #2668

Notes

I experimented quite a bit with weston, sway, mutter, and even a few others today
- weston works well....until you run in headless mode; then Gtk throws errors for no wayland "seat"
- sway seems much more limited; for instance, I don't think it supports overlapping windows
- mutter was really the only one that provided a robust testing environment
  - However, even mutter's headless mode upsets Gtk...but mutter is able to run in xvfb-run unlike the weston and sway
This approach does feel a little weird, though...because we are running a wayland server in xvfb-run...but it seems like a viable testing strategy.
- I found this strategy buried in some weston issue ticket

PR Checklist:

All new features have been tested
All new features have been documented
I have read the CONTRIBUTING.md file
I will abide by the code of conduct

rmartin16 · 2024-06-19T22:52:35Z

This is the approach I landed on today. Feel free to let me know if this strategy makes sense to you...or doesn't. From here, the failing tests just need to be addressed one way or another. May also benefit from combining the duplicated matrix fields for the two Linux jobs.

freakboy3742 · 2024-06-20T03:55:36Z

I guess the thing that doesn't make sense is the use of xvfb-run... are you sure it's actually running as Wayland? There's a bunch of functionality that should be tested (like getting an image of the current screen), but that isn't showing up as test failure as I'd expect.

Other than that (and the other miscellaneous test failures), the approach looks fine to me.

rmartin16 · 2024-06-20T04:07:55Z

I guess the thing that doesn't make sense is the use of xvfb-run... are you sure it's actually running as Wayland?

I'm relatively confident. If you run mutter without xvfb-run in a local Linux distro install, it spawns a window for the Wayland display and you can watch the testbed tests run. Furthermore, all the test failures align with the failures in a natively Wayland environment like Fedora 40.

There's a bunch of functionality that should be tested (like getting an image of the current screen), but that isn't showing up as test failure as I'd expect.

Are you thinking of this one or a different one?

FAILED tests/widgets/test_canvas.py::test_multiline_text - Failed: Rendered image doesn't match reference (RMSE==0.11368523797456337)

In general, I was surprised this ended up requiring xvfb...but all of the other approaches required running a Wayland compositor implementation in its headless mode....and Gtk just does not like that currently...but running the compositor in X via xvfb avoids the headless mode and all its problems...

freakboy3742 · 2024-06-20T04:43:56Z

There's a bunch of functionality that should be tested (like getting an image of the current screen), but that isn't showing up as test failure as I'd expect.

Are you thinking of this one or a different one?
FAILED tests/widgets/test_canvas.py::test_multiline_text - Failed: Rendered image doesn't match reference (RMSE==0.11368523797456337)

That test failure is somewhat expected - the canvas rendering tests are highly platform dependent. I was thinking more of this test - based on the implementation, that shouldn't be able to pass.

rmartin16 · 2024-06-20T04:58:33Z

ahh...in that case, pytest appears to be skipping the test.

tests/app/test_screens.py::test_as_image SKIPPED (Screen.as_image() is not implemented on wayland.)     [  4%]

freakboy3742 · 2024-06-21T05:32:12Z

ahh...in that case, pytest appears to be [skipping]

Huh - that's some forward thinking that I don't remember :-)

testbed/tests/testbed.py

rmartin16 · 2024-06-21T19:09:24Z

gtk/src/toga_gtk/screens.py

 def get_image_data(self):
- if "WAYLAND_DISPLAY" in os.environ:
+ if "WAYLAND_DISPLAY" in os.environ: # pragma: no cover
 # Not implemented on wayland due to wayland security policies.
 self.interface.factory.not_implemented("Screen.get_image_data() on Wayland")


This is left as blanket "no cover" because the probe prevents fetching an image for the screen if running on Wayland. However, I imagine that's because Toga crashes out if this code is ever actually ran in practice....because the Screen.as_image() API in core sends the returned None in to toga.Image. This may not be the best UX...especially since developers may never test on Wayland before shipping...

Agreed it's not ideal UX; returning a dummy image (blank, same size as the screen?) may be preferable. Up to you whether you roll that into this PR, or log it as a standalone issue.

I created #2673 to capture this.

rmartin16 · 2024-06-21T19:11:44Z

gtk/tests_backend/app.py

+ def assert_current_window(self, window):
+ # Gtk 3.24.41 ships with Ubuntu 24.04 where present() works on Wayland
+ if self.IS_WAYLAND and self.GTK_VERSION < (3, 24, 41):
+ pytest.skip(
+ f"Assigning the current window is not supported for "
+ f"Gtk {'.'.join(map(str, self.GTK_VERSION))} on Wayland"
+ )
+ else:
+ assert self.app.current_window == window


There is likely an older version of Gtk that works on Wayland to limit this with....but Gtk development is too opaque for me to try to track that down.

I'm OK with this. There's so little consistency in Linux testing environments that I'm OK with this being a skip. If someone wants to suggest the limit can be lower, then can submit a PR.

However, given the actual assertion is a trivial comparison, I wonder if it might be better to handle this as a feature flag on the probe (i.e., a supports_current_window_assignment), rather than an abstracted assertion. Having 5 identical implementations of a trivial comparison, and one that is the same identical implementation with a skip check seems excessive. This would also allow us to continue the tests that perform a current-window check, but in the context of other useful behavior (i.e., we could make a decision on a per-test basis whether we want to skip the test, or ignore the check entirely.=)

gtk/tests_backend/window.py

rmartin16 · 2024-06-21T19:27:18Z

Along with allowing for testing Wayland in CI, this should also allow devs to more easily run the testbed tests locally.

However, multiple monitor setups where the primary monitor is not the far left monitor will likely still fail. This kinda makes me wonder if all the CI logic to set up the environment to run the testbed tests shouldn't be in tox instead...

freakboy3742

A couple of minor details, but otherwise this looks really good. The --ci affordance is especially nice - getting a clean test run by default when we know a test has platform issues is a nice addition.

freakboy3742 · 2024-06-22T02:03:16Z

gtk/src/toga_gtk/screens.py

 def get_image_data(self):
- if "WAYLAND_DISPLAY" in os.environ:
+ if "WAYLAND_DISPLAY" in os.environ: # pragma: no cover
 # Not implemented on wayland due to wayland security policies.
 self.interface.factory.not_implemented("Screen.get_image_data() on Wayland")


Agreed it's not ideal UX; returning a dummy image (blank, same size as the screen?) may be preferable. Up to you whether you roll that into this PR, or log it as a standalone issue.

freakboy3742 · 2024-06-22T02:06:48Z

gtk/tests_backend/app.py

+ def assert_current_window(self, window):
+ # Gtk 3.24.41 ships with Ubuntu 24.04 where present() works on Wayland
+ if self.IS_WAYLAND and self.GTK_VERSION < (3, 24, 41):
+ pytest.skip(
+ f"Assigning the current window is not supported for "
+ f"Gtk {'.'.join(map(str, self.GTK_VERSION))} on Wayland"
+ )
+ else:
+ assert self.app.current_window == window


I'm OK with this. There's so little consistency in Linux testing environments that I'm OK with this being a skip. If someone wants to suggest the limit can be lower, then can submit a PR.

However, given the actual assertion is a trivial comparison, I wonder if it might be better to handle this as a feature flag on the probe (i.e., a supports_current_window_assignment), rather than an abstracted assertion. Having 5 identical implementations of a trivial comparison, and one that is the same identical implementation with a skip check seems excessive. This would also allow us to continue the tests that perform a current-window check, but in the context of other useful behavior (i.e., we could make a decision on a per-test basis whether we want to skip the test, or ignore the check entirely.=)

freakboy3742 · 2024-06-22T02:08:03Z

gtk/tests_backend/widgets/canvas.py

+ if self.IS_WAYLAND:
+ return f"{reference}-gtk-wayland"
+ else:
+ return f"{reference}-gtk"


Do we want to differentiate -x11 from -wayland in the naming here?

gtk/tests_backend/window.py

testbed/tests/testbed.py

rmartin16 · 2024-06-22T23:49:18Z

Couple questions I was thinking about:

Should we consider moving the setup logic in CI for running testbed in to tox? That would definitely simplify running it locally.
Should we consider splitting the testing sources that contain a large if block for mobile vs desktop?

freakboy3742 · 2024-06-23T00:28:48Z

Couple questions I was thinking about:

Should we consider moving the setup logic in CI for running testbed in to tox? That would definitely simplify running it locally.

I guess it could be helpful. You don't need to run the actual CI configuration very often, but when you do, it would be nice to have an easy way to replicate it without needing to spelunk through the CI YAML to replicate it.

Should we consider splitting the testing sources that contain a large if block for mobile vs desktop?

I've had the same thought myself recently. I've been breaking the dialog tests into their own modules for similar reasons; given the number of mobile vs desktop tests, it makes sense to break them out as well.

freakboy3742

Looks good to me. Splitting the mobile/desktop tests into two parts can be a separate PR - in fact, I might end up rolling it into the #2244 landing sequence that I'm in the middle of.

rmartin16 force-pushed the test-wayland branch 2 times, most recently from 0927f44 to 6e9e522 Compare June 19, 2024 22:29

Run testbed with Wayland

0ebb9ab

rmartin16 force-pushed the test-wayland branch from 6e9e522 to 0ebb9ab Compare June 19, 2024 22:35

Update test_current_window for testbed on Wayland

f6cce23

rmartin16 force-pushed the test-wayland branch from 55bf415 to f6cce23 Compare June 20, 2024 18:51

Fix window visibility and minimize testbed tests for Wayland

05e8c0e

rmartin16 force-pushed the test-wayland branch from e0b759e to 05e8c0e Compare June 20, 2024 20:04

rmartin16 added 2 commits June 20, 2024 16:26

Test Wayland on Ubuntu 22.04

10e9e65

Fix canvas testbed tests for Wayland

e51310f

rmartin16 force-pushed the test-wayland branch 4 times, most recently from 2aad2ec to d95b5f6 Compare June 21, 2024 13:28

Fix testbed coverage for Wayland on Linux

150d32a

rmartin16 force-pushed the test-wayland branch 2 times, most recently from 309a96a to 3f0d865 Compare June 21, 2024 15:55

Allow canvas test test_multiline_text to fail outside CI

4d5ac26

rmartin16 force-pushed the test-wayland branch from 3f0d865 to 4d5ac26 Compare June 21, 2024 16:09

rmartin16 commented Jun 21, 2024

View reviewed changes

testbed/tests/testbed.py Show resolved Hide resolved

rmartin16 commented Jun 21, 2024

View reviewed changes

gtk/tests_backend/window.py Show resolved Hide resolved

Minor cleanups for Wayland testbed testing

ff60b14

rmartin16 marked this pull request as ready for review June 21, 2024 19:34

freakboy3742 requested changes Jun 22, 2024

View reviewed changes

rmartin16 added 2 commits June 22, 2024 11:22

Simplify abstracted current window assertion in testbed

a1df8d4

Label original Linux multiline_text reference image as x11

ca8a068

rmartin16 requested a review from freakboy3742 June 22, 2024 16:39

freakboy3742 approved these changes Jun 23, 2024

View reviewed changes

freakboy3742 merged commit 7a0b950 into beeware:main Jun 23, 2024
32 of 35 checks passed

rmartin16 deleted the test-wayland branch June 23, 2024 03:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run testbed with Wayland #2670

Run testbed with Wayland #2670

rmartin16 commented Jun 19, 2024 •

edited

Loading

rmartin16 commented Jun 19, 2024 •

edited

Loading

freakboy3742 commented Jun 20, 2024

rmartin16 commented Jun 20, 2024 •

edited

Loading

freakboy3742 commented Jun 20, 2024

rmartin16 commented Jun 20, 2024

freakboy3742 commented Jun 21, 2024

rmartin16 Jun 21, 2024 •

edited

Loading

freakboy3742 Jun 22, 2024

rmartin16 Jun 22, 2024

rmartin16 Jun 21, 2024

freakboy3742 Jun 22, 2024

rmartin16 commented Jun 21, 2024

freakboy3742 left a comment

freakboy3742 Jun 22, 2024

freakboy3742 Jun 22, 2024

freakboy3742 Jun 22, 2024

rmartin16 commented Jun 22, 2024

freakboy3742 commented Jun 23, 2024

freakboy3742 left a comment

Run testbed with Wayland #2670

Run testbed with Wayland #2670

Conversation

rmartin16 commented Jun 19, 2024 • edited Loading

Changes

Notes

PR Checklist:

rmartin16 commented Jun 19, 2024 • edited Loading

freakboy3742 commented Jun 20, 2024

rmartin16 commented Jun 20, 2024 • edited Loading

freakboy3742 commented Jun 20, 2024

rmartin16 commented Jun 20, 2024

freakboy3742 commented Jun 21, 2024

rmartin16 Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

freakboy3742 Jun 22, 2024

Choose a reason for hiding this comment

rmartin16 Jun 22, 2024

Choose a reason for hiding this comment

rmartin16 Jun 21, 2024

Choose a reason for hiding this comment

freakboy3742 Jun 22, 2024

Choose a reason for hiding this comment

rmartin16 commented Jun 21, 2024

freakboy3742 left a comment

Choose a reason for hiding this comment

freakboy3742 Jun 22, 2024

Choose a reason for hiding this comment

freakboy3742 Jun 22, 2024

Choose a reason for hiding this comment

freakboy3742 Jun 22, 2024

Choose a reason for hiding this comment

rmartin16 commented Jun 22, 2024

freakboy3742 commented Jun 23, 2024

freakboy3742 left a comment

Choose a reason for hiding this comment

rmartin16 commented Jun 19, 2024 •

edited

Loading

rmartin16 commented Jun 19, 2024 •

edited

Loading

rmartin16 commented Jun 20, 2024 •

edited

Loading

rmartin16 Jun 21, 2024 •

edited

Loading