test_scroll fails for DetailedList & Table widgets when testbed is run locally on Android emulator

[proneon267]

Describe the bug

test_scroll fails for DetailedList & Table widgets when testbed is run locally on Android emulator.

Steps to reproduce

1. Clone the latest main branch
2. Run the testbed for Android:

briefcase run android -ur --test

3. test_scroll will fail for DetailedList & Table widgets.
4. See error:

 =================================== FAILURES ===================================
I/python.stdout: _________________________________ test_scroll __________________________________
I/python.stdout: Traceback (most recent call last):
I/python.stdout:   File "/data/data/org.beeware.toga.testbed/files/chaquopy/AssetFinder/app/tests/widgets/test_detailedlist.py", line 107, in test_scroll
I/python.stdout:     assert probe.scroll_position == pytest.approx(
I/python.stdout: AssertionError: assert 3200.0 == 2669.5238095238096 ± 4.0e+02
I/python.stdout:   comparison failed
I/python.stdout:   Obtained: 3200.0
I/python.stdout:   Expected: 2669.5238095238096 ± 4.0e+02
I/python.stdout: _________________________________ test_scroll __________________________________
I/python.stdout: Traceback (most recent call last):
I/python.stdout:   File "/data/data/org.beeware.toga.testbed/files/chaquopy/AssetFinder/app/tests/widgets/test_table.py", line 126, in test_scroll
I/python.stdout:     assert probe.max_scroll_position > probe.height * 2
I/python.stdout: AssertionError: assert 1286.095238095238 > (1061 * 2)
I/python.stdout:  +  where 1286.095238095238 = <tests_backend.widgets.table.TableProbe object at 0x7a21de3e3100>.max_scroll_position
I/python.stdout:  +  and   1061 = <tests_backend.widgets.table.TableProbe object at 0x7a21de3e3100>.height
I/python.stdout: =========================== short test summary info ============================
I/python.stdout: FAILED tests/widgets/test_detailedlist.py::test_scroll - assert 3200.0 == 266...
I/python.stdout: FAILED tests/widgets/test_table.py::test_scroll - assert 1286.095238095238 > ...
I/python.stdout: ====== 2 failed, 359 passed, 55 skipped, 32 xfailed in 165.32s (0:02:45) =======
I/python.stdout: Backfilling empty coverage stack...

Expected behavior

test_scroll should pass for DetailedList & Table widgets.

Screenshots

No response

Environment

• Operating System: Host: Windows 11, Emulator: Android 12
• Python version: 3.10.11
• Software versions:
  • Briefcase: 0.3.17
  • Toga: latest main branch
  • ...

Logs

No logs generated

Additional context

No response

[freakboy3742]

I believe this is due to the emulator you have selected.

The testbed is designed to run on a very specific emulator - in particular, a device with an Android 31 system image with a Pixel 3a skin. If you have a device with different dimensions, the tests will fail.

This isn't the same skin that Briefcase generated by default - Briefcase will use the Pixel 7 Pro skin by default. The Pixel 3a skin was the default until around the end of last year.

There's 2 changes that could be made here:

1. Update the test suite to use the Briefcase default emulator skin
2. Update the tests themselves to be resilient to different device sizes.

(1) by itself is relatively straightforward - it only requires a CI config change

(2) requires us to find a way to get the device dimensions, and incorporate that into the calculations that are failing. In making that change, we need to make certain that we're not doing a "1==1" check; i.e., we need to make sure that we're getting an independent datum, not getting a different version of the value we're asserting.

There would be an intermediate version that implements (1) but not (2), manually updating the tests to use the "new" passing values.

[proneon267]

For the time being, I'll do the intermediate version since that would be quicker and I need the tests to pass locally to check for the missing coverage errors for the Android canvas in #2484. I'll open a new PR with the changes.

[freakboy3742]

You can also make the tests pass locally by creating an Android image using the Pixel 3a skin. The CI configuration has the invocation to do this.

[proneon267]

I tried to do that with:

 briefcase run android --device '{"avd":"beePhone","skin":"pixel_3a"}' --test

But got error:

Unable to create emulator with definition '{avd:beePhone,skin:pixel_3a}'

Here is the log file: briefcase.2024_04_21-22_55_25.run.log

Looking at the log file, it seems to encounter JSON decoding error.

[freakboy3742]

Yes - because you haven't escaped the JSON for use in Windows. The CI configuration is running on Ubuntu, so it's able to use Unix quoting.

The value that is being parsed internally is {avd:beePhone,skin:pixel_3a}. That isn't valid JSON; you need to pass in {"avd":"beePhone","skin":"pixel_3a"}. The way you're currently invoking the command the "'s are being parsed out.

[proneon267]

Thanks! Here is the escaped version:

briefcase run android --device '{\"avd\":\"beePhone\",\"skin\":\"pixel_3a\"}' --test