Lona has no browser tests

[fscherf]

At the moment Lona has no browser tests. This means the whole Javascript client stack, which involves rendering and event handling, has no automated testing infrastructure.
I created fscherf/topic/browser-testing which contains a proposal on how to start a Lona app from the pytest infrastructure.

I personally played a little bit with selenium and would try to add selenium testing to this approach.

[sobolevn]

I will have time this week to contribute my vision on browser testing, I am going to use Cypress. We can compare / merge our approaches.

[fscherf]

@sobolevn Great! I am very curious how this looks in cypress. Will the tests be python or javascript code?

[sobolevn]

TypeScript 🙂

[maratori]

I have some experience with selenium tests. I used following stack:

• pytest
• selene - testing library (wrapper around selenium)
• selenoid - run browser inside docker (better alternative for selenium hub)
• allure - test reporting system

Regarding cypress. I believe tests should use Python because the whole library is about Python.

I know there is a playwright, but I didn't use it. May be it is the best choice. It should be supported well by Microsoft.

[fscherf]

@sobolevn I agree with @maratori. The whole library is about using a complete web-stack from simple python scripts. Also any test infrastructure we come up with should be a reusable. For me it's not enough to test Lona, i have to test Lona based projects too. So a Python based solution would be best, but i don't want to stop you from proposing a typescript alternative 👍

[sobolevn]

I believe tests should use Python because the whole library is about Python

Ok, but I guess I cannot help with that 😞
I've stopped using Selenium somewhere around ~2014 and never looked back!

but i don't want to stop you from proposing a typescript alternative

I think that I can showing some existing setups instead, this will take far less time (knowing that there's a low chance of acceptance) 🙂

So, let's have a look at examples that I / internet already have:

1. Setup. We tend to store all e2e files in a separate folder. Something like: https://github.com/wemake-services/wemake-vue-template/tree/master/template/docker/testcafe (it uses TestCafe, but it is almost the same thing)
2. All e2e tests are executed inside a Docker container, we can use existing image: https://hub.docker.com/r/cypress/browsers
3. This is how test looks like (https://docs.cypress.io/guides/getting-started/writing-your-first-test#Step-4-Make-an-assertion):

describe('My First Test', () => {
  it('Gets, types and asserts', () => {
    cy.visit('https://example.cypress.io')

    cy.contains('type').click()

    // Should be on a new URL which includes '/commands/actions'
    cy.url().should('include', '/commands/actions')

    // Get an input, type into it and verify that the value has been updated
    cy.get('.action-email')
      .type('fake@email.com')
      .should('have.value', 'fake@email.com')
  })
})

4. Extra docs on TS support (almost native support): https://docs.cypress.io/guides/tooling/typescript-support It is a hard requirement, because we need to catch as many bugs in test code as we can before actually running them. Since this might be a long process
5. Tests are recorded with Cypress Recorder: https://chrome.google.com/webstore/detail/cypress-recorder/glcapdcacdfkokcmicllhcjigeodacab It just writes test cases for you, as you would expect
6. The last step is CI integration: you don't want to run all your e2e suit at all times, because they are slow and can be flaky. So, the solution is to run them manually when you need to:

[fscherf]

@sobolevn Interesting! Why do you run browser tests in docker containers?
Why did you stop using selenium?

[sobolevn]

Why do you run browser tests in docker containers?

Because I run almost everything in a docker container 🙂
And it really simplifies the setup. You don't have to install different browsers / drivers / window managers. It just works, something like docker run --rm cypress:chrome path/to/your/config

You can change browsers as easy as: docker run --rm cypress:firefox path/to/your/config

Why did you stop using selenium?

It requires the whole JVM stack. It has old sync-style API with lots of tricky parts. And it is rather slow.
New runners are faster, simplier to install, and, what is the most important, have nicer user-facing API. For example, you can write quite complex test in Cypress without a single await statement (I am not even mentioning timeout 😆 )

And the last thing about Python's bindings to Selenuim (as far as I know) they are untyped. So, any bugs won't be caught by mypy. It can cost you hours to debug these things.

[fscherf]

@sobolevn Ah ok!

I am not even mentioning timeout

Good point! Do we have to deal with timing related issues? If i write a test with a button that turns red when clicked, how would frameworks like selenium or cypress know when the Lona client is done rendering before checking the buttons color? Do we have to add some kind of hooks into the javascript client?

[fscherf]

@maratori You recommended selene. Would you use it with or without pytest-selenium?

[sobolevn]

If i write a test with a button that turns red when clicked, how would frameworks like selenium or cypress know when the Lona client is done rendering before checking the buttons color?

I don't know. I guess what happens when you click a button is that some HTTP request is sent to your server. You can await this request. Wait until it is completed and the check the button's css.

[maratori]

@maratori You recommended selene. Would you use it with or without pytest-selenium?

I used it without pytest-selenium. Selene uses webdriver_manager under the hood to use correct driver for your browser. Also pytest-selenium wasn't useful for me that time.

About waits and timeouts

As far as I understand all modern tools for browser testing (playwright, cypress, selene) use the same approach.
They poll browser until desired state received. In the test it looks like ordinary assertion (function call).

Example from selene:

browser.all('.srg .g') \
    .should(have.size(10)) \  # wait until query returns 10 elements (default 4 seconds)
    .first \
    .should(have.text('Selenium automates browsers'))  # wait until the first element has text (default 4 seconds)

Hm... I tried to find something similar in playwright and failed :(
I only see that it waits before do an action like click or enter text.
But this is not a problem because there is a helper function waitFor() any condition.

[fscherf]

@maratori: how would this look like in selene? The selene API looks very promising, but it seems to be completely sync. aiohttp testing is async out of the box

[maratori]

@fscherf You are right, it is completely sync. And it was ok to me, because server didn't run in the same python process with tests.
Then my recommendation is to use playwright. As far as I understand it works well with aiohttp.

[fscherf]

@maratori do you mean this library?

https://github.com/microsoft/playwright-python

[maratori]

@fscherf Yes

[fscherf]

@maratori That looks very promising! Could you prototype how we would add the playwright dependency to the test-suite so it could run in ci? I had to run python -m playwright install to make it work, and i read conflicting information that you have to do things for headless systems.

[maratori]

@fscherf I can, but not very soon. I hope at the end of this week.

[fscherf]

@maratori I will create a pull request out of my proposal and create a simple test based on playwright

[fscherf]

I have setup #75 which contains a first playwright based. Overall i am pretty happy with this tool. I couldn't get pytest-playwright to run properly becaus pytest-aiohttp and pytest-playwright both try to start an ioloop in the main thread. Maybe this can be avoided by let both libraries share one ioloop, but i couldn't find out how. Also playwright is python3.7+. So we can't test 3.6.

[maratori]

Hm...
I see that pytest-playwright provides fixtures to work only with sync API.
Example of fixtures for async API I've found here.

[fscherf]

@maratori using pytest-playwright async seems to work fine, but it interferes with pytest-aiohttp because it raisess a
RuntimeError: Cannot run the event loop while another loop is running

[maratori]

@fscherf could you, please, share an example how pytest-playwright async seems to work fine?

Btw: I created an issue microsoft/playwright-pytest#74

[fscherf]

@maratori Oh you are right!

Running

async def test_rendering(page):
    await page.goto("https://example.com")
    await assert page.inner_text('h1') == 'Example Domain'

also raises RuntimeError: Cannot run the event loop while another loop is running. I thought that's because of pytest-aiohttp but obviously it's not.

[fscherf]

@sobolevn: @maratori and i are now done with the server part of the test-suite and added 2 simple tests based on playwright (#75 ).

Is there something we can't do with this approach? Has cypress advantages we miss here?

[fscherf]

@sobolevn Do you still want to propose a cypress based prototype?

[sobolevn]

As @maratori said the whole project is in Python, I agree that adding TS will complicate things for a wide range of contributors.

I am just not aware of some specific Python E2E tooling, because we use TS inside our company and we use TS-based products for frontend / E2E.

All this being said, https://github.com/microsoft/playwright-pytest seems like a cool solution!

[fscherf]

@sobolevn I agree. Then playwright it is! I will merge my pull-request and add some documentation on how to use the new infrastructure. Everything will be available in 1.7.1.

Thanks for your time, to both of you!