-
Notifications
You must be signed in to change notification settings - Fork 3k
Firefox iOS Test Types
Firefox and Focus iOS are tested by a combination of automated tests and manual tests on a regular basis.
The automated tests are found in the same repository with the source code. Each type of test serves a different purpose. All automated tests are run and monitored on a regular basis.
Firefox tests are found here: https://github.com/mozilla-mobile/firefox-ios/tree/main/firefox-ios/firefox-ios-tests/Tests
Focus tests are found here: https://github.com/mozilla-mobile/firefox-ios/tree/main/focus-ios/focus-ios-tests
All automated tests run on Bitrise and MacStadium’s iOS simulator except for the Robo Tests, which are run on physical devices.
Since the apps officially support the current version of iOS and the two previous versions, we run the smoke tests and the full functional tests against the current version and the two previous versions of iOS.
🛑 Note that unit tests are smoke tests are build blocking: The pull request can’t be merged if one of the tests fails. The tests are run when a pull request is open to ensure that the change does not introduce an obvious bug. See “Retry Bitrise Pipelines” for the information on debugging the test failures.
The unit tests check against the individual classes from the source code. They are the quickest automated tests. They are run on Bitrise when PRs are opened. Note that this type of automated tests is maintained by the iOS developers.
These tests are included in the UnitTest test plan for Firefox iOS and UnitTests test plan for Focus iOS respectively.
The XCUITests are end-to-end tests that launch and navigate through the app. These tests are developed, maintained and monitored by Mobile Test Engineering.
The smoke tests check the basic functionality of the app. They are run on Bitrise when PRs are opened and on MacStadium/Jenkins on a daily basis.
For Firefox iOS, the smoke tests are included in the Smoketest1, Smoketest2, Smoketest3 and Smoketest4 test plans. For Focus iOS, the smoke tests are included in the SmokeTests test plan.
The full functional tests perform a set of comprehensive checks on the functionality of the app. They are run on a daily basis on Jenkins/MacStadium.
For Firefox, iOS the full functional tests are included in the FullFunctionalTestPlan test plan. For Focus iOS, the full functional tests are included in the FullFunctionalTests test plan.
We leverage Firebase to test the app on the physical devices and some premade databases to measure the performance of the app under heavy load. The tests are run on Bitrise as scheduled builds a few times per week. These tests are developed, maintained and monitored by Mobile Test Engineering.
Robo test crawls the Firefox iOS app methodically after the app is loaded to physical devices from Google Firebase. The information about any errors and crashes are recorded if they occurs.
The performance tests load some premade databases that are filled with data, such as history, bookmarks and tabs, before the app is started. These tests are run on a weekly basis to ensure that the app is still performant when the app handles the load. The statistics such as time and memory are recorded.
The tests are included in the PerformanceTestPlan test plan.
The sync integration tests ensure that the bookmarks, tabs, history and passwords are synchronized between the Firefox iOS app and Firefox desktop using the same Mozilla account. These tests are run once a week on Jenkins/MacStadium. They are developed, maintained and monitored by Mobile Test Engineering.
The steps on the Firefox iOS app are included in the SyncIntegrationTestPlan test plan.
We use Fastlane to capture some screenshots of Firefox iOS running in different locales. Fastlane injects the locale settings to the iOS simulators and executes a set of tests. The tests tell Fastlane how to navigate the app and when to take screenshots. They are run on Taskcluster on a monthly basis. These tests are developed, maintained and monitored by Mobile Test Engineering.
The tests are included in the L10nScreenshotsTests test plan.
In addition to the automated tests, a set of manual tests are executed regularly to test the functionality of Firefox iOS and Focus iOS. The steps, the expected results and the history of the tests are recorded in TestRail. Mobile Test Engineering and QA Engineer maintain the tests.
Mobile Test Engineering evaluates from time to time whether some manual tests are suitable for automation. If a test is determined to be suitable for automation, a JIRA ticket is created for Mobile Test Engineering so that the test automation work is tracked. Once a test has been automated, we don’t have to execute it manually anymore.
We aim to have a link to TestRail test for every Firefox and Focus automated test so that we continue to keep track of the history of the tests.