Testing

Work Plan

Phase 1 (from the end of January 2011 to the end of April 2011) Done

1. Analyze areas of functionality to be tested, strategy for testing them, and identifying test cases to be created.
   • See Testsuite
2. Review automated test frameworks, and select one.
   • See Automated Test Framework. Our choice is Selenium. We plan to move to Selenium 2 as soon as it is released to gain support for Opera. Google is also providing Selenium support for mobile devices, which we will use as it becomes available.
3. Create a test plan. After preliminary discussions with Frédéric, our plan is:
   • Create Selenium ref tests for LaTeX and MathML processing, rendering, and UI testing that can be automated.
   • Create "scenario tests" for installation, configuration, API and UI testing that can't easily be tested with Selenium ref tests. The scripts in the current ./test directory can be thought of as an initial step in this direction
   • Define testing procedures.
4. Create a test suite of ref tests to be executed in Selenium. The plan here is:
   • Robert will setup a github repository dedicated to MathJax testing.
   • Robert will find people to try Selenium and the automated testing from MathJax-test github repository.
   • Frédéric will start writing reftests for LaTeXToMathML and MathMLToDisplay covering the main features.
   • Frédéric will check the issue trackers for sensitive parts of the implementation that should be tested in priority (marked with QA tags).

Phase 2 (from May 2011 to the end of December 2011)

1. Tests covering most of the LaTeX support. Done
2. Tests covering most of the MathML support. Done
3. Tests covering most of the public API. Done
4. Tests covering most of the Configuration options. Done
5. Tests covering most of the UI. Done
6. Web User Interface to control the testing instances. Done (see taskViewer.png in attachment)
7. Allow scheduling testing instances. Done
8. Upgrade Selenium and support more platforms. Done (see Migration from Selenium 1 to Selenium 2 and Platforms Supported)
9. Interface to choose/update a version of MathJax (stable, development, developer's branches etc). Done
10. Set up the webfactional machine to store the test suite, the development branches, the test runner and the QA web interface. Done
11. Set up test machines to run automated tests. In Progress
12. Connect the webfactional machine with the test machines. In Progress

Other work

• Maintain Selenium up-to-date and support more platforms.
• Complete documentation of the testing framework.
• Report browser bugs upstream.
• Maintain the testsuite:
  • Ensure that the tests work correctly on all platforms.
  • Add tests for new MathJax issues.
  • Find a way to improve simulation of user interaction via Selenium 2's "native events". Improve/Complete UI and Configuration tests that require this feature.
• Continue to improve the testing framework interface, including:
  • Interface to choose/update a browser version (stable, beta, nightly etc). For the moment this is possible with Selenium 1 by installing the programs somewhere on a test machine and setting the browserPath configuration option accordingly.
  • Set up machines to run tests and report errors regularly.