-
Notifications
You must be signed in to change notification settings - Fork 1.2k
GUI Testing Requirements
Simple wiki page to help me decide which GUI testing environment I’ll use for w3af.
Got seven requirements, each one receives a 1-10 importance multiplier. Then, each testing framework receives a 1-10 score for each requirement and finally the total score for a testing framework is calculated like multiplier_1 * req_score_1 + … + multiplier_7 * req_score_7
- Mainstream: If everybody uses X, it will be part of Python one day and receive support and even more attention (like the [http://www.voidspace.org.uk/python/mock/ Mock] module).
- Pure-Python: It’s a plus if I don’t have to learn any new (external) tool and all tests can be defined in pure-python.
- Maintainable: Tests need to be maintainable. Warning: Tools based on screenshots.
- Based on unittests: If based on unittests module it is more likely that I’ll be able to run the tests with nosetests or any other tool that runs unittests.
- Easy to code: New tests shouldn’t take more time than coding the window I’m testing
- Can be run without X windows: This is an important thing but I realize that it might not be possible. Depending on X means that when running the tests you have to leave the screen alone to avoid interfering with the test. Also, tests can not be run at the same time.
- Community support
- Does the tool have a community and/or company that supports it?
- Is the tool maintained? When was the latest release/svn commit?
| Testing framework | Mainstream(10) | Pure-Python(8) | Maintainable(7) | Based on unittests(10) | Easy to code(8) | Without X(9) | Community support(9) | Total Score |
| LDTP | 10 | 10 | 10 | 9 | 10 | 10 | 8 | 582 |
| unittest + twisted's trial | 5 | 10 | 10 | 10 | 8 | 10 | 5 | 499 |
| mago (based on LDTP) | 8 | 10 | 10 | 10 | 10 | 10 | 0 | 500 |
| guitest | 0 | 10 | 8 | 10 | 7 | 10 | 1 | 391 |
| Xpresser | 5 | 8 | 1 | 8 | 6 | 6 | 5 | 348 |
| StoryText | 5 | 5 | 8 | 5 | 5 | 5 | 4 | 317 |
| sikuli | 8 | 1 | 5 | 1 | 7 | 1 | 5 | 243 |
- guitest is too old and unmaintained, discarding as it might be a solution but won’t scale in the long run.
- unittest+trial sounds interesting, used by Canonical. Not well documented, but it works. More focused on unittest than functional testing.
- LDTP is great, but it is more functional testing than unit-testing. Maybe the solution is to merge LDTP and unittest+trial?
- mago is based on LDTP, has good documentation on how to run tests in [http://mago.ubuntu.com/Documentation/RunningOnHudson Hudson with xvfb]. Still need to understand the advantages of using mago instead of LDTP. The mago project is not supported anymore
- Xpresser tests seem too fragile, if I re-arrange the widgets in a window (but don’t change anything else) they will fail. Tests are not that easy to code since I have to take screenshots of everything I want to focus on, click x-N pixels from a location, etc.
We’ll use LDTP for functional testing of w3af’s GTK UI because:
- The active community and development
- Good documentation
- Pure-Python
- Depends on a11y which is a stable API
- Based on python’s unittest module
The only concern, which will require more research is how to run LDTP tests in xvfb. This proved to be a deal breaker that made me move to dogtail.
If needed we’ll use unittest+trial for unit testing UI modules.
Using dogtail, see the After the decision was made... section.
-
[http://ldtp.freedesktop.org/wiki/RecordHOWTO RecordHOWTO] says that we should use something called LDTPEditor which [http://download.freedesktop.org/ldtp/movies/gcalctool-record.ogg looks very cool and useful] but in the mailing list they say [http://lists.freedesktop.org/archives/ldtp-dev/2010-August/001010.html it is deprecated]. The solution is to “use the API available in LDTP (like getobjectlist, getobjectproperty etc)”.
-
Created [https://sourceforge.net/apps/trac/w3af/browser/branches/threading2/core/ui/gtkUi/tests/ldtp_wrapper/sniff.py?rev=6006 my own sniff].
-
Started to play around with LDTP and after some failures sent an [http://lists.freedesktop.org/archives/ldtp-dev/2012-October/001187.htmlemail to the mailing list]. I recommend you read through the whole thread. The summary is that LDTP doesn’t recognize any DISPLAY environment variables; which breaks my requirements of running tests on the same box using Xvfb.
-
After some more searching I found the dogtail project, which was very similar to LDTP AND it supported the DISPLAY environment variable. I'm writing a wrapper around this tool at the moment of writing these lines. The objective is to be able to run dogtail tests with nosetests, which seems to work!
- General:
- unittest + twister's trial:
- LDTP:
- If moth would have Gnome we could do this! https://www.youtube.com/watch?v=ojAN8MktFHE (awesome!) The only issue is that the code that I'm testing needs to be deployed to moth before running the test (svn up before any test is run is not good since we don't want to commit broken code - maybe scp?). The tests would live in any developer box, the code being tested would be there too, and then it is moved to moth for running the test.
- http://mago.ubuntu.com/ , deprecated (latest release: 2010). jcollado on IRC: "the reason no more effort has been put into it is that the a11y layer isn't that stable as we though, meaning that it's difficult to write tests that are reproducible". And mago uses LDTP, so it seems that LDTP should suffer from this issue too.
def calc(scores):
mult = (10,8,7,10,8,9,9)
total = 0
assert len(mult) == len(scores)
for i in xrange(len(scores)):
total += scores[i] * mult[i]
return total
print calc((0,10,8,10,7,10,1))