TODO_TESTS

Here are a collection of notes about things we want to accomplish with
the testing infrastructure.

matplotlib.testing.image_comparison decorator
=============================================

Collect all the failures in the image_comparison() decorator and raise
one failure that describes all the failed images. Right now the loop
that iterates over the images raises an exception on the first
failure, which clearly breaks out of the loop.

Put image comparison results into more hierarchical view to facilitate
knowing which test module creates which image. (Also, revive John's
move_good.py script to deal with new image locations.)

Misc testing stuff
==================

Make ImageComparisonFailure cause nose to report "fail" (rather than
"error").

Recreate John's move_good.py script once the better hierarchy of test
result images is established.

Buildbot infrastructure
=======================

Put "actual" images online from buildbots, even without failure.

Get a Windows XP 32-bit build slave.

Build nightly binaries for win32 and Mac OS X.

Build nightly source tarballs.

Build nightly docs and put online somewhere.

A note about the hack that is the failed image gathering mechanism
==================================================================

The trouble with the current actual/baseline/diff result gathering
mechanism is that it uses the filesystem as a means for communication
withing the nose test running process in addition to communication
with the buildbot process through hard-coded assumptions about paths
and filenames. If the only concern was within nose, we could
presumably re-work some the old MplNoseTester plugin to handle the new
case, but given the buildbot consideration it gets more difficult to
get these frameworks talking through supported API calls. Thus,
although the hardcoded path and filename stuff is a hack, it will
require some serious nose and buildbot learning to figure out how to
do it the "right" way. So I'm all for sticking with the hack right
now, and making a bit nicer by doing things like having a better
directory hierarchy layout for the actual result images.