RIOT's new test system: requirements and challenges

How and what should be tested by the new system

Unit tests: at least for API functions of most core/sys modules
Acceptance tests for modules/functions which require real or simulated hardware. This includes for example:
- network simulations to test the behavior of routing protocol
- hardware based tests for transceivers
Compile tests: every example application / build test for a module should compile for every platform if not stated differently in the Makefile.
Code analyzers: the code should be checked by various automated code analyzers for quality issues. For example:
- static code analyzers should be used to identify defects such as memory leaks or control flows which lead to a NULL pointer being dereferenced.
- code style checker could be used to check for any obvious violation of RIOT's coding style.
- model checker could be used to search for complex defects which are difficult to spot for static code analyzers. However, this might not be possible with open-source tools.
  - Divine might be a candidate for embedded model checking of certain modules
  - CPAchecker is another model checker for C.

We need to support various different toolchains (e.g.: clang/gcc for native and gcc for avr, arm and msp430).

we need to be able to automatically determine if a driver or a hardware dependent module works as expected on all platforms it supports.
wast architectural differences (e.g.: 8 bit atmel vs 32 bit arm cortex) between supported platforms makes a generalized emulation / simulation approach for testing difficult if not impossible to implement.
the steadily rising number of boards we support implies that a single organization cannot possibly provide a physical test platform for all of those.

the test system has to function reliable and fast. If the test system often fails for no apparent reason developers will start to ignore test results. Likewise if the system needs a long time to end a test run developers are likely to implement and use their own test system (or worse: cease to use any test system at all).
the system has be user / developer friendly. An easy to use github integration is therefore very important.