Document How The T2 Test Rig Works

[johnnyman727]

I will document my findings here and then move to a more appropriate place.

[johnnyman727]

There are three hardware components and four software components necessary to make the test rig work.

Hardware Components

1. The physical Test Rig itself, of which four are in use at Seeed. These rigs are responsible for (pulled from preliminary design doc:
   • Testing power on 5V pin
   • Testing for appropriate rails voltages
   • Programming the SAMD21 via SWD
   • Loading bootloader onto flash via USB
   • Port tests on SAMD21 (power on off, current limit, SPI/I2C/UART comms, GPIO, ADCs, DAC)
   • Testing MediaTek booting
   • Testing MediaTek image integrity
   • Testing USB Hub detect and current limit
   • Testing MediaTek RF
   • Testing MediaTek Ethernet
2. Each test rig is connected to a single Asus computer which is overseeing the test rig. This computer is responsible for:
   • Downloading the latest images from the testalator server
   • Initializing the test rig software
3. The server running at testalator.tessel.io (hosted on Digital Ocean). This server is responsible for:
   • Serving the latest images
   • Receiving logs from the testalator results

Software Components

1. The Debian image running on the Asus computer. It has modifications that set the proper USB permissions, remove screensavers, etc. so that we don't have to deal with strange configurations on boot.
2. Bootup scripts that get executed when the debian image finishes booting. These scripts essentially download the newest firmware images and initialize the test rigs. These scripts are distributed via flash drive and are not public because they contain SSH keys to the testalator server.
3. Testalator Server is running at testalator.tessel.io and it returns the latest builds and updates a database of logs.
4. Testalator Client is the "meat and potatoes" of the test rig. It provides a web UI to start tests, tracks progress, and sends instructions to the test rig hardware.

Test Rig Process

Assumptions

1. testalator.tessel.io points to a dropbox instance which is setup with supervisor to continuously run Testalator Server.
2. Each of the test rigs is plugged into a USB hub which is connected to the ASUS laptop which is powered off.
3. A router with internet access and four ethernet cords is available (it will be used on one part of the tests).

Procedure

1. The ASUS laptop is booted up.
2. The bootup scripts are executed.
3. The bootup scripts fetch the latest images from testalator.tessel.io/client and unzip the tarball. The tarball includes the latest SAMD21 bootloader, SAMD21 firmware, OpenWRT u-boot image, OpenWRT squash-fs image, and the Testalator Client code that runs the tests.
4. The bootup script then begins running Testalator Client
5. The bootup script then opens up a web browser window to the address of the client.
6. Tessels with surface mount components soldered (but without through hole components) are loaded onto the test rigs
7. The start button on the test rig is pressed, the event is sent to the Testalator Client software.
8. The Testalator Client flashes the SAMD21 firmware and OpenWRT firmware with a Python test script.
9. More Python tests are executed that perform all of the SMT testing.
10. As tests complete, Testalator Client software updates the web application via websockets.
11. Once all the SMT tests complete, the test rigs are removed from the USB hub, the next batch of SMT Tessels are placed on the test rigs and the process repeats from step 7.
12. Once all SMT Tessels have been run through SMT testing, Tessels receive through hole assembly.
13. Test rigs are removed from the USB hub.
14. Tessels are plugged directly into the USB hub.
15. The Testalator Client detects Tessels and begins through hole testing (which includes ethernet testing and reading a file over the USB port.
16. Upon completion (which is noted on the web app UI), the next batch of Tessels are loaded onto the USB cables and the tests continue until all Tessels have been complete.

Lingering Questions:

1. Can we update the Testalator Client t2-cli dependency such that it's based off of t2-cli at master and uses usb@1.1.0?
2. What is the host.json file that's on the USB sticks used by Seeed? Is this manually generated? The contents look like this:

{
"ssid" : "tmTest",
"password" : "REDACTED"
"name" : "host2"
"build": "6-25-2015"
"server": "http://testalator.tessel.io"
}

It's the build date that concerns me because I'd like to be able to update the golden image.

[johnnyman727]

Here is how I plan on improving the test rig this week:

1. Python code for flashing OpenWRT update doesn't work very often with t2-firmware @ master. Need to fix.
2. Install Node V4.x on Debian.
3. Clean up test rig changes to CLI and integrate with master (including usb@1.1.0)
4. Figure out a more simple way of bundling binaries. They are currently in two different places (client/bin, server/public/build)
5. Make sure ethernet tests are actually ethernet tests and not wifi tests
6. Improve the Speed of wifi tests (currently has a 30 second timeout)
7. Include deadbeef.hex file in client test suite
8. Assign the proper range of MAC address and store result
9. Dynamically load host address from ifconfig
10. Fully document how to use the test rig with Linux and OSX
11. General robustness improvements