Finding bugs in history

[alessandrofelder]

Charlene Bultoc has just started a post-doc at an important neuro-science institute. She is doing research on a new methodology to analyse signals in our brains detected through a combination of CT and MRI. Using image processing techniques she can simplify the whole dataset into a grid of 20x20 arrays.

Her theory is that the average of such signals through the sagittal plane is constant over time, so she has written some software to calculate this. She decided to write that software in Python so she could share it (via GitHub, rsd-sagital_average) with people from other labs. She didn't know as much Python when she started as she does now, so you can see that evolution in her program.

Charlene is an advocate of reproducibility, and as such she has been keeping track of what versions she's run for each of her results. "That's better than keeping just the date!" you can hear her saying. So for each batch of images she processes she creates a file versions.txt with a content like:

scikit-image == 0.16.2
scikit-brain == 1.0
git://git.example.com:brain_analysis.git@dfc801d7db41bc8e4dea104786497f3eb09ae9e0
git://github.com:UCL-RITS/rsd-sagital_average.git@02fd8791c93b630fc2aecd198e4e4a3a9b441eda
numpy == 1.17

With that information she can go and run the same analysis again and again and be as
reproducible as she can.

However she's found that sagital_average has a problem... and she needs to re-analyse all
the data since that bug was introduced. Running the analysis for all the data she's produced
is not viable as each run takes three days to execute - assuming she has the resources available in the university cluster, and she has more than 300 results.

In all the versions of the program, it reads and writes csv files. Charlene has improved the program considerably over the time, but kept the same defaults (specifically, there are an input file, brain_sample.csv, and an output file, brain_average.csv). She has always "tested" her program with the brain_sample.csv input file provided in the repository. However (and that's part of the problem!), the effect of the bug is not noticeable with that file.

We can then help her either by letting her use our laptops or (better) by finding when the bug was introduced and then run only the ones that need to be re-analysed.

Finding when the bug was introduced seems the quickest way. Download the repository with her sagital_average.py script and use git bisect to find the commit at which the script started to give wrong results.

Do it manually first (as explained in this section of the notes).

Steps to help Charlene:

1. Fork Charlene's repository and clone your fork.
2. Run the latest version of the code with the existing input file
3. Create a new input file to figure out what the bug is
   Hint: You can generate an input file that does show the error using the code snippet below:

   data_input = np.zeros((20, 20))
   data_input[-1, :] = 1
   np.savetxt("brain_sample.csv", data_input, fmt='%d', delimiter=',')

   You may need to create the brain_sample.csv file each time you move through the commits.
4. Use bisect manually until you find the introduction of the error. Take note of the hash and date of the commit that introduced the bug - you will need this information in class.
5. How would you fix the bug?