-
Notifications
You must be signed in to change notification settings - Fork 227
Guide for Lab Authors
Overview
Writing Autograders
Installing Autograded Labs
Example: Hello Lab
Example: Malloc Lab
This guide explains how to create autograded programming assignments (labs) for the Autolab system.
An autograder is a program that takes a student's work as input, and generates some quantitative evaluation of that work as output. The student's work consists of one or more source files written in an arbitrary programming language. The autograder processes these files and generates arbitrary text lines on stdout. The last text line on stdout must be a JSON string, called an autoresult, that assigns an autograded score to one or more problems, and optionally, generates the scoreboard entries for this submission.
The JSON autoresult is a "scores" hash that assigns a numerical score to one or more problems, and an optional "scoreboard" array that provides the scoreboard entries for this submission. For example,
{"scores": {"Prob1": 10, "Prob2": 5}}
assigns 10 points to "Prob1" and 5 points to "Prob2" for this submission. The names of the problems must exactly match the names of the problems for this lab on the Autolab web site. Not all problems need to be autograded. For example, there might be a problem for this assessment called "Style" that you grade manually after the due date.
If you used the Autolab web site to configure a scoreboard for this lab with three columns called "Prob1", "Prob2", and "Total", then the autoresult might be:
{"scores": {"Prob1": 10, "Prob2": 5}, "scoreboard": [10, 5, 15]}
By convention, an autograder accepts an optional -A command line argument that tells it to emit the JSON autoresult. So if you run the autograder outside of the context of Autolab, you can suppress the autoresult line by calling the autograder without the -A argument.
One of the nice properties of Autolab autograders is that they can be written and tested offline, without requiring any interaction with Autolab. Writing autograders is not easy, but the fact that they can be developed offline allows you to develop and test them in your own familiar computing environment.
After you've written and tested the autograder, you then use the Autolab web site to create the autograded lab. Autolab supports creating new labs from scratch, or reusing labs from previous semesters. We'll describe each of these in turn.
Create a new lab by clicking the "Install Assessment" button and choosing "Option 1: Create a New Assessment from Scratch." For assessment foo, this will create a lab directory in the Autolab file hierarchy called courses/<coursename>/foo. This initial directory contains a couple of config files and a directory called foo/handin that will contain all of the student handin files. In general, you should never modify any of these.
[CMU Only]: For course bar and lab foo, the lab directory is called /afs/cs/academic/class/bar/autolab/foo. For example: /afs/cs/academic/class/15213-f16/autolab/foo. All lab-related files must go in this directory.
-
Using the "Edit Assessment" page, turn on autograding for this lab by selecting "Add Autograder." You will be asked for the name of the image to be used for autograding this lab. Depending on how Autolab was installed, this will be either a Docker image (default) or an AWS image. Autolab is distributed with a default Docker image that contains many commonly used packages. If your class needs different software, then you will need to update the default image or create a new one.
[CMU Only]: The default autograding image at CMU is called
rhel.imgand is a copy of the CMU Andrew machines. If you need custom software installed, please send mail to autolab-help@andrew.cmu.edu. -
If you want a scoreboard, you should select "Add Scoreboard," which will allow you to specify the number of columns and their names. The "Add Scoreboard" page contains a tutorial on how to do this.
-
You'll also need to define the names and point values for all the problems in this lab, including the autograded ones.
-
Each student submission is a single file, either a text source file or an archive file containing multiple files and directories. You'll need to specify the base name for the student submission files (e.g.,
mm.c,handin.tar).
####Step 3: Add the required autograding files.
For an autograded lab, Autolab expects the following two autograding files in the lab directory:
-
autograde-Makefile: runs the autograder on a student submission. -
autograde.tar: contains all of the files (except for the student handin file) that are needed for autograding.
Each time a student submits their work or an instructor requests a regrade, Autolab
- copies the student handin file, along with the two autograding files, to an empty directory on an autograding instance,
- renames the student handin file to base name,
- renames
autograde-MakefiletoMakefile, - executes the command
makeon the autograding instance, and finally - captures the stdout generated by the autograder, and parses the resulting JSON autoresult to determine the autograded scores.
If you've created a lab for a course in a previous semester and have access to the lab directory (as we do at CMU via AFS), you can import the lab into your current course by
- copying the lab directory from the previous course to the current course,
- cleaning out the
handindirectory, then - visiting the "Install Assessment" page and selecting "Option 2: Import an existing assessment from the file system." Autolab will give you a list of all of the directories that appear to be uninstalled labs, from which you can select your particular lab.
If you don't have access to the lab directory, another option is to import a lab from a tarball that was created by running "Export assessment" in an instance of a lab from a previous semester. Visit the "Install Assessment" page and select "Option 3: Import an existing assessment from tarball." This will upload the tarball, create a new lab directory by expanding the tarball, and then import the directory.
In this section we'll look at the simplest possible autograded lab we could imagine, called, appropriately enough, the
Hello Lab
(with tarball), which is stored in a lab directory called hello in the Autolab github repo.
In this lab, students are asked to write a version of the K&R "hello, world" program, called hello.c. The autograder simply checks that the submitted hello.c program compiles and runs with an exit status of zero. If so, the submission gets 100 points. Otherwise it gets 0 points.
Autolab expects to find the autograde-Makefileand autograde.tar files in the hello lab directory, but otherwise places no constraints on the contents and organization of this directory. However, based on our experience, we strongly recommend a directory structure with the following form:
# Basic files created by the lab author
Makefile Builds the lab from src/
README
autograde-Makefile Makefile that runs the autograder
src/ Contains all src files and solutions
test-autograder/ For testing autograder offline
writeup/ Lab writeup that students view from Autolab
# Files created by running make
hello-handout/ The directory that is handed out to students, created
using files from src/.
hello-handout.tar Archive of hello-handout directory
autograde.tar File that is copied to the autograding instance
(along with autograde-Makefile and student handin file)
# Files created and managed by Autolab
handin/ All students handin files
hello.rb Config file
hello.yml Database properties that persist from semester to semester
log.txt Log of autograded submissions
The key idea with this directory structure is to place all code for the lab in the src directory, including the autograding code and any starter code handed out to students in the handout directory (hello-handout.tar in this example). The main makefile creates hello-handout by copying files from src, and then tars it up:
#
# Makefile to manage the example Hello Lab
#
# Get the name of the lab directory
LAB = $(notdir $(PWD))
all: handout handout-tarfile
handout:
# Rebuild the handout directory that students download
(rm -rf $(LAB)-handout; mkdir $(LAB)-handout)
cp -p src/Makefile-handout $(LAB)-handout/Makefile
cp -p src/README-handout $(LAB)-handout/README
cp -p src/hello.c-handout $(LAB)-handout/hello.c
cp -p src/driver.sh $(LAB)-handout
handout-tarfile: handout
# Build *-handout.tar and autograde.tar
tar cvf $(LAB)-handout.tar $(LAB)-handout
cp -p $(LAB)-handout.tar autograde.tar
clean:
# Clean the entire lab directory tree. Note that you can run
# "make clean; make" at any time while the lab is live with no
# adverse effects.
rm -f *~ *.tar
(cd src; make clean)
(cd test-autograder; make clean)
rm -rf $(LAB)-handout
rm -f autograde.tar
#
# CAREFULL!!! This will delete all student records in the logfile and
# in the handin directory. Don't run this once the lab has started.
# Use it to clean the directory when you are starting a new version
# of the lab from scratch, or when you are debugging the lab prior
# to releasing it to the students.
#
cleanallfiles:
# Reset the lab from scratch.
make clean
rm -f log.txt
rm -rf handin/*
The Malloc Lab was developed for the CMU course "15-213: Introduction to Computer Systems." In this lab, students are asked to design and implement their own storage allocator. They submit a source file whose base name is mm.c and which contains implementations of the malloc and free functions. The autograder, which is a C program called mdriver, performs a trace-driven emulation of the functions in mm.c to measure the memory efficiency and throughput of the allocator. The autograder assigns a score between 0 and 100 using a linear combination of the memory efficiency and throughput.
Autolab expects to find the autograde-Makefileand autograde.tar files in the lab directory, but otherwise places no constraints on the contents and organization of this directory. However, based on our experience, we strongly recommend a directory structure along the following lines:
Makefile Main lab makefile and README README autograde-Makefile Autograding makefile autograde.tar All of the files needed for autograding (except the student's submission) handin/ Student handins (created and managed by Autolab) log.txt Log of autograded submissions (created by Autolab) malloclab-handout/ Starter code downloaded by students. Created malloclab-handout.tar by the main lab makefile from files in src/. malloclab.rb Config files (created by Autolab, don't touch) malloclab.yml src/ Contains all source code for the lab test-autograder/ Directory for testing the autograder writeup/ Lab writeup downloaded by students
The key idea with this directory structure is to place all code for the lab in the src directory, including the autograding code and any starter code handed out to students in the handout directory (malloclab-handout in this example). The main makefile creates malloclab-handout by copying files from src:
#
# Makefile to manage the CS:APP Malloc Lab.
#
#
# INSTRUCTOR: No need to ever change anything below here
#
# Writeup files and tar files will use this name.
LAB = $(notdir $(PWD))
all: driver-code lab-writeup handout handout-tarfile
driver-code:
##########################
# Build the driver program
##########################
(cd src; make clean; make LAB=$(LAB))
lab-writeup:
########################
# Format the Lab writeup
########################
(cd writeup; make)
handout: driver-code lab-writeup
#############################
# Build the handout directory
#############################
(rm -rf $(LAB)-handout; mkdir $(LAB)-handout)
cp -p src/{mdriver.c,memlib.c} $(LAB)-handout
cp -p src/{config.h,memlib.h,mm.h} $(LAB)-handout
cp -p src/{clock.c,fcyc.c,fsecs.c,ftimer.c} $(LAB)-handout
cp -p src/{clock.h,fcyc.h,fsecs.h,ftimer.h} $(LAB)-handout
cp -p src/Makefile-handout $(LAB)-handout/Makefile
cp -p src/README-handout $(LAB)-handout/README
cp -p src/mm-naive.c $(LAB)-handout/mm-naive.c
cp -p src/mm-textbook.c $(LAB)-handout/mm-textbook.c
cp -p src/mm-handout.c $(LAB)-handout/mm.c
cp -p src/contracts.h $(LAB)-handout
mkdir $(LAB)-handout/traces
cp -p src/traces/*.rep $(LAB)-handout/traces
rm $(LAB)-handout/traces/secret*.rep
rm $(LAB)-handout/traces/seglist*.rep
cp -p src/traces/seglist.rep $(LAB)-handout/traces/
handout-tarfile: handout
########################################
# Build tarfile of the handout directory
########################################
tar cvf $(LAB)-handout.tar $(LAB)-handout
cp -p $(LAB)-handout.tar autograde.tar
clean:
#####################################
# Clean the entire lab directory tree
#####################################
rm -f *~ *.tar
(cd src; make clean)
(cd writeup; make clean)
rm -rf $(LAB)-handout
rm -f autograde.tar
#
# CAREFULL!!! This will delete all student records in the logfile and
# in the handin directory. Don't run this once the lab has started.
# Use it to clean the directory when you are starting a new version
# of the lab from scratch, or when you are debugging the lab prior
# to releasing it to the students.
#
cleanallfiles:
#############################
# Reset the lab from scratch.
#############################
make clean
rm -f log.txt
rm -rf handin/*
Filenames are disambiguated by appending -handout, which is stripped when they are copied to the handout directory. For example, src/mm.c is the instructor's solution file, and src/mm.c-handout is the starter code that is given to the students in malloclab-handout/mm.c. And src/README is the README for the src directory and src/README-handout is the README that is handed out to students in malloclab-handout/README.
Makefile clock.h fcyc.c fsecs.h mdriver.c mm-naive.c mm.h README config.h fcyc.h ftimer.c memlib.c mm-textbook.c traces/ clock.c contracts.h fsecs.c ftimer.h memlib.h mm.cThe `Makefile` contains the rules for building the autograder:
# Makefile for the malloc lab driver CC = gcc CFLAGS = -Wall -Wextra -Werror -O3 -g -DDRIVER -std=gnu99 -Wno-unused-function -Wno-unused-parameterOBJS = mdriver.o mm.o memlib.o fsecs.o fcyc.o clock.o ftimer.o
all: mdriver
mdriver: $(OBJS) $(CC) $(CFLAGS) -o mdriver $(OBJS)
mdriver.o: mdriver.c fsecs.h fcyc.h clock.h memlib.h config.h mm.h memlib.o: memlib.c memlib.h mm.o: mm.c mm.h memlib.h fsecs.o: fsecs.c fsecs.h config.h fcyc.o: fcyc.c fcyc.h ftimer.o: ftimer.c ftimer.h config.h clock.o: clock.c clock.h
clean: rm -f *~ *.o mdriver
To compile the autograder:
linux> make clean rm -f *~ *.o mdriver bash:makoshark> make gcc -Wall -Wextra -Werror -O3 -g -std=gnu99 -DDRIVER -Wno-unused-function -Wno-unused-parameter -Wno-unused-but-set-variable -c -o mdriver.o mdriver.c gcc -Wall -Wextra -Werror -O3 -g -std=gnu99 -DDRIVER -Wno-unused-function -Wno-unused-parameter -Wno-unused-but-set-variable -c -o mm.o mm.c [... some lines elided] gcc -Wall -Wextra -Werror -O3 -g -std=gnu99 -DDRIVER -Wno-unused-function -Wno-unused-parameter -Wno-unused-but-set-variable -o mdriver mdriver.o mm.o memlib.o fsecs.o fcyc.o clock.o ftimer.o linux>
To run the autograder:
linux> ./mdriver -A
Using default tracefiles in ./traces/
Measuring performance with a cycle counter.
Processor clock rate ~= 2261.0 MHz
Results for mm malloc:
valid util ops secs Kops trace
yes 87% 100000 0.005182 19296 ./traces/alaska.rep
* yes 99% 4805 0.000326 14757 ./traces/amptjp.rep
* yes 88% 4162 0.000232 17923 ./traces/bash.rep
[...some lines elided]
p yes -- 6495 0.000662 9809 ./traces/seglist.rep
yes 99% 12 0.000002 6138 ./traces/short2.rep
18 17 92% 153836 0.009039 17019
Perf index = 61 (util) & 39 (thru) = 100/100
{"scores": {"Autograded Score": 100}, "scoreboard": [100, 100, 17019, 92]}
For this particular example, autograde.tar is simply a copy of the malloclab-handout.tar file that is handed out to students.
Here is the corresponding autograde-Makefile:
all: tar -mxf autograde.tar cp mm.c malloclab-handout cd malloclab-handout; make; ./mdriver -Al clean: rm -rf malloclab-handout
The makefile expands autograde.tar into malloclab-handout, copies mm.c into malloclab-handout, changes directory to malloclab-handout, builds the autograder, and then runs it.
linux> ls -l total 26 lrwxr-xr-x 1 158399611 users 21 Apr 21 2015 Makefile -> ../autograde-Makefile -rw-rw-r-- 1 158399611 users 57 Apr 22 2015 README lrwxr-xr-x 1 158399611 users 16 Apr 21 2015 autograde.tar -> ../autograde.tar -rw-rw-r-- 1 158399611 users 23346 Dec 7 2015 mm.cTo simulate Autolab's behavior on an autograding instance, type
linux> cd test-autograder linux> make clean; make
See something wrong or out of date? Did our wording trip you up? Bad documentation sucks for everyone--be sure to let us know if there's something out of place!
-- The Autolab Team