-
Notifications
You must be signed in to change notification settings - Fork 5
devlog
I began implementation of test sweep execution, using yesterday's implementation plan as a guide.
First, I implemented a mechanism that makes the import boilerplate read-only. Next, I extended the template-fetching code to also fetch wheats and chaffs from the assignment folder (it does so by listing the contents of the folders named 'wheat' and 'chaff'). Then, I [implemented a basic dependency injection mechanism] such that the my-gdrive import would be dynamically re-interpreted as an import for a wheat. Finally, I completed an initial implementation of the one-click-multiple-runs mechanism so that when the Run button is clicked, the contents of the definitions pane are run against all of the wheats.
(Bonus: I also changed the header background color as a first step towards visually distinguishing examplar from CPO.)
What remains to be done: implementing the full staged-evaluation mechanism, where first wheats are injected, then, conditionally, chaffs. I'll need to strike a very careful balance between not revealing too many details about the assignment and presenting helpful error messages.
I'm starting work on test sweep execution, but I need to figure out how to implement it. I'm beginning by tracing what happens when the 'Run' button is clicked.
The function runMainCode in repl-ui.js is called. This function covers the UI side of running a program, and calls into restartInteractions in cpo-main.js, that does the heavy lifting. This result value represents the execution result of whatever is in the definitions window.
I think the quick-and-dirty way to get the multiple-executions behavior is to modify the call to restartInteractions in repl-ui.js to be multiple calls, while doing something to swap out the imported implementation between them.
This something could be as 'simple' doing a string replacement, or triggering some dependency injection mechanism in gdrive-locators.js. The former is easier but messes up sourcelocs, the latter is messier but doesn't.
Joe confirmed that the README was a little out of date and that a Redis database is necessary to get Google logins working. I spun up a redis instance. Google logins work. Mostly. My cookies are in an inconsistent state (I think?), so I have to use an incognito browser window for development.
Assignment template URLS work. Implements template URLs in the form of #template=GDRIVEID, where GDRIVEID is the gdrive id of a public folder containing a file named template.arr. The contents of template.arr are loaded into the editor.
Unfortunately, this is overly simplistic because, unlike usual CPO, we need to support a consistent notion of an associated assignment. For a typical programming homework, we distribute a #share URL, which graduates to a student-specific #program URL once they make changes, and the provenance of the file is forgotten. For examplar, we still want the student to be able to save their changes, but we also need to retain a notion of associated assignment so we can run their file against wheats and chaffs.
So I'm now tackling this design issue and a few related ones (how do we store student content, how do we store analytics). My first instinct was the Google Drive Realtime API, which provides realtime-access to opaque gdrive-stored blobs. Unfortunately, it was recently deprecated. Ugh.
(several hours later...)
After a false-start futzing with application data folders, the above shortcomings are resolved. User work is saved to their gdrive and is associated with the assignment using a mechanism based on custom properties. When a user navigates to a #template=GDRIVE URL, we search their gdrive for a file with the property assignment equal to GDRIVE. If such a file exists, we load it. Otherwise, the file named template.arr is selected from the folder of id GDRIVE. We copy it to the user's gdrive, set the property assignment of the copied file equal to GDRIVE, and share the copied file with a course staff google account.
First day of work! Shriram, Yanyan and I scoped out the project. I began development.
By the beginning of the semester, we will have a tool where:
- the student goes to a url, e.g.,
http://examplar.pyret.org/#template=<GDRIVEID> - a CPO-like loads with a test sweep template for that assignment. The REPL is disabled.
- upon clicking "Run", the test sweep is run against a corpus of wheats and chaffs, and one of the following outputs:
- if there are compilation errors, those are reported
- otherwise:
- if the user does not pass all wheats, some message indicating there was an issue with their test
- otherwise:
- if the user catches all chaffs, some message indicating they should move on to the rest of the assignment, and perhaps a note that some number of chaffs were not part of the exercise so they should expand their test suite for final submission
- otherwise, the fraction of chaffs they caught
This tool will serve as a drop-in for the usual test sweep writing step of homeworks. Completion of a sweep using this tool will be a requirement, but submissions will not be directly graded. We will have some mechanism to keep track of when students seek help from TAs.
We'll monitor student progress developing their test sweeps via instrumentation.
Hypothesis: Student test sweeps produced with this tool will be (1) finer-grain and (2) more effective than sweeps produced in previous semesters without in-flow feedback.
Students will navigate to a URL in the form of http://examplar.pyret.org/#template=<GDRIVEID>. GDRIVEID will point to a gdrive folder with this layout:
assignment_name
├ chaffs/
├ wheats/
└ template.arr
...where:
-
template.arris a test template file for the assignment -
chaffs/andwheats/are folders containing the compiled wheats and chaffs implementation modules
I familiarized myself with the gdrive API for listing folder contents. I'm confident that the above scheme will work just fine. Incidentally, listing the contents of even a public folder requires being logged in to a google account; this naturally enforces the requirement that students must login to use the tool!
I pulled CPO's horizon branch. The build process described in the README for the horizon branch result in a Pyret compilation error. I shifted gears and used master as a starting point, instead. This fixed the compilation error and I was able to run a local instance of CPO. However, after following the instructions for configuring google API credentials I've hit a wall: all logins fail on the login callback page which displays the error {"type":"DB error","error":{}}. The terminal I'm running the CPO server from displays a trace, too:
Error on getting user: TypeError: Cannot read property 'hgetall' of null
at process._tickCallback (internal/process/next_tick.js:61:11)
From previous event:
at Object.getUserByGoogleId (/home/john/projects/examplar/src/storage/redis-store.js:5:14)
at /home/john/projects/examplar/src/server.js:196:31
at /home/john/projects/examplar/src/google-auth.js:102:9
at /home/john/projects/examplar/node_modules/google-auth-library/lib/auth/oauth2client.js:154:5
at Request._callback (/home/john/projects/examplar/node_modules/google-auth-library/lib/transporters.js:106:7)
at Request.self.callback (/home/john/projects/examplar/node_modules/google-auth-library/node_modules/request/request.js:187:22)
at Request.emit (events.js:182:13)
Authentication failure TypeError: Cannot read property 'hgetall' of null
at process._tickCallback (internal/process/next_tick.js:61:11)
From previous event:
at Object.getUserByGoogleId (/home/john/projects/examplar/src/storage/redis-store.js:5:14)
at /home/john/projects/examplar/src/server.js:196:31
at /home/john/projects/examplar/src/google-auth.js:102:9
at /home/john/projects/examplar/node_modules/google-auth-library/lib/auth/oauth2client.js:154:5
at Request._callback (/home/john/projects/examplar/node_modules/google-auth-library/lib/transporters.js:106:7)
at Request.self.callback (/home/john/projects/examplar/node_modules/google-auth-library/node_modules/request/request.js:187:22)
at Request.emit (events.js:182:13) TypeError: Cannot read property 'hgetall' of null
at process._tickCallback (internal/process/next_tick.js:61:11)
From previous event:
at Object.getUserByGoogleId (/home/john/projects/examplar/src/storage/redis-store.js:5:14)
at /home/john/projects/examplar/src/server.js:196:31
at /home/john/projects/examplar/src/google-auth.js:102:9
at /home/john/projects/examplar/node_modules/google-auth-library/lib/auth/oauth2client.js:154:5
at Request._callback (/home/john/projects/examplar/node_modules/google-auth-library/lib/transporters.js:106:7)
at Request.self.callback (/home/john/projects/examplar/node_modules/google-auth-library/node_modules/request/request.js:187:22)
at Request.emit (events.js:182:13)
I'm not sure what to make of these errors. The README doesn't mention anything about databases except for a brief note below the heading "Setting up your own remote version of code.pyret.org with Heroku" which isn't what I'm trying to do at this moment. The above trace mentions redis, however.
I've sought help on Slack, but, at the time of writing, haven't gotten any responses.