Standardization of stimulus data property across plugins

[jessestorbeck]

We over on the Pushkin team with @jkhartshorne ran into an interesting issue this week where a particular plugin wasn't sending data to our database. The problem turned out to be that the plugin in question (unlike the seeming majority of jsPsych plugins) does not produce a stimulus data property, which our backend code expects.

For database purposes, we would like to continue to keep a dedicated stimulus column. We were wondering if jsPsych might consider adding stimulus to the list of data collected by all plugins. For plugins like animation, this would require renaming stimuli to stimulus; for others like fullscreen, stimulus would probably just get a default value.

Our temporary fix is of course to insert a stimulus property if the plugin doesn't provide one, but perhaps you'd consider standardizing this across plugins, as it already holds for most(?) of them, and this feature might be advantageous to others (besides us) who are working on backends for jsPsych.

[jodeleeuw]

Hey @jessestorbeck and @jkhartshorne,

All of the values that are collected by all plugins are things that the plugin code doesn't have to implement. They are basically core jsPsych data properties that are stored with each trial's data. Adding something like stimulus would be a departure because it would require every plugin author to implement this parameter since we can't infer what the value should be universally.

I'm also not sure there's a reason to elevate stimulus to a required parameter from a purely jsPsych perspective. My quick count is that less than half of the plugins that are part of the core repo have this property. Some of those could certainly support it but don't, while for many others it isn't clear what the value should be.

Is there a reason you need to include something like this in your backend other than supporting your current implementation? I'm trying to figure out what the advantage would be for other backend-devs and why having this particular parameter standardized would be useful.

[jkhartshorne]

Did you mean "supporting our current implementation"? Because the current Pushkin implementation is just incompatible with any plugin that doesn't have a stimulus property. Again, that property could be null (which would make sense for the few plugins that don't present stimuli).

Maybe a little background is helpful. There are a lot of reasons to want to know what the stimulus is on a particular trial. One common thing to do in a massive online experiment is have a very large database of stimuli and only present a fraction to each subject. That requires keeping track of how many subjects have seen which stimulus. It can get expensive to query all the data JSONs and parse them each time, so we have a table in the database to keep track of these things.

Another reason would be automated feedback. If you want to do any machine learning to predict something about subjects, you need to know what each item is!

Another reason would be any sort of automated analysis. So you could imagine r packages that parse and prep data coming out of jsPsych experiments, getting it into tidy format or whatever. I'm not sure there's a use-case there, but I'm not sure there's not.

Basically, any software that needs to know what stimulus a given jsPsych plugin presented needs to find this information somehow. So there either needs to be a standard name, or each software package that interacts with jsPsych needs to require users to futz with the jsPsych plugin to make it standardized. Jesse mentioned just requiring everyone who makes an experiment to ensure there is a stimulus property in the data JSON with the right information. This is not fail-safe, since you can (and frequently will) end up with a situation where one stimulus is actually displayed by the plugin and another is recorded in stimulus. It also means that you can't just drop a jsPsych timeline into Pushkin anymore; there are extra steps to make it compatible. That's clunky. And finally, it means that any software that integrates with jsPsych and needs access to the stimulus information will have to implement its own solution, which will be a pain for the ecosystem.

Basically, most (not all) jsPsych plugins have stimuli. So they are all called something. And the plugin writer has to decide what it's called. It seemed helpful to standardize that. And it's an easy enough thing to check for in the automated tests.

Of course, you have much deeper knowledge of the inner workings of jsPsych. I may be missing something.

[jodeleeuw]

Thanks, this is helpful context to have.

These are some issues that I'm worried about:

1. We can't autogenerate this, unlike all of the other universal data parameters. This imposes a constraint on plugin development that isn't currently there. This isn't necessarily a bad thing, but..
2. There are plugins where the value would be unclear. For example, a plugin that implements a survey. What should the stimulus be? What if there are 10 questions on a page? It could be null, but that's not quite right. It could be the set of 10 questions, but that's bulky and doesn't really semantically map.
3. There are plenty of plugins that don't really have the notion of a stimulus. Browser check, call function, data pipe, full screen, init microphone, init camera, etc. Having a null value is possible of course, but I worry that elevating this to required status sets an expectation for what a plugin is. A major design feature of the plugin system is that it is agnostic about whatever goes on inside, so it's OK to write plugins that simply do backend tasks, or plugins that implement an entire game, or whatever. If you have to indicate what the stimulus is when you design the plugin this is a signal that plugins should present a single stimulus.
4. The current data storage design is based on only storing relevant information for each trial. Plugins simple record whatever data is relevant to them, without worrying about what other plugins record. This does create an extra step when going to a relational database or CSV files, but it makes a lot of sense when working with document based databases or JSON output. It saves space. I understand it's not a big deal to add one additional entry for each trial, but this would be a change in the principles of the design and so then there are questions like why not store a response for every plugin as well?

I think elevating stimulus also isn't quite the universal solution.

1. Imagine a massive online experiment or automated feedback system where the relevant factor was the combination of stimulus and prompt parameters for an image-based trial, so I want to balance over the joint combination of the two.
2. Or another task where the entire stimulus property is mostly redundant info (e.g., has lots of non-informative HTML markup) and the critical bit of info is a single word that varies. stimulus would be unique in this situation, but it's not a particularly readable way of saving this information. I typically add a parameter to the data object in these situations to get cleaner output.

[jodeleeuw]

Did you mean "supporting our current implementation"? Because the current Pushkin implementation is just incompatible with any plugin that doesn't have a stimulus property.

What I was trying to get at here is that this seems like a database implementation choice. For example, the kinds of problems that need to be solved here would be different if the backend were a document-based database instead of a relational database with a JSON column. A different way of indexing the relevant information in Pushkin could also solve the problem.

[jkhartshorne]

Hmmm, I think I'm missing something. We still need a consistent variable name. I guess I'm thinking of this more like an API problem: we need to know what to call.

Maybe we should Zoom when you have some time?

[jodeleeuw]

Sure - happy to! We can schedule via email.