Unity as extension or plugin

[mgrewe]

Hi all,

I'm currently working on bringing Unity3D to jsPsych and would like to discuss how to best implement this.

Goal: Create trials in PsychoPy based on a paradigm that has been created with Unity WebGL. I plan to allow live-configuration by passing JSON messages to the app such that trials with different conditions of the same paradigm can be easily created using the same unity instance. E.g. different delays in a virtual mirror like in this project. Vice versa, events (button presses, position in a virtual world et.) captured in Unity can be passed back and recorded in jsPsych.

Background: Unity needs (1) to load typically large files containing the WebGL application, (2) some time to initialize, and (3) a (hidden) unity-canvas that it is attached to, and that is not removed before unity has been closed.

My approach: I would create a Unity Object (UO) that loads the files from a URL and creates the Unity instance & canvas. It can then be used by lightweight plugins that pass the parameters for each condition over to the unity instance on beginning of the trial, append and remove the unity-canvas to/from the DOM, and listen to events created from unity containing the experimental data.

Question: Should the UO be implemented as plugin or extension.

• As a plugin: @jodeleeuw proposed implementation similar to the preload plugin as mentioned here. This is beneficial because we can show a landing page for the user before the heavy loading takes place, and nicely visualize a loading indicator using the display_element provided by jsPsych. However, I currently do not see how such a plugin can be easily accessed by the lightweight plugins that configure the paradigm's condition, except for a global variable.

• As an extension: Implementation as an extension would allow the UO to be accessible from all plugins via this.jsPsych.extensions and follows the convention that extensions are pieces of code that are shared by different plugins. However, I currently do not see how a progress bar can be best shown since jsPsych does not deliver something like a display_element.

What do you think? Happy to hear your opinion.

[jodeleeuw]

A hybrid approach might make sense. You could use an extension to create the Unity Object, store references to the canvas, etc. Then plugins can interact with this extension to actually perform things like loading, modifying parameters, etc.

This isn't quite the use case we had in mind when developing extensions, which were really meant to be "universally compatible" and layer on top of existing plugins. It makes me wonder if we need some kind of additional system for elements that might persist across plugins.

@becky-gilbert and @bjoluc curious to hear your thoughts if you have a moment.

[mgrewe]

Thanks. I have a first draft for the hybrid extension/plugin combination, and I think this is working well. I will prepare a draft PR such that we have something to discuss.

[mgrewe]

Hey all,

I have prepared a short working example that you should be able to run and inspect. I have not created a draft PR to avoid splitting the discussion into too many branches.

Please find the jsPsych code here:
https://github.com/mgrewe/jspsych-contrib/tree/unity_integration

The example also requires the unity binary blobs. There is a separate repository for it here:
https://github.com/mgrewe/jsPsychUnityExample

I have also created a release package that you can use if you do not want to mess around with compilation in Unity. Just place the files from the zip archive next to the html file in the plugin-unity/example folder. Note that you need to run the example using a webserver because unity does not support loading through file:///.

Happy to receive your feedback!

[bjoluc]

It makes me wonder if we need some kind of additional system for elements that might persist across plugins.

I think such a system exists via jsPsych.getDisplayContainerElement(). If extension devs need to display anything persistently, the DOM is theirs – jsPsych only resets the display element itself 🙂 @jodeleeuw Did you have anything specific in mind?

[mgrewe]

Hi @jodeleeuw and @bjoluc ,

Pinging the discussion: If we implement the Unity Object (UO) as a plugin, do you see any way how this plugin can be easily accessed by the lightweight plugins? Or parsed differently: Is there a way to search the timeline for a specific plugin from within another plugin?

[mgrewe]

Hey all, we currently successfully use the combination of unity extension (holding the ref to the Unity instance) and the lightweight plugins (for configuration of the unity scene according to the different conditions) in an online experiment. I think it works nicely for us.
I will be able to work on this intensively till end of Feb. So if you are interested in integrating this stuff into the jspsych or jspsych-contrib repos, please let me know soonish.

[jodeleeuw]

Hi @mgrewe,

I think this sounds like a great contribution for jspsych-contrib. I think -contrib is the better fit because I'm not sure we're ready to take on maintenance of this feature as part of the main repo, but having an available set of tools that others can use and get inspiration from via -contrib would be awesome.

[mgrewe]

Thanks for the feedback. I will provide a pull request soon.