Feedback on design of eye-tracking calibration and validation plugins

[jodeleeuw]

As part of version 6.3 we are adding support for a new kind of jsPsych add-on called an extension. These are modular components that are layered on top of plugins. They allow you to extend the functionality of a plugin in some manner that is typically the same across all plugins.

An example is eye tracking. This is a feature that you may want to add on top of any existing plugin. As of 6.3 you'll be able to do something like this to add webgazer-based eye tracking to any trial:

var trial = {
	type: 'image-button-response',
	stimulus: 'my-scene.png',
	extensions: ['webgazer']
}

I'm currently working on the design and implementation of plugins that will allow for calibration and validation of eye tracking throughout the experiment. But, I'm not an expert in eye-tracking research. I'd love to hear from those of you who are interested in using eye tracking what kinds of features would make these plugins usable. In particular, I'm currently thinking about issues like:

1. What kinds of calibration displays / routines are important to offer?
2. What options would you want for a validation routine? I'm assuming that validation will be done periodically throughout a study.
3. What kinds of data would you want about validation? Keep in mind that precision here is not that great! Raw data is great, but I think it will be important to provide some kind of information that lets the researcher make a decision about whether to proceed or re-calibrate.

As I continue to make progress on these plugins I'll share a demo here for further feedback.

Thanks in advance for your input!

[ozika]

Hi @jodeleeuw and thanks for doing this!

I will start with the basics - I am not an eye-tracking expert but I do use it. I am actually more interested in pupil dilation (see below).

Questions/comments

1. You mention that the experimenter can decide whether to proceed manually - how is the user connected with the experimenter?

Calibration options

1. Multiple fixation sets (some studies may only care about horizontal saccades for example, so calibrating edges of the screen may not be necessary, see img below)
   
2. Validation, for start, could provide the angular distance between the initial calibration and the new measurement (per point and on average).
3. In terms of data, x/y coordinates and angular/euclidean distances would do I think.

Pupilimmetry

1. Is there an option to do this as well? I think it's possible to use openCV, I would be happy to look into this.

[jodeleeuw]

Thanks @ozika!

You mention that the experimenter can decide whether to proceed manually - how is the user connected with the experimenter?

I meant this more in a programmatic sense. I, as a researcher/programmer, could specify a rule about whether the validation is good enough to continue. I'm wondering what metrics make sense to specify that rule. I know that angular distance is going to be the norm in lab studies, but we can't easily know the distance that the participant is from the screen.

For calibration patterns the current implementation of the plugin allows the user to specify an array of points:

var calibration = {
    type: 'webgazer-calibrate',
    calibration_points: [
	[50,50], [50,25], [25,50], [75,50], [50,75]
    ]
}

That would produce the same pattern as the five layout in the image, and you could produce the others easily.

Is there an option to do [pupillometry] as well? I think it's possible to use openCV, I would be happy to look into this.

I don't think this is built into web gazer. I'm not too keen on forking web gazer and maintaining a jspsych-version of it. I think it would be better to make pull requests against the main web gazer project, if that team is open to it.

That said, the new extensions mechanism in jsPsych would make it possible to add a pupillometry-only library assuming that such data can be extracted from a webcam.

[jirilukavsky]

Hi @jodeleeuw - this sounds great!

Regarding your questions:

1. In the lab, we are using similar dot displays as @ozika described above. However, the whole calibration procedure should be quite robust (no questions and advice from experimenters are possible in online testing). I think the click-the-dots approach used in WebGazer is quite easy to explain. In this calibration, you see similar dots and you should point and click on them repeatedly (5x). Some eye-trackers are using smooth-pursuit calibration (you follow a dot moving across various locations with your gaze), but I haven't seen this in an online setting yet.
2. For validation, I would suggest showing something like 1 or 3 dots and asking the participant to click on them. I think we need to specify some interval of how often the validation appears. Maybe it could appear also if the face disappears from the camera view for some time.
3. For validation, the list of angular distances from validation targets/dots should be sufficient.

Pupillometry suggested by @ozika would be super cool! But I am not sure about the data quality with current web cameras.

[tpronk]

Version 2 of webgazer is said to be much better than version 1, but you need to build it from source. I made a little tweak in this fork do show/hide the webcam thumbnail: https://github.com/tpronk/WebGazer

You can download a build from this fork via: https://gitlab.pavlovia.org/tpronk/demo_eye_tracking2

[jodeleeuw]

@tpronk -- in the latest official version that I've been working with there are methods to hide/show camera feed. There have been some updates relatively recently to the code base.

[jodeleeuw]

@jirilukavsky - thanks so much! A couple thoughts:

1. In some early explorations I've been worried that a person's gaze and clicks don't always line up. In particular if you have to click a target x times your gaze might start to move by the end of the clicks in anticipation of the next target. So we've been doing gaze-based calibration instead of click-based, but the option for both is there. Maybe there is a way to design click-based calibration in a way that is robust. @tpronk's demo seemed quite nice with a single click per target.
2. We're thinking that validation will be like any other plugin in the experiment. So you could drop a webgazer-validate trial in at whatever interval you'd like. The current validation task is gaze-based, with configurable locations by the researcher. But we could add a click-based mode too.
3. Mentioned this in my comment above, but I'm worried about angular distances being reliable. It's hard to know the distance from the screen. There are methods like implementation of virtual-chinrest procedure #1381 that could help with this, but I wonder if we should be encouraging researchers to think more about screen-based coordinates. For example, if you know that you have 4 quadrant AOIs and that a stimulus is 600x600px, it might be reasonable to proceed if the average error is <150px. Does that seem plausible?

[jirilukavsky]

Some thoughts here:

• re single/multiple clicks - Single click per target might be sufficient. In the @tpronk 's demo, there were 16 calibration dots - if people used other calibration grids (3-, 5-, 9-dot), having more clicks per calibration target might be a good idea.
• re click/gaze calibration - If gaze-based calibration works, it is fine. I agree gaze/clicks do not always line up. Based on my lab experience (EyeLinkII and 1000), the auto-detection of the gaze being over a calibration target also brings some errors. Often you need to check the calibration responses and re-run it or turn on the manual mode where the target waits for a keystroke. I think participants get better in calibrations over time; it usually happens at the beginning of the experiment (people nervous, hasty). In my view, the main benefit of clicking is not the time-and-location synchronization signal, but slowing the participant down and encouraging more to focus on a small detail on the screen.
• re angular/pixel coordinates - You are right, using primarily pixel coordinates is probably better. The virtual chinrest procedure implementation of virtual-chinrest procedure #1381 is intriguing, but the experimenters need to think about pixel coordinates anyway when they set the size of the stimuli.

[tpronk]

Re: calibration and number of clicks and the design decisions I've made for demo_eye_tracking2

• To control for anticipation, it randomizes the location of the dots.
• Researchers can setup the location and number of dots via a configuration file. In the default config I got more dots near the edge of the screen than near the center; I noticed accuracy seems better in the center than near the edges, so this way I try to compensate for that

[tpronk]

Ah yes, before forking I discussed it with them on the forum. Lovely that they implemented it upstream!

…

On Tue, 19 Jan 2021, 15:45 Josh de Leeuw, ***@***.***> wrote: @tpronk <https://github.com/tpronk> -- in the latest official version that I've been working with there are methods to hide/show camera feed. There have been some updates relatively recently to the code base. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1379 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACESG4CVHF5PT2UQZYWA5LTS2WLJ5ANCNFSM4WH3EGZQ> .

[cainesap]

Hello,

Thank you @jodeleeuw and collaborators for jsPsych and webgazer, and others for the discussion here which is valuable to me (newcomer to eye-tracking web experiments). I'm interested in this comment you made above @jodeleeuw : "drop a webgazer-validate trial in at whatever interval you'd like" . I think that's what we'd like to do. I wonder if that feature has been implemented?

We'd like to maintain randomization across our whole stimulus set (e.g. 40 items) and drop in a validation step between item 20 and 21. At the moment I can only imagine arbitrarily dividing the stimuli into 2 trial sets of 20, then inserting validation on the main timeline between them, but this may be a limit of my imagination / failure to understand the full capabilities of jsPsych..

Any pointers would be much appreciated!
best, Andrew