A Mouse & Multi-Touch GUI Library for p5.js
by Carlos "L05" Garcia (@L05_ | L05.is)
p5.js is a wonderfully accessible way for students and creative minds alike to learn computer programming and create interactive art and experiences.
p5.touchgui is intended to extend p5.js and make it easy to add canvas-based buttons, sliders, and other GUI (graphical user interface) objects to p5.js sketches, enabling users to focus on quickly iterating ideas with easily created GUI objects that work with both mouse and multi-touch input.
As an artist and immersive experience designer, I have been researching and developing multi-person interactive experiences for 8 years, experimenting with all kinds of input sensors and mechanisms to create some pretty cool stuff.
A couple of years ago, as I was learning node.js and WebSockets, I started to experiment with p5.js as a way of creating simple, mobile touch-screen interfaces that would enable audience members to control games and installations with their phone. In this process, I created a small toolkit of p5-based GUI elements and a node.js server template for multi-person interactive games. Check out an example of one such experience here.
Sidenote: my next p5.js project is to package and release a multi-player node.js server and client template along with a tutorial!
During my graduate studies at UCLA's Design Media Arts department, I had the wonderful opportunity to work as a teaching assistant for Lauren McCarthy and Casey Reas, who both taught p5.js in their classes. I noticed that students often had difficulty with GUI elements in their projects; building a simple button in p5.js is an excellent learning exercise if the primary focus of an assignment, but it can be a blocker if the assignment goals are to express some larger creative vision.
Having already created my own personal toolkit of project-specific GUI elements, I realized that with some work, I could adapt it into a more generalized tool that may be useful to students, artists, and educators. And so the process began...
My first step in working on p5.touchgui was to research GUI design methodologies and examples. I looked at the current state of UI within p5.js. Users must either use DOM elements via p5.dom or create their own canvas-based GUI objects with p5.js.
I performed a number of preliminary tests with DOM elements and multi-touch screens, and it seemed DOM elements were not well suited to multi-touch environments: the performance was often slow and touch quality inconsistent. DOM elements seemed sufficient for simple interactions but not for more dynamic experiences such as touch-screen controllers.
While p5.dom gives users access to DOM elements, it still feels a bit clunky to use. In my classroom experiences, p5.dom often seemed a pain point for many students trying to use DOM elements in their sketches.
As my focus shifted towards designing canvas-based GUI tools, I became especially interested in learning about Immediate-Mode and Retained-Mode approaches to Graphical User Interfaces. I first learned of these categories from reading Doeke Wartena's description of his Processing GUI library, and this video by Casey Muratori on Immediate-Mode GUIs was particularly illuminating. I decided to dive into creating p5.touchgui as an Immediate-Mode GUI.
I first implemented a version of p5.touchgui using an Immediate-Mode approach, and in many ways, it felt simple and easy-to-use. The GUI Objects were not stored anywhere; they were fed a variable when drawn, and they modified that variable by reference based on user input.
Here's what the code looked like in the Immediate-Mode version:
let x;
function setup() {
createCanvas(400, 400);
createGui();
x = {val: 0.5};
}
function draw() {
background(220);
startGui();
if (button("Button 1", 50, 50)) {
print("Button 1 pressed.");
}
endGui();
}Pros
- Accessing GUI object states in the
draw()loop seemed like an exciting idea for teaching beginners. - No GUI object states had to be stored, retained, or synchronized.
- It was entirely data driven.
Cons
- Required the creation of JavaScript objects in order for pass-by-reference to work. This is an advanced concept to have to introduce to beginners.
- Would be difficult to offer custom styling.
- Multi-touch input would not work well with a strictly Immediate-Mode GUI. This was obviously kind of a deal breaker.
While I really liked some aspects of Immediate Mode GUI (IMGUI), I realized a purely IMGUI approach would not offer the functionality necessary to deliver the primary goals of p5.touchgui:
- Easy to use
- Easy to teach
- Multi-touch support
- Performant
I began to ask myself, how can I adapt some features of IMGUI (such as having access to GUI object states) into a more flexible and comprehensive GUI system?
I ended up creating a Retained-Mode GUI that mimicked some behaviors of an IMGUI. For instance, users can access GuiObject member variables such as button.isPressed, which returns a Boolean value describing if the button is in fact pressed.
By creating callback event handlers specifically for the library, I was also able to account for both mouse and multi-touch input. These custom handlers not only enable expected mouse behavior such as hover states and multi-touch control of GUI objects, but they don't interfere at all with the built-in p5.js event callbacks (e.g. mousePressed(), touchStarted(), etc.). Building the input processing around event-based handlers also made it significantly more performant than calculating inputs during every single frame.
By switching to a Retained-Mode approach, I was able to begin implementing a style system for GuiObjects, building a small set of presets, and providing user-accessible functions for modifying object styles.
Here's what the code looks like now:
let gui;
let b;
function setup() {
createCanvas(400, 400);
gui = createGui();
b = createButton("Button", 50, 50);
}
function draw() {
background(220);
drawGui();
if(b.isPressed) {
print(b.label + " is pressed.");
}
}There are a few areas in which I look forward to further developing p5.touchgui.
I learned at the p5.js Contributors Conference 2019 that p5.js, and by extension p5.touchgui, is very inaccessible to screen readers. On further reflection, this is not surprising given that p5.js is essentially a Canvas object, a buffer of pixels. This pixel buffer and the elements within it inherently carry none of the data screen readers are able to access in DOM elements.
I am interested in working with the p5.js community to find ways of making both p5.touchgui and p5.js more accessible, whether through providing off-screen shadow DOM structures or through solutions such as the Accessibility Object Model (AOM).
Style is one area where p5.touchgui needs more work; as it turns out, it's hard to build a style system that balances customizability with simplicity! The presets are a good start, and it is possible to stylize individual objects, but the number of lines of code it would take to build out a larger interface still seem extraneous.
Some features I think would help:
- Group styling - i.e. determining the style of an entire object class. This is a high priority feature to be added to the library.
- Simplified style properties - the properties currently get pretty granular, which on one hand gives a lot of color control and on the other means that a non-trivial number of properties need to be set. I am very interested in the simplery way Material UI has primary and secondary color choices, for instance.
- More presets!
There are a number of additional features on which I would like to work:
- More GuiObjects!
- Radio Button
- Color Picker
- Button Bank
- Slider Bank
- Range Slider
- Text Field
- Tags (grouping GuiObjects by tag)
- Selective Input Lock (right now all GuiObjects default to input lock)
- Migrate reference documentation to Docsify model.
- More examples!
You can also check out my development notes if you'd like to read a more informal summary of process and interests.
Please use p5.touchgui and give me feedback!
- If you use it in a project, what works and doesn't work?
- If you teach it in a class, what works and doesn't work?
Any questions pertaining to this project may be communicated via Issues on the p5.touchgui GitHub repository. Simply create a new Issue and either assign or tag me in the conversation with @L05. For anything else, don't hesitate to get in touch at carlosga@gmail.com!
I'd like to give an extra special thanks to my GSOC mentor Yining Shi for her support, guidance, understanding, and encouragement throughout the project. I'd also like to thank Lauren McCarthy for talking through some of the pedagogical applications of p5.touchgui with me during its development. Finally, thank you to all of the contributors and community members of p5.js who make it such a fantastic resource and community!