Modular, Reactive Gizmos

[viridia]

Caveat: the ideas presented here are fairly wild / brainstormy and I don't expect them to be accepted or implemented - but I do want to write them down here because they may inspire discussion.

I'm in the process of porting my game engine and editor from three.js to Bevy. Three.js has about a dozen Gizmos (which are called "Helpers" in three.js terminology), but I don't find them to be particularly useful for my needs. Instead, I've created my own Gizmo framework which has gone through several iterations. I don't use these in the game per se, but they are very helpful in the editor for things like:

• Terrain editing
• Physics colliders
• Musical and sound effects cues
• NPC pathfinding waypoints
• Navigation meshes
• Portal apertures
• Water current direction
• Use markers (named markers that tell the actor where to stand when interacting with a scenery element).

...and many others. There are about two dozen custom gizmo types that I use, all of which are based on this common framework.

Note: in the following sections, I'm going to talk about JSX. However, I am not proposing that JSX support be added to Bevy.

A typical gizmo in my three.js system looks like this:

/** Component which displays aspects of fixtures which are normally invisible. */
export const FixtureOutlinesOverlay: VoidComponent<Props> = props => {
  const toolState = useToolState();
  const fixtures = createMemo(() => props.structure?.instances.list().filter(isFixture) ?? []);
  const loader = new TextureLoader();
  const mapPin = loader.load(mapPinImg);

  const spriteColor = (config: IRegionAppearance, selected: boolean) =>
    colord(config.color ?? '#ffdd88')
      .darken(selected ? -0.2 : 0.2)
      .toHex() as ColorRepresentation;

  return (
    <SceneProvider>
      <For each={fixtures()}>
        {fix => (
          <>
            <Show when={fix.aspectConfig(waymark)} keyed>
              {config => (
                <>
                  <FixtureOrientationOverlay
                    fixture={fix}
                    config={config}
                    selected={fix === toolState.selectedInstance}
                  />
                  <Sprite
                    texture={mapPin}
                    location={() => fix.position}
                    scale={[0.25, 0.4, 0.25]}
                    color={spriteColor(config, fix === toolState.selectedInstance)}
                    center={mapPinCenter}
                    opacity={1}
                  />
                </>
              )}
            </Show>
            <Show when={fix.aspectConfig(circleMarker)} keyed>
              {config => (
                <>
                  <FixtureOrientationOverlay
                    fixture={fix}
                    config={config}
                    selected={fix === toolState.selectedInstance}
                  />
                  <FlatCircle
                    location={() => fix.position}
                    radius={fix.ensureProperties<ICircularRegionProps>().radius}
                    color={config.color as ColorRepresentation}
                    opacity={fix === toolState.selectedInstance ? 1 : 0.2}
                  />
                  <Sprite
                    texture={mapPin}
                    location={() => fix.position}
                    scale={[0.25, 0.4, 0.25]}
                    color={spriteColor(config, fix === toolState.selectedInstance)}
                    center={mapPinCenter}
                    opacity={1}
                  />
                </>
              )}
            </Show>
              <Show when={fix.ensureProperties<IWaypointsProps>().waypoints.length > 1}>
                <DashedPolyLine
                  vertices={fix
                    .ensureProperties<IWaypointsProps>()
                    .waypoints.map(wp => wp.position)}
                  dashLength={0.2}
                  lineWidth={0.09}
                  opacity={fix === toolState.selectedInstance ? 0.7 : 0.3}
                  occlusionFactor={0.6}
                />
              </Show>
            </Show>
          </>
        )}
      </For>
    </SceneProvider>
  );
};

And here's a screenshot of what that looks like:

Things I want to point out about this code:

• Gizmos are composed of multiple translucent primitives.
• Primitives can include meshes, lines, sprites and floating text.
• Gizmos can also contain reusable "components" such as FlatCircle and DashedPolyline, which are themselves Gizmos.
• Most primitives are 2D meshes like circles and rectangles.
• The attributes of the primitives are reactive: that is, when the game state changes, the Gizmo vertex buffers and material properties are automatically re-calculated. So for example in the above code, the line opacity={fix === toolState.selectedInstance ? 1 : 0.2} means that the opacity of the selected fixture is higher than fixtures that are not selected.
• In my TypeScript code, I use JSX syntax to declaratively define Gizmos. The system is similar in concept to react-three-fiber, however this system is based on Solid.js rather than React.
• Iteration and Conditions are supported via the standard Solid.js components For and Show. So if I want to display a bounding box for every physics collider, it's a simple matter to iterate over the list of colliders.

For implementing 2D primitives such as circles and rectangles, I have a 2D drawing library which generates meshes. So for example, the DashedPolyLine component calls drawShapes.strokePolyLine(), which fills in a vertex buffer and index buffer. This library also accepts a transform, so for example the Gizmo that draws portal apertures can align the generated rectangle with the portal.

Now, I know that Bevy is not a reactive framework like Leptos. However, it does have the ability to trigger behaviors on change events. So something like what I am describing is not entirely unfeasible, although it would no doubt look very different than the code that I have shown.

[alice-i-cecile]

#9400 may also be interesting to you.

[viridia]

#9400 may also be interesting to you.

Yes, I saw that. I think that I would rather have a framework for building my own gizmos than have a large set of pre-made gizmos. For example, no one (other than me) is going to provide what I need for editing terrain meshes.

Another thing I am interested in is Unity's "handles" which are gizmos that have clickable / draggable elements. I have not implemented anything like this, but it's the logical next step.

[nicopap]

You still need the primitives like "dashed line", "text", "filled shape" etc. in order to compose them and build more complex shapes/gizmos. Can't really express them in term of just lines. So #9400 is still pertinent.

I'm not sure I follow. I don't understand the bit about modularity. In rust, you'd call gizmo.draw_circle() followed by gizmo.draw_line() to get ­— say — a crossed circle. You could then define a function that takes the Gizmo struct to "compose" gizmos. Now you have a function to draw crossed circles. So composability is already taken care of.

Concerning reactivity/UI system. You aren't the only one asking for it. I've added it now to the #9400 list. I suspect the end result would look more like an immediate mode UI though.

[nicopap]

Note I totally see where you are going with the solidjs-inspired bits. I strongly suspect that dynamic queries would unlock the ability to do that kind of things.

It's not full graph reactivity, because it only goes 1 way, but an ECS + an immediate mode UI is fairly close. You'd build the UI on top of a query and the UI state would be fully coupled to the game state.

[viridia]

Immediate mode is fine. Don't pay too much attention to the syntax, the overall structure is what I'm interested in. I'm only using JSX here because that's convenient in TypeScript, in Rust you'd use something else.

Glad to hear that composability is taken care of.

I should mention something about the Solid "For" construct - it's not just a for loop, it also memoizes. Basically if you have a list of 20 items, and the 10th array item gets changed, only the 10th iteration loop body gets re-run. However, that's just a performance optimization, and may not be necessary for this use case.

Minor point: most of my gizmos use "thick" strokes, that is, strokes made out of triangles, not line primitives. In some cases, I use a hybrid of both - that is translucent filled polygons with line stroked edges. Here's an example showing the navigation mesh gizmo:

I won't be insulted if you want to close this issue. I mainly just wanted to get those ideas out there.

[viridia]

I've also thought quite a bit about reactivity, and have come to the conclusion that I need a lot more Rust experience before I'm ready to tackle that issue.

Since Rust doesn't have getters / setters, you can't do what Solid does; but you can have closures. But using closures for every property (color, opacity, line thickness, etc.) produces a lot of clutter and a therefore a bad developer experience. Immediate mode gets around this limitation by throwing away fine-grained updates - you re-build the entire tree every time.

It might be possible to build an API such that you can seamlessly mix closures with constants, using the same kind of impl magic that Bevy uses when you add systems. The result of this would a tree that gets stored in a Local and re-evaluated (but not rebuilt) every update cycle. But I'm a long way from being able to design something like that.

[MiniaczQ]

I think the major difference between this and #9400 is persistency, the current gizmos are immediate and will cause complications if not entirely block interaction.

EDIT after some more reading:

• There is a persistent gizmo PR Retained gizmos #9178