UI elements on top of react-native-skia, à la Flutter

[alexandergunnarson]

I was wondering about any plans or thoughts around implementing a React-based UI framework on top of react-native-skia, sort of like Flutter for React. It seems like it could be a revolutionary approach to creating cross-platform user experiences that are both pixel-perfect and performant.

Prior art

• Google Docs and VS Code's terminal were rewritten to use Canvas instead of DOM elements, for performance reasons.
• Flutter seems to take the approach of rendering UI elements to Skia. For accessibility, they ostensibly do what Google Docs seems to do: overlay unstyled, transparent DOM elements that they keep in sync with the Canvas element.
• react-canvas was developed by Flipboard to run 60fps dynamic list views in mobile browsers, something not possible at the time (and perhaps even now) with a pure DOM approach.
• makepad.dev, in all its text editor glory, apparently runs on WASM+WebGL in the browser.

Pros

• Performance-tuning DOM-based web apps is hard, and often brittle in the face of changes. There's the added challenge of browser vendors having implementations with sometimes wildly different performance characteristics. A Canvas-based approach sidesteps these problems.
• Ensuring that apps are pixel-identical on different platforms (regardless of whether we take this to be a worthy cause) is challenging. It's the case for native platforms vs. web, even with a React Native + React Native Web approach. It's the case among native platforms (iOS vs. Android shadows anyone?), and it's presumably the case even within the single platform of the web, since different browsers have subtle behavioral and rendering differences (even though RNW tries to paper over this). A Canvas-based approach sidesteps the differences in rendering, though admittedly not necessarily in behavior.
• Engineers don't have to use Dart, but can continue to use whatever JS-based language they've been using. In fact, any language that compiles to WASM could now be fair game.
• The same ubiquitous design language of React Native can continue to be used on any platform, but with the enhancements listed above.

Cons

• We're doing much (most? all?) of the implementation work of Flutter, which is considerable. But if we make it work with any Canvas-like GPU-accelerated backend (WebGPU eventually?), and it consistently renders at >= 60fps out of the box, and people can continue to use JS-based languages and the React Native design language, it might be worth it.
• Accessibility would have to be implemented somehow. Possibly this would be via a transparent parallel DOM that includes only elements that need to be made accessible for screen readers and the like. It would have to be kept in sync with the Canvas component, which seems tricky, but it's been done before.
• On the web, this could end up turning into, as is the case with Flutter's CanvasKit backend, a multi-MB blob that users have to wait to download. This is not great for many reasons. But perhaps it could be mitigated with an "iterative upscaling" approach: first a static site is downloaded, then a DOM-based react-native-web version, then the optimized Canvas-based version. When a next version is rendered, it can be done more or less seamlessly, with view state like scroll position or cursor state being carried over.

[wcandillon]

That's a fun discussion thread :) @chrfalch you had built a list demo; would you like to share it?

[FrancoRATOVOSON]

I spend a night thinking about all of that, thought I was the only person crazy enough to have such ideas 😅
I'd like to give it a try. Just struggling on editable text (and keyboard event) on mobile devices with rn-skia. The TextInput rn component is the only one who handles keyboad, and you can't control it programmatically.

[evelant]

I like the idea but as you pointed out it would be a ton of work. Performance (especially on Android) is really one of the biggest weak spots in react-native IMO. It's very easy to run into perf problems and there are almost no usable tools to help diagnose and solve them unlike on the web. Personally I'm hoping that bevy can become a great solution once its UI library is completed (project creator said he wanted bevy to be usable as a 2d app framework as well as a game engine). Until then it's back to hacking around android performance problems 😄