Render/View Vulkan texture (VkImageView) in widget and render UI to offscreen texture

[keithlostracco]

I'm interested in using Slint for a GUI in an application that will have two display modes. An "editing" mode and a "perform" mode. The former would need to display a texture located on the GPU (Vulkan) in a widget and the latter would be a full-screen render which would need to have the option to display some basic UI elements on top of the primary render (which is being presented the screen by the host not Slint/Winit).

Is this currently possible with Slint? On Windows the renderer is Vulkan and on Mac it will be Metal via Molten.

I did notice there are some other related discussions but I can't really see other any firm capabilities implemented other than Skia/Vulkan is now supported (just not updated in the docs).

[tronical]

There's no public API for this yet and the Skia Vulkan renderer requires additional work, as it only renders into a swap chain at the moment. But in principle I think this is something that we want to be able to support.

In your app, how do you drive the event loop?

[keithlostracco]

Hi thanks for the reply. Great news that this functionality is something you would like to support in the future.

Currently my event loop is similar to a game event loop. I use my own backend for polling incoming events (PeekMessageW on Windows) where I check for and handle events "often" then throttle my rendering to the desired framerate (if it all). I did find your example of custom event handling which is great, since I probably couldn't use Winit (I haven't looked at too deeply into though).

Regarding the interface for the Skia Vulkan renderer I have some ideas that could possibly be of benefit. If it is possible to provide a VkDevice to Skia then maybe the following could work?:

• provide interfaces for Texture, RenderBuffer, Semaphore
• pass a Texture and Semaphore to display a texture on Slint UI element (Skia renderer waits on the Semaphore)
• provide a RenderBuffer and a Semaphore to the Slint Renderer for a render target (Skia Renderer signals the Semaphore)

If Skia can't use a user specified VkDevice then I imagine the only option would be to use import and export external memory/semaphore Vulkan functionality. A RenderBuffer and a signal Semphore could be imported by the application and a Texture/wait Semaphore could be exported.

I did look into doing this myself and doing a PR but it seems all the Vulkan code is implemented in Rust and is generated for C++? Unfortunately I don't know Rust very well...

[tronical]

Ah, I'm afraid this is rather difficult to do in C++. But I also realise that I may have misunderstood your question - I thought you were looking for something the other way around (render Slint into a texture). But if you'd like to render an existing texture within a Slint UI, then perhaps this is easier.

1. We need indeed either way to construct the Skia renderer from an existing VkDevice. We do have this for a VkPhysicalDevice, queue family index, and VkSurface. Would that suffice if exposed somehow?
2. Similar to the way that we permit importing OpenGL textures, we could have a way to import a VkImage/VkImageLayout (for use in the BeforeRendering phase in a rendering notifier callback)

Another angle would be to say that we extend the existing rendering notifier callbacks with Vulkan support by providing access to the VkDevice, etc. in the RenderingSetup phase. That implies using the winit backend though.

(Sorry, this might take a few iterations before I fully understand :-)

[keithlostracco]

(Sorry, this might take a few iterations before I fully understand :-)

Happy to help! Slint is the nicest GUI library I've come across yet, I would really like to use it (and license it for commercial use).

But I also realise that I may have misunderstood your question - I thought you were looking for something the other way around (render Slint into a texture). But if you'd like to render an existing texture within a Slint UI, then perhaps this is easier.

I actually I was describing both directions... Render Slint into a host allocated texture and render a host texture in a Slint UI within a element. The first would be so you could render a GUI on top of a 3D scene the second would be so you could have an editor with a bunch of controls and have a "Viewer" of a 3D scene.

I would say rendering Slint into a texture would be higher priority in my case, I could create some workarounds to position a host render under a transparent Slint element.

We need indeed either way to construct the Skia renderer from an existing VkDevice. We do have this for a VkPhysicalDevice, queue family index, and VkSurface. Would that suffice if exposed somehow?

That would work with the import/export memory scheme. I don't think you would even need a VkPhysicalDevice or VkQueueFamilyIndex at that point, just handles to the memory and sync primitives (or provide the handles for the host to transfer a texture from Slint). I really like how Derivative has implemented texture import/export in their TouchEngine SDK. It's a C API but it's very clean and easy to use, might be worth taking a look at for reference. They don't have a Vulkan example but it's all documented. I can provide some example Vulkan code that uses it as well.

Another angle would be to say that we extend the existing rendering notifier callbacks with Vulkan support by providing access to the VkDevice, etc. in the RenderingSetup phase. That implies using the winit backend though.

Sounds like that also works (I have to look at the docs again to understand more clearly). One thing that I noticed though with the rendering notifier callback technique is that we don't need to block the CPU with Vulkan like we would with OpenGL or Direct3D11. By using VkSemaphores (in both directions - to/from SLint) both the host and the Slint renderers can run on their own timing and the GPU simply waits on the Semaphore. Ah I just realize maybe I misunderstand, I guess Slint will only render when there is a change?

I looked at the Skia documentation the other day but I must have been in the wrong spot because I couldn't really find much information regarding the Vulkan backend interface, so I'm not sure of it's limitations.