Metal 3.0 capture to file support

[vladimir-nazarenko]

Struggling to try the fantastic feature of setting MVK_CONFIG_AUTO_GPU_CAPTURE_OUTPUT_FILE alongside with MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE set to 2 to gather profiling info for later analysis with XCode tools. More or less successfully was able to use MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE under XCode debugger, but am consistently getting "Capturing is not supported" when enabling output to a file. I reckon it's due to all of my devices (Radeon 560, Intel 630 and external Vega on Mac Pro 2017) are logged as supporting metal 2.2 both on Catalina 10.15.7 and Big Sur 11.2.2 and not metal 3.0. Obviously that's not the problem of MoltenVK, but could you please point out to the hardware/software configurations you use to get Metal 3.0 support, as I was unable to find any specs on that in the apple developer manuals. Thanks!

[cdavis5e]

To use file capture, you also have to set METAL_CAPTURE_ENABLED=1 in the environment, or set a key MetalCaptureEnabled to true in your Info.plist. This should really be called out.

[vladimir-nazarenko]

Wow, I missed that point, thanks! Now I see that it was documented by Apple at https://developer.apple.com/documentation/metal/frame_capture_debugging_tools/enabling_frame_capture .
I observe the changes in mvk logs, but now I get another error message:

[mvk-info] Capturing GPU trace to Xcode.
[mvk-error] VK_ERROR_INITIALIZATION_FAILED: Failed to automatically start GPU capture session (Error code 3): Capture Destination ‘Developer Tools’ is not supported.

The application is launched using the commands:

METAL_CAPTURE_ENABLED=1 MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE=2 MVK_CONFIG_AUTO_CAPTURE_OUTPUT_FILE=/Users/user/trace.gputrace ./Contents/MacOS/App

Worth pointing out that the application was built using cmake+gcc & xcode-tools used for signing and packaging. MoltenVK version was 1.1.3, loaded to the app using volk loader directly using dlopen call.

[vladimir-nazarenko]

Well, I misspelled the env variable name, it's actually MVK_CONFIG_AUTO_GPU_CAPTURE_OUTPUT_FILE. After fixing that and running the app I actually got a 8Gb file with gputrace, but Xcode 12.4 fails to open it without any specific error.

[billhollings]

With MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE=2, you will capture the initial frame in your app.

8Gb file

That sounds very large for an initial frame capture.

Are you sure you have MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE=2?

If so, it's probably because, for some reason, your frame presentation is not stopping the capture activity.

What queue index and queue family index is your frame presentation performed on? If it's not 0:0, then you need to also set the MVK_CONFIG_DEFAULT_GPU_CAPTURE_SCOPE_QUEUE_FAMILY_INDEX and MVK_CONFIG_DEFAULT_GPU_CAPTURE_SCOPE_QUEUE_INDEX env vars as well. This is not well documented in vk_mvk_moltenvk.h. I'll make sure we clear that up.

Also, it's possible that MoltenVK might have a problem auto-capturing a frame presentation that is not on queue 0:0. If that is the case, we'll need to patch MoltenVK to fix that.

Added #1296 to address this.

[vladimir-nazarenko]

Well, I guess I have somewhat untypical setup: I'm rendering quite large models (hundreds of thousand triangles at least), using quite large textures and framebuffers (usually tens megapixels), so 8Gb does not really surprise me.

I guess the problem might be that I'm not really presenting to any queue, as I only render offscreen and copy the resulting buffers to host for further postprocessing and visualization.

I'm pretty sure that I consistently set MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE=2 and I guess that's exactly what I want, as I wish to see virtually all Vulkan/Metal calls to identify performance bottlenecks.

I've checked the queue indices and made sure that both family and index within the family are zero and only one queue is used for all vulkan command submissions.

Also using MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE=2 I managed to build a part of the app I'm profiling using XCode and run the metal frame debugger for it, also without any presentations to a queue: solely headless Vulkan rendering, but when I try to do so attaching to the app with an XCode debugger (before Vulkan initialization), MoltenVK fails to start a capture session with Capture Destination ‘Developer Tools’ is not supported error.

[billhollings]

I guess the problem might be that I'm not really presenting to any queue

This will definitely be a problem.

GPU capture requires the definition of the scope to start and stop the capture. As documented, with MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE=2, the intention is to capture the initial frame rendered by the app. The start event is the creation of the queue, and the stop event is the first presentation of a rendered frame.

The alternate MVK_CONFIG_AUTO_GPU_CAPTURE_SCOPE=1 exists for the lifecycle of a VkDevice object. It starts on the creation of the VkDevice, and stops on the destruction of that VkDevice. It is useful for headless rendering or compute activities like automated testing, where the app creates a VkDevice, uses it to perform work, and then destroys it.

Within MoltenVK, there is currently no GPU capture scope defined for what you are trying to do, and we'd have to add that. If you want to request that as a feature, submit a feature request to the MoltenVK issue list. If you do, be sure to be clear, in the description, what you want the scope of the automated capture to be.