Why obs can capture at any FPS?

[ZhengYuan-Public]

I'm recently exploring machine learning (computer vision and reinfocement learning) and capturing desktop/application is involved in many applications. Since I'm only proficient with python, so I've watched tutorials using several different methods to achieve that, such as pyautogui, win32gui, mss, and d3dshot.

The first three libs work by "taking screenshots" as fast as possible, but on my machine I can only get up to 40 FPS at 720p. Then I noticed d3dshot which uese DXGI Desktop Duplication. But according to the benchmark in its readme file, it still can't perform over 60 FPS.

My questions

• It seems obs can capture desktop/application at any FPS (Integer FPS) and I wonder how it is achieved. In the readme file of d3dshot, it explains the why FPS might not be a good metric, but I still don't know how to think about it.

  Measuring the exact performance of the Windows Desktop Duplication API proves to be a little complicated because it will only return new texture data if the contents of the screen has changed. This is optimal for performance but it makes it difficult to express in terms of frames per second, the measurement people tend to expect for benchmarks.

• I tried obs' Virtual Camera as a video stream in opencv and it out peforms those libs I mentioned earlier. I wonder how can I automate the setup of virtual camera on remote client computers (VMs)?

Advice wanted

I'm interested in understanding more about videos and streaming, can someone recommend a book/open course or other study materials?

Thanks ❤️

[flaeri]

OBS also uses DXGI Desktop Duplication for one of its "Display Capture" methods. It will only be able to generate frames at the speed the windows compositor is going, which is usually at most the speed of the monitors refresh rate. I've made an app (c++) that also uses DXGI, and it can easily go above 60fps, assuming you have a high refreshrate monitor connected and configured as such, and the content is updating at high enough speeds.

OBS also has another method called Game Capture, which is to "inject itself" into a graphics process (like games), where it can grab frames directly from the process, instead of going via windows/DWM. This is extremely fast and efficient, and cannot easily be replicated, as it relies on trust of game developers (signed certificates) etc.

As for your second question, I wouldn't personally go with trying to automate that route, and instead keep with the DXGI Desktop dupe, and find better methods of employing the buffers/textures it provides. If you need more time resolution, then you'd need to get a monitor and GPU that can do higher than 60fps. This all hinges on the python libraries being able to go at these speeds (at the resolution you're working in), but you'd run into that limitation regardless.

OpenCV should be able to go above 60fps as far as I know, at least for lower resolutions, so there is no hard limitation there, except for the computations that you need to do. As the frame rate increases, the time constraints your working in decreases, and all work would need to be done in under 8.33 ms (for 120fps), which can be quite a challenge.

I don't really have any resource that goes into all of those things, as each subject might be hundreds of hours of research, depending on your starting point. Make sure you read and understand the Microsoft docs regarding desktop dupe, and the texture formats, and how to efficiently transfer between them. Reading up on hardware accelerated scaling might also be a good idea, if the python libs don't handle this quickly enough. Video encoding and streaming are fairly seperate, and very time consuming challenges as well. Depends on the level you are needing/trying to get to.

Feel free to join the discord server if you're looking for people to have a bit more of a quicker back and forth with, for instance in tech-talk or encoding-talk.

[ZhengYuan-Public]

Thanks for the tip! Very helpful.