Questions about SceneDepth to "metric depth(distance from camera center to scene)"

[Gear-dev-sudo]

Hi, ya'all. This might be a stupid question, is the depth data rendered in the pipeline relative depth data? are there any way for me to use this as metric depth?

[jianglh-WHU]

Hi! The depth data rendered in UE is metric depth with centimeters.

[Gear-dev-sudo]

In my setup, I put a unit cube right below my camera, with the camera facing down towards the block's top surface. But in the depth image I got, the depth value of the whole top surface of the block is the same, which is not supposed to be right, (since the depth value should be the distance of thar pixel to the center of the camera in the 3d simulation), and I'm having 20-30 cm of errors across all vertices of the unit cube. My camera is right above the center of the unit cube 6 meters.

[jianglh-WHU]

Could you give me some results? Or maybe you can refer this issue #4

[Gear-dev-sudo]

My setup is this, the camera to the unit block is 300 cm, but the depth readout exr image of this is that the unit block in the middle's top surface readout

are all 300 cm, which is not correct(since the scene depth are calculated from the center of the camera to the pixel being rendered, right? If so, there should be ~3cm of difference across the whole top surface in terms of "SceneDepth" of the unit block to the camera). So, the thing is, I just wonder that is the SceneDepth in UE5 and in you guys's work are accurate enough

[yixuanli98]

Depth in UE is the distance to the camera plane, not the camera, which can transform into each other

[Gear-dev-sudo]

Depth in UE is the distance to the camera plane, not the camera, which can transform into each other

Is it the camera frustum? like the distance is calculated from the near clipping plane of the camera frustum to the pixel being rendered?

[yixuanli98]

The depth in UE is "z-depth", which refers to the distance between a point in the 3D space and the camera along the Z-axis, which is typically aligned with the camera's direction of view. This distance helps in determining which objects are in front of others and thus should be rendered on top when creating a 2D image from a 3D scene – a concept known as depth buffering.

[Gear-dev-sudo]

Ok, may I ask, is there a way to covert this into distance from camera center to the objects in the scene? I'm essentially trying to do this for a metric depth estimation DL task.

[Gear-dev-sudo]

The depth in UE is "z-depth", which refers to the distance between a point in the 3D space and the camera along the Z-axis, which is typically aligned with the camera's direction of view. This distance helps in determining which objects are in front of others and thus should be rendered on top when creating a 2D image from a 3D scene – a concept known as depth buffering.

Thank you for clarifying this.

[yixuanli98]

Ok, may I ask, is there a way to covert this into distance from camera center to the objects in the scene? I'm essentially trying to do this for a metric depth estimation DL task.

Hope this picture can help you understand. In addition, because our exr is float16, the maximum range is 65504 centimeters. So when using depth, please note that all parts equal to 65504 are invalid. Later, we will update the script loading depth and normal to determine the invalid parts and a wider range of depth maps.

[Gear-dev-sudo]

Thank you for that! This explains a lot, however, is there a way to really get the "distance from camera"?

[Gear-dev-sudo]

By using this diagram, I coded a piece of separate code to convert camera internal matrix into the theta angle per pixel (in relation to the optical center vector), by utilizing the cosine value and z depth, I got the "real distance"

Ok, may I ask, is there a way to covert this into distance from camera center to the objects in the scene? I'm essentially trying to do this for a metric depth estimation DL task.

Hope this picture can help you understand. In addition, because our exr is float16, the maximum range is 65504 centimeters. So when using depth, please note that all parts equal to 65504 are invalid. Later, we will update the script loading depth and normal to determine the invalid parts and a wider range of depth maps.