retargetting problem with laptop's built-in camera

[dbdxnuliba]

Very nice work！
And when I used the show_realtime_retargeting.py example with the laptop's built-in camera, the retargeting performed normally when my hand was facing the camera. However, as I gradually rotated my right hand, the retargeting showed a significant deviation. How can we solve this problem? Would using an RGBD camera, like Realsense, resolve it? If so, could you please provide an example code? Thank you very much.

Here is the testingmethod：
python3 show_realtime_retargeting.py --robot-name allegro --retargeting-type dexpilot --hand-type right

[yzqin]

Hi @dbdxnuliba

That is an unexpected issue. To further debug this problem, could you help to provide the original RGB video (without mediapipe markers) or image sequences so that I can take a deeper look at the problem you mentioned above?

[dbdxnuliba]

https://github.com/user-attachments/assets/3e77cb76-add3-4ca7-8d7f-11a597e9c29b
thanks a lot for your response, and this is the original RGB video record from laptop's built-in camera
@yzqin

[dbdxnuliba]

test_result.zip

and this is the test result by run dex-retargeting-0.4.0
python3 show_realtime_retargeting.py --robot-name ability --retargeting-type dexpilot --hand-type right
@yzqin ,Please inform me if there are any new developments

[yzqin]

Hi @dbdxnuliba

Sorry I find that I can not open/download your original RGB video record. Can you send me again.

[dbdxnuliba]

Hi, @yzqin
test.zip

this is the original RGB video record

Hi @dbdxnuliba

Sorry I find that I can not open/download your original RGB video record. Can you send me again.

[yzqin]

Dear @dbdxnuliba,

Thank you for sharing this example. I've successfully reproduced the issue on my system. The core problem stems from the poor quality of 3D fingertip position detection in the hand pose estimation.

In the video, even when the thumb and index fingertips are visibly touching, the detected distance between them remains quite large, approximately 0.08 units.

I've made updates to the dex-urdf repository to enhance the fingertip definition for the Allegro hand. While this offers some improvement, the issue persists.

You might want to experiment with the retargeting configuration I've provided below. It includes an increased margin for the finger touching prior. This adjustment helps to some extent, but the results are still unsatisfactory due to the significant detection errors from MediaPipe.

retargeting:
  type: DexPilot
  urdf_path: allegro_hand/allegro_hand_right.urdf

  # Target refers to the retargeting target, which is the robot hand
  wrist_link_name: "wrist"
  finger_tip_link_names: [ "link_15.0_tip", "link_3.0_tip", "link_7.0_tip", "link_11.0_tip" ]
  scaling_factor: 1.6

  # DexPilot parameters
  project_dist: 0.05
  escape_dist: 0.07

  # A smaller alpha means stronger filtering, i.e. more smooth but also larger latency
  low_pass_alpha: 0.2

I will continue investigate this issue. Maybe replace the mediapipe with a better hand pose detector in the example.

[dbdxnuliba]

Thank you for your reply. Looking forward to the progress.

[LEONliaowin]

Hello！Glad to see this great work, i also want to run the real time simulation on my laptop, however, my PC has an AMD cpu and doesn't have a NVIDIA GPU

My Question is : is my computer able to run the real time sim using the build-in camera?

THX