Skip to content

Mixed reality extension to dVRK framework, built on top of Unity

License

Notifications You must be signed in to change notification settings

jhu-dvrk/dvrk-xr

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

dVRK Mixed Reality Extension

This repository has code related to da Vinci Research Kit Mixed Reality extension.

Overview

Video Examples:

dVRK-XR in Unity Editor dVRK-XR on Windows Mixed Reality Headset
dVRK-XR with Android ARCore (3rd view) dVRK-XR with Google Cardboard (screen rec.)

dVRK-XR facilitates the integration of the da Vinci Research Kit (or da Vinci robot with research API) with extended reality platforms. To run dVRK-XR, you first need some knowledge about Unity development and the dVRK. I also wrote a blog post about the rationale behind the development of this project.

Structure of Project

  • dvrk_unity:
    • A unity project that contains prefabs, models and scripts of PSM, ECM and MTMs of da Vinci robot.
    • The virtual robots can be controlled by UDP messages that specify joint values.
    • The scene can be adapted and deployed to various mixed reality platforms.
  • sample_server:
    • A python-based sample server that sends UDP messages to the mixed reality platform.
    • Format of the JSON messages, for debugging use

Integration with dVRK

Dependencies

How to Run

  • sawIntuitiveResearchKit needs to run with sawSocketStreamer. This can be done via a component configuration file with -m option, e.g.
# Navigate to share folder
cd sawIntuitiveResearchKit/share

# Run dVRK console with a configuration file for socket streamer. 
# This example streams the MTML and MTMR joint states to remote UDP client. 
# The client IP address, UDP port number, message rate, message to send, can be specified in the \
# configuration file. Make sure the IP and port match that on the mixed reality platform. 
sawIntuitiveResearchKitQtConsoleJSON -j CONSOLE_CONFIG -m socket-streamer/manager-socket-streamer-console.json
  • Multiple component configuration files can be attached to the main program, e.g.
# This example streams both the surgeon console and patient-side arms to the remote UDP client
sawIntuitiveResearchKitQtConsoleJSON -j CONSOLE_CONFIG -m socket-streamer/manager-socket-streamer-console.json -m socket-streamer/manager-socket-streamer-patient-cart.json
  • Different configuration files can also specify different destinations (IP address), supporting multiple headsets viewing at the virtual robot at the same time.

Integration with da Vinci Robot

Dependencies

  • da Vinci robot (Classic, S or Si) with research API enabled
  • sawIntuitiveDaVinci 1.2.0 or higher
  • sawSocketStreamer
  • dvrk-xr running on a mixed reality platform
    • Unity Editor, Standalone build (Windows, Ubuntu), HoloLens, or other platforms

How to Run

Same as dVRK, but main program is sawIntuitiveDaVinciConsoleQt.

Academic Citation

@inproceedings{qian2019dvrkxr,
  title={{dVRK-XR: Mixed Reality Extension for da Vinci Research Kit}},
  author={Qian, Long and Deguet, Anton and Kazanzides, Peter},
  booktitle={Hamlyn Symposium on Medical Robotics},
  year={2019}
}

Acknowledgment

dvrk-xr was developed with support from NSF NRI 1637789.

License

JHU CISST Software License

About

Mixed reality extension to dVRK framework, built on top of Unity

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •