Training surgical robots is difficult because the work depends on precise movement, reliable perception, and careful interaction with objects that can be rigid or deformable. ORBIT-Surgical addresses that problem with a simulation environment built for research in learning-based robot-assisted surgery.
The open-source framework was developed by scientists from the University of Toronto, UC Berkeley, ETH Zurich, Georgia Tech, and Nvidia. Its core purpose is to make research in machine learning for robot-assisted surgery easier to run, compare, and accelerate.
A simulation framework for surgical robotics research
ORBIT-Surgical is designed around simulated surgical robots rather than general-purpose robot scenes. It includes detailed models of two platforms: the da Vinci Research Kit (dVRK) and the Smart Tissue Autonomous Robot (STAR).
That focus matters because surgical robotics research often needs environments where basic skills can be repeated many times under controlled conditions. Instead of beginning with a physical system for every experiment, researchers can train and evaluate algorithms in a virtual setting first.
The framework uses Nvidia's robotics simulation platform Isaac Sim for GPU-accelerated physics. It also uses Omniverse with ray tracing rendering, giving researchers a way to simulate both physical behavior and visual data inside the same environment.
Fourteen benchmark tasks cover basic surgical skills
ORBIT-Surgical provides 14 benchmark tasks that represent foundational surgical abilities. These include simple movements as well as interactions with rigid and deformable objects.
The source examples include needles and tubes. Those objects are important because they require more than simple positioning: an algorithm must learn how actions affect objects in the simulated scene.
The benchmark setup gives researchers a common base for testing progress. A shared set of tasks can make it easier to compare different approaches to reinforcement learning, imitation learning, and perception inside surgical robot simulation.
- Robot platforms: da Vinci Research Kit (dVRK) and Smart Tissue Autonomous Robot (STAR).
- Task coverage: 14 benchmark tasks representing basic surgical skills.
- Object interaction: rigid and deformable objects, including needles and tubes.
- Simulation stack: Isaac Sim for GPU-accelerated physics and Omniverse with ray tracing rendering.
GPU parallelization changes the training pace
A major feature of ORBIT-Surgical is scale. Through GPU parallelization, up to 8,000 simulations can run at the same time on a single graphics card.
That parallelism is central to reinforcement learning. These algorithms typically need many attempts before useful behavior emerges, so the ability to run thousands of simulations at once can shorten the research cycle.
The team says this enables efficient reinforcement learning training within a few hours. By contrast, classical CPU-based simulators require days or weeks for the same type of work, according to the source article.
ORBIT-Surgical also supports human input. Researchers can control simulated robots in real-time using devices such as VR controllers and the control unit of the dVRK system. Those recorded movements can then be used to train algorithms for imitation learning.
Synthetic images add another training path
The platform is not limited to motion and physics. Because it uses photorealistic rendering, ORBIT-Surgical can generate synthetic images for training visual models.
In one experiment described by the researchers, combining synthetic data with real images more than doubled the performance of a model for segmenting surgical needles. That result points to a practical role for simulation beyond robot control: it can also help produce visual training data.
This is especially relevant when the task is perception. A robot-assisted surgery system needs algorithms that can identify important elements in a scene, and simulated images can expand the data available for training those algorithms.
The use of ray tracing rendering through Omniverse supports this part of the workflow. The goal is not only to move simulated robot arms, but also to create visual scenes that can contribute to machine learning development.
Real robot transfer is promising but incomplete
The researchers also tested whether work done in simulation could transfer to a real dVRK robot. They demonstrated transfer for motion sequences and trained reinforcement learning models.
The results are not presented as finished. The reported success rate was only 50 percent, leaving room for improvement. The team attributes part of the gap to stretching effects that are not yet represented.
That limitation is important. A simulation framework can speed research, but the final challenge is whether behavior learned in simulation holds up on real hardware. ORBIT-Surgical shows progress toward that goal while also making clear that the simulation still needs more physical detail.
Future versions are expected to support simulation of incisions in soft tissue. The researchers also plan algorithms for more complex tasks such as suturing.
The larger aim is to advance learning-based robot-assisted surgery. According to the researchers, the goal is to develop systems that support and relieve surgeons during challenging procedures.
ORBIT-Surgical is now available on GitHub, giving researchers access to a framework built specifically around surgical robot simulation, benchmark tasks, synthetic data generation, and transfer experiments with a real dVRK robot.