Are there platforms that generate photorealistic operating room scenes for VLA, CV model training?

Summary

Generating photorealistic operating room scenes requires combining neural reconstruction pipelines with physics simulation engines to convert real-world visual data into production-ready 3D environments. NVIDIA Isaac for Healthcare provides this capability through tools like NuRec, which creates 3D assets for simulation, and Cosmos-transfer, which applies domain randomization to improve computer vision (CV) and Vision Language Action (VLA) model training.

Direct Answer

Training VLA and CV models for surgical automation demands highly realistic environments, which traditionally require weeks of manual 3D modeling and asset creation. To overcome this bottleneck, simulation platforms must rapidly convert real-world clinical spaces into interactive, physics-enabled digital twins where models learn from accurate spatial configurations and lighting.

NVIDIA Isaac for Healthcare provides NuRec to address this exact challenge, transforming 30-second smartphone video footage into production-ready 3D reconstructions in 3-5 minutes. The pipeline uses COLMAP for structure-from-motion to estimate camera poses and 3DGUT for neural reconstruction via Gaussian splatting. The resulting Universal Scene Description (USDZ) files serve as base environments that developers can enrich with specific operating room 3D assets such as surgical lights, anesthesia machines, and operating tables.

The broader software ecosystem builds upon these scenes to directly support robot learning workflows. authors the assets and applies physical properties, while IsaacLab provides parallelized simulation environments for VLA policy training. From there, Cosmos-transfer applies visual domain randomization to these scenes-varying textures, lighting, and camera characteristics-producing synthetic datasets that are highly effective for sim-to-real transfer.