A Mixed Reality User-Friendly Interface for Robot Teleoperation

In this paper, we present a user-friendly mixed reality (MR) interface designed for robot teleoperation. To enable untrained users to effectively operate a robot, we developed an interactive application leveraging SwiftUI, RealityKit, and ARKit, and deployed it on an extended reality (XR) headset. Our application enables users to control a robot by moving a virtual sphere that serves as a dynamic target. Using RealityKit’s support for natural hand and eye gestures, users guide the robot through easy and subtle motions, eliminating the need for controllers or exaggerated gestures. This reduces physical strain, enhances comfort, and makes teleoperation more immersive. The system has been experimentally tested and statistically validated across users with varying experience levels.