Improving Spatial Interaction with Mobile Manipulation Robots Through Mixed Reality

Research output: Contribution to book/Conference proceedings/Anthology/Report › Conference contribution › Contributed › peer-review

Abstract

Collaborative robots require intuitive interaction mechanisms that enable users to issue spatial commands and understand the robot’s behavior. However, traditional desktop interfaces rely on indirect two-dimensional representations of the robot’s three-dimensional workspace, requiring users to mentally translate between screen space and physical environment. This poses a particular problem for users without technical expertise. Mixed reality (MR) interfaces have the potential to address this limitation by enabling spatially grounded interaction through in-situ visualization and multimodal input. We present a multimodal MR interface that allows users to interact with a mobile manipulation robot using speech, gestures, and spatial visualizations of the robot’s internal state and intentions. We evaluate the interface in a within-subjects user study (N = 21) comparing MR interaction to two desktop-based interfaces with and without voice input. Results show that MR significantly reduces spatial command formulation time compared to both desktop interfaces. Perceived safety, control, usability, and workload ratings showed no significant omnibus differences, although equivalence for SUS and overall NASA-TLX was established only relative to the voice-enabled desktop baseline. These findings suggest that MR interfaces can improve the efficiency of spatial human-robot interaction, while broader claims about equivalent perceived experience require further study.

Details

Original languageEnglish
Title of host publicationVirtual Reality and Mixed Reality
EditorsMario Lorenz, Gabriel Zachmann, Rachel Farrell, Voicu Popescu, Maki Sugimoto, Patrick Bourdot
Pages284–308
Edition1
ISBN (electronic)978-3-032-42847-9
Publication statusPublished - 1 Nov 2026
Peer-reviewedYes

Publication series

SeriesLecture Notes in Computer Science
Volume17151
ISSN0302-9743

Conference

Title23rd EuroXR International Conference
Abbreviated titleEuroXR 2026
Conference number23
Duration3 - 6 November 2026
Website
Degree of recognitionInternational event
LocationUniversitat Politècnica de València
CityValencia
CountrySpain

Keywords

Keywords

  • Mixed Reality, Human-Robot Interaction, Multimodal Interaction