Exploring diminishing returns for multimodal dynamic event expression in mixed reality
Abstract: Expressing dynamic events in Mixed Reality (MR) requires users to communicate continuous spatiotemporal information, including entity trajectories, interactions, and temporal dynamics. This task is fundamentally constrained by the linearization bottleneck: human mental models preserve rich relational structures holistically, yet conventional output channels, such as speech, force them to be serialized into sequential streams. Multimodal interaction offers a promising solution by distributing information across complementary channels, but it remains unclear how different multimodal strategies perform as spatiotemporal complexity increases. This paper addresses these two knowledge gaps: the lack of a systematic analysis for decomposing dynamic events and the absence of empirical evidence comparing multimodal expression strategies under controlled-complexity conditions. We propose a structured analysis that decomposes dynamic events into fundamental information blocks and establishes a complexity taxonomy. Building on this analysis, we design sequential and parallel multimodal expression strategies and evaluate them alongside sequential (unimodal) speech as complete interaction strategies in a controlled user study using abstract geometric micro-worlds. Under error-free speech–gaze fusion provided through a Wizard-of-Oz setup, parallel multimodal input maintains a substantial completion-time advantage across the tested event levels. However, its subjective benefits decrease as coordination demands rise. When events include asynchronous timing or causal interactions, parallel input is no longer rated significantly higher than sequential multimodal input, though this does not imply equivalence. These findings suggest a boundary condition for the subjective benefits of the parallel strategy, providing design implications for adaptive multimodal MR systems.
For more details, see PDF
Team:
- Aochen Jiao
- Yuxuan Li
- Youpeng Zhang
- Hsin-Yi Wang
- Jiaxuan Wang
- Weitao Xu
- Eduardo Velloso
- Nuwan Janaka
- Shengdong Zhao