Mobility, Space and Sound Activate Expressive Musical Experience in Augmented Reality

We present a study of a freehand musical system to investigate musicians' experiences related to performance in augmented reality (AR). Head-mounted mixed reality computers present opportunities for natural gestural control in three dimensions, particularly when using hand-tracking in a creative interface. Existing musical interfaces with head-mounted displays use dedicated input devices that are not designed specifically for musical gestures and may not support appropriate interactions. We are yet to see widespread adoption of head-mounted AR musical instruments. We conducted an empirical study to evaluate musicians' (N=20) experience of performing with a freehand musical interface. The results suggest that the design of freehand musical interaction in the AR space is highly learnable and explorable, and that such systems can leverage unique aspects of mobility, space and sound to deliver an engaging and playful musical experience. The mobile musical experience with a spatial interface design allowed performers to be more bodily engaged and facilitated gestural exploration for musical creativity. This work contributes to a more developed understanding of potentials and challenges in AR-based interface design for musical creativity.