Augmented Objects as Portals into Virtual Worlds: Using Audio to Create Immersive Experiences in Extended Realities - UMBRELLA AUDIO SPATIALIZATION DEMO

<p><strong>Technical demonstration</strong></p> <p>The results of the projection mapping system in the project are clear from the <a href="https://blog.zhdk.ch/immersivearts/dreaming-of-time-and-space/">main documentation video</a>; however, the impact of the spatial audio system&nbsp;in particular, is best experienced from directly underneath the umbrellas, where one can best appreciate the various levels of mixed reality. Unfortunately, it is difficult to document these effects within the artistic context of the project, and as such, we include a brief set of examples to better demonstrate the 6 degree of freedom sound spatialization capabilities of the umbrella system.</p> <p><em><strong>NOTE:</strong></em>&nbsp;The audio in the following examples is recorded from a fixed perspective (initially underneath the umbrella) and rendered binaurally. Unfortunately, the ambisonic microphone used does not capture directionality very well when the source (in this case, the umbrella speakers) is less than ~1 meter away, and in retrospect, a single channel of pink noise was not a wise&nbsp;choice as a source material, as it appears to cause additional phasing issues. &nbsp;Additionally,&nbsp;the effectiveness of binaural audio varies from listener to listener, so <em>the perceived effect in the video is not as strong as when experienced in person</em>; nonetheless, it is possible to get the basic idea of the spatialization algorithm in action from these examples.</p> <p>PLEASE WEAR HEADPHONES IN ORDER TO EXPERIENCE THE 3D EFFECT.</p> <p>In addition to the view of the entire scene from an outside perspective, several other views of the underlying software are displayed throughout the video, including:</p> <ul> <li> <p>A radar view of the scene (umbrella and sound source) as seen by the space manager software, where the:</p> <ul> <li> <p>Blue circle = umbrella</p> </li> <li> <p>Cyan triangle, yellow square = sound source</p> </li> </ul> </li> <li> <p>A view of elements of the spatialization software running on the umbrella, specifically the:</p> <ul> <li> <p>Relative gain calculations and current output levels of each speaker</p> </li> <li> <p>Results of supporting calculations (e.g. sound location after transformation from the global to local&nbsp;coordinate system, and scaling factors used to attenuate the overall&nbsp;volume of the sound as the distance from the umbrella to the sound changes)</p> </li> </ul> </li> </ul> <p><strong>Demo #1</strong></p> <p>Stationary umbrella with a moving virtual sound source (anchored to a rigid body)</p> <p><strong>Demo #2</strong></p> <p>Rotating umbrella with a stationary sound source (anchored to a rigid body)</p> <p><strong>Demo #3</strong></p> <p>Moving umbrella with a fixed sound source (anchored to a rigid body)</p> <p><strong>Demo #4</strong></p> <p>Moving umbrella with a fixed sound source (anchored to a virtual point in space, located above the microphone); as the umbrella approaches the source, the sound first fades into the room, then collapses into the umbrella, as show in Figure 7 (&quot;Fading between umbrella and room with distance&quot;) in the main paper</p>