Truly realistic audio is something that is very hard to come by, even in this day and age. Despite surround sound and high fidelity, there’s still one thing that breaks the immersion. The audio still sounds like it’s coming from your headphone or speakers, not the position of the UI element or character in a film.
Fortunately, Microsoft thinks it has a solution to this problem. HoloLens development resulted in ‘Spatial Audio,’ a set of technologies that program manager of 343 industries Clint Rutkas labels “ground breaking.”
How does Spatial Audio Work?
According to Rutkas, Spatial Audio maps how sound reaches the ear, as well as how head shape, ear contours, the inner and brain make sense of it. All of this data helps to determine the 3D source of the sound.
Audio is then placed individually for each ear depending on physical orientation. Rutkas likens it to ventriloquism, where the artist gives the impression that the voice is coming from a puppet via both audio and visuals. The technique creates what is essentially binaural audio – a concept you may be familiar with if you’ve ever watched ASMR or that hairdresser video on YouTube.
Applications in AR and Voice Chat
Of course, the most natural fit for this is Augmented Reality. Projects such as Fragments place characters in real space, and the new technologies will make it sound like it dialogue is actually coming from their mouth.
However, it could also have an impact outside of gaming. Microsoft is also testing application in voice chat. Voices come from separate physical locations, making it easier to tell who is saying what.
It doesn’t take a genius to realize how this could be revolutionary for programs like Skype and Xbox Live chat. Businesses could have large, multi-user conversations without having to worry about talking over each other.
Rutkas also mentions use in movies, with accurate positioning of different sound sources and no need for surround sound. For music, Spatial Audio can emulate the feeling of a concert with the feeling large speakers in front of you.
If the technology takes off it could be an exciting time for the audio world. It’s applications stretch across all kinds of media, and experiences like this may become commonplace five or ten years down the line.
You can start building Spatial Audio into your apps via Microsoft’s GitHub samples. Rutkas has also provided a short tutorial on the Microsoft blog.
