Blog

We at Hugin Studios aim to ceate informative posts that promote the understanding of OpenUSD.

#1 An implementation of the Spatial Audio Schema

2025-04-04

OpenUSD has the ability to encode a multitude of 3D-graphics related objects such as light sources, cameras and geometry. Besides the more common object types of 3D graphics, USD can also describe a source of audio, that is an object in 3D space which emits sound.

The UsdImagingSpatialAudio schema was added in 2019 after the initial proposal which states all the requirements of the schema and the thought behind it. In short, a UsdImagingSpatialAudio prim holds a transform, a path to an audio file, and some attributes describing how the audio should play such as start/end time and if it should be spatial or not.

If you were to test the schema in usdview as it stands you’d find that it does in fact not play any audio. The proposal states that, although a reference implementation might be available in the future, it is not within the scope of the original proposal. It is therefore currently up to the USD consumers to follow the UsdAudio proposal and implement support for it in their respective application. From what we've gathered, there’s only one example of implementation at this time which is done by Nvidia and is available in Nvidia Omniverse. For this reason, we decided to make a prototype implementation of it as well and hopefully spark some interest in what is possible to do with OpenUSD and hydra.

Implementation

We used the open source software OpenAL-soft for handling the spatial audio playback, and libsndfile for supporting a larger range of audio formats. The rest is written as a scene index plugin which handles the bridge between hydra and OpenAL. We intended to stay as true as possible to the original proposal, however we deviated on one point in particular:

“..., we defer any consideration of triggers or interactive environments for now in the interests of simplicity and resources .”

Here, we instead decided to also include a TfNotice plugin that our scene index will listen to in order to trigger the audio playback. This enables us to start the audio playback from anywhere within the app; such as clicking the play button in usdview, or selecting a certain geometry in the viewport(playing drums in usdview, anyone?). While this does enable some nice, but arguably not critical, interaction with the sound engine, it also leaves the problem of out-of-sync sound when not rendering in real time. We currently ignore this for the purpose of this post, but plan to solve it by implementing proper time-stretch functionality in the future. For crucial in-sync audio cases, it is possible to solve this issue currently but with audible artifacts or pitch-shifting.

Using a scene index plugin, or any other hydra compliant plugin, allows us to run the audio engine from within the hydra scene, thereby gaining access to all the relevant scene events which we use to control the sound and playback of the audio sources. It also means that any hydra2.0 compliant renderer could use this plugin. In conjunction with the scene index plugin, we implement a hydra prim adapter for UsdMediaSpatialAudio prims. This enables the hydra scene composition to include all UsdMediaSpatialAudio prims, thereby automatically initializing the audio engine whenever such prim is added.

What’s left?

There’s a couple of things we’ve left out for the purpose of this prototype, such as playing audio in reverse. And some things we consider for the future; such as cone angle control, graphical representation of the prim, time-stretching, and perhaps include some of OpenAL’s audio effects. However, adding some of these features would require us to deviate from the original proposal more than we’d like to at this time.

Assets used in the example video:

Why There Is A Cat Curfue In Our House by Anselm Hollo link

Atmo of natural sounds link

Downloaded on March 11th, 2025

S: 20110503_fireplace.00.wav by dobroide -- https://freesound.org/s/120644/ -- License: Attribution 4.0

AI voice of quote from The Pink Panther Strikes Again (1976)

Mixamo model & animation