Isolating Background Music

[DeepanshCUHK]

Does anyone know if this can be used to isolate the background instrumental music (instead of vocals)? I tried multiplying attenuation by -1 but that doesn't work.

[spotlightishere]

The default functionality of AUSoundIsolation will isolate background noise using a neural network shipping with macOS (specifically aufx-vois-appl-nnet-vi-v0.plist within AudioDSP.component)! If you comment out the audio unit properties setting the paths to the custom music-oriented model (plist/base/dereverb), this should be used by default. You may want to tinker some with the denoising/tuning properties, or remove them entirely.

(However, in my experience, it's not good at fully removing loud background music. This same audio unit (or something very similar?) appears to be used for the "Voice Isolation" microphone setting, and that itself isn't always the best :)

[DeepanshCUHK]

Got it, Thanks for your reply! The quality might not be the best, but I love that it works with 0 latency. I have tried other vocal-remover solutions like librosa (Python library) and multiple deep learning solutions, where a 3 min. song can take upto 30 seconds to process depending on the CPU/GPU. (But maybe there are other better solutions I don't know of?)

So, apparently while testing it is crashing on my iPhone 11 because it cannot find the file "aufx-nnet-appl.plist" but working on my iPhone 12 Pro (Both iPhones on iOS 17+). I don't know why that is happening because Apple says that voice isolation should work on Xr and later devices (https://support.apple.com/en-in/101993#:~:text=You%20can%20use%20Voice%20Isolation,iPad%20Pro%2011%2Dinch%20models). Maybe they are using a different audio unit for this.

[spotlightishere]

Glad to hear this tool has helped :) unfortunately not familiar with other solutions, but wishing you best of luck with the search!

Both your iPhone 11 and 12 Pro should work! I've pushed some commits recently regarding model discovery under iOS 17 as the model location changed (especially 72a8734 and 09839cc) - with these pulled, are you able to run as expected?
If not, could you please send the logging of what paths it attempted to search for the model?

[DeepanshCUHK]

Great! 72a8734 fixed it.

So while testing it more, it is working for most files but I encountered an error where the plist file threw an exception for a file in .m4a format (karaoke recording with background music + vocals). Was wondering if you have any idea why this could happen? Maybe because the recording quality is bad, and background or vocals are perceived as noise?

Assertion failed: ((numInternalIOChannels == 1 || (numInternalIOChannels == numIOChannels && internalBatchSize == 1)) && "internal format must be one channel or the same number of IO channels (when internal batch size = 1)"), function CreateProcessingGraph, file SoundIsolationGraphAdapter.cpp, line 335

[spotlightishere]

Apologies for taking a little while to get back! I'm able to reproduce this with any audio that has a single channel, but unfortunately am not familiar enough to determine why.

[DeepanshCUHK]

Ok np. Thanks for all your help :) I am really enjoying this project!!

[spotlightishere]

@eric-gitta-moore

What exactly does the file and AudioDSP in the following sentence refer to? I've been doing some work on that lately.

Absolutely! From my understanding (and please note I very well could be wrong), AudioDSP is a Carbon component providing the functionality behind Audio Units. AUSoundIsolation seems to leverage another private audio unit (known as AUNeuralNet) for its functionality.

That plist configures the neural network used for sound isolation, and under macOS, can be found under /System/Library/Components/AudioDSP.component/Contents/Resources/Tunings/Generic/AU/SoundIsolation/aufx-vois-appl-nnet-vi-voice.plist.

(Note that this is its intended functionality, so you shouldn't need to configure anything else beyond the wet/dry mix parameter to leverage the audio unit for voice/sound isolation! QuietNow only has to specify their custom, secondary neural network for music isolation. You can take this project and strip out all of the neural net-related parameters to achieve its default usage.)