Comments on DESIGN_DOC.md

[ollpu]

As of 705d2f

Enumeration

• Maybe the backend could be described and selected as an enum? All variants should be available in the enum regardless if support for them is compiled in (unlike how it is in CPAL).
• Not in the design doc or API, but affects library usage: I don't think the backends should be split according to OS. In other words, I don't think the OS should be the sole discriminant of which backends are compiled in, because library users may want to configure that somehow, especially with hard-to-compile ones like ASIO. Instead I think we should dispatch directly to each specific backend if it is compiled.
• Returning all device info for all backends at once seems problematic. Some backends may take a while to "start up" and enumerate devices. Some may even hang in some cases.
  • There are also some nasty details with trying to get all of that info beforehand. I think in ALSA, there isn't really a "maximum" channel count, you just have to test a configuration and see if it works.
  • With JACK, we could provide the names of the system ports, but this doesn't limit the amount of ports that can be opened. There's also not much sense in supplying a "maximum" amount of port names – it would either be an arbitrary limit or the default 256. The idea of giving port names as strings in Config makes sense for JACK though, because the port names can also be customized, same as the client name.
  • What happens if the set of available devices changes between enumeration and startup? It might be possible to accidentally try to open a completely different device, since they are only correlated by name.
    • Finding the device with a specific name when starting up likely requires iterating over all devices in the backend again.
    • ALSA is an exception: the name can also be something else which was not enumerated, because it can have various parameters and aliases. Is this explicitly supported by the library, or should the name always be one of the enumerated ones?

Configuration

• Set this to None to automatically select the default port layout for the device.

  I think there should be an option to skip input/output functionality. Or is that indicated by an empty vector?

• The MIDI part of the API probably needs more fleshing out still.

  • Should connecting to devices from different backends be supported?
  • In ALSA/Jack we can create anonymous clients that aren't necessarily tied to any available device.
  • I suppose changing the MIDI config on the fly should also be allowed?
    • Unplugging/hotplugging are things to possibly consider

Startup/run

• Is the tuple necessary here instead of Variant(u32, u32)?

  TryNextBestWithMinMaxSR((u32, u32)),
  TryNextBestWithMinMaxSize((u32, u32)),

• Reporting estimated round-trip latency mainly serves to inform the end-user with a ballpark estimate. Sometimes that is all you can do of course, but some kind of timestamps are needed to properly correlate MIDI/other events with the audio clock.

Wild idea

Doing input and output syncrhonized in the same callback prevents a certain feature that may or may not be desired.

When recording, it is often more important that the recording is flawless, while the live playback (through effects and such) is less important and if it does have dropouts, there's not much that can be done about it. If input and output are instead run asynchronous of each other, it is possible to always keep recording without dropouts regardless of how slowly the effects chain is running.

More specifically, we could have it so that the input side runs on schedule and sends data to be stored on disk (or memory), and also sends it to the output side for further processing. The output side can then slow down and drop out as much as it likes without affecting the input side.

Something akin to this is provided by JACK2 asynchronous mode, though the recording would have to happen in a separate client from the effects. With JACK, it is explicitly stated that asynchronous mode increases latency by one period. Some additional latency is inevitable, so I suppose this should be configurable.

Earlier I thought this might be a common feature in DAWs, but my experience might have been incidental. Even with synchronized I/O, there is some room for the output to drop out before any input data is lost. From what I was able to test (and read code), none of Bitwig, Ardour or Reaper have this feature. So I guess we can happily ignore it.

[BillyDM]

• Maybe the backend could be described and selected as an enum? All variants should be available in the enum regardless if support for them is compiled in (unlike how it is in CPAL).

From the perspective of a platform-agnostic settings GUI that isn't concerned with what platform it is running on, it wouldn't make sense to expose all the backend types on every system as an option.

• I don't think the backends should be split according to OS.

I mean, isn't that exactly what a platform-agnostic abstraction does? Like CoreAudio would only make sense in MacOS. The only exception maybe could be Jack, but even then there still might be platform-specific detail differences.

• Returning all device info for all backends at once seems problematic. Some backends may take a while to "start up" and enumerate devices. Some may even hang in some cases.

Oh yeah, that is a great point. Yeah I'll probably change it so it only scans one backend/device at a time.

• What happens if the set of available devices changes between enumeration and startup? It might be possible to accidentally try to open a completely different device, since they are only correlated by name.

Do you mean we should use device IDs or something? A lot of the time the name of the device is the ID, but I suppose adding the actual ID of a device wouldn't hurt.

• I think there should be an option to skip input/output functionality. Or is that indicated by an empty vector?

Right, I suppose that could be confusing. I'll probably use a custom enum in place of Option then.

• Should connecting to devices from different backends be supported?

Hmm, I suppose this might be possible. But is it really necessary?

• In ALSA/Jack we can create anonymous clients that aren't necessarily tied to any available device.

The behavior is already to use "fake" buffers filled with silence for ports and controllers that can't be found. I suppose some backends could just register any invalid controllers and ports with unconnected ports/controllers in the audio server.

• I suppose changing the MIDI config on the fly should also be allowed?

Right, definitely. I forgot to add that one.

• Is the tuple necessary here instead of Variant(u32, u32)?

Rust marks it as an error if it's not a tuple.

• Reporting estimated round-trip latency mainly serves to inform the end-user with a ballpark estimate. Sometimes that is all you can do of course, but some kind of timestamps are needed to properly correlate MIDI/other events with the audio clock.

Right, the purpose is just to be an estimate. I suppose I can do some name changing to make it more obvious.

When recording, it is often more important that the recording is flawless, while the live playback (through effects and such) is less important and if it does have dropouts, there's not much that can be done about it. If input and output are instead run asynchronous of each other, it is possible to always keep recording without dropouts regardless of how slowly the effects chain is running.

Well, a large goal of this is to have everything synced into a single process() callback, but I do get your point. Supporting this might depend on the backend and the device. I suppose later down the road we could add an extra option to record the inputs to disk on demand in a separate thread? Another option is to leave it up to the user to spawn their own high priority thread and sync buffers between the main audio thread and their own thread (which will create some delay of course).

[ollpu]

From the perspective of a platform-agnostic settings GUI that isn't concerned with what platform it is running on, it wouldn't make sense to expose all the backend types on every system as an option.

I suppose that makes sense. The enum wouldn't be meant to be iterated over though, just something that can be more cleanly matched against programmatically (the UI might need to be slightly different for each backend or you might want to add a textual explanation like in Bitwig). My gripe with CPAL's design is that it is a nightmare to programmatically select a backend that isn't always available, as evidenced by the dance that the examples have to do to support JACK. Even string identifiers is certainly a step up from that.

I mean, isn't that exactly what a platform-agnostic abstraction does? Like CoreAudio would only make sense in MacOS. The only exception maybe could be Jack, but even then there still might be platform-specific detail differences.

Mostly they would be dictated by OS of course, but I disagree with the way you've split the code into modules by OS currently. I'm thinking specifically of ASIO. It's a pain to compile, so it probably shouldn't be enabled by default. Same with JACK to some extent. So some backends aren't necessarily available even if the OS is appropriate. I doubt there's much code to be shared between backends on the same OS. CPAL's design seems reasonable in this regard.

Do you mean we should use device IDs or something? A lot of the time the name of the device is the ID, but I suppose adding the actual ID of a device wouldn't hurt.

Possibly, but maybe it'd make sense to keep device handles open in the driver. That definitely poses its own challenges though.

[separate MIDI backends] Hmm, I suppose this might be possible. But is it really necessary?

I don't know.

Right, the purpose is just to be an estimate. I suppose I can do some name changing to make it more obvious.

My main concern with the latency thing is it's not enough: timestamps for audio input and output times are necessary, as they are for MIDI events.

I see you've now added a delta on the MIDI events "The amount of time passed, in frames, relative to the start of the process cycle." However, I think that's misguided. The MIDI events might be older than the current audio input buffer, or they may even be newer. There's no guarantee of them being inside the buffer, and live playback either needs to add systematic latency to avoid issues like this, or be slightly off. I don't think the IO lib should worry about such buffering, but rather just provide the events when they are available. Recorded MIDI events should be as precise as they can be, though.

[BillyDM]

the UI might need to be slightly different for each backend or you might want to add a textual explanation like in Bitwig

Yeah, that's fair. I suppose we could return enum values instead of the name of the backend as a String.

It's a pain to compile, so it probably shouldn't be enabled by default.

Right, backends like Jack and ASIO will be an optional feature.

The MIDI events might be older than the current audio input buffer, or they may even be newer.

Hmm, good point. I suppose this ties into whatever Bitwig calls the "MIDI clock". Do you have more insight on the proper way to do this?

[BillyDM]

Okay, I commited a new API.

Still need to figure out the MIDI clock thing though.