How to get access to the frame count / sof callback ?

[JulienBernard3383279]

I am confused about how to get access to the sof callback. I see there's a void tud_sof_cb(uint32_t frame_count) defined as a weak symbol, and that tud_sof_cb_enable is part of the application API, but even after calling tud_sof_cb_enable (or the more hidden usbd_sof_enable to specify the rhport & user consumer), no calls happen to my callback (or the weak symbol).

For context I want to use the sof to adjust audio rate before audio streaming begins, i.e for audio purposes but not to implement feedback in the UAC 2 sense.

[HiFiPhile]

Which MCU you are using ?

[JulienBernard3383279]

@HiFiPhile I'm using a STM32H750. I hacked around by adding a stray usbd_sof_enable(rhport, SOF_CONSUMER_USER, true); in audiod_set_interface; now I get calls to tud_sof_cb and can do PLL finetuning.

By the way, since you're here and given you seem very knowledgeable about USB audio as most of what I use there seems to have been written by you - many thanks for that - I have a few questions I would really appreciate if you could shed light on.

In particular I'm a little confused about the role of the feedback endpoint.
I was under the impression before reading the UAC2 specs that the device would have to adapt to the host notion of time, one way or the other, be it by adapting its clock to SoFs, to the effective arrival rate of the data, resampling to fit its clock, or drop/dupping samples; on the basis that it is not up to the master to cater to the clock of the peripheral.
I then learnt about this feedback endpoint, which if I understand correctly is essentially about telling the host 'hey, I see time go by 0.01% slower than you, so we're drifting, please adjust yourself'. (?)

What will the host truly do in response to such an 'adjustment request' ? Say a video game is playing a cutscene, I can't well imagine it slowing down the cutscene - that has hard video sync requirements - because audio is going too fast for the sake of the audio peripheral streamed to.
So what then, will the driver re-sample the audio from e.g 48kHz to 47.99kHz, just so we stop drifting ? Or will the host take care of drop/dupping samples ?
In short, what does that feedback endpoint really do in practice ? What good does it do the host to know we're drifting, will it take responsability to prevent it ?
And when is that feedback ep mandatory anyway ? UAC2 specs says it not being there for asynchronous endpoint is allowed when there is another IN channel in the same group of channels (not the case in your audio example; there is a microphone that is specified to use the same clock source in the topology transmitted to UAC2 driver so it could figure it out? But I wasn't under the impression that was allowed by the specs).
But even in the absence of any feedback path from asynchronous OUT terminals, I don't get why it's necessary. I can just look at the SoFs and then the effective data rate, and fine tune the PLL ?
Is my confusion arising from the fact that the standard is assertive that Synchronous is about SoFs, and not just any kind of clock adjustment capability from the device, but it should be, and so if I plan on looking at the data rate and adjust (have +-~0.1% adjustment capabilities from the VCOs being adjustable while running) I should signal a synchronous OUT terminal in the topology ?
Or is that remark moot anyway because the standard expects the data rate and the SoF rate to never differ in an isochronous context ? How true is that anyway, can I, in practice, fine tune my I2S clock frequency to the SoF rate before audio data streaming starts and expect no drift on modern computers ?

[HiFiPhile]

I then learnt about this feedback endpoint, which if I understand correctly is essentially about telling the host 'hey, I see time go by 0.01% slower than you, so we're drifting, please adjust yourself'.

Yes

So what then, will the driver re-sample the audio from e.g 48kHz to 47.99kHz, just so we stop drifting ? Or will the host take care of drop/dupping samples ?
In short, what does that feedback endpoint really do in practice ? What good does it do the host to know we're drifting, will it take responsability to prevent it ?

For purely audio playback (no video) what the host does is very simple, if you are slower the host will send less samples, if your are quicker then it will send more.

The host is allowed to send +-1 samples each frame, more details you can read Universal Serial Bus Device Class Definition for Audio Data Formats (Frmts20 final.pdf) chapiter 2.3.1.1.

Using 48kHz on full-speed as example, host will send 47-49 samples per ms. For 44.1kHz it becomes 44-45 samples per ms.

And when is that feedback ep mandatory anyway ? UAC2 specs says it not being there for asynchronous endpoint is allowed when there is another IN channel in the same group of channels (not the case in your audio example; there is a microphone that is specified to use the same clock source in the topology transmitted to UAC2 driver so it could figure it out

In this case it's called Implicit Feedback. You can set the Usage Type of Endpoint Descriptor of the IN EP to Implicit feedback Data endpoint, more details can be found in USB 2.0 specification Table 9-13. Standard Endpoint Descriptor (Continued)

But it's not supported by every host driver, like Windows built-in driver.

I can just look at the SoFs and then the effective data rate, and fine tune the PLL ?

Yep, then you are making a Adaptive Synchronization instead of Asynchronous Synchronization.

I should signal a synchronous OUT terminal in the topology ?

Yes

in practice, fine tune my I2S clock frequency to the SoF rate before audio data streaming starts and expect no drift on modern computers ?

Cheaper USB DACs chips have their PLL locked into USB clock so they are perfectly in sync.
Asynchronous Synchronization is used to improve audio playback quality but not a obligation, mostly used by High-End devices, since USB clock has a relatively high jitter, if we use it as PLL source then I2S will also has a high jitter and hurt DAC performance.
If you fine tune your I2S clock the impact on audio quality should also be hardly noticeable depending how you are doing it.

[JulienBernard3383279]

Using 48kHz on full-speed as example, host will send 47-49 samples per ms. For 44.1kHz it becomes 44-45 samples per ms.

But to keep frequency correctness it's not just a matter of adjusting the speed at which the audio data is transferred, but the audio data natural rate itself. Or do you mean that it would just take the 48kHz audio data stream, and send it at 47.99kHz without changing the stream values i.e without resampling from 48kHz to 47.99kHz ? Well I guess if the difference in clock frequency is so small that won't have much impact on audio quality anyway...

[HiFiPhile]

The interest of Asynchronous Synchronization is making your unadjusted I2S clock, which came from a high quality oscillator as the reference clock, PC will correct itself's drift by sending more or less data to DAC, not the DAC who adjust to meet PC clock.

There is no resampling.

[JulienBernard3383279]

But that's why I brought up cutscenes: the audio data the PC sends is related to other things. I don't expect the video playback rate to cater to the peripheral notion of time, and I don't expect video and audio to get out of sync. So how to reconcile this; if the video/audio-providing app never hears about the peripheral as manageable architecture would have it, that it produces the video/audio at frequency X, and you ask for audio at frequency X+1%, the driver buffer will underflow. Some part of the chain needs to handle the frequency gap between the video app using the PC notion of time, and the peripheral using its own. This is what I'm confused about: what part does, and how ?

[HiFiPhile]

If you bring in video then it becomes complicated. One must be the master clock, either host, DAC or monitor. The detail is out of scope here and purely software dependent, you can read more https://mpv.io/manual/stable/#options-video-sync

[JulienBernard3383279]

Wow, so the default mode of this truly scales video to audio. I would've bet on some sort of resampling or drop/dupping. At least happy to see from the listed options that I'm not going insane, the video/audio syncing decision I was going on about does have to be taken.

The next thing that puzzles me then is how the default mode here, syncing to audio, appears to assume video frequency can be fine tuned... where as far as I know, the transfer of video to screens is very much a fixed frequency thing, unless the monitor is e.g G-Sync capable. Alas, so much to understand, so little time. But yeah I guess this strays too far from tinyusb's concerns to belong here.