What does the loss graph / time (or iterations) look like for a working training setup?

[StevenSchrembeck]

I'm training a SemanticTransformer with the pre-made trainer and the loss graph isn't promising. It falls rapidly from 6 to ~5 then remains there, even 1000 iterations later. Might not be nearly enough iterations to know, but I expected it fall further, faster.

If you have a converging SemanticTransformer, what does your loss graph look like? Are you using an out-of-the-box dataset I can also test as a control?

Much appreciated!

[eonglints]

Hey, what dataset are you using? And what architectural details (number of heads, depth etc.) and feature extraction details (pre-trained model, k-means clustering model)? I got things to work reasonably well (loss falling to more like ~2 and outputs starting to move towards what you'd expect, more details here) with LibriSpeech, which is quite a small dataset in comparison to the one used in AudioLM for speech.

[eonglints]

Just saw your comment on the dataset over here but the rest would still be good to know. Very interested to know how you're approaching the semantic token extraction given that I don't know of any models pre-trained on sound effects.

[StevenSchrembeck]

Hey! thanks for getting back to me. I'll need to study up on the token extraction options and how that piece actually works, because I've been using the pieces trained on speech. I wonder if an auto-encoder could do the job.

Anyway, I'll study up and come back so that I'm not wasting time! My ML engineering is a little rusty, web dev by day, but I'm really excited to figure out generative sound effects. If I could just find an ML dev partner we'd make magic together

Appreciate the advice so far!