We also train both code encoders to enable a new kind of effect chain retrieval that ignores the parameters of the effects. For T5, we mask tokens as in standard language modeling. For BoxGraph, we randomly replace parameter nodes with learnable embeddings. At inference, either model can "mask" the parameters to create embeddings that (hopefully) represent chain topology alone. For 21 effects, this means 420 permutations of length-2 programs, so a gallery of 420 code embeddings. BoxGraph beats T5 and approaches the performance of a supervised AFx-Rep architecture trained on the same data that uses logits to select the likeliest chain. This gives me hope for "zero-shot chain retrieval": give out-of-domain audio trained on VSTs but retrieve Faust programs that produce similar audio. This will require more high-quality Faust effects to be included in the training data of models such as PEACE, so this is a high-priority for me. Check out the Faust Libraries for more context: github.com/grame-cncm/faustl….

Our method uses the non-contrastive training objective from SLAP (h/t @Juj_Guinot @howariou @elio_elioo). Like SLAP, we observe a small modality gap. We know text-to-audio relied on CLAP. What's next for SynAPSE? 🤔

If you make a design that you’re happy with, share the URL here!