Impressive new streaming transcription model from @AssemblyAI. We put a lot of work these days into creating good benchmarks and evals both for the individual models we use to build voice agents, and for complete voice agent implementations. Model launches like this are a good opportunity to talk about benchmarking! The Assembly AI launch post cites two benchmarks: one that I work on and one from our friends at @covaldev. Both benchmarks show the new AssembleAI Universal 3.6 Pro Realtime model as having truly excellent accuracy and very good latency. The accuracy of the new model is #1 in the Coval testing. It's #2 in our testing, edged out by the new Meta Muse Transcribe model. But the margin is so slim that there is unlikely to be any practical difference between the two in real-world use. These two benchmarks use different data sets and different methodologies for calculating a transcription error rate score. Which is good, because diversity in testing is really helpful! You can read detailed descriptions of how error rates are calculated in the explainer pages for both benchmarks. And both benchmarks are fully open source, so you can read the code and reproduce the results yourself. Both benchmarks also show the Universal 3.6 Pro Realtime model as having very good latency. For voice agent use cases, the important latency metric is the "time to final segment." This is how long it takes the model to deliver a complete transcript result after the user stops speaking. The Coval benchmark shows a P95 TTFS of 225ms. We report 401ms. The Coval latency numbers are continously updated and our benchmark is a one-shot aggregate from 1,000 segments run all at once. So probably the Coval number is a better representation of what you'll see from the AssembleAI API. In either case, these numbers put the new Assembly AI number on the Pareto frontier for all transcription models available today. In plain English, this means that there's no model that's better on both accuracy and latency. There are models that are somewhat faster. And (one) model that's a tiny bit more accurate. But this new model is clearly a great choice for voice agents and moves the bar upwards in the increasingly competitive speech-to-text model category.