Pinned Tweet
Someone says "I'm fine." You know they aren't.
You heard it - the pause, the flat tone. Voice AI doesn't. It all gets dropped at transcription.
So we built the Expression Measurement API. 48+ emotions, 600+ descriptors, 50+ languages, audio and video, real-time.
Conversation was always more than words.
The people behind the benchmarks. Jakub Piotr Cłapa is Hume's Head of Research.
"LLMs are only trained on what we could explain verbally, and this gap is a big one. Much larger than even for images or video."
@ElevenLabs v4 Turbo and v4 just joined the Real World VoiceEQ leaderboard at 3rd and 4th of 38.
We ran both through 18 of our human evals, three generations per prompt, with native-speaker raters for non-English passages.
Where there's room to grow:
• English long-form sits mid-table.
• Alphanumeric words, like codes and IDs: v4 is 13th of 38.
• Audio tags work on short directions but get read aloud past about ten words, so expressiveness couldn't be fully scored yet. The overall rank covers the questions they could be rated on.
A big step up from v3, which sits 15th.
Full results and the leaderboard: hume.ai/benchmark#tab-tts
Docs, both SDKs, and the Python and Node quickstarts: dev.hume.ai/expression-measu…
Currently in limited release - contact us for access.
Text-to-speech was built to read a single voice aloud. Now it's being asked to hold a conversation between two, and that's a different problem entirely.
The speakers have to stay distinct, trade turns with natural timing, and keep their emotional states from bleeding into each other. None of that shows up in word error rate.
So we built a rubric: five dimensions, 48 two-speaker scripts, and four scenarios designed to break a dialogue by pairing similar-sounding voices, forcing frequent interruptions, contrasting emotions, and running long exchanges.
Three blind human raters per clip. We ran three generations of Google's Gemini TTS models through it, alongside other multi-speaker systems.
Keeping two voices apart is the hardest part of the job. Every system handles a male and female pair easily and drops on at least one same-gender pairing. The newest Flash model made its biggest gain on exactly that.
Models with the same overall score get there differently. One leads on keeping speakers’ voices distinct, another on making the dialogue sound like one continuous recording. The overall score hides that difference.
New capabilities need new evaluations.
Full case study, with audio examples, is linked in the comments.
3/ Gemini 3.8 Flash and Flash-Lite take the top two spots on Overall, at 0.92 and 0.91.
Gemini 3.1 Flash, an older model, still leads Acting at 3.97, well clear of the 3.69 behind it.
5/ Every model on the board leads on something and trails on something else. That keeps being the finding.
Google published their own Gemini 3.8 TTS results today and cited the board: blog.google/innovation-and-a…
6/ The board on Hugging Face, with the full per-factor breakdown:
huggingface.co/spaces/HumeAI…
Same data on our site, with every factor tab:
hume.ai/rw-voice-eq
2/ Overall is now a frontier score: how close a model gets to the best possible combination of expressivity and reliability, rather than an average across factors. A model has to be both.
Voice identity is now a same-speaker rate.
A result from the Voice Controllability leaderboard we published today:
Fish's voice design model won nearly every head to head for motivational speaker voices and lost nearly every one for meditation voices. Same model, same prompt format, and the job you're casting for flipped the outcome.
6/ This is also our first public eval that goes beyond English. Thirteen languages so far. ElevenLabs v3 led in seven of them, Inworld took most of the rest, and a couple of languages went badly for models that were strong elsewhere, which is the reason we score each one separately instead of reporting an average.
7/ Details, per-model breakdowns and audio of passes and fails are on the board.
Blog: hume.ai/blog/introducing-the…
Board: huggingface.co/spaces/HumeAI…
