Tokenization Tax Report 2026 now also on @huggingface
hf.co/spaces/ellamind/tokeni…
Forgot to add: As European AI tradition demands, none of the four European tokenizers we tested beats the o200k baseline averaged across the seven EU languages we experimented with.
Forgot to add: follow @magikarp_tokens who is doing amazing tokenizer analysis incl. reproduction of Claude's vocab
Labs rarely change tokenizers. @OpenAI reached Astra-level capability with a tokenizer they shipped in May 2024 (at least the text part). No need to create your own tokenizer to go frontier. @thinkymachines Inkling also uses the o200k vocab.
Read the full article here: ellamind.com/blog/tokenizati…, includes interactive plots
/goal create a dataset for frenchmaxxing and upload to hf. Ne vous y trompez pas.
huggingface.co/datasets/maxi…
we managed to sneak some German flavor into frontier-bench, courtesy of @bla1990so
Tech report on Soofi-S-base is out: A 30B-A3B Mixture-of-Experts Model based on @NVIDIAAI's Nemotron 3 architecture, trained on 27T tokens with a strong focus on German. Supports up to 1M context length. Its a decent model and a solid first step. And SOOFI-L-base is already training.
Throughout the project, we executed more than one million benchmark task runs. During the production training run, we continuously evaluated intermediate checkpoints to track whether we are on target.
Brief 🧵 on base model evals in SOOFI, a technical blog with more details will follow soon.
📄 Releasing the Soofi S pretraining tech report: a sovereign, open foundation model for German and English
Today we’re publishing the full pretraining tech report and project page for Soofi S 30B-A3B — a Mixture-of-Experts hybrid Mamba model trained on ~27 trillion tokens with deliberately up-weighted German.
What’s in the report:
🏆 Strongest fully open model in our evaluations on BOTH the English and German aggregates — ahead of Olmo 3 32B and Apertus 70B (full methodology in the report)
📋 Radical transparency: complete per-source data accounting, all hyperparameters, training + eval code, checkpoints — everything under permissive licenses
🇩🇪 Trained end-to-end on Deutsche Telekom’s Industrial AI Cloud in Munich — sovereign AI infrastructure on German soil
Soofi S combines frontier-level capability with the highest measured aggregate long-context decode TPS, and unlike full-attention dense baselines maintains high throughput as context grows. The figure plots Capability Index versus measured aggregate decode TPS/GPU at 40K context and batch 32. The Capability Index averages five benchmark groups, i.e., Code, GSM8K, GPQA-Diamond, English aggregate, and German aggregate, after normalizing each group to the best plotted model. Aggregate decode TPS/GPU is measured with a TP=1, one-B200 vLLM latency-subtraction protocol.
Shoutout to all project partners for making this happen: @ki_verband @DFKI @FraunhoferIAIS @FraunhoferIIS @TUDarmstadt @Uni_WUE @BHT_bln @l3s_luh @LamarrInstitute @Hessian_AI @MMerantix @ellamindAI and the @BMWE_ for funding the project in the scope of IPCEI-CIS / 8ra.
Special thanks to @bla1990so @rscholz_ @bjoern_pl @jphme, not easy to stay sane in eval & looking at data land.













