Tom Burns (@tfburns) header
Tom Burns (@tfburns) profile picture

research @aleph__alpha. previously @Cornell @ICERM @OISTedu @MonashUni. opinions my own. I want this AI stuff to go well (for us all).

Joined April 2009
It turns out to be surprisingly easy to Bayesian-pill people.
5
236
Tom Burns retweeted
new red-letter edition of Elementary Applied Topology! full pdf on my website -- physical book for $15 from amazon... www2.math.upenn.edu/~ghrist/… short story: * new chapter on background material (chapter 0) * new chapter on persistence (chapter 6) * so many errors... fixed! * some new artwork... this book, begun in 2009 when the field was emerging, means a great deal to me. it was always an art project more than a textbook; indeed, the minimalist images were and are the only exercises in the text. if you can figure out what the image means, you have figured out the math behind it. i hope that keeping the pdf available on my site encourages a new generation of students to learn topology. though it might take some time, as the page count went up by a third. also, the bibliography has grown quite a bit, given the developments in the field...
11
49
4
317
12,148
Tom Burns retweeted
The largest math proof in human history is now computer-verified. Today, Axiom announces a complete Lean auto-formalization of the Classification of Finite Simple Groups, the "periodic table of symmetry." About 15,000 pages, built by over 100 mathematicians across generations.
425
1,302
656
7,163
3,754,458
The Kolibri tech report is now on arXiv: arxiv.org/abs/2610.31945. If you haven't read it yet, give it a read - it covers all steps on how to build a modern LLM.
1
6
22
1,012
There are many subtleties to getting evaluation of models right! As others have reported previously, the location of a single space can sometimes make a large difference. Why and how? Read our blog and paper if you are interested :)
The location of one space in a multiple-choice evaluation changes what it measures. It changes the probability of the correct answer by several orders of magnitude but usually barely changes the accuracy.
1
6
471
What evals do you rely on for model quality? Many BPB scores are a fixed cost divided by answer length, i.e., cross-task averages can weight tasks by answers lengths. PriorBPB cuts the correlation with length from 0.68 to 0.18. Paper and Apache 2.0 code below:
Same model, same question, same meaning, but a longer answer can improve the score ~2x. A large share of bits-per-byte (BPB), a common pre-training eval metric, reflects answer length rather than predictive quality. We propose an alternative, PriorBPB. 🧵
1
13
514
Resolution of singularities is one of the major algebraic geometry tools behind the rest of Resolution's learning theory work. We've been doing Lean formalization treating it as an axiom, but a pile of tokens later it's a Lean theorem and the downstream work is unconditional. 🧵
1/ We have autoformalized the resolution of singularities in Lean. Hironaka's 1964 theorem is one of the great mathematical results of the 20th century. It says that any singular variety is the "shadow" of a smooth one living in higher dimensions. So why did we do this? 🧵
1
6
79
5,184
Benchmaxxing is cheating in so many ways and just plain bad. If you engage in it, not only do you cheat the community out of trustworthy scores on your model, you also cheat yourself and your team. And if you don't have trustworthy metrics, how can you make evidence-based decisions? This blogpost from our team has a some great nuggets of wisdom in it. We're still learning, along with y'all :)
Benchmark text is all over training data. A model that has read the test set will nail it. That score measures memory, not skill. We scanned our mid-training pool and removed 1.5M documents. Our finding: trustworthy scores need decontaminated data.
1
1
35
1,748
Kolibri is out! I'm proud to have been part of it. I worked mostly on pre-training data curation, and also helped with evals, including of abstention. I’ll be posting some short threads on each below 👇
Small bird, fast wings, Kolibri is here. 78B parameters. 3.46B active. Up to 1M tokens of context. Built in Europe. Now the weights are yours. Run it on your own hardware, under Apache 2.0.
4
44
1,536
Evaluation of abstention: teapawt.org/tfburns/status/2106380…
Does a model abstain when it doesn't know? We evaluated Kolibri Base and comparison models using the distributional correctness score to find out 🧵
2
1
83
Evaluations in the open: teapawt.org/tfburns/status/2106474…
Kolibri Base is the best-performing model among its active param class that we tested. Evals are how we measured this, and we used an open-source (Apache 2.0) eval-framework to generate the reported scores. Here's some more context 🧵
1
28
Kolibri Base is the best-performing model among its active param class that we tested. Evals are how we measured this, and we used an open-source (Apache 2.0) eval-framework to generate the reported scores. Here's some more context 🧵
2
1
1
20
3,043
I’ve also enjoyed seeing the team building out our now many internal and external evals, including in safety and hallucination. These feed into our aggregate English and German benchmark scores and ultimately lead to consequential modeling decisions, e.g., ablation comparisons.
1
2
159
More on evals in the tech report: Section 2.4 (base model evaluation), Section 3.3 (post-trained model evaluation, including hallucination reduction in 3.3.3.2) and Appendix E (baseline model deployment details). Part of the Kolibri release: teapawt.org/Aleph__Alpha/status/21…
Small bird, fast wings, Kolibri is here. 78B parameters. 3.46B active. Up to 1M tokens of context. Built in Europe. Now the weights are yours. Run it on your own hardware, under Apache 2.0.
1
4
179
Does a model abstain when it doesn't know? We evaluated Kolibri Base and comparison models using the distributional correctness score to find out 🧵
1
2
3
13
1,765
More details in the tech report: Sections 3.1.1.4 and 3.2.2.3 for how we went further and trained the model to abstain in post-training (when the provided context doesn't contain the answer), and Appendix F for more DCS details. Part of the Kolibri release: teapawt.org/Aleph__Alpha/status/21…
Small bird, fast wings, Kolibri is here. 78B parameters. 3.46B active. Up to 1M tokens of context. Built in Europe. Now the weights are yours. Run it on your own hardware, under Apache 2.0.
1
61