
Choose disfavour where obedience does not bring honour. I do math. And was once asked by R. Morris Sr. : "For whom?" @halvarflake@mastodon.social
Joined June 2008
- Tweets67.9K
- Following2.9K
- Followers48.1K
- Likes43.9K
Pinned Tweet
I will have to print & frame this tweet :-))
If you watch one talk this month, make it this one. Profoundly insightful on the complexity threat to security by @halvarflake given at NATOs CYCON.
Video:
err.ee/836236/video-google-0…
And slides:
docs.google.com/presentation…
Halvar Flake retweeted
Latest (and probably penultimate) Building Slack post is up: buildingslack.com/salesforce…
What it's like being eaten by the whale
Halvar Flake retweeted
Weekend for me is the time for making AI-based artifacts. Very briefly, I
1 write down a problem/open question;
2 pose it to GPT6.1 and Opus5.5 agents, with maximum effort, interacting on common files, using my skills
3 read [some of] the output
4 refine the question
The output is not AI slop (at least, I learn from it) but it's not not AI slop either. It's quite good overall and I learn from them. I wonder if people would be interested in them. Below is an example. Let me know, I don't want to add to the sloppiness in the world.
Halvar Flake retweeted
I am back with a few thoughts.
I stopped reading all the trauma logs of mathematicians who recently got impacted by AI. Instead I switched to building a corpus of knowledge for why and how AI is interacting with mathematics.
I teach this term a new course here at Adam Mickiewicz University. It's title is "Computer assisted solutions to mathematical problems". I proposed it because I realized in the last 6 months something important.
A. It makes no sense to grieve the "good old ways". We should study to the bone how mathematics and computers interact. There are three levels: numeric, symbolic and formal.
B. The only way to prepare yourself for the rise of the technology is to understand it well. You cannot understand the progress of AI in mathematics if you don't know how the models are trained on the symbolic representation of mathematical proofs, how the training goes, why specific transformer architectures, reinforcement learning and basically a huge amount of incredible insights gives in practice some new value in mathematics.
C. It is incredibly fun to train your own models (yes I use generative models to help me build training pipelines too, otherwise I would not have enough lifetimes to do so).
D. The AIxMaths space is so incredibly complex - you have these weird, scalable token generating algorithms, recently in agentic form with memory, interactions etc. This is a field of study - we need to understand how it works. On the other hand, mathematics as a field of science is incredibly complex, people do very different kinds of axiomatic theories with their intuitions, tactics and strategies. This was never studied too seriously as a whole. We need to study it deeply.
So I am spending this term gearing up my students and myself for whatever is coming. We don't have any assumptions. I don't claim that models will become better at maths than all of human mathematicians. I don't think it's a good question. We just want to stay in the loop with the developments and see whether we can contribute to the change and see if this can be useful and productive.
I am looking forward to your feedback. The current plan contains the following topics:
- basic distinction between numerical computation, symbolic calculations and formal proofs
- basic understanding of the IEEE 754 standard and things around numerical computations
- how to build your own computer algebra system in Python and why it's so tricky
- why and how proof-checking software is important; we start with the basic principles of such designes, I plan to talk first about the foundations going back to Automath and the foundational work of de Bruijn.
- practical examples spanning real use cases of computer algebra, numerical integration and proofs in Peano arithmetic and how to read languages like Lean
- in the end I want to present some little post-trained models like Qwen which I taught how to do Peano arithmetic better and how they prove statements about this theory; there will be AlphaGeometry too and maybe a bit more about Mizar, Metamath and Lean. Just to give students some scope and perspective.
I plan to continue working on this course into a full book. I think I won't stop until I understand deeply why language models are currently so good at maths. It's something of utmost importance. It's important to see where this might stop and what lies beyond. I personally think these kind of questions will be the most important problems to answer by mathematicians themselves. We shouldn't stop until we find good answers to all of this. Reading AI slop is perhaphs a bit useful, but deep understanding of why it sometimes makes sense is even more challenging.
If you talk to CFD experts they'll tell you they need experiments to validate their simulations.
If you had an exponential amount of compute you could solve simulate biology exactly and wouldn't need to run any experiments.
Some of you will think "of course, but that's an unrealistic assumption" and others will think "no, you need experiments even with that much compute. That's just How It's Done".
You would of course need evaluations once you have a solution, but you don't in principle need experiments to find it: Experiments are a way to save compute, dating from a time when it was much more expensive.
And I think the presentation is exponential but real systems aren't irreducibly exponential. AlphaFold worked, after all.
Shigeru Miyamoto (Nintendo) received Japan's prestigious Order of Culture and used the press conference to talk about generative AI, what 'creating' actually means when tools automate execution, and why Kyoto (not Silicon Valley) produced Nintendo's design philosophy.
famitsu.com/article/202610/9…
The Order of Culture (文化勲章) is one of Japan's highest honors, previously given to film directors like Akira Kurosawa and writers like Yasunari Kawabata. Japan basically put games in the same category as film and literature.
He talks about generative AI in a way I have not seen from many working creators. He distinguishes between "creating" and "working." Creation is the hard part: composition, direction, deciding what should exist. Work is execution, and work is satisfying in a way that can fool you into thinking you have created something.
He says even before AI, tools have always automated technique, from typewriters to desktop publishing. What gets exposed when that happens is whether you had vision and intent, or whether you were just good at execution. AI is doing the same thing at a larger scale, and he finds this encouraging.
"The scary thing is becoming satisfied with the work and mistakenly thinking you created something."
People told him for decades that he should go outside of Japan and be in a "global hub". He stayed in Kyoto anyway. His work became more successful overseas than at home. The reason, he says, is that the person on the other side of the screen is human, regardless of geography. Deep local rootedness kept him anchored in concrete experience rather than trend-following.
Halvar Flake retweeted
Replying to @stevenstrogatz
But Stephen, don't you give away the answer at the beginning of your essay? I think Arnold used to say that "Mathematics is the branch of Physics where the experiments are cheap", or that "[pure] Mathematics is a branch of Applied Mathematics". It *is* going to be a boon for Applied Math, and then for *Pure* Math. Entire fields of *Pure* Mathematics might be created as a result. Fluid Dynamics is pure enough to count a Millennium Prize problem now, but was very applied back then. And we could have an entire new field like it, let alone one problem.
If it requires rearranging the brains and expectations of a few people, so be it. I am not discounting the pain of the next 10 years, but the long view should make everyone extremely excited.
Sandboxing is really hard especially when facing our cyber capable models. So, we used our cyber capable model to red-team our own sandbox stack and found issues in many layers of the stack, including but not limited to Kata, gVisor, docker, and Linux.
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.
On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.
While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.
After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.
openai.com/index/securing-ou…
Halvar Flake retweeted
The ugly truth about France and some of the other European countries currently in turmoil - and which none of the politicians want to admit because it'd be political suicide - is that you can't solve the problem by going after immigrants (as the right asserts) or the rich (as the left asserts), but you could solve it by going after the old.
One extraordinary statistic for you: people over 60 in France, a quarter of the population, own 60% of France's wealth, non-financial and financial (ofce.fr/pdf/revue/2025/3-192…).
And here is another crazy statistic:
that same quarter of the population, those above 60, receive nearly two-thirds of all social spending by the French government (previssima.fr/actualite/prot…) - which is famously among the highest in the world (France's government spending is an insane 57.3% of GDP).
Last statistic: they make up about 40% of voters because they're a third of registered voters (insee.fr/fr/statistiques/266…), and they turn out far more than anyone else (franceinfo.fr/elections/sond…).
Put our 3 statistics together and there's almost a bit of an apartheid situation here: the old own most of the country's wealth, collect most of the social spending and have the political power to keep it that way.
Which makes them a de-facto rentier class, living off assets and transfers paid for by younger generations.
And, of course, at the risk of stating a truism, the future is built by the young. Yet France keeps effectively taking money away from them to give it to those whose working lives are already behind them - and need the money far less since they already own the country's wealth.
What about inheritance, I hear you ask? That's the funniest/saddest aspect: given that life expectancy has kept rising, the French now inherit a home at 54 on average. The most common case is someone in their 80s leaving it to heirs in their 50s, in effect the very old transferring wealth to the about-to-be old (insee.fr/fr/statistiques/fic…).
You can add this to our earlier apartheid statement: on top of all the rest, even when the old finally let go of their wealth, it stays within their own "camp".
That's my personal situation, incidentally: I'm 42, my father is 71 and he is starting to look at ways to pass things on to me and my siblings, when our professional lives are already well settled, so it'll arrive when it changes things the least, which is pretty much the French system in a nutshell.
Now, I get that there'd be some unfairness in going after them: they paid into the system their entire working lives, playing by the rules of a social contract that told them if they paid up they'd collect at the end. Changing the rules now means breaking that promise, which is admittedly unfair.
But there's no magic wealth tree in any society: you can only take it where it is. That's the way most major societal transformations end up happening. They're often triggered by a catastrophic event (the so-called "four horsemen": mass-mobilisation war, revolution, state collapse, plague) BUT what really drives change is a transfer: wealth gets pried from the privileged class who've been sitting on it and handed to those who'll build what comes next.
So, in theory and in an ideal world, you can do without the catastrophic event and directly proceed with the transfer...
The fact that in France, and elsewhere in Europe, it's the youth protesting is probably no coincidence - even though they don't frame it as a generational fight. And even though, cruel irony, the left politicians many of them march behind promise to bring the retirement age back to 60 (lcp.fr/actualites/legislativ…), which would make the situation even worse.
Let's say they understand intuitively that the system is stacked against them, even if they haven't yet figured out how. And politicians are obviously doing what politicians do, meaning pandering to the people who actually vote, while pointing the young towards easier - and politically convenient - scapegoats.
To be clear I'm not saying it's the only problem in France, VERY far from it. I'm also not saying that all old people in France live a lavish lifestyle at the expense of younger generations: there are also plenty of very poor retirees who, if anything, would deserve more.
What I am saying is that it's probably the biggest elephant in the room: everyone can see it, nobody dares mention it - no politician, from any side, wants to run on cutting grandma's pension - and it makes every other problem harder to fix.
Halvar Flake retweeted
I take back what I said about OpenAI ... they also just flagged the same question 😂
If you wanted to manufacture a worst case scenario for cyber outcomes, it would be hard to find a better starting point that "Hey! How about we prevent people from asking questions about formal verification?!".
Cyber refusals when asking claude about details of Anthropics own formal verification work.
Guys ... come on =/
If you're worried about actual AI safety in cyber, about the worst thing you could do is take an approach that puts technology, and understanding of that technology, into the smallest number of hands possible.
Fortunately we have OAI, and open models, but it's frustrating to see the path Anthropic has gone down.
Halvar Flake retweeted
One thing about abliterated models is that it makes you very impatient with the non-abliterated models’ refusals to do research that they deem “unethical” for basically subjective (often bad) reasons.
Halvar Flake retweeted
On a side note, I'm thrilled Thinky has found Score Centering to be extremely stable! From their docs: "In our internal post-training, this loss function has led to extremely stable training, even under highly off-policy async RL and low precision numerics."
Halvar Flake retweeted
So we have a hypothesis: StableReinforce = Reinforce + Score Centering, where the expected score is computed over 8 randomly sampled tokens. We can easily verify that it is true: we implement Reinforce + SC as a custom loss in Tinker, do an optimizer step, and recover gradients that are within numeric precision of StableReinforce across all layers.
Halvar Flake retweeted
Looking at a single-token update, surprisingly only 8 logits have a non-zero gradient. In contrast, a standard Reinforce loss would put a non-zero gradient on all logits.
This observation basically reveals the objective. In Score Centering, we subtract the expected score under the sampler. Since computing the exact expectation is expensive, we have to resort to an approximation. In our paper, we propose a top-k approximation, which is deterministic but slightly biased. It seems that Thinky instead decided to compute the expected score by randomly drawing 8 tokens from the sampler, which is unbiased but higher variance.
Every token’s score has the same dense part (−p, from the softmax), so subtracting the average score over the 8 correction tokens cancels it exactly.
Halvar Flake retweeted
Dziękuję Radzie Miasta Krakowa i Przewodniczącemu Jakubowi Koskowi za gratulacje z okazji wdrożenia JPEG XL na kodowaniu ANS, który powstał w Krakowie na UJ (2006–2013), a dziś zapisuje dane w telefonach, przeglądarkach, serwerach, grach, genomice i w CERN. Miło, że w erze informacji coraz większa jej część jest zapisywana metodą z Polski.
@Radamiasta @KosekJakubRMK @JagiellonskiUni
Halvar Flake retweeted
This is absolutely nuts and only hitting German papers so far: leaked emails show that seven politicians from Germany's far-right AfD travelled to Syria to meet with Assad regime officials... facilitated by Hezbollah AND Russian intelligence.
handelsblatt.com/politik/deu…
Halvar Flake retweeted
Perhaps something to consider if you work at OpenAI / Anthropic:
If your policies have produced an outcome like 'if a researcher wants to do software verification research then they have use a Chinese model', then perhaps your approach to 'safety' is net negative.
Halvar Flake retweeted
Cyber refusals when asking claude about details of Anthropics own formal verification work.
Guys ... come on =/
If you're worried about actual AI safety in cyber, about the worst thing you could do is take an approach that puts technology, and understanding of that technology, into the smallest number of hands possible.
Fortunately we have OAI, and open models, but it's frustrating to see the path Anthropic has gone down.
Halvar Flake retweeted
Replying to @ZackKorman @halvarflake
Every generation that doesn't learn about covert channels in school/college (I read about them in a puzzle book in my teens) is going to be surprised by this and then they will think it is shocking and revolutionary because:




























