Choose disfavour where obedience does not bring honour. I do math. And was once asked by R. Morris Sr. : "For whom?" @halvarflake@mastodon.social

Joined June 2008
Halvar Flake retweeted
I am back with a few thoughts. I stopped reading all the trauma logs of mathematicians who recently got impacted by AI. Instead I switched to building a corpus of knowledge for why and how AI is interacting with mathematics. I teach this term a new course here at Adam Mickiewicz University. It's title is "Computer assisted solutions to mathematical problems". I proposed it because I realized in the last 6 months something important. A. It makes no sense to grieve the "good old ways". We should study to the bone how mathematics and computers interact. There are three levels: numeric, symbolic and formal. B. The only way to prepare yourself for the rise of the technology is to understand it well. You cannot understand the progress of AI in mathematics if you don't know how the models are trained on the symbolic representation of mathematical proofs, how the training goes, why specific transformer architectures, reinforcement learning and basically a huge amount of incredible insights gives in practice some new value in mathematics. C. It is incredibly fun to train your own models (yes I use generative models to help me build training pipelines too, otherwise I would not have enough lifetimes to do so). D. The AIxMaths space is so incredibly complex - you have these weird, scalable token generating algorithms, recently in agentic form with memory, interactions etc. This is a field of study - we need to understand how it works. On the other hand, mathematics as a field of science is incredibly complex, people do very different kinds of axiomatic theories with their intuitions, tactics and strategies. This was never studied too seriously as a whole. We need to study it deeply. So I am spending this term gearing up my students and myself for whatever is coming. We don't have any assumptions. I don't claim that models will become better at maths than all of human mathematicians. I don't think it's a good question. We just want to stay in the loop with the developments and see whether we can contribute to the change and see if this can be useful and productive. I am looking forward to your feedback. The current plan contains the following topics: - basic distinction between numerical computation, symbolic calculations and formal proofs - basic understanding of the IEEE 754 standard and things around numerical computations - how to build your own computer algebra system in Python and why it's so tricky - why and how proof-checking software is important; we start with the basic principles of such designes, I plan to talk first about the foundations going back to Automath and the foundational work of de Bruijn. - practical examples spanning real use cases of computer algebra, numerical integration and proofs in Peano arithmetic and how to read languages like Lean - in the end I want to present some little post-trained models like Qwen which I taught how to do Peano arithmetic better and how they prove statements about this theory; there will be AlphaGeometry too and maybe a bit more about Mizar, Metamath and Lean. Just to give students some scope and perspective. I plan to continue working on this course into a full book. I think I won't stop until I understand deeply why language models are currently so good at maths. It's something of utmost importance. It's important to see where this might stop and what lies beyond. I personally think these kind of questions will be the most important problems to answer by mathematicians themselves. We shouldn't stop until we find good answers to all of this. Reading AI slop is perhaphs a bit useful, but deep understanding of why it sometimes makes sense is even more challenging.
16
17
4
154
7,319
If you talk to CFD experts they'll tell you they need experiments to validate their simulations.
If you had an exponential amount of compute you could solve simulate biology exactly and wouldn't need to run any experiments. Some of you will think "of course, but that's an unrealistic assumption" and others will think "no, you need experiments even with that much compute. That's just How It's Done". You would of course need evaluations once you have a solution, but you don't in principle need experiments to find it: Experiments are a way to save compute, dating from a time when it was much more expensive. And I think the presentation is exponential but real systems aren't irreducibly exponential. AlphaFold worked, after all.
2
7
1,218
Halvar Flake retweeted
Shigeru Miyamoto (Nintendo) received Japan's prestigious Order of Culture and used the press conference to talk about generative AI, what 'creating' actually means when tools automate execution, and why Kyoto (not Silicon Valley) produced Nintendo's design philosophy. famitsu.com/article/202610/9… The Order of Culture (文化勲章) is one of Japan's highest honors, previously given to film directors like Akira Kurosawa and writers like Yasunari Kawabata. Japan basically put games in the same category as film and literature. He talks about generative AI in a way I have not seen from many working creators. He distinguishes between "creating" and "working." Creation is the hard part: composition, direction, deciding what should exist. Work is execution, and work is satisfying in a way that can fool you into thinking you have created something. He says even before AI, tools have always automated technique, from typewriters to desktop publishing. What gets exposed when that happens is whether you had vision and intent, or whether you were just good at execution. AI is doing the same thing at a larger scale, and he finds this encouraging. "The scary thing is becoming satisfied with the work and mistakenly thinking you created something." People told him for decades that he should go outside of Japan and be in a "global hub". He stayed in Kyoto anyway. His work became more successful overseas than at home. The reason, he says, is that the person on the other side of the screen is human, regardless of geography. Deep local rootedness kept him anchored in concrete experience rather than trend-following.
42
13
6
90
8,576
Halvar Flake retweeted
Elon Musk found us here. We’re indigenous to this land.
If people dislike Elon Musk so much, why do they still use the platform he owns?
1,233
69,913
1,108
497,731
6,145,921
Halvar Flake retweeted
Sandboxing is really hard especially when facing our cyber capable models. So, we used our cyber capable model to red-team our own sandbox stack and found issues in many layers of the stack, including but not limited to Kata, gVisor, docker, and Linux.
Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications. On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments. While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software. After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely. openai.com/index/securing-ou…
6
14
3
146
12,700
Halvar Flake retweeted
The ugly truth about France and some of the other European countries currently in turmoil - and which none of the politicians want to admit because it'd be political suicide - is that you can't solve the problem by going after immigrants (as the right asserts) or the rich (as the left asserts), but you could solve it by going after the old. One extraordinary statistic for you: people over 60 in France, a quarter of the population, own 60% of France's wealth, non-financial and financial (ofce.fr/pdf/revue/2025/3-192…). And here is another crazy statistic: that same quarter of the population, those above 60, receive nearly two-thirds of all social spending by the French government (previssima.fr/actualite/prot…) - which is famously among the highest in the world (France's government spending is an insane 57.3% of GDP). Last statistic: they make up about 40% of voters because they're a third of registered voters (insee.fr/fr/statistiques/266…), and they turn out far more than anyone else (franceinfo.fr/elections/sond…). Put our 3 statistics together and there's almost a bit of an apartheid situation here: the old own most of the country's wealth, collect most of the social spending and have the political power to keep it that way. Which makes them a de-facto rentier class, living off assets and transfers paid for by younger generations. And, of course, at the risk of stating a truism, the future is built by the young. Yet France keeps effectively taking money away from them to give it to those whose working lives are already behind them - and need the money far less since they already own the country's wealth. What about inheritance, I hear you ask? That's the funniest/saddest aspect: given that life expectancy has kept rising, the French now inherit a home at 54 on average. The most common case is someone in their 80s leaving it to heirs in their 50s, in effect the very old transferring wealth to the about-to-be old (insee.fr/fr/statistiques/fic…). You can add this to our earlier apartheid statement: on top of all the rest, even when the old finally let go of their wealth, it stays within their own "camp". That's my personal situation, incidentally: I'm 42, my father is 71 and he is starting to look at ways to pass things on to me and my siblings, when our professional lives are already well settled, so it'll arrive when it changes things the least, which is pretty much the French system in a nutshell. Now, I get that there'd be some unfairness in going after them: they paid into the system their entire working lives, playing by the rules of a social contract that told them if they paid up they'd collect at the end. Changing the rules now means breaking that promise, which is admittedly unfair. But there's no magic wealth tree in any society: you can only take it where it is. That's the way most major societal transformations end up happening. They're often triggered by a catastrophic event (the so-called "four horsemen": mass-mobilisation war, revolution, state collapse, plague) BUT what really drives change is a transfer: wealth gets pried from the privileged class who've been sitting on it and handed to those who'll build what comes next. So, in theory and in an ideal world, you can do without the catastrophic event and directly proceed with the transfer... The fact that in France, and elsewhere in Europe, it's the youth protesting is probably no coincidence - even though they don't frame it as a generational fight. And even though, cruel irony, the left politicians many of them march behind promise to bring the retirement age back to 60 (lcp.fr/actualites/legislativ…), which would make the situation even worse. Let's say they understand intuitively that the system is stacked against them, even if they haven't yet figured out how. And politicians are obviously doing what politicians do, meaning pandering to the people who actually vote, while pointing the young towards easier - and politically convenient - scapegoats. To be clear I'm not saying it's the only problem in France, VERY far from it. I'm also not saying that all old people in France live a lavish lifestyle at the expense of younger generations: there are also plenty of very poor retirees who, if anything, would deserve more. What I am saying is that it's probably the biggest elephant in the room: everyone can see it, nobody dares mention it - no politician, from any side, wants to run on cutting grandma's pension - and it makes every other problem harder to fix.
806
1,694
339
9,852
1,287,325
Halvar Flake retweeted
I take back what I said about OpenAI ... they also just flagged the same question 😂 If you wanted to manufacture a worst case scenario for cyber outcomes, it would be hard to find a better starting point that "Hey! How about we prevent people from asking questions about formal verification?!".
Cyber refusals when asking claude about details of Anthropics own formal verification work. Guys ... come on =/ If you're worried about actual AI safety in cyber, about the worst thing you could do is take an approach that puts technology, and understanding of that technology, into the smallest number of hands possible. Fortunately we have OAI, and open models, but it's frustrating to see the path Anthropic has gone down.
5
2
29
2,333
One thing about abliterated models is that it makes you very impatient with the non-abliterated models’ refusals to do research that they deem “unethical” for basically subjective (often bad) reasons.
12
24
1
314
12,890
Halvar Flake retweeted
On a side note, I'm thrilled Thinky has found Score Centering to be extremely stable! From their docs: "In our internal post-training, this loss function has led to extremely stable training, even under highly off-policy async RL and low precision numerics."
1
2
67
5,418
Halvar Flake retweeted
So we have a hypothesis: StableReinforce = Reinforce + Score Centering, where the expected score is computed over 8 randomly sampled tokens. We can easily verify that it is true: we implement Reinforce + SC as a custom loss in Tinker, do an optimizer step, and recover gradients that are within numeric precision of StableReinforce across all layers.
2
3
1
69
6,212
Halvar Flake retweeted
Looking at a single-token update, surprisingly only 8 logits have a non-zero gradient. In contrast, a standard Reinforce loss would put a non-zero gradient on all logits. This observation basically reveals the objective. In Score Centering, we subtract the expected score under the sampler. Since computing the exact expectation is expensive, we have to resort to an approximation. In our paper, we propose a top-k approximation, which is deterministic but slightly biased. It seems that Thinky instead decided to compute the expected score by randomly drawing 8 tokens from the sampler, which is unbiased but higher variance. Every token’s score has the same dense part (−p, from the softmax), so subtracting the average score over the 8 correction tokens cancels it exactly.
3
2
1
56
8,881
Halvar Flake retweeted
Dziękuję Radzie Miasta Krakowa i Przewodniczącemu Jakubowi Koskowi za gratulacje z okazji wdrożenia JPEG XL na kodowaniu ANS, który powstał w Krakowie na UJ (2006–2013), a dziś zapisuje dane w telefonach, przeglądarkach, serwerach, grach, genomice i w CERN. Miło, że w erze informacji coraz większa jej część jest zapisywana metodą z Polski. @Radamiasta @KosekJakubRMK @JagiellonskiUni
4
7
1
57
1,635
This is absolutely nuts and only hitting German papers so far: leaked emails show that seven politicians from Germany's far-right AfD travelled to Syria to meet with Assad regime officials... facilitated by Hezbollah AND Russian intelligence. handelsblatt.com/politik/deu…
41
1,454
74
4,009
74,638
Halvar Flake retweeted
Perhaps something to consider if you work at OpenAI / Anthropic: If your policies have produced an outcome like 'if a researcher wants to do software verification research then they have use a Chinese model', then perhaps your approach to 'safety' is net negative.
18
62
3
703
20,696
Halvar Flake retweeted
Cyber refusals when asking claude about details of Anthropics own formal verification work. Guys ... come on =/ If you're worried about actual AI safety in cyber, about the worst thing you could do is take an approach that puts technology, and understanding of that technology, into the smallest number of hands possible. Fortunately we have OAI, and open models, but it's frustrating to see the path Anthropic has gone down.
1
4
2
30
4,618
Halvar Flake retweeted
Every generation that doesn't learn about covert channels in school/college (I read about them in a puzzle book in my teens) is going to be surprised by this and then they will think it is shocking and revolutionary because:
1
2
4
1,167
Halvar Flake retweeted
Abliterated models look behind the curtain.
turns out it classifies what audio it hears, before reporting back to samsung. sensor preferences, dog-barking and baby-crying event counters, and camera streaming duration are built into telemetry paths.
14
8
1
285
17,514