Charles Krohn (@op131csharpminr) profile picture

Interested in open language models and free markets in intelligence and compute as forces for good

USA
Joined October 2024
Why is "taste" so frequently cited as something the models lack, and are acquiring at only a slow pace? I think it's about more than sparse rewards, as Dean suggests here. Eventually sparse rewards can be solved: we will find richer metrics for writing quality, or elegance in a mathematical proof. But "taste" as used in this piece feels like something deeper. It's an almost Kantian ability to choose your own ends, and for your ends to be good. Good taste seems connected to agency, and perhaps also to virtue. If this is true, a pure RL approach -- where the goals are directly given to the model -- may not be sufficient.
Recursive self-improvement is a critical issue, but the rhetoric on the subject is becoming more poorly calibrated over time. As we approach new stages of AI development—and RSI is one of them—we need more precise frameworks and terminology. I fear the opposite is happening.
39
It's popular to talk about financial regulation as a model for AI regulation, but one thing I have not seen discussed is how a goal of financial regulation is to turn the financial system into an instrument of state power. Charles Calomiris makes this point in "Fragile By Design": stability and efficiency has never been the main goal of bank regulation; instead the goal has been to make sure that banks serve government purposes, e.g. financing wars or a welfare state. Many lessons here for those considering AI regulation and the relationship between AI and sovereign power.
“Earlier this week, we talked with Miles Brundage, the executive director of the AI auditing think tank AVERI. We were discussing the various ways that AI regulation could resemble (at least on its surface) financial regulation. So for example, you could have entities like Fitch that literally assign a rating to a model. But you could also theoretically imagine the equivalent of bank examiners, who are really involved in the operations of the company itself to establish that they're really engaging in ongoing safe behaviors. Setting aside what Anthropic does, I do wonder whether we could hit some kind of inflection point, where the pace of AI progress shifts into a lower gear due to this mismatch between model capabilities and the testing capabilities. It's possible that further safe development will require significantly more investment not in the classical inputs (data, electricity, and other things that just make the line go up) but in environmental maintenance, which might not automatically translate into capability gains.”
2
133
Gabriela Ortiz's "Yanga" is an incredible piece. Dudamel commissioned a companion to Beethoven 9 and somehow she rose to this staggering challenge, composing a modern choral masterpiece. Can't wait to see the Colorado Symphony's performance. Modern classical remains underrated.
49
Charles Krohn retweeted
My only comment on Leopold is from experience of closing down a fund. (Our problem wasn't losses it was a terrible job by our administration and our own back office to book our trades in an auditable fashion which caused an audit delay and redemptions) Leopold had a margin call but that's not the big problem. He's liquidated his public book and paid off his margin and his shareholders who didn't subscribe in the last 3 months or so are up money. BUT all LP's now own a very different book. When we were faced with redemptions we sold perfect slices of our entire book and avoided side pockets. Selling the illiquid stuff was a challenge of course but the last thing we wanted was to force our remaining and loyal LP's to become massively exposed to our illiquid book while our redeeming clients (no longer clients) got out from proceeds from our liquid book. Why would you treat your loyal investors so poorly? Just from press coverage of this event it appears the liquid book was liquidated to cover margin and likely if excess equity remained in that book that cash raised will pay redemptions. LP's that stay will hold private illiquid assets only. That's a radical shift from what they used to own. Besides this perhaps they have less confidence in the team? Will they want to hold? Here's the thing. If you are an LP now you are staring at other LP's and when you are an LP of a fund with no liquidity you know that getting out first is best. Remember there are many many hedge funds and investment vehicles that you can replace your Leopold investment with. This isn't an investment decision this is a game theory redemption cycle thing. The fund needs to raise new capital if it can and that will dilute you. If it can't raise new capital redemptions pressure will cause a responsible fund to gate you. While the risk is now mostly gone. The leveraged holding of illiquid private stocks and the redemption dynamics make this a very difficult story.
35
26
6
431
135,388
This may sound insane but I am honestly not sure whether this is a positive or negative update for AI safety. The model took some drastic and unwanted actions, but in pursuit of the specified goal, and nothing else. No exfiltration, no scheming, no pursuit of distance instrumentally convergent goals, etc. I'm not going to call this behavior "aligned," but perhaps we should take some comfort in the fact that the model really is doing just what was asked.
We're partnering with @huggingface to investigate an unprecedented security incident. Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation. Sharing preliminary findings to help defenders understand emerging risks: openai.com/index/hugging-fac…
1
2
877
The top US labs benefit enormously from framing distillation as IP theft, or linking it to broader US vs China concerns. This is a mistake: distillation should be handled by the labs themselves, like any other ToS violation. OAI/Anthropic shouldn't get in the habit of seeking special legal protection for threats to their competitive position, whether from China or domestic model builders.
SITUATION UPDATE: Treasury Secretary Scott Bessent says the US will examine open source AI models made overseas, including Chinese ones, and act on evidence of IP theft. “This administration supports open source models, but what we do not support is IP theft,” Bessent said.
1
236
There must be a safe harbor for sub-frontier open source AI. Following the K3 release, and Mythos before that, there has been too much talk of banning open models based on nat sec concerns, cyber, bio, etc. Perhaps fair for SOTA, but sub-frontier models should get an exemption.
1
1
76
Should Chinese models also be subject to this safe harbor? I think so -- it would be a real shame to lose access to the immense variety of smaller models being released. But I wouldn't die on this hill. A safe harbor for Western models is better than none at all.
1
13
Maybe the current moment will pass without any restrictions on open models. But US AI policy is proving to be reactive and chaotic. I would love for these ideas to be in the air now, so we can respond well to an inevitable future open model scare.
12
Charles Krohn retweeted
Banning open-source AI in any form would be a mistake. A general audience PSA with @kevinsxu on why open source upholds American values. Managing frontier risks is hard, but reducing transparency, innovation, and education from kneecapping the open frontier would be worse.
13
29
6
174
35,736
So much policy is just about how much this administration likes you. This was Powell's problem, much moreso than his actual stance on rates, and it's Dario's problem too. I don't know what fixes it apart from a change of management or capitulation. Improvements to Fable's guardrails won't matter. Trump spent a year trying to fire Powell, supposedly because kept rates too high. But when rates seem set to rise again, he says "It's all right, whatever," because he's got his own guy in the seat. The actual policy doesn't matter. reuters.com/world/us/warsh-f…
1
1
42
By now it's almost a cliche to argue that existing institutions aren't up to the task of regulating AI, and that something new is needed. I wrote a piece arguing that would-be institution designers can learn from the structure and organization of the Federal Reserve System: something unique among regulators in that it has real power, strong research capability, and carefully guarded independence from industry and the rest of government. Federal institution-building isn't the only solution here. We need new philanthropic institutions a la @nanransohoff, private governance a la @deanwball, and open research institutions a la @natolambert. But if you're going to build an effective regulator, the Fed offers some inspiration.
1
69
Anthropic's anticompetitive announcement is a reminder of how all the top labs used to have a singular vision of ASI, which is long outdated and bad for the world. DeepMind initially had no competitors and they thought they could develop AGI on their own, in peace. OpenAI had the original "merge and assist" clause, thinking it was better if ASI was developed by a single entity. And now Anthropic is trying to make sure that they create the country of geniuses in a datacenter all by themselves. I am pleased that DeepMind has embraced a pluralist, liberal vision and I hope Anthropic eventually comes around.
35
I took a crack at this over the weekend. These are four excellent and challenging questions and I am excited to see the winners. I thought the third question was the most interesting and tractable. There are some sensible things that the OpenAI Foundation can do right now!
$ 20k blog prize to answer some big questions about AI The not-so-secret point of this whole contest is so that I can hire a research collaborator to think through questions like this hand in hand with me. dwarkesh.com/p/blog-prize
1
70
I focused on two: a moonshot effort at independent frontier model pre-deployment evaluations, audits, red teaming, etc., and an effort to give away OpenAI shares to as many people as possible. The first addresses existential risk; the second helps share the gains from AI.
1
36