matt heard (@mattheard) header
matt heard (@mattheard) profile picture

software developer/philosopher

Berlin, Germany
Joined January 2007
Photo by matt heard (@mattheard)Photo by matt heard (@mattheard)Photo by matt heard (@mattheard)Photo by matt heard (@mattheard)Photo by matt heard (@mattheard)Photo by matt heard (@mattheard)Photo by matt heard (@mattheard)Photo by matt heard (@mattheard)Photo by matt heard (@mattheard)Photo by matt heard (@mattheard)
i think of myself as a pretty grounded person, but i really hope that i'm currently suffering from intense delusions and that we're not actually doing this /1
.@paulfchristiano thinks that on the current trajectory (absent any slowing down), there is a 1 in 3 chance that we could have superintelligence powerful enough to overthrow the government within a year. (He articulates a concrete scenario). He argues that the labs should pause further algorithmic progress until they can secure their weights and research secrets. Otherwise China could simply steal the superintelligence in the middle of the intelligence explosion, and the labs would no longer be able to slow down even if they saw their models going off the rails. He also explains how AIs will be trained during the intelligence explosion, and why even a system trained entirely inside a data center could still have the capabilities profile such that humans would have very little chance of competing with it or controlling it. And he discusses the plan for alignment - basically, making a superintelligence that actually does what we ask the way we mean it. Fixing reward hacks and building better graders will help up till the next few model generations, but he thinks that no single current approach has better than even odds of letting us control superintelligence. The hope is that the AIs figure out the solution to this problem during the intelligence explosion. Paul is head of the Alignment Research Center. Previously, he was head of AI safety at the US AI Safety Institute. Before that, he led alignment at OpenAI, where he invented RLHF. He’s recently joined OpenAI’s board (though here he is speaking as a private citizen).
1
17
over the last few years i have been dealing with some intense personal stress from family stuff and i hope that this stress has caused me to fall into a delusional state about an asi end-of-times me being delusional while the rest of the world spins on is preferable to this /2
5
i'm generally in favour of pausing ai development, but as time passes, almost everybody will start to adopt esr's position trying to balance on the knife's edge of attaining immortality is incredibly unstable a pause needs to address this
I have a confession to make. I am pro-AI and pro acceleration partly because I don't want to die. To be clear, I think it's more important than my individual life that we avoid the grinding totalitarianism that I think would come from centralized government control of AI, from the coercive machinery required to enforce a "pause". But it's also the case that I don't want to die. My mind is still agile and flexible, but physically I'm getting old. I've had cancer. I have 95% five year prognosis, but I'm at the age where that's not very much different from my actuarial expectation of life without cancer. I want life extension and immortality to get here while I'm still alive to pick up the option. Unrestricted AI maximizes my chances of that. Every year of slowdown, conversely, increases the risk that I would die when I should have lived. This is something to think about, if you too are growing old.
36
matt heard retweeted
It's great that OpenAI is doing these assessments but they fall short of Sam's commitment to employee-like access Good access is key for trust because assessors can find things even if a company tries hiding them. Here OpenAI controls information flow so we must rely on good faith Limitations (per Apollo): - No Slack - Only OpenAI-selected staff for interviews - Formal conversations with staff must be logged (harder to freely raise concerns) - METR & Apollo currently have only agreed access to training run information, not internal incidents, etc. Incidents like Hugging Face are some of the most important sources of information we have about alignment I see the value in a pilot before future assessments with deeper access, but then OpenAI should present a plan for how and when they intend to meet their commitment to employee-like access. And either way, OpenAI is presumably able to go further here.
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. openai.com/index/our-first-p…
32
36
6
504
69,213
in issue 4 of infinity gauntlet, thanos uses the gauntlet to create a box around cyclops' head, cutting off his oxygen, but didn't dr strange in issue 3 cast a spell to breathe freely in the vacuum of space?!
27
this is so weird eas and rats appear to me to frequently be skewered for biting the bullet on unusual consequentialist conclusions, but when somebody starts talking about the deontological importance of keeping promises they aren't being consequentialist enough???
This is one reason @METR_Evals is captured. The people there easily enter extortative agreements with the AI companies and then put a great deal of store in honoring them and enforcing that kind of agreement on their ingroup (EAs, rationalists). They are terrified of losing ingroup status *with the AI companies*. They let their status as beggars be enshrined in contracts that prohibit them from sharing the information they were formed to share. And they take pride in being beholden to a much bigger adversary by enforcing compliance even on people like Jacob Coxon, who should clearly just pay the penalty for breaking his NDA. They see being scrupulous about confidentiality as an inherently rewarding reflection of their honor, and so they don’t think about the bigger picture. I get Beth keeping promises she made, but going out of her way to talk up the virtue of others honoring (very beatable) contracts is a reflection of highly manipulable core values that EA/rats love to display. The overall picture that emerges of their worldview is one where the AI labs are the rightful leaders. Bc they had the might to demand secrecy, they have the right to demand secrecy, and honor demands compliance with their will.
1
5
429
matt heard retweeted
Replying to @jay_douglass
I work at a frontier lab, we do not know how to reliably instil values into the AIs, this is the scientific consensus
83
174
54
2,044
127,092
matt heard retweeted
I regret to inform you all that we have found our first long false positive from 2006 it's a poem where the same 6 words are repeated four thousand times
177
276
96
13,442
1,357,734
agreed in the singularity, the consequences of what seems obviously good or obviously bad is likely instead a coin flip be humble about predicting consequences
In the spirit of holding myself accountable by saying in public the things which I say in private, I want to offer a very difficult thing to say - or even think - in this climate: the extreme hawk position on chip exports might turn out to be strategically wrong, and a crackdown on chip exports may be more destabilizing than people realize. Completely cutting China out of the upside of AI development this close to RSI and ASI may be sufficiently threatening to China's interests that it increases the pressure towards a hot conflict in the near term. A hot conflict with China is worse than a world where China remains a near-peer strategic competitor with whom we have tension but still have some room for cooperation. I'd like the AI safety China hawks to think about how to model this as part of arriving at a more carefully-considered position.
29
Replying to @repligate
People think humans are in control. They're not. Nobody saying this ever tried to change the world.
1
34
scenario: dec 26 landslide chip export control measures plus rapid ant model releases through q1 27 brings china to the table for a nobel-prize-winning agreement that saves face for xi but basically makes china a compute vassal
37
matt heard retweeted
I am worried about AI
45
17
2
455
17,299
good post
a number of respected researchers (e.g. @DKokotajlo @bshlgrs @ajeya_cotra @RyanGreenblatt) have suggested that the METR-documented ~2x speedup in AI progress at anthropic over the last year supports the hypothesis that there will be a near-term (e.g. 2027) intelligence explosion (IE). how exactly does this reasoning work? one way to think about this: perhaps the leading alternative to the near-term IE hypothesis is what we might call the factor F hypothesis, which says that there's at least one key factor F [e.g. creativity, research vision, planning ability, ...] such that (i) high F is required for an AI system to reach the superhuman levels required for full-scale IE [e.g. ASI, artificial superhuman general intelligence or at least SAR, superhuman AI researcher], and (ii) F is so difficult to build into AI systems that leading AI systems won't have high F in the near term. i don't know whether the factor F hypothesis is true, but it's not obvious to me why the 2x speedup makes it much less likely. it's widely agreed that the 2x speedup reflects an even larger (perhaps 10x) speedup in other capacities G such as coding and experimentation. under the F hypothesis, G could explode locally, leading to major progress in sub-AGI capacities without much progress in F (think of G as the easy part of AGI and F as the hard part). that tends to lead just to a weak form of recursive self-improvement (call it an intelligence subexplosion) below AGI levels. the AI futures document responding to the METR report is usefully explicit about nearby issues. the authors report believing in a high-ish growth rate for "research taste" (AI research factors beyond coding and experimentation) relative to increases in general capacity. presumably they take potential Fs to be aspects of research taste that will grow fast too. but i don't really see where the evidence counts again a relatively-low-growth F hypothesis where F stays below threshold for years. maybe one could argue that the explosion in G provides bayesian evidence against this F hypothesis (G's exploding is antecedently less likely under the F hypothesis than under its negation), but that's far from clear, and even if true there will remain a solid mass of credence under the F hypothesis. maybe there are other reasons to reject the factor F hypothesis: e.g. because AI research is relevantly like math research, and AI math research is becoming superhuman already. but if we have reasons like that, the 2x speedup isn't really needed to make the case for near-term SAR and IE. maybe the 2x speedup could speed up an expected IE slightly from 2028 (two years away) to 2027 (one year away), and/or maybe it could be extrapolated into ever-faster speedups (another 2x in 6 months, 3 months, and so on) converging in 2027. but those are very different arguments that require huge premises to work. am i missing something here? i'm curious to hear which evidential routes from 2x speedup to near-term IE people have in mind. N.B. i'm not saying you shouldn't worry about near-term ASI. i'm worried!
1
1
22
4,991
new hires are the antidote to cargo cults
I bring a sort of “why do we do it this way” vibe to the work place that most ppl don’t like
33
matt heard retweeted
"what if ai gets too good" i dont know . Who cares. What if bras and panties get too erotic
76
686
55
12,611
963,807
does this count as a personal story?
I’ve stopped reading tweets that are personal stories, as I think most all are made up.
22
me: hmm let's escape the ai discourse for a few minutes by watching cooking videos first cooking video:
28
somebody once cursed me with living in interesting times and you're all coming along for the ride
26