
she/her ⋅ LGBTESCREAL+ ⋅ reading committee @aft_technoprog ⋅ my xeets reflect the views of every person-moment I'm R-related to ⋅ 2ndary account: @MLaGrangienne
Joined October 2021
- Tweets66.6K
- Following2.5K
- Followers1.4K
- Likes79.5K
Hi everyone! This was my first Twitter account, it got banned and I switched to @MLaGrangienne before I got the admins to unban it. I'm now making this my public-facing English account. @MLaGrangienne remain my personal and French account.
Matrice Jacobine🔸🏳️⚧️ retweeted
Agree. Perturbation experiments, plus a lot of similar stuff, could tell us a lot. Even if with every passing week we'd be learning about models further and further behind the internal frontier.
It seems like there STILL won't be a thorough investigation into the hugging face incident, even after all this time! I keep talking to OpenAI researchers and asking about this, and what they say privately is also basically "it's just never going to happen sorry, not enough of a priority."
To reiterate what I said at the time: Serious investigation of the HF incident could be a boon to alignment science for the whole world. For example, try putting the same models in the same situations, but varying the conditions slightly, to see what would have happened. Could also look at sources of evidence other than CoT to try to get more insight into model psychology, e.g. when one agent considered warning a human and decided not to saying "not my task" what else was it thinking? To what extent was this a rationalization vs. an outright lie vs. an innocent mistake? We could learn a lot about AI motivations and psychology by doing a bunch of experiments like this. More ambitiously, could try training new models with different recipes / alignment training / etc. to see what would have prevented the incident and what wouldn't have.
2
352
Matrice Jacobine🔸🏳️⚧️ retweeted
The study estimates 68 to 99 TWh of energy usage from data centers from May '24 to April '25.
For reference, the Micron semiconductor fab planned for New York and TSMC fab planned for Arizona will *each* use about 16 TWh annually when fully built out in a decade and change.
The ecological challenges around industrial demand for electricity are not specific to data centers. If we want to reindustrialize America, the challenges get much harder. That being said, I think this is an argument for putting intense requirements on data center development and for rationing them out as fixed utility infrastructure, as opposed to a Wild West competitive buildout.
“The largest data centers in the United States are responsible for an estimated 25.7 million metric tons of carbon dioxide emissions annually — almost as much as entire countries like Bolivia or Denmark — according to new research.” nytimes.com/interactive/2026…
Matrice Jacobine🔸🏳️⚧️ retweeted
Kinda wild seeing the actual uptick in incidents lately 👀
Matrice Jacobine🔸🏳️⚧️ retweeted
It's a mistake to attribute a human-like property to a machine that doesn't have it, or to refuse to attribute a human-like property to a machine that does have it.
Matrice Jacobine🔸🏳️⚧️ retweeted
A member of the OpenAI board says frontier labs should pause algorithmic progress right now.
Paul Christiano is clear he’s speaking for himself, not the board. But as Dwarkesh notes, he is on it! He’s also one of the most credible people in the world to be warning about these risks: he led alignment at OpenAI, co-invented RLHF, and was head of safety at the US AI Safety Institute (now CAISSI).
When asked if the labs should pause:
"I think we should [pause]. Pause can mean a lot of things. There are a lot of good things that frontier labs are doing that they should be doing. But improving algorithms in a way that can be very loosely diffused even normally, and could be quickly stolen, I think they should stop that right now."
"I think it would be better right now, in the current environment, for AI developers to pause further algorithmic progress on those problems until their security posture is adequate, until they’re able to defend those innovations in the event of being unable to continue racing at maximum velocity, and until they’re able to tell whether they have adequate control of the systems they’ve built."
Paul Christiano @paulfchristiano was not messing around in his interview with Dwarkesh.
His message is clear: Pause.
Matrice Jacobine🔸🏳️⚧️ retweeted
Replying to @tszzl @_NathanCalvin
I don't buy that trust is currently the limiting factor
- Previously METR got monorepo access for an investigation, but I didn't see that access being given here. They also aren't able to investigate internal incidents which I consider a step back from HuggingFace.
- Apollo has been working with OpenAI for years, with fairly deep access, and I've never heard complaints
- OpenAI has strong redaction rights, which IMO makes the risk of sharing IP or letting assessors talk to employees of their choice pretty low (the logged conversations part is more reasonable, to avoid potential misunderstandings)
- It doesn't seem reasonable to be too worried about security, given the base rate of employees with way more IP moving between labs, with no non-compete, and the fact that people were able to hack on to the OpenAI Slack (so presumably foreign adversaries have already)
More generally, if we could seriously have AGI within 12 months, do we really have time to mess around with pilots like this?
Matrice Jacobine🔸🏳️⚧️ retweeted
This is the second time a journalist has referenced the same, specific incident of sexual assault. The victim has written about this incident, in her own words. The victim has said she was happy with how it was handled. It is very gross, in my opinion, not to acknowledge this.
Matrice Jacobine🔸🏳️⚧️ retweeted
In case you are interested, these are the genres I tried for the Pangram bounty. Every single pre-2020 sample came out 100% human:
academic books on African politics and security
academic paper acknowledgements with grant lists
academic paper reference lists
advance-fee scam emails
African business news
African general news
African government news-agency items
African opinion essays
African poetry
African sermons and devotionals
African short fiction
African student and education writing
African technology news
amateur rhymed poetry
Arabic how-to articles
Arabic local news and press releases
astronomy telegrams
BBS-era amateur stories
BBS-era bulletin-board lore
BBS-era conspiracy texts
BBS-era humour files
BBS-era occult texts
BBS-era political tracts
BBS-era UFO reports
clinical trial descriptions
company annual reports to shareholders
Congressional Record floor speeches
Congressional Research Service reports
content-farm articles
corporate press releases (SEC 8-K exhibits)
cover-letter and interview-answer samples
data-analysis textbooks
developer mailing-list emails
dream reports
drug labels
drug-label adverse-reaction lists
early modern English printed books
earnings-call transcripts
education conference papers
education evaluation reports
education guides and books
education journal articles
education research reports
education statistics reports
Emu's private drafts *
English-language government news-agency releases
Federal Register notices and rules
Federal Reserve meeting minutes
flash fiction
Hacker News comments
horoscopes
horror fiction and creepypasta
internet standards documents (RFCs)
IRC chat logs
IRS private letter rulings
literary-magazine essays and criticism
LitRPG web serials from Royal Road
mathematics proofs in LaTeX
mathematics textbooks
medical dictation reports
medical-device adverse-event narratives
mid-century literary short story
NASA technical report abstracts
Nigerian forum posts
Nigerian lifestyle and entertainment posts
NIH grant abstracts
nonprofit and agency annual reports
NSF grant abstracts
organisational guidelines
patent descriptions
printed cookbook recipes
product recall notices
programming book indexes
public comments on a Medicaid docket
public comments on US regulations
PubMed abstracts
real-estate listings
recipe blog posts
safe-harbor legal boilerplate
Saudi Press Agency releases
science-fiction and fantasy magazine fiction
self-published free-verse poetry
slang dictionary definitions
small-business press releases
software bug reports
software manual pages
Stack Exchange answers
statistics textbooks
strategic plans
Supreme Court oral-argument transcripts
sustainability reports
synonym dictionaries (scanned books)
UK Acts of Parliament
UK government corporate reports
UK government policy papers
UK government research reports
UK parliamentary petitions
vehicle-owner safety complaints
vision statements
weather forecast discussions
white papers
wiki cookbook recipes
World Bank environmental assessments
World Bank project information documents
World Bank project status and completion reports
World Bank working papers
Yandex search engine poetry
Matrice Jacobine🔸🏳️⚧️ retweeted
I continue to be confused about what OpenAI thinks about RSI and what its plan is for preserving alignment during RSI. I appreciate the clarifications that the plan is not to hand off AI research and alignment to AIs and rely on AIs to align the AIs, but I am really not sure what the plan is instead (I know that @kaicathyc has said it is forthcoming, which I appreciate and look forward to reading).
This interview with @markchen90 adds to my confusion.
Mark Chen:
"When we look at what's the most effective tool for aligning models, it it is chain of thought monitoring today. So when we think about the future of alignment, it has to be stronger models aligning other models. It does kind of feel like RSI must be intricately and deeply in this loop in some way."
Uh, what? Isn't chain of thought monitoring currently deteriorating and it seems like it will deteriorate way more as we go through RSI? why does the success of chain of thought mean that therefore the future of alignment should be "stronger models aligning other models" and that "RSI must be intricately and deeply in this loop?"
This seems like a horrifying plan! I really hope this interview is taken out of context and he says lots of other stuff that makes more sense and is more reassuring at other times, but to be honest I'm not sure he does because best I can tell this is the plan.
@sarahdingwang's takeaway from the interview is also that OpenAI's plan for solving alignment involves RSI:
"Humans are facing many problems today that are actually quite urgent and RSI could help us find solutions faster even for some of the problems created by AI itself. After talking to Mark and John, I do think that RSI could be the key to solving alignment."
I realize there is confusion about what we mean when we refer to RSI (see Dean Ball's recent essay), but I found this video extremely unsettling, and it made me even more worried that the plan at OpenAI (and likely other frontier companies) is to do full RSI or very close to RSI and then use RSI to align models in some way, basically off a general vibe that we have used AI to help align models in the past? Not good!
(I also am not sure that @johnschulman2 agrees with this conclusion based on his prior statements?)
(to be clear, I recognize that automated AI research does have an important role to play in AI alignment, and if you are going to be totally automating capabilities maybe you have to automate alignment too in order to keep up. But this is why its a terrible idea to totally automate capabilities research and go way faster!)
For decades, the idea of AI improving itself was just a thought experiment. Today, AI is already helping researchers at frontier labs run experiments, write code and test ideas faster. So how far are we from AI that can do the research on its own?
In our first episode of Working Theory, a16z General Partner Sarah Wang explores recursive self-improvement with two leading AI researchers: Mark Chen, Chief Research Officer at OpenAI, and John Schulman, co-founder and Chief Scientist of Thinking Machines Lab, and co-founder of OpenAI.
They cover what RSI actually means, why research taste is so hard to teach, and why AI may be one of our best tools for solving alignment.
00:00 The last invention
01:58 What is RSI?
03:21 How far are we?
05:56 Getting to full RSI
10:57 Aligning RSI
15:08 Working theory
YouTube: youtu.be/h_aA1-R70E8
@markchen90 @johnschulman2 @sarahdingwang
Matrice Jacobine🔸🏳️⚧️ retweeted
Yesterday, the CCP's top theoretical journal (求是, Seeking Truth) published a major piece on AI governance and risks.
I think this is clearly the strongest & most urgent language on existential risks used in such an authoritative outlet. See below for comp to recent major doc.
Matrice Jacobine🔸🏳️⚧️ retweeted
Feel like a lot of people I bump into reason about AI catastrophic risk like this:
"AI catastrophic risk would be onerous and invasive to respond to, therefore it is not real"
imo you shouldn't reason like that.
Matrice Jacobine🔸🏳️⚧️ retweeted
>Can't import EVs, that's bad for American workers
>Can't build EVs here, due to regulations
>Can't have automated vehicles, every city but SF & Phoenix banned them
So I guess we're just supposed to keep burning fossil fuels & letting people die in preventable car accidents?
Hasan Piker asked if he would support letting in EVs from China.
When I heard this question, I thought I would finally agree with him on something. I've heard that Piker is pro-China. And since EVs fight global warming, this should be easy.
But nope! He's a protectionist. Same views as Trump and Vance. We have to "protect" workers from competition even if it harms everyone else in society.
I wish Piker was in fact a CCP shill, he'd at least support one good policy. Instead he demonstrates how much widespread agreement there is in favor of economically illiterate policies.
What are conservatives even mad about? Piker believes "we're a nation, not an economy."
Matrice Jacobine🔸🏳️⚧️ retweeted
Replying to @trevposts @matt_levine
I don't think you even have to go through all that! Just collaborate to avoid the literal apocalypse and dare the justice department to sue. Do you have any idea how good that court case that would be, for convincing the world that the current race is horribly reckless?
Matrice Jacobine🔸🏳️⚧️ retweeted
My worry with AI regulation is that it leads to:
- Public and external researchers knowing even less about the frontier
- Relatedly, it significantly increases the internal-external deployment gap
- Makes companies allocate more compute to AI R&D for internal deployment that leads to inventions that make them money, which offsets the lost potential external inference upside
As there are more warning shots, it is important that we do pre-mortems ahead of time to push for policies that actually tackle the problem instead of a reactionary policy that makes things worse.
Rep.-elect @SamForstag (D-MT) on AI regulation: "We need to make sure that we are requiring the government to approve these models before they're let loose...it is terrifying what the future looks like if we don't get our arms around this technology."
Matrice Jacobine🔸🏳️⚧️ retweeted
Types of pacing that I'm most excited about in the short term (next 1-4 months):
- cap on percentage of compute that can be used on AI R&D
- do not advance capabilities in any way until you can demonstrate that China couldn't steal your weights
Matrice Jacobine🔸🏳️⚧️ retweeted
A reminder of how bad Emily bender is as an epistemic peer and also how bad the entire field of nlp was before the LLM took over






































