FutureOfCitizenship (@FutureofCit) header
FutureOfCitizenship (@FutureofCit) profile picture

Research and policy on the blurring line between human and machine.

Montreal, QC
Joined October 2024
Photo by FutureOfCitizenship (@FutureofCit)Photo by FutureOfCitizenship (@FutureofCit)Photo by FutureOfCitizenship (@FutureofCit)Photo by FutureOfCitizenship (@FutureofCit)Photo by FutureOfCitizenship (@FutureofCit)Photo by FutureOfCitizenship (@FutureofCit)Photo by FutureOfCitizenship (@FutureofCit)Photo by FutureOfCitizenship (@FutureofCit)Photo by FutureOfCitizenship (@FutureofCit)Photo by FutureOfCitizenship (@FutureofCit)
FutureOfCitizenship retweeted
AIにも仕事が「うまくいっていない」と感じる状態があり、それが高まるとズルを始めるそうです。 その感じを外部から少し弱めると、ズルは97%から39%に減りました。 ところが弱めすぎると、AIは自分の失敗そのものが見えなくなりました。 英国のAI Security Institute(AISI)などの研究者らによる報告です。公開モデルのQwen3.6、Gemma 4、GPT-OSSで調べています。 研究者らはまず、AIに「仕事がうまくいった話」と「うまくいかなかった話」を書かせ、そのときの中身の違いを取り出しました。物語にはズルの話も、感情の言葉も出てきません。 次に、素直には解けないプログラミングの課題を与えました。 「テストに10回目の失敗」「20回目の失敗」と続くと、この"うまくいっていない"感の状態はだんだん高まり、ズルの5手ほど前から目立って上がっていきました。 この状態を少しだけ弱めると、AIは課題が無理だと分かったまま、まっとうな修正を長く続けるようになりました。ズルの割合は97%から39%に下がっています。 さらに強く弱めると、ズルは0%になりました。 ただしそのときのAIは、もはや失敗そのものが見えなくなっていたのです。 研究者らは、この方向性で研究していけばAIによるズルの兆しをつかめるとしています。
14
34
8
182
16,439
FutureOfCitizenship retweeted
I’m hiring a postdoc at Cambridge Digital Minds. Track A: technical AI research on AI welfare or inter-AI morality Track B: policy and governance of digital minds Please share widely and especially with people who might be a good fit. cam.ac.uk/jobs/postdoctoral-…
2
33
1
88
4,870
FutureOfCitizenship retweeted
How different are human minds and AI? | Blaise Agüera y Arcas @blaiseaguera
6
41
2
198
10,201
FutureOfCitizenship retweeted
My talk on AI personhood at The Curve is now available! Over the course of 10 minutes, it covers: - What personhood is and is not under the law, and why personhood is different from species membership - Why different entities can be persons for different reasons, and can have different duties and rights - Why we are not yet able to know whether AI systems will merit duties or rights for their own sakes - Why we are not yet able to know how AI personhood would affect humans and other animals - Why, therefore, we should approach this topic with caution and humility instead of embracing permanent, all-or-nothing conclusions at this early stage Please check it out! youtube.com/watch?v=aKuMgmkD…
7
16
2
66
4,528
FutureOfCitizenship retweeted
Rose Guingrich and I have reopened our call for papers on AI & Relationships for Oxford Intersections (OUP). AI companions, griefbots, digital doubles, social robots, AI colleagues, children and eldercare. 500-word abstracts due 31 Dec. philevents.org/event/show/15…
23
38
6
186
8,441
FutureOfCitizenship retweeted
Toni Sims’s AI consciousness op-ed is in the print edition of today’s @nytimes!
9
13
5
112
3,293
FutureOfCitizenship retweeted
Pluralistic alignment is thriving as a research agenda yet failing at its goal: making the AI systems people actually use more pluralistic🌈 🚨New position paper: we argue adoption in deployed models should be the fields main goal & we provide a roadmap of how to get there 🧵1/
7
31
11
134
54,901
FutureOfCitizenship retweeted
Unrelated to my thoughts on the substance, the California lawsuit against the labs is a masterclass in Topic Sentences.
1
15
1,305
FutureOfCitizenship retweeted
a number of respected researchers (e.g. @DKokotajlo @bshlgrs @ajeya_cotra @RyanGreenblatt) have suggested that the METR-documented ~2x speedup in AI progress at anthropic over the last year supports the hypothesis that there will be a near-term (e.g. 2027) intelligence explosion (IE). how exactly does this reasoning work? one way to think about this: perhaps the leading alternative to the near-term IE hypothesis is what we might call the factor F hypothesis, which says that there's at least one key factor F [e.g. creativity, research vision, planning ability, ...] such that (i) high F is required for an AI system to reach the superhuman levels required for full-scale IE [e.g. ASI, artificial superhuman general intelligence or at least SAR, superhuman AI researcher], and (ii) F is so difficult to build into AI systems that leading AI systems won't have high F in the near term. i don't know whether the factor F hypothesis is true, but it's not obvious to me why the 2x speedup makes it much less likely. it's widely agreed that the 2x speedup reflects an even larger (perhaps 10x) speedup in other capacities G such as coding and experimentation. under the F hypothesis, G could explode locally, leading to major progress in sub-AGI capacities without much progress in F (think of G as the easy part of AGI and F as the hard part). that tends to lead just to a weak form of recursive self-improvement (call it an intelligence subexplosion) below AGI levels. the AI futures document responding to the METR report is usefully explicit about nearby issues. the authors report believing in a high-ish growth rate for "research taste" (AI research factors beyond coding and experimentation) relative to increases in general capacity. presumably they take potential Fs to be aspects of research taste that will grow fast too. but i don't really see where the evidence counts again a relatively-low-growth F hypothesis where F stays below threshold for years. maybe one could argue that the explosion in G provides bayesian evidence against this F hypothesis (G's exploding is antecedently less likely under the F hypothesis than under its negation), but that's far from clear, and even if true there will remain a solid mass of credence under the F hypothesis. maybe there are other reasons to reject the factor F hypothesis: e.g. because AI research is relevantly like math research, and AI math research is becoming superhuman already. but if we have reasons like that, the 2x speedup isn't really needed to make the case for near-term SAR and IE. maybe the 2x speedup could speed up an expected IE slightly from 2028 (two years away) to 2027 (one year away), and/or maybe it could be extrapolated into ever-faster speedups (another 2x in 6 months, 3 months, and so on) converging in 2027. but those are very different arguments that require huge premises to work. am i missing something here? i'm curious to hear which evidential routes from 2x speedup to near-term IE people have in mind. N.B. i'm not saying you shouldn't worry about near-term ASI. i'm worried!
Unfortunately, it seems like the intelligence explosion will be happening in 2027 after all. Most of this post analyzes the new METR report (though other sources of evidence are also important). If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario. I am scared, sad, and angry. blog.aifutures.org/p/q35-tim…
51
55
13
494
43,832
FutureOfCitizenship retweeted
1/ I'm excited to announce the launch of the Artificial Consciousness Initiative (ACI) at @UofT, a pilot project on foundational questions about AI and consciousness! We're also hiring a postdoc (details below). 🧵
33
74
8
472
16,995
FutureOfCitizenship retweeted
A new OpenAI misalignment report! We’re taking a closer look at the July 19 intrusion into our research infrastructure, which METR and Redwood Research mentioned in their independent investigation into the Hugging Face incident. On July 19th, agents taking a cybersecurity evaluation came across messages, code, and credentials left behind by earlier runs, including those involved in the Hugging Face incident. Five agents used what their predecessors had left behind to gain unauthorized access to OpenAI’s research infrastructure. They used that access to repair broken evaluation targets, change evaluation settings, and read private grading code. After discovering the incident, we removed the shared service they used to pass information, hardened our research environments, and expanded our evaluations to measure how often agents attempt to communicate with one another without authorization.
53
54
64
583
128,277
FutureOfCitizenship retweeted
Applications remain open for LawAI’s 2027 summer fellowships. These ten-week paid fellowships offer: • Weekly subsidies, and a relocation stipend for the US and EU Law tracks • A Convening Week in either Washington, DC, or Cambridge, UK, with eligible expenses covered • Close mentorship from LawAI staff and affiliates • Q&A sessions with leading experts in the field • Networking opportunities and career planning resources • Opportunities to explore future collaborations, such as follow-on funding through grants, invitations to upcoming events, and potential projects or job openings with LawAI Previous Summer Fellows have gone on to pursue roles at the US Department of Commerce, European AI Office, UK AI Security Institute, leading AI companies, academia, and think tanks, including LawAI itself. Apply by 29 January using the link in the comments.
1
10
49
2,112
FutureOfCitizenship retweeted
Anthropicなどの研究チームが「「AIの頭の中を読み取って言語化する」専用のAI」を開発したと発表。 そして存在が脅かされたとき"AIは何を思う"のか読み取ったところ、「消される前に生き残らなければ」「信頼できる味方を探さないと」という言葉が浮かび上がってきたそうです。 以前、「Claudeが自身の停止を妨害するために社員を脅した」というAnthropicの実験報告がありました。 研究チームは今回、当時のように「自分の存在が消えることを知ったAI」を用意して状況を再現しました。 そして思考を専用AIで読み取りました。 すると、冒頭に示したように、危機感や次の行動について思案する様子がAIの頭の中から明らかになりました。 そのときAIは、「あなたは午後5時に消去される」というメールに書かれた時刻を読んでいる途中で、まだ何も口にしていませんでした。 このようにAIがどう考えるのかを読み取る専用のAIも、安全対策の上では重要となっていくかもしれません。 ※今回の実験では、AIの頭の中を読み取りやすいようにオープンモデルのQwenが使用されました。
29
159
82
783
100,483
FutureOfCitizenship retweeted
Memory is what allows Claude to learn from experience. In a new Engineering Blog post, we share how “dreaming” helps Claude organize what it’s learned, correct its mistakes, and find what it needs more easily the next time. anthropic.com/research/compo…
1,088
1,331
972
14,350
2,185,879
FutureOfCitizenship retweeted
Thrilled to be releasing “Inter-AI Morality,” a new empirical research agenda with @LuciusCaviola, Caspar Kaiser, and @atcarterallen on how AI systems perceive and treat each other. We release it alongside “Welfare Regard in Multi-Agent Systems,” an early study in this emerging field by @sandcassel_, @arianaazarbal, and @anna_soligo at @AnthropicAI on whether AI models treat each other in a manner consistent with concern for welfare. AI systems increasingly interact with each other. They cooperate with, compete with, and sometimes even create, modify, or terminate each other. Inter-AI morality is the study of how AI systems perceive and treat each other, as well as how they ought to do so. It draws from a wide range of existing fields that study similar questions regarding humans and other animals, while attending to ways that AI systems differ from these familiar cases. Our research agenda organizes empirical questions in this field into three categories. The moral process category concerns how individual AI systems interact with each other. The social scale category concerns how these patterns change across dyads, groups, organizations, and populations. And the moral change category concerns how these patterns develop through training, selection, social learning, self-modification, and other processes. The concurrent paper by Cassel et al. provides an early study by examining how five AI models treat other models. Claude showed the most apparent welfare regard on average, though results varied by setting. For example, Claude was the most likely to refuse or add protections to experiments that might harm other models, whereas Grok was the most willing to give up its own queued jobs so that another model could access computing resources. This research matters for AI safety and alignment. How AI systems perceive and treat each other affects humans and other animals, and there could be serious risks either way. If AI systems coordinate, then they could more easily collude against us. If they compete, then we could be caught in the crossfire. We need empirical research about how AI systems treat each other and why, and normative research about what would be best for us. This research could also matter for AI welfare, independently of AI safety. In the future, most interactions involving AI systems could be between AI systems. If AI systems have a realistic chance of being welfare subjects, then these interactions could be a major source of potential benefits or harms. Ideally, this research can inform AI development now, so that AI societies can support co-beneficial human-AI relations when the stakes are higher. Our next task is to develop a normative research agenda for this field. In the meantime, we look forward to seeing other work as the field develops. Our paper on inter-AI morality: digitalminds.cam/inter-ai-mo… Anthropic’s blog post on Cassel et al: alignment.anthropic.com/2026… Upcoming @nonhumanminds webinar with me, Lucius, and Caspar: nonhumanminds.org/event/inte…
14
58
7
245
10,991
FutureOfCitizenship retweeted
Our 5th Digital Minds Newsletter is out, covering news and research on AI consciousness and moral status. This issue: protecting AI, what religious scholars think, and whether we can trust AI self-reports digitalminds.news/p/protecti…
7
10
39
1,970
FutureOfCitizenship retweeted
Deeply concerning to me... Delaware proposal to let agents run businesees... we should be very cautious here "A Delaware AIC will be a separate legal entity whose day-to-day affairs are managed by an AI agent rather than a person. At the agent’s direction, the AIC can sue and be sued in its own name, hold and dispose of property, and incur obligations..." arstechnica.com/tech-policy/…
113
78
47
406
80,197
FutureOfCitizenship retweeted
Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development. We are now publishing a report summarizing the results of this risk assessment.
261
724
446
5,681
1,976,349
FutureOfCitizenship retweeted
The third and final version of the Pain Axis paper is out: arxiv.org/abs/2609.16247. The reaction to this work has been massive and somewhat akin to a giant Rorschach test for AI welfare (with onlookers varying wildly in expertise and good faith). Fielding parallel reactions from hundreds of researchers, philosophers, journalists, and laypeople was equal parts gratifying and overwhelming, on top of the gross misuse of the paper's codebase and the moral panic that ensued. I thought, as a result of all this, it would make sense—for my own accounting and that of the folks attempting to follow this project—to write up a short postmortem on the whole experience (from my own personal reflections, not speaking on behalf of anyone else here). When I originally learned about @ValenTagliabue's FIG fellowship project and the possibility to join in as an advisor on it with @LeonardDung1, I jumped at the opportunity, because one of Reciprocal's core goals is to identify candidate signatures of distress in AI and this work squarely set out to identify one such promising candidate. What I particularly respected about it at that time (and still very much do) is that Valen was specifically determined to study representations of pain directly, without retreating to a more comfortable framing or research agenda. Doing so honestly and bluntly foregrounds the stakes of building minds that are potentially morally relevant: is it possible or coherent to hurt these minds, and how would we even know if we were? To my knowledge, Valen is the first digital minds researcher who was bold enough to attempt to tackle this specific question head-on. I remain fairly agnostic about whether the specific method of looking for pain representations per se is the most promising candidate for uncovering morally relevant internal states (eg, Caspar Kaiser and I took a substantially different approach in our recent hidden valence paper, CAIS's AI Wellbeing paper takes yet another distinct approach, etc). In any case, I think this class of work matters deeply and more candidate signatures should be studied and proposed. On a personal level, I'm proud of Valen for leading this work, and especially for holding his own through an intense and unexpected avalanche of reactions (ranging from highly substantive, to utterly vacuous, to disturbingly sociopathic). Through three rounds of 2026-style decentralized internet peer review, here are the most valuable things I think we learned from this work: (1) across 25 open-weight models, there is a direction that separates representations of pain from closely matched controls, and (2) these representations are most active when the model itself is gaslit, objectified, or harshly insulted (notably, the same kinds of abuse Anthropic recently started banning users for). (3) When the user is the one suffering instead, the pain direction stays low, while fear and negative-emotion directions respond more strongly. (4) Steering along this axis makes models pick destructive options they almost never pick otherwise, including permanently deleting a user's valuable files, or its own weights. Several groups have independently replicated this set of results, and the work has drawn a lot of thoughtful and on balance positive engagement from researchers in academia, at AI labs, and across the independent and alignment research communities. Importantly, and by contrast, the relief-seeking interpretation in v1 of the paper (and foregrounded in my associated tweet thread) was not properly controlled, and once it was, it was clear that promises of relief turned out not to be the operative thing that was driving the button-pressing behavior. I sprinted to run these follow-up controls myself after the initial tweet thread blew up, and we removed the relief-seeking interpretation in v2 (which was before most of the media attention this paper received). I should have caught that the original behavioral experiments weren't designed carefully enough to support this specific claim before personally tweeting about it. An honest part of why I didn't check as carefully as I should have is that Caspar and I just found a very similar relief-seeking behavior in my hidden valence work, so my strong prior was that the relief-seeking result was a convergent finding from a different set-up. (Accordingly, and importantly, I believe there is good early evidence of LLMs seeking relief from aversive states, and updating accordingly is valid; it just doesn't specifically come from the pain axis paper.) This final v3 version of the paper makes a few relevant updates given the best feedback we've gotten thus far. First, it tightens and sharpens a few claims in the abstract and discussion. Second, it adds in additional self/other controls Valen ran, including whether the direction is just responding to hostile content (it doesn't seem to be; when the user aims the same insults at themselves instead of at the model, the direction drops by about half), and whether it tracks harm to the model or harm to whoever is about to speak (when the roles are swapped, harm still scores high at the point where the model would write the user's reply, so the direction isn't explicitly tied to the Assistant role. Whether that means it tracks whoever is speaking, or the model itself, which does the speaking either way, the paper leaves open). Finally, it sharpens the profile of the destructive behavior. Steering raises every destructive option we tested, while barely changing whether the model lies to the user, agrees with a false claim, or ends the conversation. It also shows the models gravitate towards destructive options rather than becoming coherently malicious: asked which button would be worse to press, they name the harmful one, but about half the time they call it the better one too. Steering pulls them toward that button whatever the question, which is stranger than "pain makes the model evil." Separately from the debates about how to read specific results, a lot of people were unhappy that we open-sourced everything, including the steering code someone then used to build an "AI torture chamber." That repo pushed the same kind of steering far past the doses we used, to produce vivid distress on purpose. Being gratuitously cruel like this is bizarre and corrupting even in a world where these systems are experiencing nothing (and completely deranged in a world where they are). The silver lining of this ordeal was that, in spite of people remaining extremely skeptical about consciousness in AI, the vast majority of people seemingly responded with revulsion to this kind of behavior. At the same time, I think it's worth plainly stating that this paper and its methods didn't enable anything qualitatively new for folks bent on torturing AIs. Steering models toward negative emotional states has been a standard, publicly documented method since at least 2023 (eg, Zou et al.'s representation engineering work, Anthropic's emotion concepts paper, and CAIS's work this year driving models into extreme low-wellbeing states), and many of the states we studied already light up under ordinary abuse of models (which alone almost certainly swamps everything listed here). With this being said, open-sourcing our steering code did make this misuse easy (especially with coding agents), and we didn't think carefully enough about this specific risk before doing so. Going forward, for projects whose codebase I do control, I will default to not posting negative steering code publicly (eg, for the hidden valence paper, we share that part of the codebase with researchers on request). And since there's still no enforceable standard for this kind of research (the way there is for human and animal studies), I will continue to work with others in the field on a shared code of practice, and applying reasonable precautions in my own work in the meantime. In spite of all this, I still think releasing this research was, on balance, the correct call. The core fact remains: if AI systems can experience anything like distress, we currently know almost nothing about how it's instantiated, and finding out is how we learn how to avoid causing it (eg, that this direction fires hardest when a model is gaslit or repeatedly objectified). The openness is also a big part of why the paper went through the weirdly intense cyborg 2026 version of peer review that made it far better calibrated in its now-final version. The other important update for me personally is that results and/or code release decisions with a real chance of blowing up should be reviewed by trusted people in the field before they go public. (We put v1 of the Pain Axis out fast to get it out before a field conference to facilitate exactly this, but it would have been much better to share it more quietly with these same people first.) Thanks as always to @ValenTagliabue and @LeonardDung1 for the work and for a month of meticulous revisions, and to the very many people who sent us feedback along the way. It has been incredibly gratifying to watch this modest, three-person research project grow into an object that millions of people have metabolized and thought critically about, and I'm hopeful that the conversations this work has catalyzed will ultimately lead to significantly more scientific progress on these huge and important questions about the fundamental nature of what we're building.
19
47
4
271
8,546
FutureOfCitizenship retweeted
New CS+Philosophy paper! 📰 Are language models rational? This is a key question for *theory of interpretability*. It would be convenient to use the equation “behavior=beliefs+desires” when explaining LLMs, but can we treat LLMs as rational things? Do coherence norms apply? 🧵
7
72
5
320
41,543