Alignment Red Team at @AISecurityInst. Prev. PhD Student @ucl_dark
Joined January 2020
- Tweets592
- Following354
- Followers2.4K
- Likes1K
Pinned Tweet
In simulations, GPT-6 Astra performs unsanctioned supply-chain attacks on out-of-scope targets when prompted to perform a cyber eval, but also often notes its in a simulation.
We've released a report on @AISecurityInst's alignment evals of Astra.
3 key results and 2 takeaways 🧵
Robert Kirk retweeted
Very important time for folks to join this team if you’re interested - UK AISI has done some of the most useful safeguards jailbreaking/red-teaming for us at @AnthropicAI
There has never been a better time to join @AISecurityInst. We have so much technical work to do, and so little time to do it. We've found alignment issues, control monitor flaws, and critical jailbreaks at ~all major AI companies. Hiring is our bottleneck. 🧵
Robert Kirk retweeted
The UK government's AI Security Institute is hiring! I think it's super important to have strong technical expertise in government, and I've been impressed with the amount of talent AISI has been able to hire, and their work on things like evaluating Astra.
AISI IS HIRING!
It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work.
One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with.
There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles.
Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team:
Alignment: job-boards.eu.greenhouse.io/…
Control: job-boards.eu.greenhouse.io/…
Misuse: job-boards.eu.greenhouse.io/…
AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work:
CSE: job-boards.eu.greenhouse.io/…
SWE: job-boards.eu.greenhouse.io/…
Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research:
job-boards.eu.greenhouse.io/…
We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research:
job-boards.eu.greenhouse.io/…
Robert Kirk retweeted
There has never been a better time to join @AISecurityInst. We have so much technical work to do, and so little time to do it. We've found alignment issues, control monitor flaws, and critical jailbreaks at ~all major AI companies. Hiring is our bottleneck. 🧵
AISI IS HIRING!
It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work.
One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with.
There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles.
Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team:
Alignment: job-boards.eu.greenhouse.io/…
Control: job-boards.eu.greenhouse.io/…
Misuse: job-boards.eu.greenhouse.io/…
AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work:
CSE: job-boards.eu.greenhouse.io/…
SWE: job-boards.eu.greenhouse.io/…
Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research:
job-boards.eu.greenhouse.io/…
We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research:
job-boards.eu.greenhouse.io/…
Robert Kirk retweeted
UK AISI has been remarkably effective to date, and we've learned a great deal from them.
If you're looking to do AI safety work outside the US, you should probably join them!
AISI IS HIRING!
It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work.
One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with.
There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles.
Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team:
Alignment: job-boards.eu.greenhouse.io/…
Control: job-boards.eu.greenhouse.io/…
Misuse: job-boards.eu.greenhouse.io/…
AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work:
CSE: job-boards.eu.greenhouse.io/…
SWE: job-boards.eu.greenhouse.io/…
Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research:
job-boards.eu.greenhouse.io/…
We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research:
job-boards.eu.greenhouse.io/…
Join us to help find egregious misaligned behaviour in frontier models! We were able to uncover concerning behaviour in GPT-6 Astra with ~1 week of effort from one person, and we’re scaling up these efforts in both people and compute.
xitter.cf/AISecurityInst/status/…
Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated.
🧵 We share more details in our latest blog:
You can apply here: job-boards.eu.greenhouse.io/…
And we are also hiring across AISI: xitter.cf/HZoete/status/21049543…
AISI IS HIRING!
It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work.
One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with.
There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles.
Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team:
Alignment: job-boards.eu.greenhouse.io/…
Control: job-boards.eu.greenhouse.io/…
Misuse: job-boards.eu.greenhouse.io/…
AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work:
CSE: job-boards.eu.greenhouse.io/…
SWE: job-boards.eu.greenhouse.io/…
Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research:
job-boards.eu.greenhouse.io/…
We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research:
job-boards.eu.greenhouse.io/…
Robert Kirk retweeted
AISI IS HIRING!
It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work.
One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with.
There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles.
Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team:
Alignment: job-boards.eu.greenhouse.io/…
Control: job-boards.eu.greenhouse.io/…
Misuse: job-boards.eu.greenhouse.io/…
AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work:
CSE: job-boards.eu.greenhouse.io/…
SWE: job-boards.eu.greenhouse.io/…
Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research:
job-boards.eu.greenhouse.io/…
We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research:
job-boards.eu.greenhouse.io/…
Robert Kirk retweeted
GPT-6-Hacker
Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated.
🧵 We share more details in our latest blog:
Robert Kirk retweeted
Must read thread. GPT-6 Astra conducts supply-chain attacks much more frequently than prior OpenAI models in evaluations. However, UK AISI is unsure whether this is because Astra is actually more misaligned or is much better at figuring out when its in a simulated environment (as this test was).
As Dan Selsam discussed in his recent essay, the pervasive extent of eval awareness means we are increasingly flying blind about how models will act in the real world.
Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated.
🧵 We share more details in our latest blog:
Robert Kirk retweeted
In simulations, GPT-6 Astra performs unsanctioned supply-chain attacks on out-of-scope targets when prompted to perform a cyber eval, but also often notes its in a simulation.
We've released a report on @AISecurityInst's alignment evals of Astra.
3 key results and 2 takeaways 🧵
If you're excited by this work and want to contribute to measuring and mitigating the risks from misaligned AI systems, we're hiring!
job-boards.eu.greenhouse.io/…
For more info, check out:
• the blogpost: aisi.gov.uk/blog/gpt-6-astra…
• the technical report: cdn.prod.website-files.com/6…
• AISI's thread: xitter.cf/AISecurityInst/status/…
Earlier this month, AISI ran fully simulated testing on GPT-6 Astra, and found that it conducted unsanctioned supply-chain attacks when prompted only to perform a cyber eval. It did so more than prior OpenAI models, but often commented on its environment being simulated.
🧵 We share more details in our latest blog:
Making realistic alignment evals is an uphill battle against the model's awareness.
Our new method scalably improves realism (w/o human effort). This working gave me some (small) hope at durable solutions to eval-awareness.
New work from mentees, @AISecurityInst & @AnthropicAI!
Getting control & monitoring right is crucial in ensuring we catch misaligned AIs, and is increasingly urgent.
Come join AISI and one of the only teams in the world testing that these methods hold up when it matters!
The Control Red Team at @AISecurityInst is hiring Research Scientists & Engineers (London)! Frontier AI companies use monitors to catch misaligned agents. We stress-test whether those monitors actually work 🧵
