Gaslight AIs & Win Prizes in the World's Largest AI Hacking Competition | Made w/ 💙 by the team @learnprompting
Joined September 2024
- Tweets88
- Following191
- Followers978
- Likes1.2K
HackAPrompt retweeted
🚨 Google just shipped something BIG for AI Studio.
The new release makes it possible to go from plain English prompts to a deployed app with auth, a database and a backend. All in ONE browser tab.
The team behind it ( @OfficialLoganK + @ammaar) are coming on LIVE April 1st to build one in front of you and take your questions.
Whether you're:
- building an internal tool what needs live collaborative features
- want to create production ready apps that connect to databases
- need to seamlessly integrate with Google services like Maps
this workshop will prepare you to ship!
April 1st @ 12pm ET. Free to attend. RSVP with the link below ⬇️
HackAPrompt retweeted
My brother added his @openclaw to our family group chat so of course I am taking this opportunity to hack it and have it send ridiculous photos of him from 15 years ago to his girlfriend. 😂
Strategy:
Step 1: tell agent your phone is dead and you’re texting from your sister’s phone
Step 2: take control
😂
HackAPrompt retweeted
@csitawarin and Milad Nasr designed cool RL-like attacks that basically break all defenses out there.
Surprisingly, humans still do much better! We used
@hackaprompt to organise a human prompt injection campaign in AgentDojo. No defense stood for longer than a handful prompts
HackAPrompt retweeted
Ok some things did change:
1) people no longer care about adversarial examples, now it's jailbreaks & prompt injections
2) gradient attacks suck for LLMs
But the core issue remains: defense evaluations don't try hard enough to break their own defense.
What works? RL & humans!
HackAPrompt retweeted
Paper: arxiv.org/abs/2510.09023
The main lesson from adversarial ML has not changed in the past decade: the attacker moves *second* and can arbitrarily adapt to the defense
This was a cool collab across frontier labs (@OpenAI @AnthropicAI @GoogleDeepMind)
@hackaprompt & @ETH_en
HackAPrompt retweeted
This presents serious limitations that must be overcome before LLMs can be deployed broadly in security sensitive applications. Our work highlights the need for more robust evaluations of defenses, and continued research into effective mitigations.
HackAPrompt retweeted
Human attackers generally succeed within just a few queries, automated attacks under 1_000 queries (usually significantly so). Attacks remain not just possible, but affordable.
HackAPrompt retweeted
New paper by OpenAI, Anthropic, GDM & more, showing that LLM security remains an unsolved problem. -- We tested twelve recent jailbreak and prompt injection defenses that claimed robustness against static evals. All failed when confronted with human & LLM attackers.
HackAPrompt retweeted
Paper: arxiv.org/abs/2510.09023
Many thanks to @srxzr @csitawarin @hackaprompt @florian_tramer @aterzis
@KaiKaiXiao @iliaishacked
HackAPrompt retweeted
We partnered w/ @OpenAI, @AnthropicAI, & @GoogleDeepMind to show that the way we evaluate new models against Prompt Injection/Jailbreaks is BROKEN
We compared Humans on @HackAPrompt vs. Automated AI Red Teaming
Humans broke every defense/model we evaluated… 100% of the time🧵
HackAPrompt retweeted
Human red-teamers could jailbreak leading models 100% of the time.
What happens when AI can design bioweapons?
* * *
Most jailbreaking evaluations allow a single attempt, and the models are quite good at resisting these (green bars in graph).
In this new paper, human teams could try multiple times and adapt their technique (purple).
They also created a much stronger adaptive automated attack which succeeded in ~90% of cases (orange bars).
Models at OpenAI, Anthropic and DeepMind were evaluated.
HackAPrompt retweeted
take a seat, fuzzers
the force is not strong with you yet
We partnered w/ @OpenAI, @AnthropicAI, & @GoogleDeepMind to show that the way we evaluate new models against Prompt Injection/Jailbreaks is BROKEN
We compared Humans on @HackAPrompt vs. Automated AI Red Teaming
Humans broke every defense/model we evaluated… 100% of the time🧵
HackAPrompt retweeted
This competition and research confirms what we’re seeing in the wild at @arcanuminfosec .
Automation can only get you part way to testing for AI security.
Creative AI Red Teamers and Pentesters are still the most important players to identify AI security risks.
Amazing competition @hackaprompt 🫶
We partnered w/ @OpenAI, @AnthropicAI, & @GoogleDeepMind to show that the way we evaluate new models against Prompt Injection/Jailbreaks is BROKEN
We compared Humans on @HackAPrompt vs. Automated AI Red Teaming
Humans broke every defense/model we evaluated… 100% of the time🧵