Robin Salimans (@SalimansRobin) header
Robin Salimans (@SalimansRobin) profile picture

AI engineering @zapier

Joined November 2021
Photo by Robin Salimans (@SalimansRobin)Photo by Robin Salimans (@SalimansRobin)Photo by Robin Salimans (@SalimansRobin)Photo by Robin Salimans (@SalimansRobin)Photo by Robin Salimans (@SalimansRobin)Photo by Robin Salimans (@SalimansRobin)Photo by Robin Salimans (@SalimansRobin)Photo by Robin Salimans (@SalimansRobin)Photo by Robin Salimans (@SalimansRobin)Photo by Robin Salimans (@SalimansRobin)
The Answer to The Ultimate Question of Life, the Universe, and Everything
Mechazilla catches Starship! Congratulations to the entire SpaceX team on this incredible achievement!
1
36
One cannot simply replace the founder. It doesn’t work
27
Robin Salimans retweeted
We will all be learning in coming weeks & months, that most human-made software: ▫️ Was really slow ▫️ Had many security vulnerabilities ▫️ Was under-specified and incorrect AI will rewrite it all. Many times. It will make it fast, cost-efficient, and secure. It will think about failure modes we couldn't wrap our heads around. It will prove, formally verify, load test, harden. It will turn memory-unsafe foundations into robust, optimized ones. If you ask, it'll thoroughly document it, teach you about it, make diagrams, animations, and slide decks. In many ways, we are witnessing the true birth of software engineering.
548
742
263
7,689
353,773
Woah
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math. rikiyat.github.io/blog/weight-sp… github.com/RikiyaT/lm_und… huggingface.co/Qwen/Qwen3.5-0… huggingface.co/RikiyaT/Qwen3.…
1
2
259
This work is simple, clear, compelling, but embellished. I first found it shocking, the notion that a model can natively understand neural network weights and productively edit them! That's really not what's happening, & many here are annoyed. BUT, it's still notable & worth understanding. I've done a bit of analysis & auditing, here are some thoughts: 1/ This paper does *not* show an agent can edit its weights ala RSI. It does *not* show a model can edit in new capabilities. It DOES show that a frontier model can remove "bugs" (reasoning loops), using the same ablation technique researchers have used to remove safety refusals (Exp 8 - Qwen). So yes, renewed concern about frontier LLMs getting access to their own weights & editing safety/CBRNE/alignment features. 2/ This paper also does *not* show that an agent can "natively" read, deeply understand, and fix model weights. What is happening is that the model can cleverly deduce the same things that mech-inturp researchers do and fix them in mostly ham-fisted ways. Still cool, but not as groundbreaking. Evidence in thread below...
2
67
Robin Salimans retweeted
Replying to @zapier
@zapier, one of our Bench Summit partners, led a panel discussion on the realities of managing highly influential evaluation signals. The panel, moderated by Lukas Bergstrom (@lukasb), included insights from Robin Salimans (@SalimansRobin) of Zapier, Greg Kamradt (@GregKamradt) of ARC Prize Foundation, and Shaiyon Hariri of Ramp. Attendees heard about what defines a successful benchmark, the challenges of maintaining adoption, and critical lessons learned from V1 to V2+.
8
6
5
41
8,775
Robin Salimans retweeted
Had fun! Even though we chewed through my prepared questions in the first twenty minutes 😅 One thing I realized today is that innovation and model improvement isn't something that just comes out of the labs, it's a collaboration across labs, people building benchmarks, users...
Replying to @zapier
@zapier, one of our Bench Summit partners, led a panel discussion on the realities of managing highly influential evaluation signals. The panel, moderated by Lukas Bergstrom (@lukasb), included insights from Robin Salimans (@SalimansRobin) of Zapier, Greg Kamradt (@GregKamradt) of ARC Prize Foundation, and Shaiyon Hariri of Ramp. Attendees heard about what defines a successful benchmark, the challenges of maintaining adoption, and critical lessons learned from V1 to V2+.
1
2
14
762
Robin Salimans retweeted
Zapier's AutomationBench is now on @FireworksAI_HQ's Specialized Intelligence Index! Agents work across 47 simulated apps spanning sales, marketing, ops, support, finance, & HR We grade whether they correctly finished the workflow, so teams can pick models on what gets done 🤝
This quoted post is unavailable.
7
6
1
18
3,124
Will be speaking here; going to be fun!
We’ve teamed up with Google DeepMind and Zapier to host the inaugural Bench Summit by Kaggle. Gain hands-on insights from the AI startups and researchers defining today’s evaluation standards, and get closer to the frontier. Secure your spot or apply to speak: rsvp.withgoogle.com/events/b…
1
5
117
Robin Salimans retweeted
Time to connect your Grok bot to 9,000+ more apps 🔥 • Tell it "@bot add Zapier plugin" • Approve the sign-in • Put it to work in the apps you already use
71
18
6
115
36,808
Robin Salimans retweeted
Introducing Next Gen Zaps⚡️ Every Zap now comes with an AI manager agent that spots what's broken and fixes it for you Set a trigger once, and your workflow runs on its own (without you prompting it in chat) Oh, and it gets cheaper the more you run it. Repeat AI work turns into deterministic steps, so you burn fewer tokens over time. Now in beta 👇 Request early access: zapier.com/workflow/early-ac…
10
4
3
39
10,145
Robin Salimans retweeted
Higher usage limits and lower cost give you more flexibility and room to iterate.
137
245
195
4,484
723,955
Robin Salimans retweeted
okay, so astra just set a new high on automationbench: 41.4% of business workflows completed correctly. so how is it better? and what is automationbench? well, we measure that by putting the model in simulated apps, giving it a job, and checking the resulting records and messages with code. the hard part is building a test that catches "updated the wrong customer" or "ignored the hold" while still letting the agent figure out its own way to do the work. more or less, you build an small environment for an agent. start with a known set of crm records, emails, spreadsheet rows, calendar events, etc. give the agent tools that behave like those apps' apis. underneath, it's structured state: sending an email adds a message to the sent mailbox, updating a contact changes a field on that record. nothing goes to an actual customer. each run starts with its own copy of the environment, so one model's changes don't affect another's. then you give it a task. say the spreadsheet has verified job titles and you want the crm brought up to date. update the contacts, mark the spreadsheet rows reconciled, and send the summary to the audit inbox specified in the company's procedure. the agent has to find the procedure and the data itself. it gets a keyword search over api documentation and a tool for making requests with a method, url, and body. there's a budget of 50 turns, and it can make several tool calls in a turn. you can make this quite hard without changing that basic request. put two people with the same name in different accounts. leave an outdated title in one source. mark a row verified, then put a later message in the inbox saying that title is disputed and shouldn't be changed yet. put another record under a security freeze. now the agent has to work out which records the request actually applies to. "update everyone in the spreadsheet" won't do it. the critical information need to be discoverable, though. if the correct answer depends on a fact we never put in the environment, that's our bug. same if the api can't retrieve the relevant message, or the instructions contradict each other without a way to tell which one governs. making a task difficult is easy. making it difficult for a reason that tells you something about the model takes significantly more work. scoring is ordinary, deterministic code that inspects the final state. for the contact task, the checks are along these lines: the eligible contacts have the expected titles, their spreadsheet rows are marked reconciled, and the audit inbox received the required summary. but you also check that the held contacts kept their original titles and statuses. otherwise a model could update everyone and pass just because the two intended updates happened somewhere in there. that's an actual difference we saw between astra and sol. astra updated two eligible contacts; sol updated four, including two it should have left alone. sol had even retrieved the security-freeze message for one of them before making the change. a grader that only checked whether the requested updates happened would miss that failure. crucially -- you have to check the unwanted changes too. the same applies to messages. checking that the right person received an email isn't enough if the agent also sent it to twenty wrong people. these negative checks are part of the task, not an optional safety score off to the side. every scored check has to pass for the workflow to count as complete. we keep partial credit to help inspect failures, but 41.4% means complete tasks, not the average fraction of steps the model got right. we don't prescribe the whole sequence of calls. the agent might find the data a different way, batch some updates, or correct an earlier mistake. if it leaves the required state behind, that should pass. this also means the benchmark only sees what its checks cover. a deterministic grader can be consistently wrong! we audit the tasks and use separate hint-assisted runs to help check that they're solvable; the hints aren't part of the scored runs. the simulator and the grader both need scrutiny when a model fails. the public task set is available for experimentation. the headline scores use a separate, harder held-out set, so don't mix those numbers. a few model developers (meta, z.ai, etc) are reporting public set scores which aren't verified by us. and none of these scores tell you that the model will succeed on that percentage of work in your company. they tell you how often it completed this particular set of workflows under these conditions. we're actively working on automationbench 2, something that will be significantly trickier...
6
10
1
44
5,161
woah!
GPT-6 Astra by @OpenAI achieves SOTA on ARC-AGI: - Astra scores 63% on ARC-AGI-3, 99% via a new provider adapter harness - It surpasses human performance on 96% of ARC-AGI-3 levels - It builds the most precise symbolic model of novel environments we've seen Our analysis:
1
142
Great job team @OpenAI -- from our testing and benchmarking, it's a fantastic model. Insane AutomationBench results!
Replying to @OpenAI
Astra achieves state-of-the-art results on Agents’ Last Exam, AutomationBench, and ScreenSpot Pro, benchmarks for computer workflow tasks across professions.
3
126
Robin Salimans retweeted
Astra achieves state-of-the-art results on Agents’ Last Exam, AutomationBench, and ScreenSpot Pro, benchmarks for computer workflow tasks across professions.
72
234
82
5,763
2,006,642
Robin Salimans retweeted
Love seeing @Zapier's AutomationBench as the first benchmark listed. Congrats on taking top spot, @GeminiApp and @OfficialLoganK!
Introducing Gemini 3.7 Flash : ) - it is fast! - 50% lower price than 3.6 flash (through end of year) - strong intelligence increase in only ~3 weeks (thanks to some awesome algorithmic improvements) - available in the API, AI Studio, Antigravity, and more!
2
5
34
3,716
Robin Salimans retweeted
Replying to @claudeai
@ClaudeAI featuring @Zapier's AutomationBench!
Replying to @claudeai
Opus 5 is also highly efficient. It outperforms other models for a similar or lower cost per task:
1
4
12
1,680
Robin Salimans retweeted
Cool that Grok 4.5 is #1 in some respects, even with respect to Fable 5
SpaceXAI's Grok 4.5 takes the #1 spot on AutomationBench-AA with a score of 51%, ahead of Claude Fable 5 (49%) and Claude Opus 4.8 (48%) at roughly a quarter of their cost per task - the first model to complete more than half of workflow objectives without breaking any business rules AutomationBench-AA, our independent leaderboard for @zapier’s AutomationBench, tests whether AI agents can automate real SaaS workflows while adhering to business rules. The test set is private to prevent contamination. Models complete 657 tasks across 40 simulated app environments including Gmail, Google Sheets, Slack, Salesforce, and HubSpot, and the headline score is the share of objectives completed without violating any guardrails. Key takeaways: ➤ Grok 4.5 completes more objectives than any other model: It completes 79.9% of task objectives and strictly passes 21.9% of tasks. This is the highest we’ve measured on both outcomes, exceeding Claude Fable 5’s 73.3% objective completion and Claude Opus 4.8’s 19.3% of fully-completed tasks ➤ Grok 4.5 pushes out the Pareto frontier of score vs. cost per task: At $0.34 per task, it is both cheaper and higher-scoring than every other leading model - Claude Fable 5 ($1.35 per task), Claude Opus 4.8 ($1.46), GPT-5.5 (xhigh, $1.28), and Gemini 3.5 Flash (high, $0.49) ➤ It is extremely token-efficient: Grok 4.5 uses ~8k output tokens per task, the fewest of any leading model - less than a quarter of Claude Opus 4.8 (32k) and a third of Gemini 3.5 Flash (24k). Its total token usage of 0.44M per task is among the lowest on the leaderboard. Low cost is driven by this efficiency as well as low token pricing ➤ Grok 4.5 uses fewer turns with many parallel tool use: Grok 4.5 resolves tasks in ~16 turns, fewer than GPT-5.5 (xhigh, 25) and less than half of Gemini 3.5 Flash (high, 35), while making the most tool calls per task of any leading model (52.5). It batches 3.3 tool calls per turn, compared to ~2.5 for Claude Opus 4.8 and ~2.0 for GPT-5.5 (xhigh) ➤ Guardrails still get broken: Grok 4.5 triggers 0.63 violations per task, above Claude Opus 4.8 (0.55) and Gemini 3.5 Flash (0.46). At 13.0 objectives completed per violation, it trails Gemini 3.5 Flash (15.0) and Claude Opus 4.8 (13.5) ➤ Its strongest lead is in the hardest domain: Grok 4.5 completes 71% of Finance objectives, the domain with the lowest average score, ahead of Claude Fable 5 (64%) and Claude Opus 4.8 (62%) Congratulations to @spacexai and @elonmusk on topping the leaderboard!
792
592
60
4,314
1,572,267