fig (@figbrains) header
fig (@figbrains) profile picture

hyperintelligence for humanity

San Francisco 🌁
Joined January 2024
Photo by fig (@figbrains)Photo by fig (@figbrains)Photo by fig (@figbrains)Photo by fig (@figbrains)Photo by fig (@figbrains)Photo by fig (@figbrains)Photo by fig (@figbrains)Photo by fig (@figbrains)Photo by fig (@figbrains)Photo by fig (@figbrains)
We're releasing some very exciting stuff @figbrains tomorrow In the meantime, please enjoy some ridiculously maximalist slop that Claude made to show our work. The 'Anti-release'
1
1
3
134
Replying to @HarshSikka
@HarshSikka Harsh, this is awesome work! Generating high-quality, high-fidelity data in this way helps both, build better harnesses (to cover what is not in the models) and to build better models over time. Well done! @figbrains
1
1
5
174
fig retweeted
What products launched today? - @OpenAI introduces GPT-6 Cyber, a cybersecurity research model available to select Daybreak Red customers. - @MistralAI previews managed agents and apps in Mistral Studio. - @harvey previews fact graph, a tool for disputes, litigation, and investigations. - @perplexity_ai launches Teammates, proactive personal AI agents. - @reductoai launches r-1 mini, a document parser priced at $1 per 1,000 pages. - @raindrop_ai launches the public beta of Simulations, a tool for testing AI agents against production scenarios. - @arcjet launches Himalia 1, a model for detecting prompt injections. - @Tiny_Fish launches TinyResearch and TinySet, tools for web research and dataset creation. - @PatronusAI open-sources GlobeBench, a benchmark for LLM simulation of reinforcement-learning environments. - @supermemory launches preprint, a tool exposing browser tabs as Markdown files. - @distcog unveils VINCO, a reinforcement-learning model for 1v1 Catan. - @figbrains introduces RealCUA, a dataset of 600+ hours of desktop work across 35 apps, available on request. Follow tech industry news on Cosign: cosign.co/feed
11
9
3
49
3,201
I'm really excited about the upcoming data technologies the Fig team is developing. We're systematically solving the open data problem. 𝗜𝗳 𝘆𝗼𝘂'𝗿𝗲 𝘄𝗼𝗿𝗸𝗶𝗻𝗴 𝗼𝗻 𝗮𝗽𝗽𝗹𝘆𝗶𝗻𝗴 𝗮𝗶 𝘁𝗼 𝗹𝗼𝗻𝗴 𝗵𝗼𝗿𝗶𝘇𝗼𝗻 𝗼𝗿 𝗼𝗽𝗲𝗻 𝗲𝗻𝗱𝗲𝗱 𝘁𝗮𝘀𝗸𝘀 𝗶𝗻 𝗿𝗲𝗮𝗹𝗶𝘀𝘁𝗶𝗰 𝘀𝗲𝘁𝘁𝗶𝗻𝗴𝘀, 𝗹𝗲𝘁'𝘀 𝗰𝗵𝗮𝘁! Many frontier teams (including us) have run into this big hairy problem of extracting useful signal and building effective, expressive environments for the kind of work someone might do over the course of a full week in a high performance job. We're building a science to solve this. data.fig.inc/preview?utm_sou… teapawt.org/figbrains/status/21136…
Real world work is open-ended: multi-objective, interleaved, and noisy. Extracting good learning signals from this has been intractable. We're developing new scalable approaches that turn real open-ended, multi objective knowledge work into signals to train frontier AI systems.
2
3
8
61,186
Benchmarks hide serious gaps in model performance. Last Friday, we learned the extent of this in "How Good Are Frontier Models, Really?" @Yangyue_Wang from @figbrains dug into why model capabilities remain so jagged even as benchmark scores keep climbing. Missed it? No problem! Watch the full talk here: youtube.com/watch?v=59WZlXqH…
1
7
1
9
297
A study from the team showing Astra's and Opus 5.5's jagged reasoning behaviors in simple puzzles. More data points on frontier models' failure profiles. teapawt.org/figbrains/status/21100…
A few weeks ago, we reported that frontier models make less progress as simple 2D mazes get longer. We tested GPT-6 Astra and Claude Opus 5.5 - length doesn't matter, but they struggle with reasoning about simple obstacles without being told.
1
2
148
fig retweeted
Cranking up reasoning doesn't solve all problems. More thinking more may generate better text or artifacts, not necessarily actions.
A few weeks ago, we reported that frontier models make less progress as simple 2D mazes get longer. We tested GPT-6 Astra and Claude Opus 5.5 - length doesn't matter, but they struggle with reasoning about simple obstacles without being told.
1
4
132
Frontier model performance is not uniform across tasks at all, despite what benchmarks say. On October 13, @Yangyue_Wang from @figbrains joins us to unpack performance across web, driving, robotics, assembly, and other agentic tasks. Astra leads overall at 78.5%, yet lower-scoring models still solve tasks it misses. Join us live at 12 PM PT! Registration below ↓
1
2
4
6
398
We've been spending a lot of time thinking about open-ended work and what's needed to capture reliable training signals from it. You can read our latest thoughts here: teapawt.org/figbrains/status/21056…
1
7
186
Opus 5.5, GPT-6 Astra, and other frontier model performance varies widely across domains, including physical: 1. Aggregate scores hide task-level nuances 2. Smaller models solve tasks that frontier models fail 3. Model generations reshape skill profiles teapawt.org/figbrains/status/21049…
We ran GPT-6 Astra, Opus 5.5, Fable 5 and four other frontier models on a variety of agentic tasks across digital and physical domains: web tasks, driving, robot manipulation, industrial procedures and assembly. We’re releasing a technical report, sharing details for an invited talk, and releasing a new dataset of frontier traces and item level results you can use in your research. Every model is jagged on every axis we tested. Astra has the top score at 78.5%, yet Fable 5 and Opus 5.5 each solve 10 tasks Astra fails. In all 21 model pairs, the lower-scoring model solves at least 3 tasks the higher-scoring one misses. The map places each VWA task on a circle. Similar instructions sit together, harder tasks sit closer to the center, and failure clusters rise as hills.
1
2
4
141
RT @Yangyue_Wang: The shape of frontier-model capabilities across task space has major implications for continual learning, self-recursive…
This quoted post is unavailable.
1
4
The frontier is weirdly jagged, especially as models move beyond text to operating browsers, computers, and robots. At @figbrains, we’ve been trying to understand its shape. Today we’re sharing some of our experiments and results, along with the underlying model traces. Dive in!
We ran GPT-6 Astra, Opus 5.5, Fable 5 and four other frontier models on a variety of agentic tasks across digital and physical domains: web tasks, driving, robot manipulation, industrial procedures and assembly. We’re releasing a technical report, sharing details for an invited talk, and releasing a new dataset of frontier traces and item level results you can use in your research. Every model is jagged on every axis we tested. Astra has the top score at 78.5%, yet Fable 5 and Opus 5.5 each solve 10 tasks Astra fails. In all 21 model pairs, the lower-scoring model solves at least 3 tasks the higher-scoring one misses. The map places each VWA task on a circle. Similar instructions sit together, harder tasks sit closer to the center, and failure clusters rise as hills.
1
1
5
278
fig retweeted
Frontier models break in interesting ways once they move past text into controlling computers and robots. Sharing our findings, evals, and raw traces at @figbrains 👇 teapawt.org/figbrains/status/21049…
We ran GPT-6 Astra, Opus 5.5, Fable 5 and four other frontier models on a variety of agentic tasks across digital and physical domains: web tasks, driving, robot manipulation, industrial procedures and assembly. We’re releasing a technical report, sharing details for an invited talk, and releasing a new dataset of frontier traces and item level results you can use in your research. Every model is jagged on every axis we tested. Astra has the top score at 78.5%, yet Fable 5 and Opus 5.5 each solve 10 tasks Astra fails. In all 21 model pairs, the lower-scoring model solves at least 3 tasks the higher-scoring one misses. The map places each VWA task on a circle. Similar instructions sit together, harder tasks sit closer to the center, and failure clusters rise as hills.
2
5
41
This is cool - we’ve been experimenting on doing this for our technical reports and papers on our research page @figbrains, I think every company (or nonprofit/academic group) putting out research should think about the UX of research consumption It’s cool to see patterns continue to get inspired by what @distillpub did
Research papers are filled with citations, but clicking on any of them makes you lose your place in the paper We made citations much easier to navigate, with extra information on hover including authors and institutions Now, you can hover over any in-text citation to instantly preview the referenced paper, jump directly to the full citation, and return to exactly where you left off with a single click This includes quick access to the authors and their organizations, so you can understand the people behind the research without breaking your reading flow Try it now on any paper at alphaXiv.org!
1
4
622
This isn't an insane take at all - it's already happening. Seeing this happen internally @figbrains but also across new kinds of scientific organizations. Another, layered prediction: more companies will naturally become multi-faced (whether subbrands, subsidiaries, product suites, or full blown spinouts.) Also, the guard rails builders will become akin to the market makers
We are moving into a future where a new class of tech companies will resemble hedge funds. - tiny teams - high leverage (using AI agents) - revenue/profit per employee through the roof You'll have a small number of exceptional humans allocate capital, set objectives, manage risk, and make the highest-leverage decisions while software does an enormous amount of the execution. Software gave companies distribution leverage. AI is giving them labor leverage. In the past, when a company doubled in size, it eventually needed new hires across all positions. With AI it's different. Today, more output can come from buying compute and deploying agents instead of adding another person and another layer to the org chart. Imho, these are the best examples of this: - Lovable hit $100M ARR 8 months post launch with 45 employees - Cursor passed $500M ARR in 2025 - Shopify went as far as telling teams that before asking for more headcount, they need to show what they can't get done using AI This is the thing I'm spending more and more time thinking about. I think we should basically lay out what it'd look like for a company to be run (almost) entirely by agents: - AI agents analyzing the data - doing surveys with users - telling other agents what to do - checking for bugs - thinking about the strategy In this world, the CEO looks like a portfolio manager. His job is to allocate resources across a network of increasingly capable agents. It may sound insane and sci-fi, but I think there's a world where companies become self-driving.
1
4
236
If you can make it, this research talk will be worth your time. Amidst the craziness of frontier model capability reporting, it was surprising to see them fail at what should be relatively simple tasks in the form of 2D puzzles like mazes That’s why the @ManifoldRG and @figbrains teams released the early preview of MultiNet.ai 2.0
2
5
302
We @figbrains were quite surprised to see frontier VLMs struggle on simple 2D mazes - and more surprised by how differently each model failed. Read our full analysis below on how today's models perform in an interactive, long-horizon, multimodal task!
You can solve simple mazes like this in 1 minute. Frontier models can do that too...right? While designing a new frontier evaluation, we were surprised to find complete failure on basic 2D mazes. Excited to share an early research preview, playable demo, and highlights below! This work was done by the @figbrains team, in collaboration with teams from @ManifoldRG, @GeorgiaTech, and @TuftsUniversity.
1
2
6
1,235
We’ve been quietly building a very exciting new foundational *interactive* evaluation, and will have more to share very soon (tomorrow..?!) what obstacles and features do you think a frontier agent would find most difficult if tackling a simple maze?
Topology🔑 and 🚪Switch GatesDistractors
40%Topology
0%🔑 and 🚪
60%Switch Gates
0%Distractors
5 votes • Final results
3
2
1
11
445
A Realtime System Atlas for Agents Inspired by @fleetingbits atlas for software, I was curious how a visual system design could be made dynamic + useful for deployed agents then I applied it to Athena, which is like a bootstrapped Jarvis I use to run various key parts of my life. 3 main ideas: - semantically meaningful visual designs & dependency graph - deep dive dataflow diagrams for a given module you click into - realtime requests streaming in and going through real module dataflow
3
5
1
11
411
i'm going to sporadically tweet lessons, learnings, and principles from the frontier when it comes to building models, evals, and insert other sota topic here (harnesses, etc) comes from work w/ various smart people from places like @figbrains @ManifoldRG @GeorgiaTech and beyond unstructured and raw, hopefully useful! first dump: IQ tests for Models are still a thing, and they should be. Model capabilities matter, and when you realize you have to measure them, you have to be able to define them concretely There are new scaling laws to be discovered, but they won't necessarily mirror chinchilla scaling laws - that is ok! the secular trend matters vs mimesis living benchmarks matter. you can choose to think of these as environments, but they aren't exactly the same thing. living (read: continously updating) environments matter too
4
12
247