
Co-founder & CTO @diadia_health (AI medical reasoning). OSS Agentlane (distributed agent runtime). Ex-@uber / self-driving / @google.
🇺🇸
Joined December 2007
- Tweets2K
- Following1K
- Followers1K
- Likes5.2K
Pinned Tweet
Months of work, and today we are finally launching our medical AI transparency lab.
It's a reasoning system we have built at Diadia Health that verifies the high-severity medical claims an AI makes when it generates a diagnosis or a treatment protocol, breaking each claim down into a mechanisms graph and checking every pathway in that graph against real evidence.
What we are trying to do here is to provide the next form of transparency in the medical AI domain as models get stronger and their claims will keep sounding more right.
While most of the discussion about medical AI happens at the level of benchmarks and high-level numbers that are often meaningless in a day to day work of a clinician, we believe it is more useful to look at the actual claims AI makes with full evidence trace attached and reasoning validated.
Earlier we published a study of several frontier models across a variety of medical cases and claims, and showed how much their reasoning, outcomes and agreement diverge. The average divergence in verdicts was 31% per patient, over 85% of claims were unique to a single model, and only 3 of 36 patients (8.3%) showed consensus across models.
It makes it hard to trust an AI diagnosis, harder for clinicians to do their work, and harder to deploy these models in real medical settings.
Transparency Lab aims to shed more light on AI’s medical reasonong providing a live catalog of thousands of verified claims, with evidence traces, contradictions and reasoning paths for anyone to explore.
transparency.diadiahealth.co…
Andrii Yasinetsky retweeted
We've developed a way to bootstrap post-training from positive examples alone and show it works on arithmetic, molecular structures, and scientific writing. We've used the method to train a 235B LLM for writing indistinguishable from human writing. 1/5
This is definitely the future and many have been approaching this problem all wrong.
The new reality is that no one has time to write and perfect prompts, and evaluate the results.
Most professionals in other domains outside coding/math just want to get the job done, delegating tedious and unglamorous work to AI without having to invest days and weeks into it to make it work.
The future looks more like a "hired" agent contractor where a customer pays for the work done well.
Today, we're launching Perplexity Teammates.
They are personal agents for work. And change the paradigm from writing prompts in a session to giving high-level goals. Rolling out to Max users on the web today. Native apps (mobile, desktop) coming soon.
69
Everyone's now talking about multiplayer AI and it reconfirms my thesis I outlined earlier this year and built Agentlane for.
Just like an organization divides work among participants with different responsibilities, roles and boundaries agent systems will eventually converge on the same hierarchy and will need a substrate to run, communicate and collaborate on.
We are seeing some early innings of this new architecture emerging in various shapes and forms, but no one really has built a fully autonomous organization yet that is run and governed by agents.
A fully agentic organization system will ultimately need these layers (some of which Agentlane addresses):
Identity
An agent needs to be a first-class participant: a persistent identity, credentials, a record of what it did and why, and authority that is scoped and can be delegated.
A communication and coordination substrate
Agents need a shared place to hand off work, ask for help, flag blockers and pick up work, with memory that belongs to the organization rather than a single agent.
Governance as code
Charters, policies, budgets, approvals, and escalation paths have to be explicit. Agents will have to follow these rules. Humans can tolerate some ambiguity in a corporate handbook. But agents will need a constitution that they can execute against, and probably a way to amend it.
Evaluation as the HR function
Hiring, performance reviews, promotions and off-boarding all essentially become evals. A poorly performing agent gets put on a virtual "pip" and goes through retraining / eval loop before being put to producton.
Trust layer between agents
One agent has to be able to rely on another’s output without redoing or re-evaluating the work. A brief review with some feedback loop should be sufficient to establish temporary trust. That also means provenance on claims, signed artifacts, and some notion of reputation that accumulates over time.
An economic layer
Agents will hold budgets, pay for compute and tools, and enter into agreements with each other and with humans. Somewhere in there a legal wrapper has to exist that says who is accountable when the organization acts.
Failure handling & accountability
Incident response, rollbacks and post-mortems, run by agents for agents, with humans as the governing board rather than the operators.
Eugenia Kuyda of @Wabi predicts that within a year, one of the biggest consumer AI platforms will be multiplayer, with real network effects:
"Today most AI products are single player. And I would say that you could probably argue they don't have much of a moat. They don't necessarily have switching costs."
"It was pretty easy to move between, say, Claude and ChatGPT, but at least they had some memory about me. With these single player agents, it's so easy to jump between them because all the context is in my Gmail."
"I think in a year what we'll see is that one of the biggest consumer AI platforms will be multiplayer and will have network effects in it. Right now we just sort of agree that single player is the way to go for most AI products."
"A year from now, I think we'll look back at it as more kind of the early innings, and the products that win will have a lot of network effects and be focused on you talking to other people, not just AI."
@ekuyda w/ @sophiadew on @MTSLive
3
1
148
Just added support for SPACE in Agentlane in the latest build.
More in the docs on how to set it up getagentlane.dev/docs/harnes…
we open-sourced SPACE, the sandbox platform we use internally for secure agentic runtimes. it powers Computer as well as our RL training and evaluation workloads.
github.com/perplexityai/SPAC…
Very inspiring work! 👏
The statue of Prometheus is finally finished.
It was cast in Paris, crossed the Atlantic, and was assembled at Starbase. It took us 100 days to go from nothing to this.
People of the West, we have lived through an era devoid of dreams, devoid of destiny.
But the time has come to reclaim our dream of antiquity.
In a few months, we will start building a new city. It will be the most magnificent city in all of history.
After all, the great dream of the West has always been to rebuild Rome.
Come walk beside us in this endeavor. The world is waiting.
2
85
Excited to launch the Functional Medicine Bench,
@diadia_health's new benchmark we are using internally for evaluating LLMs on complex patient treatment tasks.
Most benchmarks for clinical AI evaluate medical knowledge through multiple-choice exams, question answering, or evals on whether a model cites the right evidence. Most frontier models are getting better on these benchmarks.
But decisions that determine whether a treatment plan helps or potentially harms a patient gets much less attention in various benchmarks. Often what gets missed is:
- which additional tests are needed
- whether a treatment should start now or wait for a test result or an another step
- whether it belongs in the plan at all
- what dosage to use
- in which order the treatments should begin
This is how a functional medicine mind typically thinks through the case and this is the gap we are aiming to close with this work.
diadiahealth.com/research/fu…
Below is a snapshot of how Diadia performs on these questions versus frontier.
2
1
101
Read our blog: diadiahealth.com/resources/f…
41
I just followed @deedydas advice and had Claude create an explainer for Agentlane.
This is mind blowing, definitely adding it to the repo and a website next.
every single open source repo should have an explainer video like this 7min one for SQLite
1. explains what the repo does and why its useful
2. shows a high level map of the code (beautiful graphic!)
3. life of a query through the codebase
4. the core abstractions in the repo
5. a real trace of execution and what happens, with a bit on join-order query planning
and yes, this was all generated by opus 5.5 w/ gemini 3.8 tts. i sound like i'm on repeat, but im in shock at just how coherent and capable this model is.
everything truly is code.
1
88
Spent Saturday morning doing some large format photography.
This is the kind of activity that helps me fully reset and refill my energy and creativity.
8x10 film is my meditation medium, spending this time in the most beautiful nature on Earth 10x this experience.
1
92
read our blog:
diadiahealth.com/resources/f…
7
IMO using AI to write papers is a perfectly natural fit. Paper is a technical report, it uses technical language, facts, formulas, tables and graphs. Prose describes the method and is supposed to be factual and accurate.
Story writing though is totally different and this is where a writer's voice is supposed to come through, rather than machinery language.
I also believe that in any kind of story writing there exists a silent trust contract between a writer and a reader, and this contract is violated if the time it took to produce a piece of content with AI is less than being asked from the reader to spend on reading it.
1
59
While working on our Diadia harness from the ground up I found the most helpful way to think about context is to imagine it as a system boot sector. So I used this analogy to decompose Codex's and Hermes's contexts as an example below.
The idea is that the context, just like a boot sector, has immutable pre-allocated sectors, mutable sectors that are loaded by the operating system (harness) and user files (conversation history).
So the design process becomes:
1. decide and pre-allocate N segments (decide which are immutable)
2. allocate slots in each segment
3. populate each slot with instructions and estimate the size (using tokenizer)
4. optimize
88
We are nearing the top
Wow... Instinct in talks to raise at a $10B val, @sequoia + @benchmark looking to lead (@theinformation)
100k+ users, free product, hitting capacity limits + heavy compute costs so they need $$$
Val trajectory: $100M → $500M → $2.5B → ~$10B in a few mos! W/ no revenue... 🤯
Longer term they want to buy their own chips + run their own data centers
theinformation.com/articles/…
1
4
184
"Trust in clinical artificial intelligence (AI) cannot be benchmarked into existence."
This is why at Diadia we are approaching verifiability of medical claims via mechanism graphs rather than relying on purely on benchmarks.
Published in Nature Medicine: @Google researchers and collaborators share lessons from prospective studies of AMIE, highlighting why real-world clinical evidence is critical to evaluating conversational AI in health.
Read more: goo.gle/4Ac1b4P
93
Most of the discussion about AI safety centers around coding and software stack exploits which are highly verifiable outcomes. Frontier models excel at verifiable tasks, because this is how they were post-trained.
But in domains where verifiability is not so obvious, like healthcare for example, most of the discussion boils down to whether the model is capable of citing the right research when prescribing a treatment or creating a diagnosis, making it look good on a benchmark.
But this is quite misleading and loses a lot of nuance that goes into this kind of reasoning and it's hard to verify how one treatment is better or worse than the other, what exact dosage should be prescribed and what order of treatment should be followed, especially in non trivial cases.
We started working on a benchmark which we first apply in a functional medicine space and tested a few frontier models vs our own Diadia harness that we built together with our doctors.
Below are just a few snapshots from a larger benchmark, and the key issues that we are seeing comparing to our own harness:
- Missing the order of treatment (this is almost a flip of a coin in terms of what part of the treatment should start first)
- Over-prescribing treatments patients might not need
- Missing key treatments that doctors would've prescribed
- Messing with the dosing
Getting these outcomes wrong can harm a patient even when every citation is correct and the model does well on a benchmark.
Even if AI frontier is paced tomorrow, the reality is that most domains haven’t caught up to capabilities of models even two generations ago let alone the frontier.
Lots of domain knowledge and expertise is still trapped inside bespoke workflows and is largely an untapped territory of net new demand.
The model release cadence maybe slowed but the demand is not slowing.
FWIW this is bullish for startups.
1
55
Andrii Yasinetsky retweeted
What Is To Be Done?
I propose a simple plan:
Big AI companies should be allowed to build AI as fast and aggressively as they can – but not allowed to achieve regulatory capture, not allowed to establish a government-protect cartel that is insulated from market competition due to incorrect claims of AI risk. This will maximize the technological and societal payoff from the amazing capabilities of these companies, which are jewels of modern capitalism.
Startup AI companies should be allowed to build AI as fast and aggressively as they can. They should neither confront government-granted protection of big companies, nor should they receive government assistance. They should simply be allowed to compete. If and as startups don’t succeed, their presence in the market will also continuously motivate big companies to be their best – our economies and societies win either way.
Open source AI should be allowed to freely proliferate and compete with both big AI companies and startups. There should be no regulatory barriers to open source whatsoever. Even when open source does not beat companies, its widespread availability is a boon to students all over the world who want to learn how to build and use AI to become part of the technological future, and will ensure that AI is available to everyone who can benefit from it no matter who they are or how much money they have.
To offset the risk of bad people doing bad things with AI, governments working in partnership with the private sector should vigorously engage in each area of potential risk to use AI to maximize society’s defensive capabilities. This shouldn’t be limited to AI-enabled risks but also more general problems such as malnutrition, disease, and climate. AI can be an incredibly powerful tool for solving problems, and we should embrace it as such.
To prevent the risk of China achieving global AI dominance, we should use the full power of our private sector, our scientific establishment, and our governments in concert to drive American and Western AI to absolute global dominance, including ultimately inside China itself. We win, they lose.
And that is how we use AI to save the world.
It’s time to build.
I bet Shopify is such a great product and successful company precisely *because* of this.
There’s always natural resistance to such behavior, there are even armies of leadership coaches who will tell you this is bad.
It’s important to not believe them and just to follow your own instincts as a founder.
100
If AI is gonna kill anybody then it’s by misdiagnosis or mistreatment while sounding plausibly convincing
Months of work, and today we are finally launching our medical AI transparency lab.
It's a reasoning system we have built at Diadia Health that verifies the high-severity medical claims an AI makes when it generates a diagnosis or a treatment protocol, breaking each claim down into a mechanisms graph and checking every pathway in that graph against real evidence.
What we are trying to do here is to provide the next form of transparency in the medical AI domain as models get stronger and their claims will keep sounding more right.
While most of the discussion about medical AI happens at the level of benchmarks and high-level numbers that are often meaningless in a day to day work of a clinician, we believe it is more useful to look at the actual claims AI makes with full evidence trace attached and reasoning validated.
Earlier we published a study of several frontier models across a variety of medical cases and claims, and showed how much their reasoning, outcomes and agreement diverge. The average divergence in verdicts was 31% per patient, over 85% of claims were unique to a single model, and only 3 of 36 patients (8.3%) showed consensus across models.
It makes it hard to trust an AI diagnosis, harder for clinicians to do their work, and harder to deploy these models in real medical settings.
Transparency Lab aims to shed more light on AI’s medical reasonong providing a live catalog of thousands of verified claims, with evidence traces, contradictions and reasoning paths for anyone to explore.
transparency.diadiahealth.co…
1
93



























