Founder & CEO @Olirainc • Biostatistics PhD • Previously @Harvard @GoogleDeepMind @Penn • Digital phenotyping and ML • Uncertainty motivates discovery
Joined March 2020
- Tweets214
- Following234
- Followers492
- Likes5.3K
Patrick Emedom-Nnamdi retweeted
My favorite insight from this work is that monitorability problems are just way easier if you make use of the trajectories and rewards generated during RL. They have so much to tell you about how your model has learned to (mis)behave. Our reward description training does this.
Super excited to share our new paper: Reward misspecification degrades chain-of-thought (CoT) monitorability!
During RL training, when the model is prompted with one task intent (e.g., response ≤ 900 words) but rewarded for a different or even contradictory behavior (e.g., response > 900 words), the CoT keeps reasoning about the task intent while the output violates it.
We also propose a simple mitigation method that requires no prior knowledge of the true reward or the reward hack behavior. 🧵
Patrick Emedom-Nnamdi retweeted
Great conversation with @BrendanFoody talking about the main bottleneck to enterprise adoption of AI and AGI: knowing whether AI is doing what you want it to. The hard part is creating a measurable specification for qualitative tasks in the real data context, i.e., an eval.
We’ve automated a very low percentage of enterprise tasks with AI.
That’s @jefrankle’s estimate, and he thinks the models are already smart enough to do significantly more.
Jonathan is the Chief AI Scientist at Databricks, co-founder of MosaicML, and my latest guest on The Frontier.
We talked about why evals are the biggest bottleneck to enterprise AI adoption. The intelligence is increasingly there, but enterprises need to figure out how to define what they want agents to do and measure whether they’re doing it well.
Full episode below 👇
Choosing an agentic memory architecture is a design problem that centers on how teams to structure context in their model harnesses.
We’re excited to release our MEMOIR benchmark to help teams evaluate how their approach to context management impacts performance on queries designed around longitudinal patient data.
We’re excited to share MEMOIR, an open-source benchmark for evaluating representations of longitudinal clinical memory.
MEMOIR: Memory Evaluation over Multi-year Oncology Inference & Retrieval includes 117 synthetic oncology patients, 65,377 clinical events, and 3,617 questions spanning factual retrieval, rare-event detection, clinical episode reconstruction, and longitudinal reasoning.
Clinical memory has a different structure from conversational memory. A patient’s history is assembled from events across many systems, at very different frequencies, over months or years. Answering a question may require finding a single event, reconstructing a time series, or connecting observations across an entire treatment episode.
We evaluated several memory and retrieval approaches, including vector-based memory (mem0), temporal knowledge graphs (Graphiti), agent-managed memory (Letta), direct retrieval over Olira’s typed clinical events, and materialized patient state (Olira State).
The clearest differences emerged when an agent had to reason across time. Reconstructing a time series, connecting events across a treatment episode, or understanding how a patient changed over time made the underlying representation much more consequential.
Blog: olira.ai/blog/the-wrong-shap…
GitHub: github.com/olira-ai/memoir-b…
Had an awesome time hosting a Healthcare AI Happy Hour on our office rooftop this week with Dria Ventures! Loved hearing from a diverse group of founders, investors engineers, and researchers about what’s exciting them and what they’re thinking for the future of healthcare AI. Excited to have more of these!
First successful event in the books. Our Healthcare AI Happy Hour last week, co-hosted with Dria Ventures, was everything we hoped it would be.
We brought together founders, engineers, researchers, operators, and more working on AI in healthcare for an evening on our office rooftop. Our biggest takeaway: the best ideas in healthcare AI come from builders talking to other builders.
A huge thank you to the Dria Ventures team for co-hosting and to Cooley for supporting the evening. Looking forward to the next one.
Patrick Emedom-Nnamdi retweeted
As reasoning models consume more tokens and AI systems become more expensive to run, understanding what those tokens actually buy is becoming increasingly important. In this episode, @Stanford professor and Big Spin co-founder @ChrisGPotts joins us to discuss AI tokenomics and his research into “tokenflation”—the possibility that token usage is growing faster than the measurable value those tokens produce.
We explore how to measure the return on AI spending, why benchmarks alone provide an incomplete picture of model progress, and what inference-time scaling means for the economics of increasingly capable models. Chris also explains why expert AI users tend to get better results by challenging and iterating with models, how AI fluency affects outcomes, and why more efficient architectures could change the underlying economics. We also discuss DSPy, interpretability, the limits of today’s transformer architectures, and where Chris sees opportunities for more fundamental innovation in AI.
🗒️ Full show notes: twimlai.com/go/776.
📖 CHAPTERS
===============================
00:00 - Introduction
05:38 - Linguistics in the Age of Language Models
09:26 - Scale Limitations in NLP Research
12:54 - Challenging the Bitter Lesson Mindset
15:12 - Relationship Between Data, Mechanistic Interpretability, and Efficiency
17:03 - DSPy
21:32 - Prompt Optimization and Model Variability
24:35 - Tokenomics and the Rising Cost of AI
28:13 - Measuring Token Purchasing Power with a CPI
32:18 - Inference-Time Scaling
35:36 - Defining Value Across Different AI Tasks
38:33 - AI Value Creation
40:38 - Predicting AI Costs
42:25 - Tokenflation
46:22 - AI Fluency
50:12 - Key Lessons of AI Fluency Work
54:44 - Future Directions
Insanely cute and innovative! This is why I love working in ML. Can’t wait to own one
Sharing some fun experiments with Microduck
Basically, once you have an open-source robot with so many sensors, the sky's the limit
Here we vibe-coded an image detector integration to detect and follow a laser pointer.
More at pollen-robotics.com/microduc…
Amid Epic’s recent AI announcements, it’s most exciting to see a new class of healthcare foundation models starting to enter clinical practice. Epic’s Curiosity (their generative medical event model) is built on 118M patient histories and 115B medical events, and evaluated across 78 clinical and operational tasks. Curious to see how health systems respond and how aggressively Epic doubles down here. Separately, I believe the initial model was developed in collaboration with Microsoft, so interested to see whether that continues or whether Epic pursues this more independently moving forward.
fiercehealthcare.com/health-…
Patrick Emedom-Nnamdi retweeted
Checkout our latest work on instruction steered parameter synthesis. This work is a step towards the North Star of achieving flexible and accurate inference time generation of model weights that target specific tasks / capabilities.
0/ With @derylucio, we co-led the development ofcSkillSmith, an LLM that can perform instruction steered parameter synthesis and composition by treating weights as a native modality. SkillSmith outputs PEFT parameters for a target task given text knowledge and related task weights 🧵
Really clean way of thinking about the harness @hwchase17
A big part of the harness is deciding what context an agent should see, and how it should work with that context for a specific task. It’s also where domain specialization across agents lives. In healthcare, for example, context could be structured against questions like
> What perspective of the patient should the agent focus on?
> What chronology of clinical events matters?
> What medical or supportive information should it lean on for guidance?
Each of these is a pretty deep problem on its own and can be abstracted as a set of tools an agent can call on when appropriate.
gave a talk "owning your intelligence" - ty @sequoia @sonyatweetybird for having me
talked about harnesses and evals and the role they play in owning your intelligence
TLDR:
> agents = model + harness + context
> model - own the weights using something like @FireworksAI_HQ
> context - memory needs to be portable
> harness - needs to be model agnostic. also needs to be good at bringing right context to llm. "right" context may depend on your use case, which is why an open/configurable harness helps
> how to use middleware in langchain/deepagents to configure your harness
> how to use langgraph to fully own your cognitive architecture
> why evals/obs matters - some quotes from @satyanadella
- “Create your private evals, because evals define what “good” looks like inside the organization”
- “retain ownership of your organization’s memory, traces, feedbacks, decisions, and institutional context”
- “you create your own continuous learning loop (i.e. hill climbing machine) that will allow your AI investments to compound the value of your firm”
> how to use harbor for evals
> tracing is important
> evals + observability only matter so you can set up a data flywheel
> data flywheel = run agent -> collect traces -> find interesting traces -> use those to improve
> demo of langsmith engine which does exactly this!
full video: youtube.com/watch?v=HI2q3ci3…
Every healthcare organization already lives with fragmented patient data. CMS’s ACCESS model is the first time that fragmentation shows up directly in what an organization gets paid.
Under fee-for-service, a care team could work around missing context and still get paid for the visit. Under ACCESS, payment depends on actually knowing what happened to the patient.
Data fragmentation was always an operational tax. ACCESS is the first model that turns it into a payment risk.
Very useful breakdown. The detail worth sitting with is that 70% of the estimated value comes from lower medical expenses, not administrative savings. Documentation and claims automation get much of the attention today. The larger opportunity sits somewhere harder: changing care delivery and what happens to the patient.
NEW: A new NEJM paper estimates full AI adoption could cut up to $811B from annual US healthcare spending within ~5 years (~10% of total spend)
The largest opportunities by annual value:
> Private payers: $205B–$357B
Avoidable readmissions, prior auth, claims auto-adjudication, & fraud, waste, and abuse.
> Hospitals: $131B–$220B
OR block optimization, workforce productivity, staffing, supply chains, quality, & patient safety.
> Public payers: $62B–$107B
Care management, provider oversight, & Medicare and Medicaid claims operations.
> Physician groups: $29B–$90B
Documentation, missed-visit prediction, workflow support, continuity of care, & clinical analytics.
Across the system:
• 70% came from lower medical expenses
• 30% of the value came from admin savings
• +50% of the opportunity involved genAI
The line in the caption is the one to sit with. "Decision authority is a property of deployment, not of the model..." And much of what determines it is the reliability and completeness of the patient state the system actually has to work with.
The same model can sit at L4 in one organization and L2 in another. Model capability matters, but it does not determine authority on its own. The quality of the data and system around it determines where it can safely operate.
Our perspective on decision authority in health AI was just published at @Nature Health, w/ Aiden Gu
free access rdcu.be/fwDdv
nature.com/articles/s44360-0…
An understated idea in this letter, and perhaps the most important, is that organizations should be able to own the value they create with AI. Today, most of that value sits around the model: data, context, environments, tools, and evals. Over time, more may move into the model through continual learning and domain-specific adaptation. Healthcare makes this especially important. We still do not know whether durable advantage will live in the data, the harness, the weights, or all three. Open weights preserve the ability to discover that, rather than inheriting one provider’s fixed definition of useful intelligence.
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-W…
Really interesting implication for domain-specific systems: experts often know what success looks like without being able to specify the decomposition. Give the model context, tools, and a success signal, then let it learn through the harness to reduce novel problems to in-distribution steps. This feels like a very natural direction.
So on point @caltchek. The CMS ACCESS Model is a signal of where the industry is moving. Reimbursements for this are intentionally thin to attract parties that have cost structures in-line with low cost, tech-enabled, outcome-based care
This week on Lifers, I sat down with @caltchek, founder and CEO at Cadence. We talked about what media taught him, why healthcare isn't the slog everyone says, and how Cadence is quietly changing chronic care.
Timestamps:
(00:00) Lead In
(00:59) Intro
(02:40) Early life and trading
(05:04) Why investing is hard
(09:05) Founder mindset and grit
(12:54) From media to healthcare
(14:41) Mic lessons and platforms
(17:10) Simple messaging in health
(18:35) Sponsor: Granola
(19:29) Avoiding EHR dependency
(23:16) Cadence business model
(26:24) Unit economics and scale
(28:13) Getting the data flywheel
(30:28) Winning health systems early
(34:59) CMS access model strategy
(40:53) AI for deflationary care
(45:09) Ten year healthcare vision
(48:18) Wrap
McKinsey's 2025 State of AI survey found 88% of organizations now use AI in at least one business function and 62% are experimenting with agents ...but only 7% report fully scaling agents in any individual function. Most of the market is still early enough that the default architecture holds up.
Curious how that translates in healthcare - one of the highest-stakes, most regulated, most data-complex environments agents are being deployed into. The architecture that served early experimentation doesn't automatically survive contact with longitudinal clinical care. Most teams won't find that out until it matters.
The more time I spend on clinical AI, the more convinced I am that trust is not a model problem. It is a patient context problem.
The default agent architecture (e.g. vector database, semantic retrieval) is a reasonable bet for most of what's being built today. Cheap, fast, genuinely sufficient for most current use cases. It breaks the moment it meets longitudinal clinical care, because retrieval answers a different question than clinical care asks. Retrieval finds what looks relevant. Clinical care needs the best-supported state of this patient right now, what evidence supports it, and what's still unknown.
Three axes determine whether a patient-context layer can actually deliver that: how it handles time, whether it understands clinical meaning, and how honestly it represents what it hasn't observed. The current market addresses at most one.