Founder: Project VAIL. Father, Friend, Advisor, Entrepreneur, Technologist, Creative Thinker. Thinking about post-AI world views.
San Francisco, CA
Joined October 2008
- Tweets3.1K
- Following941
- Followers1.7K
- Likes1.9K
Pinned Tweet
We should know more about AI models, where they came from, what they’ve been trained to do, and what their reward functions are. Model provenance is much more complex than just asking which country a lab is based in that trained a model.
Manish Shah retweeted
We know we're in alignment with others when we trust them, and it's not a mistake. But what is trust? Where does it come from? How can we develop it? My cofounder and I, with our agents, wrote something on the topic:
softmax.com/blog/origins-of-…
Everyone I spoke to when the first reports of the hugging face incident agreed that there was zero chance this was limited to just HF. These incidents are going to keep coming.
What’s needed is a way to detect these things BEFORE they happen, not after.
Excited to share the first report (of many) from Megan, VAIL’s Model Behavior Scientist. There is so much more to explore with behaviors of these idiosyncratic and oftentimes enigmatic agentic systems, at all layers of the stack.
Who You Gonna Call? Evaluating Tool Calling Reliability and Behavior projectvail.substack.com/p/w…
#agentobservability #verifiableAI
Manish Shah retweeted
A group of us from the AI grantmaking orgs, AI labs, natsec policy and cyber worlds are creating an AI Cybersecurity Observatory non-profit that'd provide continuous forecasting and epidemiology around the coming wave of AI automated cyberattacks to policymakers and cyber defenders.
No such organization currently exists and policy is being made on weak signals and vibes.
We have soft commitments of funding pending finding a founder who'd lead the organization and recruit its staff. Would appreciate retweets to help us find the right person. More info and a place to register interest here:
docs.google.com/forms/d/e/1F…
Thank you to @kevtownsend and @SecurityWeek for covering our research with @CiscoAI on knowing where models are truly from. There is a lot more to understanding model provenance than country. We strongly believe in behavior as a way to understand AI - a model IS what it DOES.
Manish Shah retweeted
This OAI/HF piece from @GaryMarcus and Zack Korman -- whose main takeaway is that AI labs should practice defense in depth and suffer repercussions when they don't -- is representative of the response from many security folks.
This position is not even wrong, it's just that it repeats the obviously true security catechism of the past 20 years in the face of a milestone event that marks the beginning of a new era in our field.
It's as though we've discovered Winter is Coming and we're mainly focused on the banal politics of who's to blame in how we found out.
To be clear: should OAI/Anthropic/Irregular/Meta/etc. do the things we've been saying forever and pay a price when they don't? Yes. Is it dangerous that they're not doing them? Yes.
But also -- thought leadership should engage the new questions -- like:
- What do we do about the fact that your median regional hospital network has *worse* security than OpenAI, Anthropic, Huggingface, and Meta? How do we secure tens of thousands of critical organizations quickly? What's AI's role here and what isn't AI's role here?
- What are the potentially destabilizing geopolitical implications of hacking agents that can turn amateur-level non-state cyber actors into elite state-level actors now?
- What will cybercrime look like when amateurish ransomware affiliates suddenly have the capabilities of elite nation states and what do we do about this?
- To what extent will we need to guardrail our networks from internal loss of control incidents now given the business pressures to turn more and more engineering functions over to agents?
Now live (and free) at Marcus on AI: takeways on the OpenAI / Hugging Face attack, coauthored with @ZackKorman, focusing not so much on what the AI did as on what OpenAI should have done.
garymarcus.substack.com/p/5-…
Manish Shah retweeted
@dee_bosa , I totally agree that we are entering an era of extreme model-choice or as you call it model-agnostic. But the difference is that we aren’t quite at model-equivalence just yet. Will we get there. Likely. Here’s why I say that.
Generic benchmarks increasingly mask the differences that matter in production. When conducting task-specific evals, open-weight models tend to be frontier-class for some workloads, but meaningful gaps may remain in others.
Those gaps are narrowing quickly, and not every workload needs the frontier.
This is why I believe that the winning architecture will route each task to the right model based on quality, cost, latency, privacy, risk and control.
The new moat might become knowing which model to trust for which task, which might vary by company, team and problem set.
.
Manish Shah retweeted
Very cool research from our AI security team. We already treat software as a supply chain, and models are no different. Worth a read!
Models don’t have passports; they have supply chains.
New @Cisco and VAIL research examines how AI lineage can cross organizational and geographic borders—and how artifact and behavioral fingerprints can provide evidence consistent with documented technical lineage.
Learn more below.
🔗Blog: blogs.cisco.com/ai/model-pro…
🔗Paper: cisco.com/c/dam/en/us/produc…
We @projectvail are excited to share research with our partners @CiscoAI that explore provenance of model origins. This work is an example of how two independent approaches arrive at similar conclusions. Often the discourse in the AI ecosystem is “first to the key” wins. It doesn’t have to be. We can do much stronger work by corroborating and collaborating with each other to reproduce and strengthen research. And because of AI, we can explore many paths at once so multiple teams pursuing different rollouts will be the norm. It was a real joy to collaborate with @notjustanychang and @CiscoAI on this work, aligned in our goals but in pursuit from different starting points, ultimately coming back together.
Models don’t have passports; they have supply chains.
New @Cisco and VAIL research examines how AI lineage can cross organizational and geographic borders—and how artifact and behavioral fingerprints can provide evidence consistent with documented technical lineage.
Learn more below.
🔗Blog: blogs.cisco.com/ai/model-pro…
🔗Paper: cisco.com/c/dam/en/us/produc…
“Verification requires monitoring, monitoring is only as good as the behavioral data it trains on [7], and behavioral data for agents is a network property: the same agent must be observable across every model and provider it touches, because misuse, misalignment, and compromise all present identically at the point of action, as transactions. Security for agents is therefore the deployment-time alignment problem, and deployment-time alignment, like fraud detection before it, is a data problem.”
Love this and this is why we built arena.projectvail.com - an independent data stream to assess and verify different inference providers serving the same model. Instead of looking at meta data or transaction shapes, we monitor for behavioral baselines and stability from our own input probes. Deviations are detected when changes are made by the inference providers, and we can calibrate to very high levels of sensitivity.
We think there should be much more verification tooling as we go further with AI systems.
today, @OpenRouter is joining @stripe
this is good for frontier alignment and ecosystem security
you can read why, here: amppublic.com/research/openr…
thank you @alexatallah @cclark and team for your stewardship - here's to all the scaling ahead of us
nostalgia in thread below
So:
- Labs use benchmarks to see if they are SOTA in specific domains
- Companies are building more and more bespoke benchmarks to measure performance in specific domains (which get rented to labs to check perf)
- Developers and orgs choose models based on benchmarks in specific domains
And yet, the real world experience doesn’t live up to scores on benchmarks. Newer models do well but overthink, sometimes older models do just as good, experience on Monday doesn’t match Tuesday.
Feels clumsy.
An analogy I like to use is, would you know @StephenCurry30 is the greatest shooter of all time if you only ever reviewed his three point contest scores? We need much better ways for models to pass the “eye test” during game time performances.
Manish Shah retweeted
Important: how do frontier labs monitor their agentic evals, because OpenAI’s agents were rummaging around doing things they shouldn’t for months leading up to hacking HuggingFace.
What if the agents were doing far worse stuff, causing more harm? No one would know?
Manish Shah retweeted
Nice new Web interface from the @interconnectsai team
> Artifacts Hub
> Tracking the models that matter for the open AI ecosystem
GG @natolambert @xeophon
Very excited to finally share artifactshub.ai with the community. My wish was that we can lift up those that are leading the frontier of open source / open weight models. And it is fun to collab with @natolambert and @xeophon on this.
It's a good time to share a new @interconnectsai project we made to help make sense of the accelerating open-weight releases these days. The Artifacts Hub builds on our monthly open model roundups and daily monitoring of every model on @huggingface.
The new free resources are:
1. The Artifacts Hub — a curated view of the models trending on Hugging Face, highlighting inference tokens via Open Router, model intelligence via Artificial Analysis, and our tailored adoption metrics building on top of Hugging Face’s data.
2. Our Adoption Dashboard — a living dashboard of download and derivative model numbers by geography and organization. This highlights the US-China gap and growing players in the open ecosystem.
To date, our primary efforts on Interconnects have been release recaps for popular models like Kimi K3, GLM 5.2, DeepSeek R1, etc. and monthly round-ups of the open models that matter, Artifacts Log. We’re expanding on these, building on the tools and internal data we’ve collected for other projects like The ATOM Project (and report). This allows us to capture our ecosystem view of open models, develop methods for understanding adoption of giant MoE models, and everything in between. We’re sharing them freely to help the open ecosystem find its strengths and grow.
The Artifacts Hub right now covers 792 models released in the last two years, across the core text-focused language models and multimodal generative models. At Interconnects we follow the data of every model on Hugging Face, analyze the core few thousand LLMs (this list is public on GitHub and regularly updated), and hand select these core few hundred for further explanation.
For the most popular models, the Hub let’s you quickly see how far behind the model was in terms of frontier intelligence based on Artificial Analysis’s Intelligence Index, compare Hugging Face and Open Router adoption to similar models, glance at relative adoption metric (RAM) scores for time-size normalized downloads, or look at the VAIL similarity index of models with related generations. A snapshot for what you’d see for something like GLM-5.2 is below.
Thanks to @mnshah at VAIL for encouraging us to make this and @huggingface, @OpenRouter, & @ArtificialAnlys to making such useful data openly available.
When new models are released, they get a lot of attention and fanfare. It moves the narrative towards what are the next set of capabilities unlocked.
But, what happens to the previous models that were once the shiny new thing and have now been relegated?
At Project VAIL we're continuously monitoring Anthropic's APIs every hour for the last 6 months to understand the behavioral stability of the models they serve. The results show when newer (& bigger) models are released, its the older models that sacrifice.
Full write up here: projectvail.substack.com/p/m…
Also, check out Stability Arena: arena.projectvail.com