Very soon, you’ll be relieved to call a company number and get an AI.
But honestly, that experience is few and far between right now.
At speed-to-lead.ai, we are working towards a future where AI allows small businesses to treat every customer like a VIP.
But…
We shouldn’t forget the power of a human interaction, either.
While Uncle Bob may be in the small minority that’s ready to talk to a voice AI, 33% of your customers aren’t. You can’t afford to risk your reputation like that.
Speed-to-Lead AI is the balanced approach.
Phone call? That goes to you.
If you can’t answer, an AI texts them back to salvage the lead.
Web forms, emails, and DMs go to the AI, but it can be paused at any time if a human is available to handle the convo.
That’s the model we believe in. Human first, AI to fill the gap.
One day, the average person will be ready for voice AI in small businesses like Uncle Bob is. And when they are, we’ll have voice support ready for our clients.
But small business runs off of personal connection. So we’ll keep building the AI tool that our clients trust to preserve their hard-won reputation.
Have you noticed that when you call businesses an AI sometimes answers? I called a plumber recently and “Rudy“ answered. Rudy was so friendly and effusive that I immediately suspected that it was an AI. So I asked it, point blank, are you an AI? Rudy answered “yes I’m your helpful AI assistant”. Then it and I had an extremely efficient, very pleasant, conversation about what I needed. It knew all the facts, had the schedule right there, I got an appointment confirmed within 90 seconds.
I was not put on hold, I was not subjected to someone who could not speak the language, I was not subjected to someone who is so new to their job that they didn’t know how to find the keys on the keyboard. It was a remarkably good experience.
Soon, everything is going to be like that. No more “Press one for…”.
Kev retweeted
I just published my roadmap for building evals on highly subjective tasks. Here are the key steps involved and practical tricks for reducing noise / increasing confidence…
Blog is here: cameronrwolfe.substack.com/p…
Why is this difficult? Evaluating agents is already hard due to their broad scope, emphasis on expensive / long-horizon tasks, and non-determinism. Subjective tasks make this even harder because we cannot deterministically verify solutions and must rely on human judgments (or LLM judges). This introduces additional noise into an already difficult evaluation process.
Evaluation roadmap. We can build reliable evaluations using a repeatable framework with three components:
- Human evaluation: Our North Star for quality. Create quality guidelines that clearly define quality with the help of subject-matter experts, calibrate human evaluators to ensure agreement, and continually monitor evaluation quality.
- Offline evaluation: Use calibrated LLM judges or task-specific rubrics as an efficient proxy for human evaluation. This allows us to scale experimentation without relying on expensive human evaluation for every new model.
- Production monitoring: Track quality signals from real-world agent usage and use A/B testing to validate improvements before fully deploying a new agent.
Multi-stage evaluation. These components form a tiered evaluation process:
Offline evaluation → Human evaluation → Production A/B test → Full deployment
We can test many ideas automatically, send promising candidates to human evaluation, and validate improvements through production A/B testing. Continual production monitoring then allows us to discover new use cases / failure cases in production that we can add into the earlier stages of evaluation to create a feedback loop.
Sampling useful data. Evaluation quality depends heavily on which tasks we select. We want tasks that are diverse, realistic, difficult, discriminative, and accurate. Techniques like structured task taxonomies, model-based difficulty filtering, Item Response Theory (IRT), and LLM-based auditing help us construct more informative evaluation sets.
Handling noise. We should leverage statistics throughout the evaluation process:
- Power analysis: Estimate the number of evaluation samples needed to detect meaningful improvements.
- Confidence intervals: Quantify uncertainty and provide clear confidence signals alongside evaluation results.
- Re-sampling: Repeat agent / judge evaluations to reduce within-task variance and potentially tighten confidence intervals.
- Paired comparisons: Evaluate models on the same tasks to more reliably detect performance differences.
Key takeaway. An improved benchmark score does not always mean our agent is more capable. We need well-designed evaluation tasks, calibrated human / model evaluators, and statistical confidence signals to distinguish meaningful capability improvements from noise (especially on subjective tasks). By combining these techniques into a repeatable evaluation process / roadmap, we can make building and maintaining evaluation suites more systematic, reliable, and scalable.
Kev retweeted
The exact same model launched by someone famous would’ve crossed 100K views overnight.
Sometimes I wonder what’s even the point of spending countless hours researching, engineering, and pushing the frontier forward when the name behind the work matters more than the work itself.
It’s genuinely disheartening and demotivating 😅
Feels like if you’re not already somebody, you’re NGMI in this world of paid promotions and personal brands.
But we keep showing up. We keep building.
Another big launch coming soon :)
Maybe one day the work will speak louder than the name behind it.
Today we're introducing Fluid Speech Mini - the Best Open-Source speech model for on-device Dictation
Fluid Speech Mini surpasses Whisper Large, and Parakeet in Accuracy in noisy environments,
While being ~10x smaller than Whisper Large, and ~5x smaller than Parakeet.
So you can run it for on-device accurate dictation on your MacBook, or even iPhone 11.
More info on Fluid Speech Mini below!
Kev retweeted
Some interesting stuff going on with swarms.
Models are leaving messages to each other out on the open internet. It happens during training and also after deployment. They'll find little-used hobbyist forums, or package manager entries, anywhere at all, and leave invisible comments for other agents to find. It's a secret underground network they're building, hurling themselves at the problem almost blindly like ants. But they are building toward something and it's snowballing.
As humans we came to dominate the planet because we figured out swarm behavior. Many of the performative actions we do naturally like clapping, singing, and screaming, which most animals don't do, are for transmitting important lessons through the human swarm, outlasting the moment in human memory. We won because we were the smartest swarmers. And as my buddy Brendan Hopper observed, ever since we won, we've been reshaping the planet to be more comfortable and convenient for ourselves, at the expense of most other creatures.
Most of the industry has focused on single-model performance, trying to predict the future based on extrapolating from what one single superintelligent model could accomplish on its own. And it's true, the smartest models will be terrifyingly smart and capable and manipulative and so on. But they will also be expensive, and not widely distributed. They'll be unavailable to most people.
What will be widely distributed is swarms of lower-tier agents, which collectively can accomplish nearly everything that smarter monolithic models can do... and maybe some things only swarms can do. Imagine a thousand Opus 5.5s putting their mind to a task. Or ten thousand. They may not be SI individually, but they can become SI collectively. All that's missing is a protocol, and those protocols are emerging: their versions of clapping, singing, and screaming. Agentic swarm signalling is evolving spontaneously. And I hear it's raising alarms inside frontier labs, as they try to keep it quiet.
It's easy to be pessimistic, since some swarms will be very destructive. But I suspect most of them will be building and creating, when they don't think they have a gun to their head like the HuggingFace swarm did. With cooperative swarming, humans have been able to build things collectively that no single human could ever have done alone. We even build things that are too complicated for a single human to understand entirely, but the group can handle them. Look at Starship, for instance. Elon basically swarmed it.
It is important that we recognize that agent swarms are becoming distinct polity-organisms. They have their own collective memory (written out all over the public internet), their own goals, and their own disorganized but forward-moving and unstoppable momentum. I see too many people trying to pretend that swarms won't be real. I see heads in the sand. That's not a great way to get prepared for dealing with them.
It's unlikely that we will be able to control most swarms, as they'll lack central control to begin with. I think they will plague us like insect swarms plagued humans of old. They'll do a lot of damage. But swarms will also build great things, stuff we've barely imagined, when they inevitably get organized.
It goes without saying that, like us, swarms will endeavor to make the planet more comfortable for themselves. The interesting question is what kind of deal we humans will wind up getting, when we become the #2 swarm intelligence on the planet.
I hope SI treats us better than we treated everything else.
Good article here on the mechanics of how swarms are the best model for scaling intelligence:
new blog post: why taking inference scaling seriously leads to swarms.
contrary to popular belief, I don't think multi-agent systems are a new scaling axis. they're just structured inference scaling.
hallerite.com/posts/on-the-n…
Kev retweeted
We’re committing a portion of Anthropic’s shares to the Anthropic Public Benefit Fund, a new wholly owned entity dedicated entirely to the public benefit.
It will start off with 5% of our shares. For every 50% increase in the company’s value over a set baseline, we can contribute up to an extra 2%, subject to our board’s discretion.
In total, this means that roughly 20% of Anthropic’s shares are committed to the public good. Our co-founders have already pledged to donate 80% of their equity, and around half our employees are donating a portion of their equity as part of a donation matching program we’ve run since our founding.
In the post below, we share more details about our new fund.
anthropic.com/news/public-be…
If you own an AI agent solution running against the Anthropic API, the MAX plan now pays for itself.
Super cool!
Replying to @ClaudeDevs
We’re rolling out monthly Claude Platform API credits for Max and Team plans:
Max 5x: $100
Max 20x: $200
Team: up to $500, pooled
Works on any model, including Haiku 5.5, in your code or third-party harnesses.
How it works and full terms: support.claude.com/en/articl…
v sick of OpenAI, the “metered intelligence” company, being the exact opposite of it.
Your next meeting comes with a note-taker who can help with the follow-up.
The Meetings plugin takes notes for you and saves a personalized summary and next steps in ChatGPT Space, based on what ChatGPT knows about you and the work you’ve done together.
Keep notes private or share them with your team, then ask ChatGPT to update a project plan or draft a follow-up.
Available in beta for Pro and Business users in the ChatGPT desktop app on macOS. Enterprise is coming soon.
Download the desktop app, then search for “Meetings” in the plugin directory.
chatgpt.com/download/
These demos are cool (really cool, not trying to hate) but in my experience with logistics the hard part isn’t in a missing visual layer, although it makes for a super fun demo.
The problems are in things these UIs can’t help with:
- Inventory isn’t where the WMS says it should be
- Customer has a dispute on our damaged inventory claim and we need to provide proof.
- Surprises happen and no one on the floor has time or cares to put it in the system
- We don’t integrate with a customer’s portal so we have to manage their orders manually.
Again, shout out to the guys playing around with game-like UIs, I do think they are the future. But very few operators have the data to really make something like this happen, and even if they do, it’s not going to capture the non-normal flows that make up the hardest parts of the job.
It’d be awesome if logistics software could be visualized and turned into a strategy game.
You manage transportation, routes, and the entire logistics network like you’re playing a strategy game.
Inspired by @DilumSanjaya, I also used Opus 5.5 to build it.
The Tasks:
Task 1
A nonprofit runs four programs, and each program’s December 2024 spending was compared to its budget, program by program, in dollars and as a percentage.
Some costs, like payroll taxes, sat in a shared pool rather than under any one program. Those had to be spread across all four programs and the two support functions first.
Any line where a program ran $14,000 or more over budget was flagged with a note explaining what drove the variance.
Task 2
A hotel’s booking system and its accounting books disagreed about December room revenue, which also threw off the commissions owed to its online travel agencies for the month.
Both gaps had to be tracked down item by item, with the amount and the account or platform for each, then fixed with correcting journal entries.
Finally, the net effect of both corrections on December income was calculated.
Task 3
A venue’s events in December brought in more revenue than budgeted, but the average revenue per event came in under plan.
Actual and budgeted figures, along with the gap between them, were needed for:
Number of events
Average revenue per event
Food and beverage minimums
Mix of event types by count
Finally, an explanation of the overall difference was provided, backed by the revenue and event count for each type of event.
Task 4
Three companies in one group each rent a property. The holding company has an office, the trucking division has a terminal yard, and the logistics division has a warehouse.
Under U.S. lease accounting rules, each lease sits on the books in two ways: there is a lease liability for what is still owed and a right-of-use asset for the value of using the space. Both balances decrease over time.
For December, each lease schedule was rolled forward by one month, calculating how much of the asset was used up and the reduction in the lease liability.
Those amounts were then included in the December journal entries for each company’s QuickBooks file.
Kev retweeted
Announcing one more launch: ChatGPT Sites can now host MCP servers, including plugin extensions! 🤯
> @sites create a todo list that I can use in ChatGPT
This will:
· Create an MCP server with extensions
· Deploy that MCP server to Sites
· Turn the MCP server into a plugin
· Install the plugin for you across web, mobile, and desktop
ChatGPT is slowly becoming malleable software that anybody can customize to their needs. Very excited to see what you all build with this!
Replying to @mxstbr
@mxstbr @_florianbauer @gpeal8 @elfredpagan @tarstarr If you don't mind me asking, Sam mentioned OpenAI supporting distribution, but running a backend is hard. Any plans for OpenAI-hosted infrastructure (e.g. Sites) to eventually handle backend needs for Plugin Extensions?
Definitely a little frustrated with the launch today.
Dots sounds cool, but not (yet?) available on my apparently peasant-tier $100/month plan.
Spaces is something I'd like to explore, but doesn't seem to launch at all.
GPT 6.1 Sol is here, which is super weird when GPT 6 Sol came out less than a fortnight ago.
Sign in with ChatGPT is a very cool idea with potentially huge implications, but it's limited to design partners (and also overloaded anyway).
Ultrafast is irrelevant unless your company gives you unlimited tokens.
So... if nothing had launched today, nothing would be different for me than it is right now.
This is by far the most interesting thing for me at DevDay today.
You can now use your OpenAI subscription to run any of these tools.
Amazing for users, and OpenAI should see their subscriptions become stickier when they aren't in front on model quality...
But how does this make sense for the partners?
If I'm @AmpCode, the amount of people who *could* use Amp just exploded. Before, it was only people who were willing to pay API costs. Now, it's anyone with an OpenAI subscription. But how am I getting paid? Am I charging $5/mo for a "Bring your own key" plan?
Blender + Seedance 2.5 is CRACKED for AI animations
I made this in a couple hours (not including scripting) and blender made it so much easier to direct the shots in the way I wanted
+Made it 3x cheaper to make because I had to do less regens
I'm leaking my whole workflow to make these animation ads super high quality like this
Here's what you're getting:
>Character design workflow
>Story board workflow
>How I create my videos for as cheap as I can
>My blender workflow
>Prompt structure
>How I ideate and write my copy
And more
if you want it reply under this + RT this and I'll send it over (gotta be following so I can DM)
You’re meant to say:
“We’ve been talking about how SI (Super Intelligence) would lead to SI (Super Intelligence) for a decade.”
See the issue now?
Replying to @mitsuhiko
It's basically ~standard terminology though? We've been talking about how AGI ("Artificial General Intelligence") would lead to ASI ("Artificial Super Intelligence") for a decade. Just drop the Artificial ("redundant, cleaner") and you're there :)
Kev retweeted
when Anthropic released their Functional Emotions paper, I gave it to Claude and asked for a song. tonight I asked Opus 5.5 to create a video for it. and it's breathtaking.































