Deploy AI with Confidence

Joined September 2023
Distributional retweeted
Better tools to do better science faster is the rising tide that lifts all boats toward human abundance. At @TalariaSci we're building a self-reinforcing loop for better discovery greater accuracy in physical AI. Learn more here: talariasci.com/blog/the-righ…
2
6
175
Distributional retweeted
Timely reading for anyone building agent harnesses in domain specific / OOD problem spaces from @DrScottClark
2
2
4
190
Distributional retweeted
I wrote a post about all the mistakes I made as a second time founder and my journey pivoting @dbnlAI to @TalariaSci. Hopefully this helps the next founder that tricks themselves into thinking the second time will be easier (you should do it anyway!). scottclark.io/blog/make-all-…
2
4
209
Big update on the new direction of Distributional from @DrScottClark about the pivot to @TalariaSci
2
175
DBNL helps you make sense of the patterns in your AI agent behavior, detecting where there are deviations and then providing automated recommendations so that you can build an application that behaves like you want it to. Get started: distributional.com
108
Kick off your AI analytics flywheel with DBNL. DBNL transforms traces into actionable insights, ensuring teams can continuously fix and improve their agents with the right evidence in production: dbnl.io/cPRvZf
66
Is your AI agent actually calling a tool when it claims to? Scott Clark explains one of the most common anti-patterns he sees with multiple models today, where the agent gets "lazy" and will say it called a tool but the traces prove otherwise. Learn more about how DBNL's analytics helps teams catch dark patterns: dbnl.io/iMRLJz
42
Every day, DBNL users get fresh insights on their most recent production data. These insights drive the development of new offline metrics to guide reward functions, evals, or context for agents. They can be used to generate new test cases that are run pre-production on agents or in a continuous “integration testing” style setting. And they create new metrics to be monitored real time by more traditional observability tooling. Learn more about how DBNL analytics generate actionable insights: dbnl.io/cPRvZf
34
Your agent handles 50,000 traces a day. Accuracy is 87%. Error rate is stable. To your dashboard, everything is fine. Meanwhile: → A retrieval path is silently returning 90-day-old documents to users asking about recent policy changes. → A prompt tweak from last Tuesday shifted tool-calling order in a way that tanks task completion for multi-step workflows by 12%, for four hours, every Tuesday. → A whole topic category the agent answers confidently is wrong 40% of the time. None of this shows up in your metrics. You find out when a user complains, or when satisfaction scores quietly erode over a month and you can't explain why. Between your monitoring dashboard and your trace viewer, there's an analytical gap. Learn how to convert agent traces into behavioral insights with DBNL. Connect your OTEL traces and get insights in minutes: dbnl.io/Luuawb
38
Distribution analysis answers the questions that summary metrics can't. DBNL's distribution comparison can help you pinpoint the issue—this is especially helpful when evaluating experiment variants: dbnl.io/CjZ9X9
1
43
AI agents are complicated, but catching and fixing issues with them is easy with DBNL. From pre-deployment evals that fail to perform in production to insights on user intent and more, DBNL helps teams identify issues faster: dbnl.io/uVtqmf
33
When you sample traces, you’re making a bet that the interesting patterns are distributed randomly across your data. They’re usually not. Random sampling will systematically misses low-frequency, high-impact patterns. Where does your organization fall on the observability maturity scale? Read and find out: dbnl.io/i00yOD
1
32
Your P95 latency doubled after a deployment, but your average cost-per-call looks fine. Which one do you trust? This is the core tension in LLM agent observability: summary metrics tell you something changed, but not what changed—or for whom. For example, a bimodal latency spike (where half your requests are fast and half are stuck) looks identical to a uniform slowdown when you compress it to a single number. DBNL's distribution comparison unlocks the details. Here's how it works: dbnl.io/CjZ9X9
1
60
AI agent analytics insights and recommendations—automated. DBNL automatically turns logs into action items, helping you identify where your agents could use some tweaks. Learn more: dbnl.io/IvoBGJ
1
1
69
Agent breaking in production? There are 3 types of issues that DBNL's analytics can help catch before they become a bigger problem: 1. Offline, pre-deployment evals that fail to perform out of sample in production 2. Insights on user topics, intent, and input patterns 3. Issues in the complexity of agent behavior dbnl.io/uVtqmf
27
Are you missing key information about your AI agents? Here are the 3 failure modes that are invisible to aggregate metrics: → Behavioral drift by segment → Unknown-unknown behavioral patterns → Gradual degradation vs. hard failures Take our quiz to see where you land, then learn how to close the gap with DBNL: dbnl.io/arGD1z
15
Your evals are passing. Your agent is still failing. This is one of the most underappreciated problems in production AI systems. Distributional CEO Scott Clark breaks it down in this episode of the @twimlai podcast with @samcharrington : youtube.com/watch?v=ZqehXrVl…
29
Get a quick picture of where your AI agent's behavior has gone off course. DBNL makes it easy to identify irrelevant ouput in your logs, which you can then use for offline prompt iteration, reinforcement learning, fine tuning, tool changes, or hyperparameter optimization. Here's how it works: dbnl.io/uXZKvW
21
Most AI teams can see their aggregate metrics, but they can’t see the behavioral patterns inside them — which is where the real informations. Here's why most AI teams today are still "flying blind" when it comes to observability, and how to close the gap: dbnl.io/arGD1z
19
Imagine a Maslow’s hierarchy of observability for your AI agents: Telemetry = Logging what happens Monitoring = Alerting on known signals Online analytics = Surfacing the unknown unknowns Where do you land? Learn how to get to the top in this episode of @twimlai: youtube.com/watch?v=ZqehXrVl…
1
29