Adam J. Gramling

Running long-horizon, agentic data projects is the best preparation for building robust enterprise AI apps bc it forces you to build thoughtful expert <> agent interactions and optimize LLM usage. Data companies have a massive advantage in serving the enterprise. We now spend more on LLMs than on headcount. That does not make human operators, and especially experts, less valuable. It actually makes them more productive and more valuable. Purely synthetic data without experts embedded in the authoring process doesn't work, and data companies that don't dig into their own data will ship slop. They won't produce the nuance, realism, or complexity the task demands, and the labs struggle to catch all these issues because they rely on agentic graders that have the same blind spots. But the token spend is significant, and it goes to three places: (1) hybrid synthetic + expert environment / task authoring, (2) rollouts to validate task complexity, and (3) QC on the data. You cannot produce a high-quality long-horizon agentic task without LLMs across all three buckets. We also use agents to automate internal processes, but that is a small fraction of the bill. Any data company telling you the bulk of its LLM spend is internal automation is lying. Producing this data at high quality and reasonable cost comes down to two problems. (1) the expert-agent interaction paradigm - how the expert creates, reviews, interprets, and audits the LLM's outputs (synthetic input files, prompts, rubrics, and so on). Solving this means knowing precisely where agents fail or underperform, designing the interface through which experts validate those outputs, and building the tools that let experts steer the agent going forward. (2) optimizing every QC check for the right tradeoff between cost, quality, and latency. That means constructing complex and diverse eval sets, configuring an agentic system with the right models (often chained prompts, guardrail LLM calls, and deterministic checks), running the eval, then iterating on the harness, prompts, chaining, and model choice until the mix is optimal. Both are now core skills of our SPLs. Enterprise deployments follow the exact same pattern. Understand the workflow and where models fail or underperform, design the user-agent interaction to mitigate those failures, build evals that test the agentic system across the workflow, and optimize the system against those evals. Running data pipelines forces us through this loop hundreds of times a month across a wide range of project types, and we have to build product and infrastructure to do it fast and repeatably. You cannot build an optimal enterprise deployment without these muscles, and the largest data companies have already done it hundreds of times.