The demo works in a week. Three sites, clean data, meeting concludes this is straightforward. Correct, and irrelevant. Building was never the expensive part, year two is.

70% of the web had a rough Wednesday. Sites went offline worldwide to patch PHP2Shell. Your scraper saved "We'll be back soon!" as a product title and logged it as 200 OK. Patch your servers. Then check your data supply. Link in the replies. #PHP2Shell #DataForAI

Every data team has one collection job nobody has touched in years. No owner, no alert, no dashboard. It is usually fine. Occasionally it has been quietly returning a subset for eighteen months and the volume looked plausible enough that nobody checked.

Compliant collection is four decisions. What is the purpose, stated narrowly. What is the permission basis per source. What happens when a control is hit, which should be stop. What trail is attached per record. Public availability is not a basis.

If collection were banned tomorrow: price comparison dies in a week, salary benchmarking follows, and academic research takes the quietest hit and the longest to notice. Search engines are built on the exact mechanism being argued about. That part never makes the argument.

Crashes are cheap. Blocks are cheap. You find out immediately. The expensive one is a job that silently stops covering a subset of pages. Volume looks normal. Totals look normal. A whole category quietly leaves your analysis.

One of our valued customers came to us looking for more data sources. But the real challenge was elsewhere: their engineering team was spending most of its week rechecking data they already had. Their pipeline had no way of flagging when a source silently changed its structure.

The part of in-house collection that never makes it into a budget is silent failure. A run that breaks at 2am and alerts nobody does not appear as a cost. It appears six weeks later as a decision made on stale data.

Most collected websites, 2026: TikTok, Google, ChatGPT, Naver, Perplexity, Make.com, Target, Amazon, GitHub, Stack Overflow. Three survive from last year. Teams stopped asking what people search for. They are asking what the answer engines say back.

From the access report: provenance per record is becoming a procurement requirement, and most providers cannot produce it on request. That is the finding people argue with. It is also the one their legal team will raise next year.