
Building Shepherd w/ @StanfordNLP – git for agent execution.
Stanford, CA
Joined April 2007
- Tweets242
- Following333
- Followers316
- Likes2.5K
A friendly reminder that OpenAI still has not disclosed the identity of Books1 and Books2, two datasets used to train GPT3. They deleted the datasets. But... they're shaped like LibGen.
Anthropic is taking a real beating in recent news cycles. Still not sure if this is sus 🤔
🦔AI companies are bulk-buying rare books, scanning them through high-speed machines that cut the spines off, and shredding the originals. A service called ISBNdb facilitates orders of up to a million books and keeps buyers anonymous. Pre-2022 books are premium because they're free of AI-generated text. A federal judge ruled the practice is fair use because eliminating the original means only one copy exists at a time. Anthropic hired the former head of Google Books partnerships to obtain "all the books in the world."
My Take
This got to me. A bookseller told 404 Media that rare books with almost no surviving copies are being fed into this pipeline. Books that survived wars, fires, and centuries of handling are being shredded so an AI can learn to write a better marketing email.
ISBNdb's website literally says "'AI company destroys two million books' is not a headline that generates sympathy," and they still built an entire business around making it happen quietly. They offer NDAs as a feature. They coach clients to call it "digital preservation."
I've covered AI companies scraping the internet, torrenting libraries, and stealing music. This is worse because it's irreversible. You can re-upload a website. You can reprint a bestseller. You can't replace the last three copies of an 18th-century botanical text once someone shreds them for training data. And the judge said it's legal. So it's going to accelerate.
"We shred rare books and offer NDAs so nobody finds out" is a legitimate business model in 2026. What a timeline.
Hedgie🤗
Replying to @RosarioTechSolu @RoundtableSpace
Author here. It lets you revert! But full disclosure, it's still quite a young framework – it works great for research needs, but we might need a few weeks before it's robust enough for production.
Attached: this is the kind of syntax we're building towards. Rollback is as natural as discarding variables in nested scopes!
1
1
140
Replying to @feulf @_avichawla
For DB forks/rollbacks, we have a proof of concept that seems to be working using Postgres savepoints. But it's definitely not production grade yet.
On the larger question – we're building something like a clean surface / encapsulation layer for agent execution. It's a little hard to express the full vision in words, but we're working very hard on the roadmap – please revisit the GitHub in a few weeks, and you might get a clearer view!
Little preview attached.
2
50
Replying to @benellsworth01 @_avichawla
We have something cooking on this front too: teapawt.org/dch/status/20738153158…
We've got some early proofs of concept of the above working for a Postgres instance, a GitHub login (raising PRs, etc.), and actually for email too!
Screenshot relevant. The agent within thinks the emails are going out – but they get captured into a Git-like "dev branch" instead for human review.
Replying to @_avichawla @Timur_Yessenov
Great point – we actually have plans here too! We're writing components that let you snapshot external systems (email, DB, etc.) like a local git pull, let agents make changes freely, and "git push" back to the real world on human approval! Coming soon...!
1
35
Replying to @shi_weiyan @karpathy
It's still surprising and unintuitive to me that multi-turn dialogue simulation works as well as it does!
People believe that LLMs are bad at multi-turn for many reasons. Was mode collapse just driving them out of distribution the whole time?
2
588
Replying to @GeorgeCheng65 @shi_weiyan
That's odd – what prompt did you use? We definitely verified that it works with Gemini 2.5 Pro; screenshot related. Different prompt variants work best with different models, though. For example, we occasionally get confused refusals with Claude if the instruction isn't in the system prompt!
84
Replying to @jtotheren @alex_prompter
Amazing! That was part of what motivated this project – I suspect it'll work unusually well!
We're going to do a more academic facing deep dive next week. This went completely under the radar, but distributional accuracy on multi-turn dialogue is actually very surprising...!
2
3
151
Replying to @NateCodez @alex_prompter
Surprisingly no – mode collapse in RLHF makes models put almost all their probability mass in one answer, so temp stops helping!
Example: I just measured the logprobs of "Name a U.S. state" with GPT-4o (GPT-5 stopped offering logprobs)
1
1
64
Replying to @dch @alex_prompter
(*Not exactly the same, but unusually similar – the below are from Claude Opus)
36
Replying to @alex_prompter
We definitely run into this in every model we try, and confirmed it happens locally, so this isn't caching!
FYI sometimes the words you get will be different, but the underlying story will be the same. Try asking for a few stories about a bear, for example!
1
41












