Likes breaking ideas and systems, writing, picking systems apart @confluentinc Ex @Splunk, @VMware https://xitter.cf/t.co/voJWmL5iBE Credit: ESO/B. Tafreshi
Barcelona, Spain
Joined November 2016
- Tweets1.8K
- Following252
- Followers5K
- Likes1.1K
Jack Vanlightly retweeted
Recent security incidents are fundamentally changing how I think about open source.
The original idea behind open source was great: bring people together to build better software through collective intelligence.
But in the AI era, things are moving in a dangerous direction. Contributors are no longer humans, but AI agents.
As AI becomes more capable, malicious contributions will inevitably become more sophisticated. And with an overwhelming volume of AI-generated pull requests, thorough human code review simply cannot scale. This could undermine the very foundation of open source.
I see two possible paths forward:
- Stop accepting code contributions. Accept PRDs and specifications instead, reviewed by trusted humans. Let AI agents handle the implementation.
- Build superintelligent code review agents capable of comprehensively detecting security vulnerabilities, malicious code, and subtle attack vectors.
Open source is fundamentally a good thing. But until we achieve at least one of these, I think we need to be extremely cautious about its future.
My TLA+ spec of Chroma's S3 impl of wal3 is done. Next weekend, the write-up where I'll find commonalities and differences across the surveyed WAL-on-S3 designs, maybe come up with some kind of taxonomy.
I might come back later and cover the rest of the designs in my todo list, assuming the Interest Goblin is still interested by then.
github.com/Vanlightly/s3-wal…
I think I've largely failed to make the wal3 spec generalized and abstract. I think I went too far trying to make it faithful to the implementation and so it has become a monster. Still, it did detect a real bug. I can't spend more time making it beautiful though. Day job. Will publish it over the weekend, then I'm calling it a day on writing more specs. I'll do the final write-up and classification of all the WAL-on-S3 designs I have surveyed.
Well, I got there too late. I just found a correctness bug in Chroma's wal3 this morning, and was working on an integration test to prove what my TLA+ spec found, and on reading the fork, I see that just 2 days ago, the bug got fixed!
github.com/chroma-core/chrom…
Another day, another WAL-on-S3 system encoded in a TLA+ spec. This one is Shared Stream Consensus (S2C) which is more of a shared log for SMR, but fits closely enough to a WAL. After all, a WAL on shared storage is a shared log. My spec confirmed a theoretical consistency bug in the design, related to GC freeing a log address so a stale writer could do a put-if-absent of a log entry behind the snapshot boundary.
A second spec adds a fencing record in the log, SlateDB style, to avoid the scenario.
github.com/Vanlightly/s3-wal…
I'm documenting the fix for the consistency issue I found in Shared Storage Consensus (S2C), and I'm referencing SlateDB (which I copied the solution from). That's what is neat about learning how varied systems work, you can easily steal what you already know works somewhere else! Will publish the latest spec over the weekend after more model checking.
cc @Mo69796244, I'll let you know when you can review ths spec and the variant with the fix.
When reverse engineering a TLA+ spec from code, the hardest thing is understanding the code. While AI can write syntactically correct TLA+, I can't say it's good at writing specs. So I write the spec and use AI to help me review it. But AI is really useful when reading the code. It helps me understand what each class does, the roles of certain variables and structures, things that without AI can be quite time consuming and laborious. But I still read the code, I want a good handle for its structure and organization. Then bit by bit, as I walk the flows in the code, I build up a more simplified model in TLA+. My goal in the end is understanding.
This morning I decided to look at Shared Storage Consensus (S2C) and one glaring correctness bug already stands out before I have started writing a spec. GC can delete log segments that allow a stale writer to make progress (resulting in unreadable data aka data loss). SlateDB uses fencing records which are not GCed for this purpose. S2C does not have this and so GC introduces a consistency problem.
Whenever the WAL is defined by a sequential address space guarded by put-if-absent, GC introduces the risk of allowing a stale writer to make progress resulting in data loss.
Another WAL-on-S3 design verified, Objwal.
That's 4 single-writer WALs and 4 multi-writer WALs so far. Gonna add a few more before I do a classification and write-up.
github.com/Vanlightly/s3-wal…
Even though I've specified 6 WAL-on-S3 designs (plus 2 variants), the backlog keeps growing. Luckily, I'm starting to see more and more overlap and I can reuse major parts of already written specs. So maybe, I might be able to speed things up, though I only get 2-3 hours a day to work on this. This morning I've started on objwal, which on the surface looks like my OpenData Buffer single-writer variant, so hopefully I can lean on that spec for this one.
The Walgit spec is done, bringing the number of designs covered to 6 plus 2 variants. More to go before I start writing up the survey results and start classifying WAL-on-S3 patterns. github.com/Vanlightly/s3-wal…
Which next?
Jack Vanlightly retweeted
Unfortunately, you can’t outsource understanding any more than you can outsource sleeping. You can, of course, pay someone else to do it. But that doesn’t accomplish your goal.