Likes breaking ideas and systems, writing, picking systems apart @confluentinc Ex @Splunk, @VMware https://xitter.cf/t.co/voJWmL5iBE Credit: ESO/B. Tafreshi
Barcelona, Spain
Joined November 2016
- Tweets1.8K
- Following252
- Followers5K
- Likes1.1K
Jack Vanlightly retweeted
Recent security incidents are fundamentally changing how I think about open source.
The original idea behind open source was great: bring people together to build better software through collective intelligence.
But in the AI era, things are moving in a dangerous direction. Contributors are no longer humans, but AI agents.
As AI becomes more capable, malicious contributions will inevitably become more sophisticated. And with an overwhelming volume of AI-generated pull requests, thorough human code review simply cannot scale. This could undermine the very foundation of open source.
I see two possible paths forward:
- Stop accepting code contributions. Accept PRDs and specifications instead, reviewed by trusted humans. Let AI agents handle the implementation.
- Build superintelligent code review agents capable of comprehensively detecting security vulnerabilities, malicious code, and subtle attack vectors.
Open source is fundamentally a good thing. But until we achieve at least one of these, I think we need to be extremely cautious about its future.
Jack Vanlightly retweeted
Unfortunately, you can’t outsource understanding any more than you can outsource sleeping. You can, of course, pay someone else to do it. But that doesn’t accomplish your goal.
Jack Vanlightly retweeted
Want to get better at (or started with) formal methods? Surround yourself with experts @hillelogram @DominikTornow @k0nn0v @muratdemirbas @lemmster @ilyasergey @ankushpd @Leonard41111588 @MarcJBrooker @vanlightly @immadnaseer
You're welcome.
Jack Vanlightly retweeted
The Consensus tracks over 120 of the best developer blogs, and that number grows each week. There's now a dedicated feed page for this group. And RSS as well. No subscription needed.
theconsensus.dev/feeds/devel…
Jack Vanlightly retweeted
This is a wonderful write-up by @vanlightly on the work we did together at Confluent; I particularly like the connection he establishes to the fantastic post by @muratdemirbas on modeling vs. modular abstraction, which Murat pointed me at just a couple of days back.
There’s been some interesting discussion recently around WALs on S3, and separately around abstraction as reduction. I’ve done a write up on the LogDrive OSDI paper: how it extends the shared log protocol Virtual Consensus by decomposing the Loglet into separate sequencing and durability abstractions.
This is research that came out from Confluent's Kafka-on-S3 work and next-gen K2 engine (which powers Freight Clusters). The interesting part is that reducing durability to a smaller abstraction makes it much more composable, making it easier to adapt the log to changing storage services/prices with only minimal code changes.
jack-vanlightly.com/blog/202…
Jack Vanlightly retweeted
Modularity abstraction versus Modeling abstraction
muratbuffalo.blogspot.com/20…
Jack Vanlightly retweeted
New blog post!
"Lorenz and Little: How Much Does Your Tail Cost?" takes a look at how much tail latency contributes to the concurrency in a system, and how much that contributes to cost.
brooker.co.za/blog/2026/07/2…
Jack Vanlightly retweeted
📝 "A Fast Path for Fixed-Length Lists in Parquet"
New blog dropped, discussing how we optimize reading effectively fixed-length lists (think vector embeddings) in Hardwood. Up to 3.9× faster, getting to the level of flat columns for larger lists.
Link below 👇
Jack Vanlightly retweeted
The lines between OLTP, OLAP, and HTAP are getting incredibly blurry.
In his latest piece, @vanlightly tackles this head-on and proposes a brilliant, unified taxonomy for modern storage/workload architectures.
Let's break down the 5 main categories 👇 🧵
Jack Vanlightly retweeted
A fundamental problem with extending Codex/Cowork/Code to all knowledge work is that they remain very "software-brained" where the end result (the software) is what is important & that code serves as a source of truth.
For a lot of other knowledge work, the process is at least as important as the outcome. This includes researching what is known, an exploration of alternatives, failed efforts, prototype branches, experiments, etc. All of those things are valuable, so you cannot use the PowerPoint at the end the way you can use a codebase, nor is progress on a to-do list sufficient context post compaction. You work in learning loops, refining your perspectives as you go.
In some ways, this makes long-running models like Fable hard to use for deep knowledge work, since they are designed to deliver product to you in the end. You can prompt your way around this problem, but everything about the Codex and Code harnesses want you to be a software developer and you have to fight them. There is a real disconnect between how a manager or analyst thinks about problems and how the agentic software tools approach solving them. Addressing this is critical to breaking out of the coding niche for these tools.
Jack Vanlightly retweeted
It May be time … for the May edition of Interesting Links in the Data & AI World!
rmoff.net/2026/05/28/interes…
Thanks to the folk continuing to publish great content that I have the pleasure to link to, including @vanlightly, @VictorRentea, @MichelTricot , @jthandy, @marklit82, @DataMozart, @pdrmnvd, @AMdatalakehouse, @teivah, @jamessewell, @BenjDicken, @shanselman, @croloris, @MikeMcQuaid, and many more :)
Jack Vanlightly retweeted
📝 "Hardwood Reaches Beta: S3, Predicate Push-Down, CLI, and More"
Happy to announce Hardwood 1.0.0.Beta1, adding S3 support, predicate push-down for both local and remote files, Avro bindings, a CLI for inspecting Parquet files, a new website, and much more. Links below 👇!
Jack Vanlightly retweeted
Hello, hardwood-cli 🖥️!
Happy to share the latest addition to the #Hardwood project: a command line tool for inspecting Parquet files. Think of it as a Swiss Army knife for Parquet; use it to get general file info, schema, encodings, compression ratios, page details, and more. You can take a quick look at some records, convert to JSON, etc.
Early access builds for Linux/macOS/Windows available from the GitHub repo (link below); a first tagged release of the CLI is planned for the next week.
A big, big shout-out to @brbrown for spearheading this effort. While I had planned this for only after the 1.0 release, he didn't hesitate and made it happen right now. That's the magic of open-source in action!
Jack Vanlightly retweeted
And the old group coordinator implementation is gone from Apache Kafka - love it when open-source projects can delete large chunks of complex code.
github.com/apache/kafka/pull…
Jack Vanlightly retweeted
If you are looking for formal models of a real-world distributed system, DeepSeek @deepseek_ai released P specifications for their new distributed file system (3FS): github.com/deepseek-ai/3FS/t…
Jack Vanlightly retweeted
There's a tiny set of Great Explainers in distributed systems (@muratdemirbas, @adriancolyer, @martinkl come to mind) who can describe state-of-the-art research better than the original papers did -- @vanlightly is definitely in that set!
New distributed systems protocol write-up!
Virtual Consensus (Delos) heavily inspired the new architecture in Confluent’s Kora engine, powering Freight Clusters. This write-up dives into Virtual Consensus in Delos paper and why it makes sense as the default log replication protocol in the era of object storage and hybrid environments.
jack-vanlightly.com/blog/202…
Jack Vanlightly retweeted
🎃The October issue of #CheckpointChronicle is now out 🌟
It covers @Ververica's Fluss, Flink 2.0, @iggy_rs_, @strimziio's support for Kafka 4.0, tons of OTF material from @vanlightly, @chollinger93's write up of @ngrokHQ's data platform, and more!
dcbl.link/cc-oct240
Jack Vanlightly retweeted
One of the main reasons why there is so much burnout in this space. Everyone is copying everyone, pretending to be ticking all the boxes instead of trying to build a cohesive great product that solves some hard problems in ways others don’t solve.
