
Everett Roeth retweeted
in a world where data interoperability is a must, it's absurd to see the games dbx plays...
Everett Roeth retweeted
When someone tells you "we use Unity Catalog," what do you think they mean?
Turns out there are two!
• Unity Catalog OSS, an LF AI & Data sandbox-stage project
• Unity Catalog, Databricks' proprietary, closed-source governance service
I decided to dig into the data.
Everett Roeth retweeted
Why is Apache Arrow becoming the universal columnar foundation for modern data and AI infrastructure?
Apache Arrow is great because of its language-independent columnar memory format, zero-copy data interchange, efficient in-memory processing, and broad multi-language ecosystem.
That’s why projects like Apache Spark, Apache Parquet, ClickHouse, Dremio, Polars, Pandas, InfluxData, OpenObserve, Greptime, LakeSail, Spice AI, IOMETE, Daft, Apache Doris, and so many other great systems use it.
For data and AI systems, Arrow is not just another format. It is becoming a foundational layer for moving and processing data efficiently across the entire ecosystem.
RisingWave, a unified data platform for real-time and Iceberg-based lakehouse workloads, also uses Arrow extensively, particularly for interoperability with Apache Iceberg, Apache DataFusion, Parquet, UDFs, and some connectors.
Shoutout to our partner @columnar_tech for advancing Apache Arrow and ADBC. They're really doing great work and redefining database access and connectivity. 🙌
A data platform that's simple, transparently priced, and ridiculously fast.
The LakeSail Platform runs your Spark SQL and Python jobs on a Rust-native runtime. It's faster than Spark, drastically cuts hardware costs, and scales from a single node to a multi-node cluster. It reads and writes your Delta Lake and Iceberg tables where they already live, so your lakehouse stays open and your data stays in place.
For those who want to run Sail in production most efficiently, with all the operations handled for you, this platform was built for you.
Sail ran derived TPC-DS at 3TB and 1TB across a distributed cluster, and the results are the latest evidence for our vision in rethinking distributed compute.
Hadoop pioneered open-source distributed processing, built around disk. Spark moved intermediate data into memory, though its shuffle stayed disk-first. When we started Sail, we envisioned another shift in distributed compute, toward doing even more of the work in memory, with an engine built entirely in Rust.
We built the first version of Sail's distributed architecture in November 2024, with Arrow Flight carrying shuffle data between workers, and redesigned the cluster control plane earlier this year.
Arrow Flight shuffle can look surprising at first. A TPC-DS query with dozens of stages and a few hundred partitions can have more than a million logical shuffle streams active in the cluster. Arrow Flight is built on gRPC and HTTP/2, so one TCP connection per pair of workers carries them all. The same approach has since come up in the DataFusion Comet and Riffle communities as well, and we are glad to have started proving this direction two years ago.
The full benchmark ran in 1,289 seconds at 3TB and 444 seconds at 1TB, with no shuffle data written to disk. On similar hardware and settings, Sail outperforms the published TPC-DS results for Spark and Spark accelerators at these scales.
The post covers the results, the design, and the full setup: lakesail.com/blog/tpcds-benc…
Sail is the first open-source engine to ship with built-in Jev support! Check out the docs to get started:
docs.lakesail.com/sail/lates…
Sail now includes Jev built natively into the engine, the first of its kind.
Any Spark SQL or DataFrame query can ask Jev about every row of a table, with nothing extra to install. The answers come back as typed columns you can filter, join, and aggregate like any other data. The same query can run on your laptop or across a cluster.
Check out the full release and some examples of it in action. Link in thread.
Sail 0.7.1 adds the first ever Rust implementation of Apache Celeborn as a shuffle backend.
If you already run Celeborn for your Spark workloads, Sail uses that cluster as it stands. The same masters, same workers, nothing is changed on the Celeborn side.
Read more on the engineering in our blog:
lakesail.com/blog/sail-celeb…
Everett Roeth retweeted
All ahoy— @LakeSail team is coming to Amsterdam! Speaking at @pydataamsterdam and our own meetup with @adyen on Wednesday: bay.news/amsterdam1 — join us for food, drink, Rust, AI and Data!
The interesting part of rebuilding Spark in Rust wasn’t the Rust.
@shehab_amins joined Daniel Beach on Data Engineering Central to talk about why Spark compatibility turned out to be much harder, and more interesting, than expected. And what changes when the thing querying your lakehouse is an agent instead of an analyst.
Delta vs. Iceberg, streaming vs. batch, Arrow and DataFusion. Full episode below.
dataengineeringcentral.subst…
Ever wondered if your Spark code runs on Sail?
Find out it in seconds with our new Spark compatibility scanner: migrate.lakesail.com
Sail 0.7 has been released, with two highly requested features: blocking shuffle and checkpoint.
In Spark and most engines, shuffle and checkpoint data sit on the workers themselves, so a lost worker loses data and a running job pins its cluster. Sail 0.7 moves that state out to object storage: failed tasks retry by reading persisted data back, finished stages hand their workers to the next one, and iterative workloads checkpoint instead of recomputing every loop.
Check out the full release: lakesail.com/blog/sail-0-7-b…
Everett Roeth retweeted
If you’re curious and love to learn, there has never been a better time to be alive.
Everett Roeth retweeted
is it over for the JVM?
👉 join me as I talk to "Jordan has no life", a cracked distributed systems youtuber with >100k subs, about all things data infra & agent coding:
• columnar processing and how the industry is standardizing on it (Apache Arrow) 🏹
• the proliferation of new file formats (Vortex, Lance, Nimble, F3) 🎁
• object storage as the backbone for modern data systems 💪
• the open core business model (e.g the Tabular acquisition and Iceberg's PMC dominance) 👀
We also touched on all the second-order effects from AI coding revolution:
• English becoming the working abstraction for code
• human reviewers becoming the bottleneck (especially in OSS)
• focus on benchmarks, tests and design more than implementation
• testing practices for the AI era - triaging flaky tests with AI, fuzz/chaos testing a-la TigerBeetle
Last but not least, we touched on Jordan's two latest projects he's been working on: StreamFusion and IceStream 🌟
• StreamFusion accelerates Apache Flink by offloading Flink SQL operators to Apache DataFusion using Rust and Apache Arrow 🐿️
• IceStream, which optimizes streaming Iceberg upserts through an Apache Paimon-backed index to asynchronously convert Iceberg equality deletes 🧊
TIMELINE
0:00 the future of the Data Ecosystem: new file formats, columnar batching & object storage
8:00 Jordan's Background (HFT trading firm, Google)
13:21 OSS vs proprietary infra
22:25 the future (or present) of engineering \w AI
29:03 the asymmetry of code review & testing (\w AI again)
33:49 jordan's StreamFusion project - accelerating Flink in Rust with DataFusion and Arrow
50:16 jordan's IceStream project - optimizing streaming upserts with a Paimon-backed index to async convert equality deletes
57:52 table formats (Apache Paimon, Iceberg & more)
1:11:35 object storage as the foundation of new infra
Everett Roeth retweeted
Use Grust, the universal fast graph API in Rust:
pair.rs/grust
Grust Book on First Pair Press:
fp.ms/grust
Foundation of the QueryGraph stack:
QueryGraph.ai
Supports 15 backends with GQL/Cypher, including the fastest and Rust graph databases such as
@LakeSail — the AI native Rust Lakehouse
@helixdb
@SurrealDB
@tursodatabase
@evokoa_ai pgGraph
Postgres 19 with graph extensions
@falkordb
@cocoindex_io for graph change streaming














