sign up link: onehouse.ai/events/webinars/…

Spark on Kubernetes at scale is hard. Onehouse runs a lot of it, and tomorrow we're sharing what we've learned. In most places, the knowledge required to get all of this right lives across the data team and the Kubernetes/infra team. We've collected a list of things that looked simple at first, but hard in production: - OOM kills - Shuffle storage - Spot interruptions - Autoscaling - Long-running drivers - Cluster upgrades - Keeping streaming jobs alive while the infra underneath changes Most of it lives in the gap between the data team and the infra team. Webinar tomorrow. Bring your worst production problem. Link in comments.

This has got to be the most hype that gzip has had in a decade or more.

Alright . That is in fact pretty freakin' cool.

I feel this is AI generated. 🙂 What tool are you using?

Last week, I posted on why warehouses still default to closed formats. What I learned: they won't flip the default. Open formats are opt-in. So the fix is with the users. Give every open format (Delta, Hudi, Lance, Vortex, ClickHouse, DuckLake) the same consideration on performance and flexibility that you give the closed ones. Your data will thank you. Back to building on open formats.

Does the technical problem simply boil down to a very large table on object storage with fast writes and fairly fast analytical and point lookups?

I like that the ChatGPT Mac app has a built-in markdown editor. It definitely reduces friction in review cycles.

My wrists are happy that I have to type less. This feeling might be same elsewhere too

Not at all. Customers are stuck. We speak to dozens every week. They have to pay hefty egress costs to do the migration. The question I raised is more about first principle and forward-looking. Why not make open formats the default for any new table created from this point forward? That costs the vendor nothing. Btw the LinkedIn counterpart for this x post has a lot more thoughts if you are curious about this topic .

My thoughts and receipts are in the blog onehouse.ai/blog/snowflake-b…

Last week, Databricks CEO Ali Ghodsi and I debated Iceberg and open table formats on a post. Then I caught myself. Why are we debating each other while Snowflake, BigQuery and Redshift still default customers to closed formats? Two years of Iceberg announcements. Still no public retirement date for their closed formats. Open-source projects get scrutinized for interoperability on every release. The largest warehouses should meet the same bar. We’ve spent a decade building open formats and data lakes. I think it’s time we asked.

lol it's actually pretty basic when you see your servers only have 50% CPU utilization.

To squeeze the performance out of our runtime components, we implemented a good amount of concepts from - SEDA: an architecture for well-conditioned, scalable internet services dl.acm.org/doi/10.1145/50205… The seminal work at MIT around writing high-performance server software and the gift that keeps on giving.

I don't have extreme opinions on unit tests, but I hate the false sense of security they give developers, who only later realize the end-to-end functionality doesn't work at all.

Muse, Dots, Grok bot. Eagerly awaiting the consumer AI/agent app explosion, to watch agents melt our databases at scale.

Impressive numbers.

What if you need a super innovative idea to collect the $0.01 ?

Consumer lives and livelihood improving will the ultimate test for this AI wave? Once we start to run out of ideas to use AI to make enterprises efficient.

Not really. more like tradeoffs around madness