Spark on Kubernetes at scale is hard. Onehouse runs a lot of it, and tomorrow we're sharing what we've learned. In most places, the knowledge required to get all of this right lives across the data team and the Kubernetes/infra team. We've collected a list of things that looked simple at first, but hard in production: - OOM kills - Shuffle storage - Spot interruptions - Autoscaling - Long-running drivers - Cluster upgrades - Keeping streaming jobs alive while the infra underneath changes Most of it lives in the gap between the data team and the infra team. Webinar tomorrow. Bring your worst production problem. Link in comments.

Last week, I posted on why warehouses still default to closed formats. What I learned: they won't flip the default. Open formats are opt-in. So the fix is with the users. Give every open format (Delta, Hudi, Lance, Vortex, ClickHouse, DuckLake) the same consideration on performance and flexibility that you give the closed ones. Your data will thank you. Back to building on open formats.

Last week, Databricks CEO Ali Ghodsi and I debated Iceberg and open table formats on a post. Then I caught myself. Why are we debating each other while Snowflake, BigQuery and Redshift still default customers to closed formats? Two years of Iceberg announcements. Still no public retirement date for their closed formats. Open-source projects get scrutinized for interoperability on every release. The largest warehouses should meet the same bar. We’ve spent a decade building open formats and data lakes. I think it’s time we asked.

🚀 Introducing the Onehouse AI Gateway: model routing + MCP tool governance, without lock-in, wherever your data lives. 🔧 What is it? A unified inference + MCP gateway with shared access controls and built-in observability at no extra-costs. Get blazing-fast SQL tool access to any open table. Swap models, connect any catalog, and deploy the gateway right next to your data or your agents. 💡 Why it matters Snowflake’s Cortex AI Gateway routes only to Snowflake-hosted models. Databricks supports external providers, then charges $0.10/GB for inference logs and usage tracking. With Onehouse, you can use any model provider and store traces in open formats on infrastructure you control. ⚙️ How you use it Add provider credentials (OpenAI, Anthropic, etc.), define model aliases, set fallback and traffic-splitting rules, and apply user/group access controls. Then point Claude Code or Codex at the gateway and go. 📊 We’re already running it on our own data lake. The blog has the full details.

NetApp previewed AI Data Engine integrations with Onehouse at Insight today. Read together with Novus, their new fast file system for AI, it brings modern lakehouse compute to data centers and Neoclouds. The last decade of lakehouse innovation has mostly stayed locked inside the three major clouds. BigQuery, Snowflake, Databricks all require their managed services. Onehouse runs on open formats and containerized on any Kubernetes cluster. Same speed or faster, with maximum portability. This integration unlocks three patterns: 1️⃣ Data centers running Hadoop-era stacks finally get a path to faster Spark and interactive queries on existing storage, without migrating. 2️⃣ Neoclouds generate massive volumes of training and trace data but have no real data platform. Agent traces need to be joined with outcomes and curated into training datasets. That's exactly what a lakehouse handles well. 3️⃣ NetApp storage systems hold decades of enterprise documents and archives. AI can analyze unstructured data at GPU speeds, without blocking on human data entry. What would you build if you could query all the files in your filers and NFS mounts alongside your business data?

The thing that makes AI agents useful ("non-deterministic reasoning") is exactly what makes them impossible to debug. Traditional automation was boring because it was predictable. Same input, same code, same output. AI is useful precisely because it breaks that model. It can "think on its feet", at 1000x human speeds. So, if you give an agent the same goal twice and it may take a different path, call different tools, retrieve different context, and produce a different answer. That autonomy is the value prop. The balance lies in ensuring that when something goes wrong, you can reproduce what actually happened. Teach it, Coach it. For that, versioning the model is not enough. You need the exact state of everything that influenced the result: • source data • transforms • embeddings • indexes • policy/config • model checkpoint Miss one, and you are not replaying the same run. AI makes systems non-deterministic. Versioning is how we make that non-determinism debuggable, auditable, and reproducible.

Caching is supposed to make queries faster. But on lakehouses, simply moving Parquet files into memory or fast SSDs often barely moves the needle. The common assumption is that data access is network-bound. With modern cloud networks offering amazing throughput, the bottleneck often isn’t the network anymore. It’s the CPU.

AWS EMR and GCP Dataproc both ship Hudi 1.x now. Most AI apps require four systems: Structured data → lakehouse Embeddings → vector DB Binary objects → object storage Semi-structured → document store Every write has to propagate across all four. Keeping them consistent is brutal. Hudi 1.2 collapses that. Native VECTOR, BLOB, and VARIANT types in one table. Vector search, binary storage, hierarchical queries—same transactional guarantees. The consistency problem disappears because there's nothing to sync.

🔥 📣 Your lakehouse already has the data. Your training pipeline doesn't know how to use it. We're announcing AI training data pipelines on Onehouse for @baseten , @FireworksAI_HQ , and @togethercompute Training data curation today feels like carrying lake water in a leaky bucket. Most teams "version" training inputs by dumping JSONL files into a cloud folder. When a label changes or an evaluation surfaces a gap, there's no way to trace which examples came from where, or rebuild the dataset from updated source tables. Consider a support agent learning from past conversations. Useful examples require joining each conversation to its outcome: Was the issue resolved? Was the answer correct? Did someone intervene? The conversation alone isn't enough. The evidence lives in operational tables. When the model handles a class of requests poorly, the next training dataset needs to reflect updated labels, corrected data, and new selection logic. Teams need the selection query, the source-table versions it read, and the history of those corrections. 🔄 These are familiar data engineering requirements: joins, versioning, lineage, backfills. The lakehouse already handles them. Here's what we built: 1️⃣ Quanton curates examples from lakehouse tables using SQL selection logic—join conversations to resolution status, filter by outcome quality, batch vector search for good/bad examples at scale. It writes versioned exports as training datasets. 2️⃣ Onehouse submits the training job to Baseten, Fireworks, or Together AI and tracks status in the UI. The provider trains the model and returns the fine-tuned weights or adapter. 3️⃣ Lakegres serves point-in-time context during training or eval—queries lakehouse tables for the exact documents, records, or state needed to replay a past trace. The dataset remains reproducible. Engineers can inspect the selection query, trace examples to source records, and rebuild the dataset when labels change or evaluation surfaces gaps. Go to quanton.dev for a free account and give it a spin, let us know!

Companies spend billions on Spark compute every year. Most assume open-source accelerators can close the gap to commercial engines. We ran TPC-DS 10 TB across 5 engines on the same 11-node cluster to find out. OSS Spark: 12,200s (baseline) Comet (tuned): 9,122s (25% faster) Gluten (tuned): 8,563s (30% faster) Databricks Photon 18.2: 2,550s (5× faster) Quanton: 2,384s (5× faster) Open-source accelerators can get you partway there—under ideal conditions.