OSS Engineer | Speaker • Trainer | #DatabricksMVP | Author @OReillyMed | ❤️ #ApacheSpark. ❤️ #Dogs. #DatabricksMVP. Views are my own
California, USA
Joined November 2008
- Tweets2.8K
- Following940
- Followers873
- Likes5.1K
Scott Haines retweeted
As a longtime @neondatabase user, there is one feature that I have always wanted.
Well...
Scott Haines retweeted
We experimented with TypeSafe AI's Jev the only way we know how: using Protobuf. That turned into protoc-gen-jev, an experimental plugin that generates typed Jev clients for Go, TypeScript, and Python.
buf.build/blog/asking-jev-qu…
Scott Haines retweeted
Join us tomorrow, Oct 13 at 9 AM PT for the Delta Lake Community Meetup! 🎉
On the agenda:
🔹 Parquet FILE data type protocol support
🔹 Concurrent identity column generation RFC
🔹 delta-rs integration with Unity Catalog Delta APIs
🔗 RSVP: luma.com/w6gqy0ul
#DeltaLake #OpenSource
Scott Haines retweeted
connect-go v2 is here.
We took four years of production experience and redesigned the API: simpler method signatures (no more generic Request/Response wrappers), pluggable transports, a unified interceptor model, and a migration tool to automate the upgrade.
Learn more: buf.build/blog/connect-go-v2
Scott Haines retweeted
Join us next Wednesday for Schema-on-Read Without Regrets!
Geir Alstad (Gabler AS) and @newfront (@databricks) will walk through Spark Variant shredding in a real pipeline.
🗓️ Wed, Sept 30 · 8:00 AM PT
🎟️ Register: luma.com/open-z92a
#ApacheSpark
Scott Haines retweeted
How well do you actually know Protocol Buffers? We put together a quick interactive quiz covering schema design, field rules, and wire format mechanics.
protobuf.com/quiz/
The questions are now randomly selected from a question bank so retrying will give you a few new questions.
#protobuf #grpc #quiz
Scott Haines retweeted
Omnigent v0.14.0 is now available! 🎉
This release makes it easier to work across GitHub repositories, configure sessions, and keep track of ongoing work.
🔀 Multi-repo GitHub PRs and sandboxes
🎛️ Redesigned composer
🪟 Claude SDK /compact
🗄️ Gensee, Kubernetes, CockroachDB
🔁 Approvals that survive restarts
Learn more 👉 omnigent.ai/releases/0.14.0
#Omnigent #AIAgents #OpenSource
Scott Haines retweeted
At Airflow Summit, Meni Shmueli from DataFlint covers how #ApacheSpark and @ApacheAirflow run together in production, including splitting CPU work (like reading Parquet) off GPU machines so AI jobs aren’t paying GPU rates for CPU-only steps.
Apache Spark 4.2 is out: auto CDC and Real-Time Mode for streaming, DataSketches as first-class operators, and Spark Connect so client/server queries come back in milliseconds instead of spinning up a full app.
#SparkConnect #DataSketches #ApacheAirflow
What’s New in @ApacheSpark 4.2: Spark Connect, DataSketches, and Airflow youtu.be/ePDTF5BZd5k?si=8h8h… via @YouTube
RT @ApacheSpark: 📣 Schema-on-Read Without Regrets: Lessons from Apache Spark™ Variant shredding in Production
Join Geir E. Alstad (Gabler…
Scott Haines retweeted
The Replit-Databricks integration is now generally available, with new native Lakebase support!
Now, you can use @Replit to build full-stack Databricks Apps on live, governed enterprise data. Replit Agent can automatically provision a Lakebase database, while AI-proposed schema changes require human approval.
Build, deploy, and iterate in one flow.
See it in action: replit.com/blog/databricks20…
Scott Haines retweeted
What’s New in MLflow: September 2026 Roundup xitter.cf/i/broadcasts/1kKzDPqQY…
Scott Haines retweeted
3 weeks out: Open Lakehouse + AI Meetup is coming to Paris! 🇫🇷
Agenda:
🔹 @unitycatalog_io (You Can Go Your Own Way....Go Your Own Way)
🔹 @omnigent_ai — A Meta Harness for AI Agents
🔹 AI Agents at Every Scale: Personal, Team, and Data Workflows
Then networking, light bites & swag.
📅 Wed, Sept 23 | 6:00–9:30 PM CEST
📍 La Fondation | Paris, FR
🎟️Register: usergroups.databricks.com/ev…
#OpenLakehouse #AI
Scott Haines retweeted
Delta Lake 4.4.0 is out! 🙌 It simplifies engine integrations for catalog-managed tables via Delta Kernel and the UC Delta APIs.
Delta Spark now defaults to Spark 4.2. Identity columns in CREATE TABLE. Flink primary-key upserts.
🔗 Learn more: delta.io/blog/2026-08-20-sim…
#DeltaLake #ApacheSpark #OpenSource
Scott Haines retweeted
Apache Spark is built for large, fault-tolerant jobs: query plans, stages, tasks, and retries. On small local data, such as about 20 MB of Parquet or JSON, those scheduling steps can add roughly 100 milliseconds each, and the iteration loop gets slow.
Project Feather is a SPIP to make that laptop and local-mode loop faster without a new engine or API changes. Spark committer Daniel Tenedorio, a co-author of the proposal, walks through the work here 👉
youtu.be/p9ABo1BK6Zc
#ApacheSpark #OpenSource #DataEngineering #Spark
Scott Haines retweeted
Mark your calendars! 🗓 The Delta Lake Community Meetup is back on Tuesday, September 8 at 9:00 AM PT.
Here’s what we’re covering:
🔹Delection Vectors Support in CDF using Kernel – Stephen Carman (@Databricks) breaks down technical updates & key challenge fixes
🔹delta-rs 1.0 – Tyler Croy (Buoyant Data) highlights the latest features and major updates
RSVP 👇
luma.com/delta-7kbm
#deltalake #opensource #oss
Scott Haines retweeted
DevConnect is back on tour for H2 2026, built for experienced Databricks builders who want a closer look at what’s worth shipping next.
Expect deep dives into:
• Lakehouse//RT, LTAP, Disaster Recovery, Multi Table Transactions, Catalog Managed Commits, and open table interoperability
• MCPs, Unity AI Gateway, ZeroOps, and Omnigent to cut boilerplate, reduce context switching, and keep costs in check
• Apps, Genie, and custom chat dashboards to put more builds in the hands of non-data teammates
Find a stop near you and register: bit.ly/databricks-devconnect
Scott Haines retweeted
Introducing protobuf.com a visual, interactive guide to Protocol Buffers.
Scott Haines retweeted
1 week away ⏳ delta-explain: Making Delta Lake File Pruning Visible, and Testable in CI
See how Delta Lake file pruning is attributed to partition pruning and data skipping from transaction-log metadata, then turn that analysis into a CI contract before regressions hit production.
📅 Aug 18 | 9AM PT (virtual)
🎟️ Register: luma.com/delta-odhl
#DeltaLake #OpenSource
Scott Haines retweeted
Apache Spark 4.2 is now available and it continues maturing Spark Connect. 🙌
It's a thin client over gRPC and Arrow: the client builds the plan, the server runs it, results return as Arrow batches, no full runtime or JVM on the client. 4.2 also keeps closing the compatibility gap with Spark Classic.
🔗Download Spark 4.2: spark.apache.org/downloads.h…
#ApacheSpark #SparkConnect #PySpark #ApacheArrow