AI agents and real-time apps increasingly need analytical answers from data that was written seconds ago. Pointing those queries straight at an operational database slows production and widens security risk. The usual alternative, an ETL pipeline into a warehouse, is costly to operate and can lead to data that's stale by hours or days. Spice 2.0 offers a new approach, and is now available. It stands up a sandboxed analytic replica beside your operational store and keeps it current with high-throughput change data capture directly from those sources. You point Spice at a database and start running analytical queries in minutes, while production never carries the analytical load. High-throughput CDC replication is complemented by the rest of the 2.0 platform: multi-node distributed compute on @ApacheDataFusio Ballista for petabyte-scale queries, Spice Cayenne columnar acceleration on the open @vortexdotdev format, a cluster-sidecar architecture that pairs isolated local sidecars with a shared cluster, and enterprise security enforced in the engine with OIDC, Cedar policies, PII masking, and mTLS. "Spice is the data plane behind Summation. Every connector we ship, from Snowflake to a generic REST-as-a-table, collapses into one SQL surface, which is what makes our AI agents portable across a customer's stack. 2.0's distributed query takes the scaling concern off the table." -Ramachandra Ramarathinam, CTO at @summation   It's open source, portable, scalable, and fast. Read the launch blog for more details: hubs.ly/Q04p5pFw0 And, check out the thread ⬇️ for breakdowns of the major features in 2.0.

The Research Behind Modern Data Compression & @vortexdotdev When we chose Vortex as the storage layer for Spice Cayenne (the data accelerator engine in Spice), we were betting on decades of database research finally reaching production-ready maturity. Here's the research behind Vortex: 📄 BtrBlocks (SIGMOD 2023) - The core algorithm from the Technical University of Munich. Cascading multiple lightweight encodings outperforms monolithic compression. Optimize for decompression speed, not just compression ratio. 📄 FastLanes (VLDB 2023) - Hardware-friendly integer compression. Structures bit-packing to maximize SIMD utilization across AVX-512, AVX2, and ARM NEON. Near-memory-bandwidth decompression. 📄 FSST (VLDB 2020) - Fast Static Symbol Table for strings. Near-LZ4 ratios at 5-10× faster decompression. Critical for string-heavy columns. 📄 ALP (CWI Amsterdam) - Adaptive Lossless floating-Point compression. Exploits real-world float patterns (prices with 2 decimals, sensor readings with limited precision). 📄 MonetDB/X100 + Morsel-Driven Parallelism - Foundations for vectorized, NUMA-aware query execution that Vortex builds on. The result? Compression that is tailored to your data: • Integers via FastLanes bit-packing • Floats via ALP adaptive encoding • Strings via FSST symbol tables • Timestamps via delta encoding • Sorted columns via run-length encoding Why does this matter for production systems? 1️⃣ Query performance scales with decompression speed. Focus on decode performance translates directly to faster queries. 2️⃣ Automatic encoding selection means zero configuration. The algorithm samples your data and picks optimal strategies per column. 3️⃣ SIMD acceleration is baked in. FastLanes was designed for vectorized, hardware accelerated execution from day one. 4️⃣ Zero-copy Arrow access. Data decompresses directly to Arrow arrays with no intermediate copies. Vortex is now a Linux Foundation AI & Data project, and researchers are building on it (Anyblox, F3). You get SOTA research in production systems. The future of data storage is exciting. To learn more about our Vortex implementation, check out the blog: hubs.ly/Q04bGfvf0 #datafusion #ai #data #vortex #spiceai #arrow #parquet