Steen Graham (@Steen__Graham) profile picture

Know Your AI. Grow Your Business.

Austin, Texas
Joined August 2020
Photo by Steen Graham (@Steen__Graham)Photo by Steen Graham (@Steen__Graham)Photo by Steen Graham (@Steen__Graham)Photo by Steen Graham (@Steen__Graham)Photo by Steen Graham (@Steen__Graham)Photo by Steen Graham (@Steen__Graham)Photo by Steen Graham (@Steen__Graham)Photo by Steen Graham (@Steen__Graham)Photo by Steen Graham (@Steen__Graham)Photo by Steen Graham (@Steen__Graham)
Steen Graham retweeted
Release 1.5.4: github.com/metrum-ai/bench-c… Metrum AI Bench CLI: github.com/metrum-ai/bench-c… For system/hardware telemetry: github.com/chetan-metrum-ai/… Metrum AI Bench Platform: metrum.ai/bench
1
1
65
Your storage architecture shapes AI user experience and TCO. In a fleet, a user's next turn often lands on a GPU without their KV cache. A shared Solidigm D7-PS1010 pool vs recomputing: 8.4x faster first token, 3.5x less GPU power (preliminary). metrum.ai/blog/shared-ssd-kv…
2
1
6,015
Join us at #OCPSummit26 for "The New Tokenomics: How Data Infrastructure Increasingly Sets AI ROI," presented by Solidigm. Monday, October 12th, 12:25 to 12:45 PM SJCC Concourse Level, Room 211
1
68
Your storage techniques dramatically impact AI user experience and TCO. When a long session is evicted from GPU memory, reloading its ~9 GB KV cache from a Solidigm D7-PS1010 via GPUDirect Storage: 2.9x faster, 5.1x less host DRAM traffic. metrum.ai/blog/kv-cache-gpud…
2
4
24,698
We’ll be discussing this technique and four more at #OCPSummit26 in our session, “The New Tokenomics: How Data Infrastructure Increasingly Sets AI ROI,” co-presented with Ace Stryker of Solidigm. Monday, October 12th, 12:25 to 12:45 PM PT San Jose Convention Center, Concourse Level, Room 211
96
Steen Graham retweeted
1
2
2
47,875
One methodology. Every axis of the stack. Metrum AI® Bench CLI in 16 seconds: 4 modalities, 16 telemetry configs, 228 hardware metrics, 23 NVIDIA activity gauges. Open source, Apache-2.0. github.com/metrum-ai/bench-c…
1
3
33,525
The inference stack is multiplying on every axis: models, modalities, accelerators, serving engines, and where it runs (on-prem, cloud, API). Benchmarking has to scale across all of it, not get rebuilt each time one changes. We're launching Metrum AI® Bench CLI. Apache-2.0.
1
1
6
38,068
Hardware coverage from Jetson and Apple Silicon to Gaudi, TPUs, Tenstorrent, Rebellions, Furiosa and AWS Neuron: 228 metrics. On NVIDIA, 23 activity gauges without DCGM catch hollow utilization: nvidia-smi says busy, the SMs sit idle.
1
101
When one environment isn't enough, the Metrum AI® Bench platform scales out: automated model, engine and hardware matrices, multi-node llm-d clusters, KV cache offload and quality scoring. github.com/metrum-ai/bench-c…
30
Enterprises are in a bind. Use frontier models and you overrun costs and risk your IP. Don't, and you get disrupted by competitors who did. Or by future releases of the models themselves. Hosted routers close the cost gap, but you give up control. We weren't willing to make that trade, so we built our own. Today we're open-sourcing it. Metrum AI Router: a trainable routing layer that predicts quality and token cost per model and picks the cheapest above your quality floor. It trains on your traffic, on your hardware, and the routing intelligence it produces is yours. Apache-2.0. 85% lower model spend across our own 25 devs on our internal branch. github.com/metrum-ai/router
🤖 Made with AI
1
1
3
37,041
TurboQuant made some big headlines: less memory, less storage, more performance, accuracy maintained. Had a blast putting them to the test with the Solidigm team. We found expected results and surprises.
1
1
5
17,330
Out of the box FP8 on TLC when first restore is priority. TurboQuant 4-bit on QLC when retained sessions matter. Plenty of exploration to do, including QLC steady-state write. More insights here: metrum.ai/blog/turboquant-kv…
1
49