LithosAI (@lithos_ai) profile picture

The fastest agentic inference on the planet. https://teapawt.org/t.co/S0G0RsbcCk

Joined December 2025
Photo by LithosAI (@lithos_ai)Photo by LithosAI (@lithos_ai)Photo by LithosAI (@lithos_ai)Photo by LithosAI (@lithos_ai)Photo by LithosAI (@lithos_ai)Photo by LithosAI (@lithos_ai)Photo by LithosAI (@lithos_ai)Photo by LithosAI (@lithos_ai)Photo by LithosAI (@lithos_ai)Photo by LithosAI (@lithos_ai)
LithosAI is pushing the Pareto frontier of agent inference speed, latency, and cost, and our latest results on Artificial Analysis proves it. Unlike most providers, offering more tiers improves our utilization. Read the full story:
2
7
3
35
479,332
It was standing room only today during @JiaZhihao's talk at SGLang Summit, Agent Inference at Hardware Speed: Co-Designing GPU Kernels and Compilers. We're excited to see so much interest in our team’s approach to pushing the frontier on inference speed and cost with full model quality. 🚀 Missed the talk but want to know why agents need 10x faster inference than conversational AI? You still have a few more hours to catch Zhihao and @dimitriosSkar at SGLang Summit and get their thoughts on why software-hardware co-design is key to agent speed.
1
3
24
1,535
Should time per task become the new standard for inference performance? Our co-founders @dimitriosSkar and @JiaZhihao think so - it’s why the Lithos engine is designed to put end-to-end response first, including for Kimi K3.1 Pro Ultra. As our team’s said over the last few months: the most important metric for agent inference is how long it takes to get useful work. Excited to see this change from @ArtificialAnlys and the shift in how the industry thinks about inference and agent speed!
Announcing the next version of Artificial Analysis API Performance Benchmarking: Time per Task is now our headline performance metric, measuring how long a model takes to complete a task rather than how fast it streams The update applies to the ~1,000 serverless endpoints we benchmark multiple times a day for speed, latency and caching. Key changes: ➤ Time per Task is now the headline metric to compare models: as workloads become more agentic, the time to complete a task matters more than speed per token. A model that is fast per token but verbose can take longer than a slower model that uses fewer tokens. Time per Task divides a model's output tokens per Intelligence Index task by its End-to-End Output Speed. ➤ End-to-End Output Speed covers the full request: speed is now measured from sending the request to receiving the final token. That includes the wait before the first token (prefill, and reasoning for models that hide it) as well as generation. Output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer, and we monitor for discrepancies in reported counts. ➤ Agentic workloads are now part of every measurement: we benchmark short, document-length and long-context prompts, plus multi-turn sessions across coding and knowledge work. Headline figures are the average across these workloads over the past 5 days. ➤ API default effort is now tracked alongside highest effort: most developers run models at the effort level each lab sets as its API default, so we now benchmark key models at that setting as well as at their highest effort. Effort names are not comparable across labs, so we use each lab's own default rather than one shared setting. Key takeaways: ➤ OpenAI models make up most of the Intelligence vs. Time per Task Pareto frontier despite slower output speeds: across effort levels, GPT-6 Astra and GPT-6.1 Sol hold the frontier from ~45 to ~55 on the Intelligence Index. At max effort, GPT-6 Astra runs at 51 tokens/s but finishes a task in ~17 minutes, against ~26 for Claude Opus 5.5 (max, 111 tokens/s) and ~29 minutes for Gemini 4 Carbon (high, 71 tokens/s), because it uses ~50k output tokens per task against ~120k to ~170k. ➤ API default effort cuts Time per Task by ~3 to 5x and Cost per Task by ~2 to 4x for GPT-6 Astra, GPT-6.1 Sol and Claude Opus 5.5: all three default to medium effort, which scores ~5 to 11 points lower on the Intelligence Index in exchange. Claude Opus 5.5 falls from ~26 minutes and $10.04 per task at max to ~6.8 minutes and $2.45 at medium effort. ➤ Endpoints that optimize for decode speed alone see the largest drop in End-to-End Output Speed: high decode speeds can come at the cost of latency, for example by queuing requests to run at lower batch sizes, or by holding output and sending it in a single burst. Both lengthen the wait before the first token, which End-to-End Output Speed now counts.
2
2
7
757
Kimi K3.1 Pro on LithosAI: 4.7 min per Intelligence Index task 🚀 ranked #1 and up to 230% faster than the rest of the top 5 The headline metric for inference speed is changing (for the better), and LithosAI is still the fastest provider for Kimi K3.1 Pro. The K3.1 Pro weights and our Ultra tier inference launched 2 days ago, just in time to test it against @ArtificialAnlys’s new API performance benchmark: time per task. LithosAI now ranks first in time per task for Kimi K3.1 Pro at full model quality. Nothing’s changed for our team. We’re still focused on speeding up what matters most: how fast your agents complete useful work.
1
3
24
1,654
LithosAI is headed to @sgl_project Summit 2026 in SF! Nov 12–13 in SF Our co-founder and CTO @JiaZhihao will be leading an Open Table session on optimizing for the highest inference speed possible while maintaining model quality. Catch @dimitriosSkar and @JiaZhihao there to ask all your inference optimization, AI architecture, and agent latency questions. We’re excited to hear more about what you’re building (or want to build) with more intelligence per second in the loop. See you there!
1
1
17
1,086
🚀 We’re thrilled to be a launch partner with @Kimi_Moonshot! Our new Ultra tier delivers up to 800 output tokens/s/user, bringing blazing-fast inference to the world’s most powerful open-weight model. Frontier intelligence. Ultra-fast inference. Available now. ⚡ console.lithosai.cloud Huge congratulations to the Moonshot AI team on this incredible launch!
Releasing the model weights of Kimi K3.1 Pro: huggingface.co/moonshotai/Ki… We'll keep advancing open intelligence.
4
4
2
23
4,808
Does faster inference actually matter? Isn’t 200 or even 100 tok/sec fast enough for agents? Those are questions we’ve been seeing and hearing as we’ve shared our performance results over the past few weeks, especially as we’ve hit and then surpassed 1500 tok/sec on both NVIDIA and AMD GPUs. That’s why @dimitriosSkar, @JiaZhihao, and the team were excited to sit down with engineering and AI leaders from DoorDash, Uber, ResolveAI, Cline, and more in SF this week. Everyone came ready to talk about what faster inference actually unlocks for AI agents. But the conversation quickly moved beyond the inference layer. The reality is that just like output speed isn’t all there is to inference speed, faster inference isn’t all there is to fast agents. What matters, at every layer of the AI stack and every point in the agent loop, is the impact to end-to-end response time. With every optimization tradeoff we make at LithosAI, we’re thinking about and discussing whether it will actually improve agent performance. At the end of the day, an AI agent only has value once it’s completed acceptable work. Everything that delays its path to useful work reduces its value; so while a faster in-between step might look impressive, it does nothing for the teams that are building and relying on agents. During the dinner, we got into everyone’s experiences of how inference speed, latency, and reliability actually affect performance in production: → Peak inference speed is just a signal of which models and inference engines are worth paying attention to. The real test is how the agent performs with that inference under production traffic and with long-context windows. → Faster inference speed and low latency aren’t always worth having to choose a smaller model; often that leads to retries that make multi-agent systems just as slow or even slower. → Once you have ultra-low latency inference with enough intelligence per second in the loop, you can bring interactivity back to coding, you can rely on SRE agents, and you can trust that your CX agents aren’t frustrating users, even when given complex requests → With open models catching up to the frontier, the next generation of AI applications and industry competition will be about making the agent loop faster and AI stacks more efficient. A big thank you to @thatniamhod and @Unusual_VC for putting on an amazing event. It was great to hear this group’s perspectives on what faster intelligence unlocks. Excited to see how our teams keep building what’s ahead!
7
4
22
2,390
2,100+ tokens per second per user on DeepSeek V4.1 Flash, at full model quality, on eight AMD Instinct MI355X GPUs. In our benchmarks, the Lithos engine delivers: → Up to 3.4× faster inference for DeepSeek V4.1 Flash → Up to 2.5× faster inference for Kimi K3 Compared with tested open-source engines on the same eight GPUs. The key challenge is reducing the coordination tax: time spent waiting between kernels and across GPUs. Our approach combines persistent megakernels, overlapping execution, and communication inside computation. Our technical deep dive covers: - Decode speed, cost per token, and latency up to 1M context. - How Lithos engine reduces launch overhead and overlap computation with communication across GPUs. - How MI355X’s register capacity and direct instruction-level control enable hardware-specific optimizations. - What our roofline analysis reveals about opportunities for further speedups. Read the full analysis: lithosai.com/blog/ultra-fast…
22
8
10
181
34,969
Stay tuned, Kimi K3.1 ultra fast is coming to Lithos! 🚀
2
1
18
1,103
Kimi K3.1 is coming to @lithos_ai ultra-fast!
1
7
512
Fast inference without full model quality actually increases how long agents take to deliver useful work. If you're interested in seeing LithosAI in action, try it on representative end-to-end tasks under the load you expect and measure your agents' task completion rate. Our users have shared that even agentic development use cases with large context windows and multi-turn loops are running as much as 5-6x faster end to end. Learn more: lithosai.com/blog/need-for-s…
2
3
14
1,237
LithosAI retweeted
Speed matters. Especially for today's AI agents, which can make thousands of LLM calls to complete a single task. Super excited to enable 1,500+ tokens/second/user for both DeepSeek V4.1 Flash and GLM-5.3-Flash on just 8 standard GPUs! 🚀 We're pushing the boundaries of inference speed to make AI agents faster than ever!
4 models, 12 #1s on Artificial Analysis - including GLM 5.3 Flash running up to 1,543 tok/s. LithosAI ranks first for output speed, latency, and E2E response time, without sacrificing model quality, for: → GLM 5.3 Flash → Kimi K3 → DeepSeek V4.1-Flash → GLM 5.3 Check out our full benchmark breakdown and bookmark it. lithosai.com/blog/lithosai-t… More models and more cost-efficient, low-latency inference coming soon!
6
5
29
2,218
1,500 tok/s is just the beginning. We have a surprise coming next. Stay tuned 👀
4 models, 12 #1s on Artificial Analysis - including GLM 5.3 Flash running up to 1,543 tok/s. LithosAI ranks first for output speed, latency, and E2E response time, without sacrificing model quality, for: → GLM 5.3 Flash → Kimi K3 → DeepSeek V4.1-Flash → GLM 5.3 Check out our full benchmark breakdown and bookmark it. lithosai.com/blog/lithosai-t… More models and more cost-efficient, low-latency inference coming soon!
1
2
18
1,990
4 models, 12 #1s on Artificial Analysis - including GLM 5.3 Flash running up to 1,543 tok/s. LithosAI ranks first for output speed, latency, and E2E response time, without sacrificing model quality, for: → GLM 5.3 Flash → Kimi K3 → DeepSeek V4.1-Flash → GLM 5.3 Check out our full benchmark breakdown and bookmark it. lithosai.com/blog/lithosai-t… More models and more cost-efficient, low-latency inference coming soon!
3
3
3
27
5,741
DeepSeek-V4.1-Flash-1017 is now live on LithosAI at up to 1,500 tok/s 🚀 Great model upgrade from @deepseek_ai
🚀 DeepSeek-V4.1-Flash-1017 is here! Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API. 🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model. 1/4
5
4
2
28
1,797
Fast tokens, fast shipping! 🚀 DeepSeek-V4.1-Flash-1017 is now live on @lithos_ai
3
1
12
925
Replying to @lithos_ai
Proud to be building LithosAI with this team! 🚀
1
1
2
138
Optimizing local inference for ultra-low latency is both easier and harder than optimizing for a cloud service. The unavoidable hardware variety, memory constraints, and thermal throttling automatically make each optimization more complicated. But knowing which challenges to tackle first is easier because you’re dealing with users who: 1) know exactly what they’re looking for from inference performance for their setup and 2) are predisposed to build in the open and share their findings Here’s how we know: @JiaZhihao announced that we were open-sourcing lithos-metal, our local inference engine for Apple silicon, 5 days ago. In that time, we’ve made a ton of progress because everyone who let us know they tried lithos-metal (or even just explored the code or blog) gave us the platonic ideal of technical feedback: immediate, specific, actionable. Thanks to all of you who shared your questions and suggestions, including insights into your actual use cases and testing approach. Our day job is still primarily building ultra low-latency agent infrastructure on standard GPUs, but it’s important to us to keep lithos-metal going and build the community. Internally, we talk a lot about taking a co-design approach to AI efficiency. That means simultaneously thinking about how the hardware and software layers impact each other. Only focusing on one area for improvement leaves performance gains on the table. That logic applies to where your inference engine lives, too. Whether it’s in the cloud, on-prem, or on your laptop, we’re excited to keep working to solve agent latency for our users.
🤖 Made with AI
2
6
16
1,405
In the last 5 days, we’ve rolled out a ton of hardware-software optimizations and production-ready operations for lithos-metal. Here are the highlights: 1️⃣ Prefill phase optimizations Improving TTFT on M5 Max chips and reducing latency without altering model quality; fixing a memory buffer reuse bug to prevent GPU page faults; fixing token output discrepancies 2️⃣ Hardware-specific kernel optimizations Adding native engine backends for M3 Max, M3 Ultra, M4 Pro, and M6 chips; an improved DSpark head optimized for coding; fixes for FP8 ALU bottlenecks and matrix tile layouts specifically for M3/M4 chips; and refined kernel selection logic and hardware-level dispatch strategy 3️⃣ Production-ready serving and runtime operations Expanding the server CLI with standard enterprise protocols (OpenAI Chat Completions, Anthropic Messages, Ollama), SSE streaming, tool execution, and stateless request routing; adding SSD-backed caching for prompt prefixes to avoid repeated prompt context; implementing idea time-to-live to auto-unload models from unified memory when inactive; adding pre-compiled sampling routines during startup to avoid runtime compilation stutter; shipping dynamic fallback kernel selections for chips without pre-measured performance profiles; adding support for serving 4-bit quantized versions of Qwen3.8-27B Stay up to date: github.com/lithos-ai/lithos-…
1
2
523
Our team still has more work in the queue, including adding support for: → More chips: M2, M5 Ultra chips are up next → More device configurations: Including Macs running Sonoma → More models: Starting with Qwen 3.8 Flash-Next Don’t see the update or fix you’re looking for? Help us and other lithos-metal users out - submit an issue: github.com/lithos-ai/lithos-…
1
1
138