#swahilijs retweeted
Why KV cache stores K and V vectors but never Q?
(a popular technical LLM interview question)
An LLM writes text one token at a time, and it predicts each new token from all the tokens before it.
Without a cache, the model would process the whole prompt again at every step. The KV cache exists to skip that repeated work, and what it keeps shows how attention actually works.
Inside each attention layer, every token produces three vectors. The query (Q) describes what the token is looking for, the key (K) describes what it offers to other tokens, and the value (V) carries the content it passes along.
After a forward pass, every position ends with a hidden state, its final vector. Only the last position's hidden state becomes logits, the scores for the next token.
So the real question is what that last hidden state depends on. Here's how a 4-token prompt produces token 5 (follow along with the visual below):
β³ In prefill, the model reads all 4 tokens in one parallel pass, and every layer computes a Q, K and V for each token.
β³ In the last layer, only the last token's row feeds the prediction, so its query Qβ is compared against every key from Kβ to Kβ.
β³ Softmax turns those scores into weights, which blend Vβ to Vβ into a single output row.
β³ The feedforward block processes each position on its own, so that row becomes the last hidden state, and the next token is picked from it.
That final vector needs only the newest query, along with every key and value.
Now token 5 joins. Each layer computes Qβ
, Kβ
and Vβ
for it, then compares Qβ
against Kβ to Kβ
.
Kβ to Kβ and Vβ to Vβ are exactly what the model computed in the previous step. The causal mask stops any token from looking at tokens after it, so adding token 5 can't change them.
Qβ only ever served row 4, and that row is already finished. No future step will read it again.
So the cache keeps every K and V, while each step creates one fresh query, uses it once, and discards it.
The visual below shows the complete flow.
- Only the last hidden state predicts the next token.
- That state needs the newest Q plus every K and V.
- Old K and V never change, so they're cached, while each Q is used once and discarded.
I've quoted my full breakdown of KV caching below if you want to go deeper.
#swahilijs retweeted
Introducing Qwen3.8-27B-pi-adapted π€
We adapted Qwen's model specifically for the pi coding agent using reinforcement learning and a LoRA adapter.
The results, measured as pass@1:
π SWE-bench Pro (Hard 51): 29.4% β 62.7%
π» Terminal-Bench 2.1: 39.5% β 55.8%
β‘ 55β62% fewer tool calls per task
This is the power of open models: you can adapt them to YOUR tools and YOUR use cases. Letβs build! π
Open weights: huggingface.co/harness-hub/Qβ¦
Methodology: huggingface.co/blog/harness-β¦
#swahilijs retweeted
System Design: The Ultimate Learning Roadmap
βββ 01 Fundamentals
β βββ Client-Server Architecture
β βββ HTTP, DNS & Networking
β βββ Latency, Throughput & Availability
β
βββ 02 Data Structures & Algorithms
β βββ Arrays, HashMaps & Trees
β βββ Time & Space Complexity
β βββ Searching & Sorting
β
βββ 03 Database Fundamentals
β βββ SQL vs NoSQL
β βββ Indexing & Query Optimization
β βββ ACID & Transactions
β
βββ 04 API Design
β βββ REST, GraphQL & gRPC
β βββ API Versioning & Pagination
β βββ Authentication & Rate Limiting
β
βββ 05 Scalability & Performance
β βββ Horizontal vs Vertical Scaling
β βββ Load Balancing
β βββ Caching Strategies
β
βββ 06 Distributed Systems
β βββ CAP Theorem & Consistency
β βββ Replication & Sharding
β βββ Consensus & Fault Tolerance
β
βββ 07 Messaging & Event-Driven Systems
β βββ Kafka, RabbitMQ & SQS
β βββ Pub/Sub & Event Streaming
β βββ Retries & Idempotency
β
βββ 08 Storage & Data Processing
β βββ Object Storage & CDN
β βββ Relational vs Distributed Storage
β βββ Batch & Stream Processing
β
βββ 09 Microservices Architecture
β βββ Service Discovery & API Gateway
β βββ Saga & Circuit Breakers
β βββ Service-to-Service Communication
β
βββ 10 Security & Reliability
β βββ OAuth2, JWT & RBAC
β βββ Encryption & Zero Trust
β βββ Disaster Recovery & Backups
β
βββ 11 System Design Patterns
β βββ CQRS & Event Sourcing
β βββ Strangler Fig & Sidecar
β βββ Leader-Follower Architecture
β
βββ 12 Observability & Infrastructure
β βββ Logs, Metrics & Distributed Tracing
β βββ Docker & Kubernetes
β βββ CI/CD & Infrastructure as Code
β
βββ 13 Real-World System Design
βββ Design URL Shortener
βββ Design WhatsApp
βββ Design YouTube
βββ Design Uber
βββ Design Netflix
System Design isn't just about drawing architecture diagrams.
It's about building scalable, reliable and fault-tolerant systems that handle real-world traffic.
Save this roadmap.
Learn one concept at a time.
Practice designing real-world systems.
The goal isn't just to design systems. It's to build systems that scale.
"And he informed me, and talked with me, and said, βO Daniel, I have now come forth to give you skill to understand.β" Daniel 9:22
May God give us the skill to understand, and the grace to fulfill our purposes.
30
#swahilijs retweeted
We entered the era of custom inference engine vibe coding.
None of this would be possible without projects like SGLang, vLLM, and llamacpp.
We are so lucky they are open source.
#swahilijs retweeted
NVIDIA released a new looped transformer:
Instead of stacking more layers, it runs the same layers multiple times so a 4B model can think like a 9B one.
Here's how it works:
In a standard transformer, every token passes through a stack of decoder blocks once, and each block has its own weights.
So if you want a deeper model, you add more blocks, and the parameter count grows with the depth.
NVIDIA's Dory1.0-4B decouples the two by reusing a chunk of its layers.
The model is split into three parts:
> An input block of 8 blocks runs once and turns the tokens into a hidden state x.
> A recurrent block of 20 shared blocks runs k times, where k can be anywhere from 1 to 4.
> An output block of 8 blocks runs once on the final state, followed by the LM head.
Before every pass through the recurrent block, x is added back to the current state and the sum is L2-normalized.
This keeps the original token representation in view on every pass, instead of each pass only seeing what the previous one produced.
Both models in the diagram below store 36 blocks of weights.
The standard transformer applies each block once, while Dory applies 36 blocks at k=1, 56 at k=2, and 96 at k=4, i.e., 72, 112, and 192 physical layers from the same checkpoint.
The extra depth is actually helpful on reasoning and agentic tasks:
> SWE-bench Verified:
- at 1 loop β 50.3
- at 2 loops β 59.5
- at 4 loops β 61.8
> 12-benchmark average:
- at 1 loop β 35.17
- at 2 loops β 39.67
- at 4 loops β 40.60
- and with 2 or more loops, it beats Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B.
More loops implies more compute per token, but the 4-loop model also writes 1.3x to 2.5x fewer output tokens on agentic tasks.
So 2 loops end up costing only 1.03x to 1.25x the compute of 1 loop end to end, which is why NVIDIA recommends 2 as the default.
By default, Dory keeps a separate KV cache for every loop, so at 4 loops the recurrent layers store 4x the cache.
To avoid this, it can keep only the cache from the last loop, which uses the same KV memory as a normal model with only a small change in accuracy.
One thing to note here is that looping helps the model reason over what it already knows, but it does not add knowledge.
For instance, on tasks like AA-Omniscience, accuracy stays at 10.0 whether you run 1 loop or 4, and on knowledge-heavy tasks like GPQA, Dory still trails Qwen3.5-4B.
Here's the paper: arxiv.org/abs/2611.03998
That said, how much memory looping costs comes down to how the KV cache is handled, so I wrote a full breakdown of how KV caching works.
It covers what gets cached during prefill, how the cache grows during decoding, and why it becomes the main memory bottleneck at long context.
Read it below.
#swahilijs retweeted
Nothing exposes a fool faster than a conversation with someone who actually knows what they're talking about.
#swahilijs retweeted
If you can complete Stanfordβs CS336 (Language Modeling from Scratch) and CS230 (Deep Learning) assignments, you can build and train custom LLMs from the metal up.
The full lecture playlists are public on YouTube.
Cost: $0.
#swahilijs retweeted
Former Scale AI founder and newly appointed @Meta Chief AI Officer @alexandr_wang dropped a bombshell during his YC conversation with @garrytan, and you could almost hear the tech leadership world go quiet.
He said Meta has already seen this internally:
Build the right agentic loop, give it an evaluation system and metrics that let it optimize itself, and a group of AI agents can complete more work than a team of 100 senior engineers.
And they do it βvery easily.β
But the most interesting part wasnβt the 100-engineer comparison.
It was how simple the system underneath it actually is.
Youβd expect some insanely complex, almost alien architecture powering a swarm like this.
Instead, Wang described it with a few almost comically basic words:
βMarkdown files, cron jobs, goal, metrics, data.β
Once you strip away the hype, the implications for traditional software engineering become pretty clear:
1οΈβ£ Itβs not that the models are magically smarter. The eval loop is doing the heavy lifting.
Traditional approach: humans write prompts, run the code, inspect the output, and hope nothing broke.
Metaβs approach: turn the business goal into something a machine can score automatically.
The agent submits its work. The system runs tests, calculates metrics, finds whatβs wrong, and sends it back for another pass. Repeat until it passes.
Nobody has to babysit every step. The metric becomes the supervisor.
2οΈβ£ The real alpha is burning 1,000x more tokens inside the feedback loop.
A lot of people are still optimizing for the cost of a single AI call.
The frontier labs are playing a different game: spend 1,000x or even 1,000,000x more tokens in the background so agents can constantly review, rerun, challenge, and verify each otherβs work until they reach a reliable business outcome.
Token cost is fixed. The payoff is a pipeline that keeps running.
3οΈβ£ Memory doesnβt need some fancy database.
Persistent memory can live in Markdown files.
Scheduling can be handled by the serverβs built-in cron jobs, running overnight.
The simpler the scaffolding, the more robust the system can be. Less infrastructure also means fewer ways for context to fall apart.
This is a pretty brutal change in how technical organizations work.
The ceiling for a tech lead used to be partly about how many people they could manage, how many meetings they could sit through, and how many teams they could coordinate.
The leverage for the next generation of technical leaders may look very different:
Can you turn a messy business objective into a rigorous set of metrics that an AI can evaluate automatically?
If 100 peopleβs output can be replaced by a few cron jobs, Markdown files, and a well-designed eval loop, the era of βjust take the ticket and write the codeβ is coming to an end.
The people who can design the evals and orchestrate the swarm arenβt just holding a new tool.
Theyβre effectively running a virtual company.
#swahilijs retweeted
Google Brain founder Andrew Ng:
"Prompting will be dead in 6 months"
"Loops and graphs are replacing it"
In just 2 hours, he shows how to build agents that work, learn, and improve on their own
LLMs β Agents β Loops β Graphs β Self-Improving Systems
The first 15 minutes alone are more valuable than most $500 AI courses
Most people are still learning prompts while AI engineering is already moving beyond them
Watch the lecture first
Then read the full guide to loops and graphs below β
Use PyTorch-native libraries within the NVIDIA NeMo Framework to customize models to hit your exacting requirements for latency, speed, memory, and compute.
@NVIDIA's blog demonstrates how quantization-aware distillation (QAD) improves the Nemotron 3.5 Lightning model using NVIDIA Model Optimizer. We walk through the entire training pipeline, and show how QAD consistently outperforms post-training quantization (PTQ) compression on agentic benchmarks, ensuring high quality while reducing memory usage.
Read it here: developer.nvidia.com/blog/deβ¦
#swahilijs retweeted
NVIDIA CEO, Jensen Huang:
"Nobody writes prompts anymore. The new job is building Loops and Graphs."
In 50 minutes he explains why the engineers who stopped prompting are already years ahead, and what they're building instead.
The missing piece most people skip: loops handle the work, graphs handle the loops.
Watch it, then read the full guide on loops and graphs below.
almost no ai engineers are talking about claude skills yet.
andrew ng just released a 2 hour collab with anthropic that walks you through building agentic skills from zero.
β’ 00:00 - getting started with agentic skills in claude
β’ 22:29 - using claudeβs ready-made skills
β’ 41:04 - skills vs mcp and subagents
β’ 1:06:03 - designing skills for long-running agents
β’ 1:51:59 - live build: creating agentic skills end to end
prompting isnβt the main game anymore. claude skills is what serious builders are using to ship real agents.
i watched it last night.
somewhere in the middle it hit me: iβve been treating claude like a chatbot, not an agent platform, for years.
save this and check the article below.
#swahilijs retweeted
INSTEAD OF WATCHING AN HOUR OF NETFLIX TONIGHT.
This 1-hour Stanford lecture by Joel Peterson will teach you more about negotiation and getting what you want than most people learn in years.
Bookmark it and give it an hour, no matter what.
#swahilijs retweeted
Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub!
This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people.
Thanks also to everyone who shared ideas and opened PRs with improvements!
Of course, I plan to keep adding new material, including new attention variants and architectures (while bigger projects like RL and Reasoning From Scratch live in their separate repositories).
I am also currently working on a larger applied custom βsmallβ LLM project. It has been keeping me super busy this month, but I will share more on that soon.
If you are new to it, some of the highlights include
1. Of course, the complete code path from tokenization and attention to pretraining, classification, and instruction fine-tuning, etc. All of it FROM SCRATCH, of course! (RL lives in a companion repo.)
2. From-scratch implementations of Llama, Qwen, Gemma, and Olmo (smaller variants that run locally and can be plugged into the training scripts).
3. From-scratch implementations of attention alternatives and other architecture components, such as GQA, MLA, sliding-window attention, Gated DeltaNet, DeepSeek Sparse Attention, cross-layer KV sharing, and mixture-of-experts
4. Materials on KV caching, training performance, memory-efficient weight loading, DPO, evaluation, and LoRA
So, if you donβt have any weekend plans yet, happy tinkering!





























