#swahilijs (@Mwandi_) header
#swahilijs (@Mwandi_) profile picture

Entrepreneur building safer AI | DevSecOps x LLMs

Tanzania
Joined November 2020
Photo by #swahilijs (@Mwandi_)Photo by #swahilijs (@Mwandi_)Photo by #swahilijs (@Mwandi_)Photo by #swahilijs (@Mwandi_)Photo by #swahilijs (@Mwandi_)Photo by #swahilijs (@Mwandi_)Photo by #swahilijs (@Mwandi_)Photo by #swahilijs (@Mwandi_)Photo by #swahilijs (@Mwandi_)Photo by #swahilijs (@Mwandi_)
Learn. Unlearn. Relearn. Forgive. Love
1
11
29
Why KV cache stores K and V vectors but never Q? (a popular technical LLM interview question) An LLM writes text one token at a time, and it predicts each new token from all the tokens before it. Without a cache, the model would process the whole prompt again at every step. The KV cache exists to skip that repeated work, and what it keeps shows how attention actually works. Inside each attention layer, every token produces three vectors. The query (Q) describes what the token is looking for, the key (K) describes what it offers to other tokens, and the value (V) carries the content it passes along. After a forward pass, every position ends with a hidden state, its final vector. Only the last position's hidden state becomes logits, the scores for the next token. So the real question is what that last hidden state depends on. Here's how a 4-token prompt produces token 5 (follow along with the visual below): ↳ In prefill, the model reads all 4 tokens in one parallel pass, and every layer computes a Q, K and V for each token. ↳ In the last layer, only the last token's row feeds the prediction, so its query Qβ‚„ is compared against every key from K₁ to Kβ‚„. ↳ Softmax turns those scores into weights, which blend V₁ to Vβ‚„ into a single output row. ↳ The feedforward block processes each position on its own, so that row becomes the last hidden state, and the next token is picked from it. That final vector needs only the newest query, along with every key and value. Now token 5 joins. Each layer computes Qβ‚…, Kβ‚… and Vβ‚… for it, then compares Qβ‚… against K₁ to Kβ‚…. K₁ to Kβ‚„ and V₁ to Vβ‚„ are exactly what the model computed in the previous step. The causal mask stops any token from looking at tokens after it, so adding token 5 can't change them. Qβ‚„ only ever served row 4, and that row is already finished. No future step will read it again. So the cache keeps every K and V, while each step creates one fresh query, uses it once, and discards it. The visual below shows the complete flow. - Only the last hidden state predicts the next token. - That state needs the newest Q plus every K and V. - Old K and V never change, so they're cached, while each Q is used once and discarded. I've quoted my full breakdown of KV caching below if you want to go deeper.
29
27
2
246
14,061
why the name logistic regression? shouldn't be
1
35
Introducing Qwen3.8-27B-pi-adapted πŸ€— We adapted Qwen's model specifically for the pi coding agent using reinforcement learning and a LoRA adapter. The results, measured as pass@1: πŸš€ SWE-bench Pro (Hard 51): 29.4% β†’ 62.7% πŸ’» Terminal-Bench 2.1: 39.5% β†’ 55.8% ⚑ 55–62% fewer tool calls per task This is the power of open models: you can adapt them to YOUR tools and YOUR use cases. Let’s build! πŸ’› Open weights: huggingface.co/harness-hub/Q… Methodology: huggingface.co/blog/harness-…
263
561
145
5,185
324,034
#swahilijs retweeted
System Design: The Ultimate Learning Roadmap β”œβ”€β”€ 01 Fundamentals β”‚ β”œβ”€β”€ Client-Server Architecture β”‚ β”œβ”€β”€ HTTP, DNS & Networking β”‚ └── Latency, Throughput & Availability β”‚ β”œβ”€β”€ 02 Data Structures & Algorithms β”‚ β”œβ”€β”€ Arrays, HashMaps & Trees β”‚ β”œβ”€β”€ Time & Space Complexity β”‚ └── Searching & Sorting β”‚ β”œβ”€β”€ 03 Database Fundamentals β”‚ β”œβ”€β”€ SQL vs NoSQL β”‚ β”œβ”€β”€ Indexing & Query Optimization β”‚ └── ACID & Transactions β”‚ β”œβ”€β”€ 04 API Design β”‚ β”œβ”€β”€ REST, GraphQL & gRPC β”‚ β”œβ”€β”€ API Versioning & Pagination β”‚ └── Authentication & Rate Limiting β”‚ β”œβ”€β”€ 05 Scalability & Performance β”‚ β”œβ”€β”€ Horizontal vs Vertical Scaling β”‚ β”œβ”€β”€ Load Balancing β”‚ └── Caching Strategies β”‚ β”œβ”€β”€ 06 Distributed Systems β”‚ β”œβ”€β”€ CAP Theorem & Consistency β”‚ β”œβ”€β”€ Replication & Sharding β”‚ └── Consensus & Fault Tolerance β”‚ β”œβ”€β”€ 07 Messaging & Event-Driven Systems β”‚ β”œβ”€β”€ Kafka, RabbitMQ & SQS β”‚ β”œβ”€β”€ Pub/Sub & Event Streaming β”‚ └── Retries & Idempotency β”‚ β”œβ”€β”€ 08 Storage & Data Processing β”‚ β”œβ”€β”€ Object Storage & CDN β”‚ β”œβ”€β”€ Relational vs Distributed Storage β”‚ └── Batch & Stream Processing β”‚ β”œβ”€β”€ 09 Microservices Architecture β”‚ β”œβ”€β”€ Service Discovery & API Gateway β”‚ β”œβ”€β”€ Saga & Circuit Breakers β”‚ └── Service-to-Service Communication β”‚ β”œβ”€β”€ 10 Security & Reliability β”‚ β”œβ”€β”€ OAuth2, JWT & RBAC β”‚ β”œβ”€β”€ Encryption & Zero Trust β”‚ └── Disaster Recovery & Backups β”‚ β”œβ”€β”€ 11 System Design Patterns β”‚ β”œβ”€β”€ CQRS & Event Sourcing β”‚ β”œβ”€β”€ Strangler Fig & Sidecar β”‚ └── Leader-Follower Architecture β”‚ β”œβ”€β”€ 12 Observability & Infrastructure β”‚ β”œβ”€β”€ Logs, Metrics & Distributed Tracing β”‚ β”œβ”€β”€ Docker & Kubernetes β”‚ └── CI/CD & Infrastructure as Code β”‚ └── 13 Real-World System Design β”œβ”€β”€ Design URL Shortener β”œβ”€β”€ Design WhatsApp β”œβ”€β”€ Design YouTube β”œβ”€β”€ Design Uber └── Design Netflix System Design isn't just about drawing architecture diagrams. It's about building scalable, reliable and fault-tolerant systems that handle real-world traffic. Save this roadmap. Learn one concept at a time. Practice designing real-world systems. The goal isn't just to design systems. It's to build systems that scale.
6
11
51
1,924
"And he informed me, and talked with me, and said, β€˜O Daniel, I have now come forth to give you skill to understand.’" Daniel 9:22 May God give us the skill to understand, and the grace to fulfill our purposes.
30
We entered the era of custom inference engine vibe coding. None of this would be possible without projects like SGLang, vLLM, and llamacpp. We are so lucky they are open source.
42
36
7
669
16,995
#swahilijs retweeted
NVIDIA released a new looped transformer: Instead of stacking more layers, it runs the same layers multiple times so a 4B model can think like a 9B one. Here's how it works: In a standard transformer, every token passes through a stack of decoder blocks once, and each block has its own weights. So if you want a deeper model, you add more blocks, and the parameter count grows with the depth. NVIDIA's Dory1.0-4B decouples the two by reusing a chunk of its layers. The model is split into three parts: > An input block of 8 blocks runs once and turns the tokens into a hidden state x. > A recurrent block of 20 shared blocks runs k times, where k can be anywhere from 1 to 4. > An output block of 8 blocks runs once on the final state, followed by the LM head. Before every pass through the recurrent block, x is added back to the current state and the sum is L2-normalized. This keeps the original token representation in view on every pass, instead of each pass only seeing what the previous one produced. Both models in the diagram below store 36 blocks of weights. The standard transformer applies each block once, while Dory applies 36 blocks at k=1, 56 at k=2, and 96 at k=4, i.e., 72, 112, and 192 physical layers from the same checkpoint. The extra depth is actually helpful on reasoning and agentic tasks: > SWE-bench Verified: - at 1 loop β†’ 50.3 - at 2 loops β†’ 59.5 - at 4 loops β†’ 61.8 > 12-benchmark average: - at 1 loop β†’ 35.17 - at 2 loops β†’ 39.67 - at 4 loops β†’ 40.60 - and with 2 or more loops, it beats Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B. More loops implies more compute per token, but the 4-loop model also writes 1.3x to 2.5x fewer output tokens on agentic tasks. So 2 loops end up costing only 1.03x to 1.25x the compute of 1 loop end to end, which is why NVIDIA recommends 2 as the default. By default, Dory keeps a separate KV cache for every loop, so at 4 loops the recurrent layers store 4x the cache. To avoid this, it can keep only the cache from the last loop, which uses the same KV memory as a normal model with only a small change in accuracy. One thing to note here is that looping helps the model reason over what it already knows, but it does not add knowledge. For instance, on tasks like AA-Omniscience, accuracy stays at 10.0 whether you run 1 loop or 4, and on knowledge-heavy tasks like GPQA, Dory still trails Qwen3.5-4B. Here's the paper: arxiv.org/abs/2611.03998 That said, how much memory looping costs comes down to how the KV cache is handled, so I wrote a full breakdown of how KV caching works. It covers what gets cached during prefill, how the cache grows during decoding, and why it becomes the main memory bottleneck at long context. Read it below.
62
222
18
1,764
108,134
langChain vs langGraph
24
Nothing exposes a fool faster than a conversation with someone who actually knows what they're talking about.
7
18
1
84
11,035
#swahilijs retweeted
If you can complete Stanford’s CS336 (Language Modeling from Scratch) and CS230 (Deep Learning) assignments, you can build and train custom LLMs from the metal up. The full lecture playlists are public on YouTube. Cost: $0.
2
23
241
4,779
#swahilijs retweeted
Replying to @elonmusk
Thank you Jensen
12
18
1
632
66,209
#swahilijs retweeted
Jensen
"β€ŠYou're gonna be remembered forever because some things are eternal."
6,230
13,566
1,294
152,833
12,877,676
taking cs 229, am sometimes getting lost but am not giving in. forward ever!
152
Former Scale AI founder and newly appointed @Meta Chief AI Officer @alexandr_wang dropped a bombshell during his YC conversation with @garrytan, and you could almost hear the tech leadership world go quiet. He said Meta has already seen this internally: Build the right agentic loop, give it an evaluation system and metrics that let it optimize itself, and a group of AI agents can complete more work than a team of 100 senior engineers. And they do it β€œvery easily.” But the most interesting part wasn’t the 100-engineer comparison. It was how simple the system underneath it actually is. You’d expect some insanely complex, almost alien architecture powering a swarm like this. Instead, Wang described it with a few almost comically basic words: β€œMarkdown files, cron jobs, goal, metrics, data.” Once you strip away the hype, the implications for traditional software engineering become pretty clear: 1️⃣ It’s not that the models are magically smarter. The eval loop is doing the heavy lifting. Traditional approach: humans write prompts, run the code, inspect the output, and hope nothing broke. Meta’s approach: turn the business goal into something a machine can score automatically. The agent submits its work. The system runs tests, calculates metrics, finds what’s wrong, and sends it back for another pass. Repeat until it passes. Nobody has to babysit every step. The metric becomes the supervisor. 2️⃣ The real alpha is burning 1,000x more tokens inside the feedback loop. A lot of people are still optimizing for the cost of a single AI call. The frontier labs are playing a different game: spend 1,000x or even 1,000,000x more tokens in the background so agents can constantly review, rerun, challenge, and verify each other’s work until they reach a reliable business outcome. Token cost is fixed. The payoff is a pipeline that keeps running. 3️⃣ Memory doesn’t need some fancy database. Persistent memory can live in Markdown files. Scheduling can be handled by the server’s built-in cron jobs, running overnight. The simpler the scaffolding, the more robust the system can be. Less infrastructure also means fewer ways for context to fall apart. This is a pretty brutal change in how technical organizations work. The ceiling for a tech lead used to be partly about how many people they could manage, how many meetings they could sit through, and how many teams they could coordinate. The leverage for the next generation of technical leaders may look very different: Can you turn a messy business objective into a rigorous set of metrics that an AI can evaluate automatically? If 100 people’s output can be replaced by a few cron jobs, Markdown files, and a well-designed eval loop, the era of β€œjust take the ticket and write the code” is coming to an end. The people who can design the evals and orchestrate the swarm aren’t just holding a new tool. They’re effectively running a virtual company.
47
58
13
506
134,027
#swahilijs retweeted
Google Brain founder Andrew Ng: "Prompting will be dead in 6 months" "Loops and graphs are replacing it" In just 2 hours, he shows how to build agents that work, learn, and improve on their own LLMs β†’ Agents β†’ Loops β†’ Graphs β†’ Self-Improving Systems The first 15 minutes alone are more valuable than most $500 AI courses Most people are still learning prompts while AI engineering is already moving beyond them Watch the lecture first Then read the full guide to loops and graphs below ↓
5
52
1
217
27,330
#swahilijs retweeted
Use PyTorch-native libraries within the NVIDIA NeMo Framework to customize models to hit your exacting requirements for latency, speed, memory, and compute. @NVIDIA's blog demonstrates how quantization-aware distillation (QAD) improves the Nemotron 3.5 Lightning model using NVIDIA Model Optimizer. We walk through the entire training pipeline, and show how QAD consistently outperforms post-training quantization (PTQ) compression on agentic benchmarks, ensuring high quality while reducing memory usage. Read it here: developer.nvidia.com/blog/de…
2
3
2
51
16,792
#swahilijs retweeted
NVIDIA CEO, Jensen Huang: "Nobody writes prompts anymore. The new job is building Loops and Graphs." In 50 minutes he explains why the engineers who stopped prompting are already years ahead, and what they're building instead. The missing piece most people skip: loops handle the work, graphs handle the loops. Watch it, then read the full guide on loops and graphs below.
59
292
20
2,314
1,242,806
#swahilijs retweeted
almost no ai engineers are talking about claude skills yet. andrew ng just released a 2 hour collab with anthropic that walks you through building agentic skills from zero. β€’ 00:00 - getting started with agentic skills in claude β€’ 22:29 - using claude’s ready-made skills β€’ 41:04 - skills vs mcp and subagents β€’ 1:06:03 - designing skills for long-running agents β€’ 1:51:59 - live build: creating agentic skills end to end prompting isn’t the main game anymore. claude skills is what serious builders are using to ship real agents. i watched it last night. somewhere in the middle it hit me: i’ve been treating claude like a chatbot, not an agent platform, for years. save this and check the article below.
28
29
170
17,953
#swahilijs retweeted
INSTEAD OF WATCHING AN HOUR OF NETFLIX TONIGHT. This 1-hour Stanford lecture by Joel Peterson will teach you more about negotiation and getting what you want than most people learn in years. Bookmark it and give it an hour, no matter what.
49
1,179
13
7,107
701,222
#swahilijs retweeted
Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub! This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people. Thanks also to everyone who shared ideas and opened PRs with improvements! Of course, I plan to keep adding new material, including new attention variants and architectures (while bigger projects like RL and Reasoning From Scratch live in their separate repositories). I am also currently working on a larger applied custom β€œsmall” LLM project. It has been keeping me super busy this month, but I will share more on that soon. If you are new to it, some of the highlights include 1. Of course, the complete code path from tokenization and attention to pretraining, classification, and instruction fine-tuning, etc. All of it FROM SCRATCH, of course! (RL lives in a companion repo.) 2. From-scratch implementations of Llama, Qwen, Gemma, and Olmo (smaller variants that run locally and can be plugged into the training scripts). 3. From-scratch implementations of attention alternatives and other architecture components, such as GQA, MLA, sliding-window attention, Gated DeltaNet, DeepSeek Sparse Attention, cross-layer KV sharing, and mixture-of-experts 4. Materials on KV caching, training performance, memory-efficient weight loading, DPO, evaluation, and LoRA So, if you don’t have any weekend plans yet, happy tinkering!
82
551
27
3,937
172,935