200 people murdered in Benue, 50 killed in Plateau

A 12GB RTX 3060 is reportedly generating around 50 tokens/s with Qwen3.8-35B-A3B, a model with 35 billion parameters. The same setup is also reporting approximately 550 tokens/s in prompt processing. No 24GB GPU or expensive multi-GPU setup. Just an RTX 3060, 16GB of system RAM and some careful inference optimisation. The hardware and configuration The reported setup uses: GPU: RTX 3060 (12GB VRAM) System RAM: 16GB Model: Qwen3.8-35B-A3B Quantization: Q4_K_M Generation speed: ~50 tokens/s Prompt processing: ~550 tokens/s Configured context: 262K tokens Runtime: llama.cpp Optimisations: Flash Attention and Q8 KV cache The two speed figures measure different stages of inference Generation speed measures how quickly the model produces new tokens. Prompt processing measures how quickly it processes incoming text before generating a response Both are useful, but they shouldn’t be treated as interchangeable How does a 35B model run on 12GB VRAM? The explanation starts with the model’s architecture Qwen3.8-35B-A3B uses a Mixture-of-Experts (MoE) design Although it contains approximately 35 billion parameters, only around 3 billion are activated for each token Instead of running every part of the model during every generation step, the routing system selects particular experts to perform the computation This reduces the amount of computation required per token compared with a dense model of similar total size However, there’s an important distinction: 3B active parameters does not mean the remaining 32B parameters disappear The full model weights still need to be stored and accessed. That’s where hybrid inference becomes important The CPU and GPU work together The setup uses llama.cpp to distribute the model across GPU and system memory According to the reported configuration, 26 MoE layers are offloaded to the CPU This allows system RAM to accommodate parts of the model that cannot fit entirely within the RTX 3060’s 12GB VRAM The GPU handles the portions assigned to it, while the CPU and system memory take care of the offloaded components Three optimisations help make this arrangement practical: 1. Q4_K_M quantization The model weights are stored in a 4-bit quantized format, significantly reducing their memory footprint compared with higher-precision versions The trade-off is that quantization can affect model quality, so coding performance still needs to be evaluated 2. Flash Attention Flash Attention improves how attention operations are computed by reducing unnecessary memory movement It can also help control memory requirements, particularly when processing longer sequences 3. Q8 KV cache The KV cache stores attention keys and values from previously processed tokens Using Q8 stores this information at 8-bit precision rather than a higher-precision format, reducing memory consumption as the context grows Together, these techniques allow the system to run a model that would otherwise be difficult to accommodate entirely in GPU memory What about the 262K context? The reported configuration supports a context window of up to 262K tokens But a configured context limit doesn’t mean the system can process that much text at the same speed or without memory pressure The KV cache grows as more context is processed. With only 16GB of system RAM and 12GB of VRAM, longer prompts could introduce substantial memory pressure and reduce performance. The actual usable context will depend on the runtime configuration, cache allocation and available memory The real test: coding agents Getting 50 tokens/s from a local model is encouraging, but generation speed alone doesn’t tell us how useful it is for software development A coding agent needs to: Inspect files and understand an existing codebase Identify dependencies and trace how components interact Modify multiple files Execute commands and tests Diagnose errors Recover from failed attempts Verify that the final changes actually work