Mohamed

Let's make KV cache great again :) What if you could run your LLM on almost arbitrarily long text without caring about the KV cache, and still match a full read? ⬇️ New preprint: Periscope: Extending Frozen Language Models Beyond Their Context Window 🔴 tl;dr: a new training-free inference method that relies on short local and strided reads to approximate the global context. A frozen model served at a window of W tokens can now read texts of W²/c tokens (e.g., a 27B model reads 4.5M tokens on one 80GB GPU) with the KV cache size of only a few chunks, at accuracy comparable to reading everything. (image: results on LongBench v2)

When a benchmark saturates, we write a harder one: MMLU → MMLU-Pro, MMMU → MMMU-Pro. What if the harder benchmark was already sitting inside the old one? 🤔 🔴New preprint: Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling ➡️tl;dr: We introduce Answer Pooling, a label-free way to make multiple-choice benchmarks harder with zero human intervention. Take questions that share a context, pool all their options into one list, and ask the model to match every question to its answer.