Quite interesting findings. It makes sense to me that LR should be designed along with data, as LR can be viewed as a form of data reweighting.

🚀 Very excited to introduce TTR-Compressor -- A test-time regression framework for sequence compression! I particularly like this work for its fundamentally different perspective on sequence compression. Unlike popular pooling and projection, TTR-Compressor formulates sequence compression as a test-time regression problem. Instead of directly aggregating tokens, we fit regression keys derived from the input tokens to learnable targets, using the fitted coefficients as the compressed representation. This leads to a bi-level optimization with an exact closed-form solution to the inner test-time regression. The closed-form solution captures second-order inter-token correlations, allowing TTR-Compressor to reduce redundancy and better preserve distinctive information, going beyond conventional first-order aggregation in pooling and projection. Strong gains in two applications: ✨ Block-wise sparse attention: +12.24% in RULER retrieval accuracy over attention-based compression. ✨ Visual token merging: Comparable multimodal performance with 55.6% fewer visual tokens than standard MLP merging. 🤔 Takeaway: Sequence compression can be viewed as input-adaptive regression rather than static aggregation. This provides new insights to building more efficient language and multimodal models. 📄 Paper: arxiv.org/abs/2610.31574 🌐 Project: spherelab.ai/ttr-compressor/ 💻 Code: github.com/Sphere-AI-Lab/ttr…

Can LLMs manage their own context natively without any external harness? We are teaching LLMs the ability of Native Context Editing. Our method achieves considerable performance gain on popular long-horizon agentic benchmarks (Terminal-Bench, SWE-bench) while reducing token usage significantly. 🤔 This can be viewed as a form of verbalized context management, which leverages LLMs' inherent capabilities to optimize context. Our prior work, Verbalized Machine Learning (arxiv.org/abs/2406.04344), similarly explores context optimization to enhance downstream task performance. Project: dengyl20.github.io/native-co… Paper: arxiv.org/pdf/2610.25550 Code: github.com/dengyl20/native-c…

It is always inspiring to see great professors who genuinely love what they do and bring such passion to teaching.