‼️ new paper https://teapawt.org/t.co/qQco7fO3Dy visiting @meta, nyu phd @agentic_ai_lab prev. voleon group, @UMich
New York, NY
Joined December 2022
- Tweets203
- Following667
- Followers1.2K
- Likes1.3K
Pinned Tweet
Labs disagree on vision encoders for MLLMs! ⚔️
Gemma 4 12B scrapped the encoder.
Kimi K3 dropped encoder pretraining.
Inkling 2 Small revived the encoder Inkling killed.
We trained 200+ models in a controlled study to settle it:
✅ Keep vision parameters, skip the pretraining. And they don't have to be an encoder! 🤔
Paper: arxiv.org/abs/2611.17334 🧵
Chris Hoang retweeted
AI视觉模型的核心设计
来自Meta和纽约大学的研究，做视觉多模态模型的可以参考。核心结论：
1）视觉编码器不用提前单独训练：跟着主模型从零一起训练，最终看图问答的准确率完全不输预先单独训练的版本，而且能省下大笔算力预算。
2）必须保留专门处理图像的独立参数：如果直接删掉视觉部件、让图片和文字共用参数，看图能力会大幅掉队，需要多花3倍算力才能追平。
3）视觉专属参数放进主模型前八层最有效：把视觉独立模块直接嵌进语言模型的前八层内部，不仅识别文字和图表的效果与外挂编码器打平。放进中后层则会拖后腿。
4）嵌入式设计让AI看图出字更快：省去了图片在外挂模块排队处理的环节，1.7B规模的模型在H200显卡上的首字响应时间从64毫秒缩短到37毫秒，反应更快。
项目地址：unbundling.github.io/
Chris Hoang retweeted
LLMs have made remarkable progress, yet MLLMs still face foundational challenges on conflicting training recipes and scaling: how should visual information flow through a LLM?
We show that neither the standalone ViTs that start from the vision side, nor encoder-free designs start from the language side is all right. In our controlled study, we revisit the architectural choices that reshape the choice of whether encoder or not to: 'encoding' and where to train encoding. We hope this work opens up new directions for both vision and multimodal research.
Super excited to work on this with
@_ellisbrown @choang333 @JasonQSY @ShangwenLi1 @ml_perception @gh_marjan @LukeZettlemoyer @giffmana @mengyer Jakob Verbeek @sainingxie @AIatMeta
Labs disagree on vision encoders for MLLMs! ⚔️
Gemma 4 12B scrapped the encoder.
Kimi K3 dropped encoder pretraining.
Inkling 2 Small revived the encoder Inkling killed.
We trained 200+ models in a controlled study to settle it:
✅ Keep vision parameters, skip the pretraining. And they don't have to be an encoder! 🤔
Paper: arxiv.org/abs/2611.17334 🧵
Chris Hoang retweeted
[CV] Unbundling Multimodal Vision Encoding
E Brown, C Hoang, S Qian, S Li… [Meta FAIR & New York University] (2026)
arxiv.org/abs/2611.17334
Chris Hoang retweeted
This work builds on years of research at @metaai, from curating MetaCLIP data to rethinking vision encoding.
Although the data is more invariant and offers observations about the physical world, how to train an architecture that carries those observations into an LLM is still an open question. In a strictly controlled study with total compute, we explored both standalone ViT encoders and encoder-free designs. Such binary decision on vision alone is not enough. We rethink how vision encoding should place its capacity and flow naturally with the LLM.
Super excited to work on this with @_ellisbrown @choang333, @JasonQSY
@ShangwenLi1
@ml_perception
@gh_marjan
@LukeZettlemoyer
@giffmana
@mengyer
Jakob Verbeek
@sainingxie
Chris Hoang retweeted
Last couple years I've been asked what I think about decoder-only vs encoder-decoder for multimodality, and never really given an answer.
Ellis and Chris take a good step towards answering that question in this study.
Two big points:
(1) Recent encoder-free models like Gemma4-12B, Inkling1, TUNA-2, etc are all a single "point estimates": they show this is possible but tell us absolutely nothing about how it scales, or doesn't! -> this paper does.
(2) Encoder-free or not is actually a false dichotomy. It hides three questions which Chris and Ellis nicely disentangle, and answer separately:
(a) separate pre-training,
(b) separate vision params, and
(c) vision params before or inside decoder?
Answer in pic below, and full thread and paper.
Labs disagree on vision encoders for MLLMs! ⚔️
Gemma 4 12B scrapped the encoder.
Kimi K3 dropped encoder pretraining.
Inkling 2 Small revived the encoder Inkling killed.
We trained 200+ models in a controlled study to settle it:
✅ Keep vision parameters, skip the pretraining. And they don't have to be an encoder! 🤔
Paper: arxiv.org/abs/2611.17334 🧵
Vision needs its own parameters, not its own pretraining or its own encoder.
Still open: how should multimodal architectures change for agents, beyond VQA? Curious what you think! 😄
Paper: arxiv.org/abs/2611.17334
Blog, FAQ: unbundling.github.io
ALT The paper's Figure 1 splits a vision encoder into three choices, each with a verdict. Separate pretraining is optional, dedicated vision capacity is key, and that capacity can sit inside the LLM (in-fusion).
1
13
1,159
Had a blast working on this with @_ellisbrown at @AIatMeta! We got to pretrain all kinds of multimodal models as PhDs :)
Huge thanks to @JasonQSY @ShangwenLi1 @ml_perception @gh_marjan @LukeZettlemoyer @giffmana @mengyer Jakob Verbeek @sainingxie @Hu_Hsu for their support and mentorship!
11
1,000
Can you drop the encoder? Not for free.
Encoder-free trails an encoder on VQA at every size from 1B to 6.4B. Catching up takes about 3x the compute!
ALT Encoder-free models score below models with an encoder on VQA at every size from 1B to 6.4B. They need about 3 times the training compute to catch up.
However, the encoder doesn't need pretraining!
We pretrained four encoders with contrastive, SSL and captioning objectives. When pretraining comes out of the same compute budget, none of them pays for itself. At best, they tie an encoder trained from scratch, and more pretraining makes VQA worse.
ALT Pretraining the encoder doesn't pay off at matched total compute. Pretrained encoders tie an encoder trained from scratch when pretraining takes a small share of the budget, and score lower on VQA when it takes a large share.
3
57
Vision needs its own parameters, not its own pretraining or its own encoder.
Still open: how should multimodal architectures change for agents, beyond VQA? Curious what you think! 😄
Paper: arxiv.org/abs/2611.17334
Blog, FAQ: unbundling.github.io
2
31
Chris Hoang retweeted
Context Tuning (ICML 2026) helped language models learn from a few examples by refining their stored context, with model weights held fixed.
Developed by Courant PhD student Jack Lu (@Jacklu_me) and collaborators.
nyudatascience.medium.com/he…
1/2
Chris Hoang retweeted
CDS Asst. Prof. Mengye Ren (@mengyer) and his lab released @outerloop_sci, open-source software whose AI agents open pull requests only when their experiments improve a benchmark.
The agents got a GPT model to its target in 77% fewer training steps.
outerloop.science/
