Erchi Wang retweeted
With large image patches, plain DiTs can succeed with clean-data prediction while velocity/noise prediction can fail dramatically. Why?
In our new blog, we explore this puzzle through the lens of residual-stream burden.
The key idea is that prediction targets determine what information the residual stream must carry. Noisy targets require preserving noise-dependent input variation through depth for the final readout, forcing subsequent layers to compute on representations that still carry this information. This creates friction in representation learning.
As patches grow larger, each token contains more noise-dependent variation that must be preserved through depth, putting greater pressure on the residual stream. In contrast, clean targets are concentrated in fewer directions, reducing this preservation burden and leaving more residual-stream bandwidth for learning useful representations.
This perspective also helps explain why architectures that provide separate information pathways or expand residual-stream bandwidth via Hyper-Connections can alleviate the friction.
Read our blog:
tongtongliang.github.io/resi…
🤖 Made with AI
Erchi Wang retweeted
#NeurIPS results are out! Whether your paper is accepted or rejected, you still have a chance to present at InfPriv @ #NeurIPS2026, Sydney! Consider submitting your work via our NeurIPS fast-track with minimal effort, where the deadline is 9/25 AOE.
beyond-private-training.ai.s…
How do we push AI privacy beyond standard DP training? 🤔 Join our #NeurIPS2026 Workshop in Sydney to tackle the biggest open questions in privacy for AI at inference time and AI for privacy!
Learn more: beyond-private-training.ai.s…
Erchi Wang retweeted
InfPriv @ NeurIPS 2026, Sydney deadline is approaching (about 4 days left)! Submit your work by 9/7 AOE! Join our discussion on AI for privacy and privacy for AI!
How do we push AI privacy beyond standard DP training? 🤔 Join our #NeurIPS2026 Workshop in Sydney to tackle the biggest open questions in privacy for AI at inference time and AI for privacy!
Learn more: beyond-private-training.ai.s…
Erchi Wang retweeted
It's my pleasure to organize this workshop with @ruihan_w @kamalikac @yuxiangw_cs @niloofar_mire @erchi_w @JiachenWang97 @ahonkela !
We also have a list of great speakers: @lin_zinan @ChulinXie @MihaelaVDS ,Dinh Thai Hoang, and Saeed Mahloujifar!
Can frontier models reasoning about Differential Privacy?
We are excited to release DPrivBench, a benchmark curated by DP experts to evaluate LLMs’ reasoning ability on Differential Privacy. @pengrun_huang @chien_eli @omthkkr @kamalikac @yuxiangw_cs @ruihan_w
DPrivBench contains two complementary tracks:
Category 1 (preliminary): fundamental DP mechanism questions focused on sensitivity calculation and noise calibration.
Category 2 (advanced): research-level DP algorithms from the literature that require advanced, algorithm-specific mathematical reasoning.
Our findings are both promising and cautionary: frontier models are strong on Category 1, yet still struggle with research-level DP algorithms requiring nuanced reasoning about assumptions, privacy accounting, and algorithm-specific guarantees.
We warmly welcome comments on interesting DP topics/problems for dataset extensions, as well as ideas for using this benchmark to build LLM-powered DP pipelines. Excited to see what the community builds from it!
Please check out our paper and play with the dataset. We’d love to hear your feedback!
Paper: arxiv.org/pdf/2604.15851
GitHub: github.com/erchiw/DPriv-Benc…
Dataset: huggingface.co/datasets/erch…
1
240





