3D Gaussian Splatting is great, but can it work without the pre-computed camera poses? Introducing: COLMAP-Free 3D Gaussian Splatting Our recent work shows not only it can, but 3D Gaussians make camera pose estimation easy (compared to NeRF) along with reconstruction. 👇🧵

Congratulations to Jeff Dean, Greg Corrado, & co-authors of the paper “Distributed Representations of Words and Phrases and their Compositionality”, for winning the #NeurIPS2023 Test of Time Award! This prize recognizes a highly impactful paper published at NeurIPS 10 years ago.

Excited to share our results on Efficient Coding of Natural Images using Maximum Manifold Capacity Representations, a collaboration with @KuangYilun @EeroSimoncelli and @s_y_chung to be presented at #NeurIPS2023 1/n

tl;dr: Maybe learning simple things (basic knowledge, heuristics, etc) actually lowers the loss more than learning sophisticated things (algorithms associated with higher cognition that we really care about), and the sophisticated things will eventually be learned as scaling further drives down the loss. Some rough thoughts on this issue: As we know, LLMs are (pre)trained to minimize next-token prediction loss across human text. Ignoring fine-tuning, RLHF, etc., what they learn will be determined both by (1) what is __possible__ to learn and (2) what is __optimal__ to learn under this objective. So what do models learn under this objective? The low-hanging fruit might be to learn token frequencies, then common bigrams, then to learn how to repeat subsequences in context (transformer-circuits.pub/202…) then simple grammatical rules, then all sorts of knowledge, and so on and so on. With a small network you can only learn the low-hanging fruit (arxiv.org/abs/2303.13506), but of course the *hope* with scaling is that eventually, to reduce the loss slightly more, you'll need to learn the more sophisticated algorithms for reasoning or planning or drawing connections between ideas or coming up with new explanations (gwern.net/scaling-hypothesis). So why might current models have not learned the algorithms associated with higher cognition yet? It could be that simply (1) they are not possible for the models to learn. But one defense of scaling would be that (2) is the bottleneck: It could be possible that the (loss improvement) ÷ (network capacity) of learning the algorithms/circuits we associate with higher cognition is still worse than for learning many additional pieces of knowledge or statistical patterns in language. Like maybe the model could have learned the "making new connections" algorithm but actually that is suboptimal and you reduce the loss more by learning lots of dumb things instead. Presumably though eventually the algorithms for reasoning, planning, making new connections, etc. would be the optimal things to learn to further reduce loss, and they would be learned with further scaling. This explanation feels slightly odd to me since I would have expected that learning the sophisticated general parts of cognition would be very frequently useful when predicting human text and therefore would have been learned by now. But maybe it's just a fact about predicting language that there is a *lot* of low-hanging fruit in learning the dumber things: xitter.cf/conitzer/status/167362… Explanation (1) -- there's a fundamental limitation -- is complicated. Maybe there are so many steps or so much compute needed to implement certain algorithms that it's just really hard to implement them in a forward pass of a transformer. Further scaling could maybe solve this (maybe you just needed to make the network deeper), but maybe you'd need such a massive network that it's practically impossible. Maybe it's not about expressivity but instead about whether SGD can learn the algorithms we care about and whether we have the data needed to learn them: xitter.cf/xuanalogue/status/1666…. So a rebuttal to your point would be about (2), but personally I don't know if that explanation is right! The bottleneck might really be about expressivity or learnability and we're just using the wrong architectures or objectives or data. Super interesting question.

I'm excited to see more paper walkthroughs exist!

Everyone is using RAG, but most of the retrieved context is noisy! 🚨 Introducing FilCo: “Learning to Filter Context for Retrieval-Augmented Generation” TL;DR: Get rid of the irrelevant content using FilCo, and you'll get better outputs. Preprint: arxiv.org/abs/2311.08377

Very excited to release the consistency decoder for DALLE-3, a consistency model that transforms VQGAN latents to images with improved quality at remarkable speeds!