Ferdinand Kapl (@KaplFer) profile picture

Ferdinand Kapl

@KaplFer

PhD Student @TU_Muenchen @helmholtz_ai

 Munich, Germany
Joined July 2023
  • Tweets34
  • Following460
  • Followers115
Looped LLMs are everywhere right now! But why are they interesting? Do they model language differently than regular LLMs? Grown and looped transformers are better sequence models! With @kaitlinmaile & @oswaldjoh (@Google, Paradigms of Intelligence Team) and @stefanAbauer. 🧵
Image from Ferdinand Kapl (@KaplFer)
8
27
3
142
8,195
1/ We pretrain models up to 3B (SmolLM3) from scratch by progressively duplicating layers (Growing) or looping middle layers a fixed number of times (Looping). We compare both to a full-depth baseline at equal inference FLOPs, with the looped model having fewer parameters.
Image from Ferdinand Kapl (@KaplFer)
1
1
1
15
941
This project started with a literature search on "growing" work in modern vision pipelines, and I was surprised that there was not much, although there have been more recent works in language. Since I started my PhD by working on vision (object-centric?!) I was interested.
Amir Karimi (@AMK6610) profile pictureAmir Karimi@AMK6610Oct 26
Excited to share “Growing to See”: growing SSL vision models provide better representations with up to 25% fewer FLOPs! Co-led with @KaplFer. With @Google’s Paradigms of Intelligence team (@kaitlinmaile & @oswaldjoh), @andrea_dittadi, Karl H. Johansson & @stefanAbauer. 1/13
Image from Amir Karimi (@AMK6610)
1
4
10
442
Although modern vision pipelines are similar to language (e.g., architectures), the most popular approaches differ in their training objectives: AR LLMs vs. Masked/Self-Distillation in vision. This has some surprising effects!
1
2
71
SOTA LLMs are trained as large monolithic models. But looping or depth growth can beat that. How? Co-led with @AngelisManos and @TobiasHppe1. Incredible collaboration with Paradigms of Intelligence Team @Google 🙌 @kaitlinmaile @oswaldjoh @ninoscherrer and @stefanAbauer. 🧵🔽
Image from Ferdinand Kapl (@KaplFer)
1
12
1
19
7,665
Summary. 🔎 We find that training transformers by growing in depth leads to more effective usage of model depth and formation of permutable computational blocks. 🥇 LIDAS, our growing strategy, results in even better reasoning performance while matching NLL of the baseline.
1
4
249

No more items