PhD student in Computer Vision for plants and certified frenchmaxxer
Joined October 2013
- Tweets1.1K
- Following681
- Followers268
- Likes2.1K
Pinned Tweet
I finished training JEM for 100 epochs, and the results are actually better than I expected. It's only 1 point away from a fine-tuned DINOv3 (which has seen WAY more data). I'm pretty sure it could beat it with more training.
It took a bit more than 20 H100 hours to train for 100 epochs on the dataset (~250k training images), and as you can see, my LeJEPA is still undertuned (Claude is currently working on it to get a better baseline).
The main problem, though, is with rare classes, but that's expected. It might be a sampling issue. For example, I could continue training while oversampling rare classes, but that's not really what I want.
All plants kind of look the same, so I'm wondering if we could make the algorithm better at capturing fine-grained details and the patch-level loss already helps a lot imo. But tbh, the dataset is so hard that I'm not even sure there's much room for improvement on rare classes.
I'm running a small experiment to see how they compare for species classification.
I'm training both methods from scratch on a fairly small dataset, Pl@ntNet-300K (github.com/plantnet/PlantNet…): 244k training images of 1,000 plant species with a heavy long tail. I want to see which SSL method best separates rare classes.
For LeJEPA I used the official minimal implementation (MINIMAL.md). For JEM I reimplemented the paper myself and try to match its settings as closely as I could. I use opus and astra to triple check but there's still a possibility that I did something wrong
I used a ViT-B/16 backbone as the best dataset baseline for now is a fine-tuned DinoV3 ViT-B. So far JEM beats LeJEPA but it's looking too good to be true and I'm pretty sure my LeJEPA is under-tuned. But it's still better than nothing I guess, I'll have to retrain a new LeJEPA
Also comparing at equal epochs is unfair because JEM uses ~2x more compute per image (due to teacher pass). But JEM is still ahead at equal compute: LeJEPA's 100 epochs took ~23h on one RTX PRO 6000, the same as ~40 epochs of JEM, and at that point JEM's linear probe is already higher (76.6% vs 73.1% top-1). On rare species (<20 training images) they're close at equal compute (10.1% vs 9.2%).
Furthermore this task probably suits JEM better due to its patch-level objective that helps capture the small details separating species.
For reference, the released Pl@ntNet-300K-v2 ViT-B checkpoint (DINOv3 pre-trained on 1.7B images, then fine-tuned) reaches 84.5% top-1, but only ~32% on those rare species.
So it's not the end of the story, I'll continue the experiments and see if the results change with better tuning. For rare species I obvly don't expected any of them to beat a fine tuned dinoV3 as it has a better feature space, but I'm curious to see how far SSL can get from scratch with this "little" data.
Is there a word for when you're confident in your paper until 3 days before the deadline, and then suddenly you think it's shit and want to rewrite everything?
1
6
567
idk who made the tikz rl env for Opus5.5 and Fable5.2 but you have a special place in my heart
4
303
CVPR paper nearly finished, so let's continue working on this side project. Previously I compared LeJEPA and JEM on PlantNet300K, so I thought: why not combine both?
I always thought teacher/student EMA was kind of a “bizarre thing,” so I wanted to see if JEM could work without it (they already say that the EMA is not needed in the paper, but I wanted to go further).
JEM uses a teacher to produce the targets and two regularizers to avoid collapse. LeJEPA has no teacher at all, and SIGReg is the thing that prevents collapse. So the idea is just to keep JEM's crops and patch-level matching, train both sides with the same network, and let SIGReg replace JEM's regularizers.
Introducing LeJEM! (you can see it as a kind of a modified VICRegL too, so call it whatever you want. I'm not saying this is a novel idea anyway arxiv.org/abs/2210.01571)
It's a single network with no EMA or stop-gradient. The loss matches the same patch seen in two crops, and SIGReg is applied twice, once on the whole image like in LeJEPA and once on the patches inside each image. Otherwise, all the patches of an image end up with the same code.
I gave this idea to claude to implement, and voilà! Also, one thing claude did on its own was remove masking. Without the stop-gradient, predicting masked patches makes the patches collapse inside each image, so thanks Claude for figuring that out, I guess.
After 100 epochs with the same ViT-B/16, LeJEM gets 83.9% top-1 with an attention probe (which is on par with DinoV3 !), vs. 82.8% for JEM and 80.5% for my tuned LeJEPA. It never collapsed, and it's actually ahead of JEM early in training (67.4% vs. 62.7% at epoch 10 for the online val probe), but JEM catches up at the end. Also, the linear probe is a bit worse on LeJEM than on JEM without LS.
tbh, I expected it to be faster to train, but it's not. The teacher pass is forward-only, so it was never that expensive. It still takes ~20 H100 hours.
I can't say it's significantly better than JEM, but imo it's cool that it works. Also, I haven't looked at segmentation yet, which is probably where removing masking hurts the most, and I have no idea if it scales (which is the point of JEM). So stay tuned (or not. I've already used too much compute on this side project. If someone wants to give me an internship with lots of compute I'll be down).
(The figure credits go entirely to Claude and TikZ.)
Finally, here’s a visualization of the patch features.
I compare LeJEPA, JEM with and without LayerScale, and the supervised DINOv3 checkpoint from Pl@ntNet as a reference.
Interestingly, even though LeJEPA performs worse, its features still look crisp, especially since it’s only trained through its global embedding and never gets a loss on its patch tokens.
Also, removing LayerScale from JEM was the right move. Without it, the maps are smoother and segment leaves and flowers cleanly.
Simon retweeted
Finally, here’s a visualization of the patch features.
I compare LeJEPA, JEM with and without LayerScale, and the supervised DINOv3 checkpoint from Pl@ntNet as a reference.
Interestingly, even though LeJEPA performs worse, its features still look crisp, especially since it’s only trained through its global embedding and never gets a loss on its patch tokens.
Also, removing LayerScale from JEM was the right move. Without it, the maps are smoother and segment leaves and flowers cleanly.
I finished training JEM for 100 epochs, and the results are actually better than I expected. It's only 1 point away from a fine-tuned DINOv3 (which has seen WAY more data). I'm pretty sure it could beat it with more training.
It took a bit more than 20 H100 hours to train for 100 epochs on the dataset (~250k training images), and as you can see, my LeJEPA is still undertuned (Claude is currently working on it to get a better baseline).
The main problem, though, is with rare classes, but that's expected. It might be a sampling issue. For example, I could continue training while oversampling rare classes, but that's not really what I want.
All plants kind of look the same, so I'm wondering if we could make the algorithm better at capturing fine-grained details and the patch-level loss already helps a lot imo. But tbh, the dataset is so hard that I'm not even sure there's much room for improvement on rare classes.
Simon retweeted
here is the code: github.com/saulane/jem-plant…
and also the weights: huggingface.co/saulane/jem-v…
Finally, here’s a visualization of the patch features.
I compare LeJEPA, JEM with and without LayerScale, and the supervised DINOv3 checkpoint from Pl@ntNet as a reference.
Interestingly, even though LeJEPA performs worse, its features still look crisp, especially since it’s only trained through its global embedding and never gets a loss on its patch tokens.
Also, removing LayerScale from JEM was the right move. Without it, the maps are smoother and segment leaves and flowers cleanly.
I finished training JEM for 100 epochs, and the results are actually better than I expected. It's only 1 point away from a fine-tuned DINOv3 (which has seen WAY more data). I'm pretty sure it could beat it with more training.
It took a bit more than 20 H100 hours to train for 100 epochs on the dataset (~250k training images), and as you can see, my LeJEPA is still undertuned (Claude is currently working on it to get a better baseline).
The main problem, though, is with rare classes, but that's expected. It might be a sampling issue. For example, I could continue training while oversampling rare classes, but that's not really what I want.
All plants kind of look the same, so I'm wondering if we could make the algorithm better at capturing fine-grained details and the patch-level loss already helps a lot imo. But tbh, the dataset is so hard that I'm not even sure there's much room for improvement on rare classes.
here is the code: github.com/saulane/jem-plant…
and also the weights: huggingface.co/saulane/jem-v…
I retuned LeJEPA for a fairer comparison, but it's still not better than JEM, (even though I didn't tune it much). One could argue that LeJEPA is still undertrained, looking at the compute-equivalent plot, but it's really plateauing now so not sure I could make it much better without changing the recipe too much.
I'm also currently running another JEM experiment without LayerScale, as I noticed it was causing the attention residual contribution to dip for unknown reasons (and also because it's only a ViT-B, which doesn't need LayerScale for stability). It's already beating everything I've tried so far.
Again, this is a "smallish" dataset, and JEM was designed to scale, so it's nice to see it perform well here, and with just simple tricks I'm sure it could be really data efficient. LeJEPA also has the advantage of being efficient and easy to train and tune (even though, as @gabriberton pointed out teapawt.org/gabriberton/status/211…, it performs worse in practice most of the time).
But sometimes you just don't have access to 10k H100s or a B300 rack, so compute efficient methods are still nice. Not everyone can embrace the bitter lesson after all.
I'm running a small experiment to see how they compare for species classification.
I'm training both methods from scratch on a fairly small dataset, Pl@ntNet-300K (github.com/plantnet/PlantNet…): 244k training images of 1,000 plant species with a heavy long tail. I want to see which SSL method best separates rare classes.
For LeJEPA I used the official minimal implementation (MINIMAL.md). For JEM I reimplemented the paper myself and try to match its settings as closely as I could. I use opus and astra to triple check but there's still a possibility that I did something wrong
I used a ViT-B/16 backbone as the best dataset baseline for now is a fine-tuned DinoV3 ViT-B. So far JEM beats LeJEPA but it's looking too good to be true and I'm pretty sure my LeJEPA is under-tuned. But it's still better than nothing I guess, I'll have to retrain a new LeJEPA
Also comparing at equal epochs is unfair because JEM uses ~2x more compute per image (due to teacher pass). But JEM is still ahead at equal compute: LeJEPA's 100 epochs took ~23h on one RTX PRO 6000, the same as ~40 epochs of JEM, and at that point JEM's linear probe is already higher (76.6% vs 73.1% top-1). On rare species (<20 training images) they're close at equal compute (10.1% vs 9.2%).
Furthermore this task probably suits JEM better due to its patch-level objective that helps capture the small details separating species.
For reference, the released Pl@ntNet-300K-v2 ViT-B checkpoint (DINOv3 pre-trained on 1.7B images, then fine-tuned) reaches 84.5% top-1, but only ~32% on those rare species.
So it's not the end of the story, I'll continue the experiments and see if the results change with better tuning. For rare species I obvly don't expected any of them to beat a fine tuned dinoV3 as it has a better feature space, but I'm curious to see how far SSL can get from scratch with this "little" data.
One thing I found interesting about JEM is that the attention LayerScale dips after each supervised layer. JEM supervises training at multiple depths, not only at the output layer; this is discussed in the supp of the paper.
However, one thing I did not know was that a side effect of doing so is that the attention LayerScale dips. I also computed the total attention residual contribution to x (‖γ ⊙ layer output‖ / ‖x‖), and it's also significantly down after each supervised block. It's well known that attention LayerScale grows with depth, but I didn't know it kind of "resets" after intermediate losses.
I may need to study a bit more to understand why it happens. @maxseitzer, do you know anything about it?
1
320
This is a super cool paper that makes me believe even more that there must be a way to make SSL simpler and more theoretically grounded.
I asked Opus to make a recap video in Manim using ElevenLabs voices. It’s not perfect, but it’ll give you a nice overview before diving deep into the paper.
opus 5.5 code is usually not what a human would call "pretty" (especially the 'while True: ... break' here wtf), but at the same time the formatting kind of reminds me of @__tinygrad__ code which I really like (tinygrad still better though)
1
153
