Pretty AI for solving real-world problems
Toulouse, France
Joined September 2016
- Tweets792
- Following250
- Followers270
- Likes753
Jolibrain retweeted
New paper by Joey Hong shows how we can train LLMs with value-based RL for multi-turn tasks *just by turning probabilities into Q-values*! This provides an algorithm that can be used for LLMs, VLMs, robotics tasks, etc. with one simple loss function. Thread👇
Jolibrain retweeted
🚀Now it is the time, Nov. 11 10:24! The perfect time for our best coder model ever! Qwen2.5-Coder-32B-Instruct!
Wait wait... it's more than a big coder! It is a family of coder models! Besides the 32B coder, we have coders of 0.5B / 1.5B / 3B / 7B / 14B! As usual, we not only share base and instruct models, we also provide quantized models in the format of GPTQ, AWQ, as well as the popular GGUF! 💖
👉🏻Blog: qwenlm.github.io/blog/qwen2.…
👉🏻Tech Report: arxiv.org/abs/2409.12186
👉🏻Hugging Face: huggingface.co/collections/Q…
👉🏻ModelScope: modelscope.cn/collections/Qw…
👉🏻Kaggle: kaggle.com/models/qwen-lm/qw…
👉🏻GitHub: github.com/QwenLM/Qwen2.5-Co…
👉🏻Demo [chat]: huggingface.co/spaces/Qwen/Q…
👉🏻 Demo [Artifacts]: huggingface.co/spaces/Qwen/Q…
The flagship model, Qwen2.5-Coder-32B-Instruct, reaches top-tier performance, highly competitive (or even surpassing) proprietary models like GPT-4o, in a series of benchmark evaluation, including HumanEval, MBPP, LiveCodeBench, BigCodeBench, McEval, Aider, etc. It reaches 92.7 in HumanEval, 90.2 in MBPP, 31.4 in LiveCodeBench, 73.7 in Aider, 85.1 in Spider, and 68.9 in CodeArena!
Jolibrain retweeted
This is the most important paper in a long time . It shows with strong evidence we are reaching the limits of quantization. The paper says this: the more tokens you train on, the more precision you need. This has broad implications for the entire field and the future of GPUs🧵
[1/7] New paper alert! Heard about the BitNet hype or that Llama-3 is harder to quantize? Our new work studies both! We formulate scaling laws for precision, across both pre and post-training arxiv.org/pdf/2411.04330. TLDR;
- Models become harder to post-train quantize as they are overtrained on lots of data, so that eventually more pretraining data can be actively harmful if quantizing post-training!
- The effects of putting weights, activations, or attention in varying precisions during pretraining are consistent and predictable, and fitting a scaling law suggests that pretraining at high (BF16) and next-generation (FP4) precisions may both be suboptimal design choices!
Joint work with @ZackAnkner @spectorb @blake__bordelon @Muennighoff @mansiege @CPehlevan @HazyResearch @AdtRaghunathan.
Jolibrain retweeted
Excited to share our NeurIPS 2024 Oral, Convolutional Differentiable Logic Gate Networks, leading to a range of inference efficiency records, including inference in only 4 nanoseconds 🏎️. We reduce model sizes by factors of 29x-61x over the SOTA.
Paper: arxiv.org/abs/2411.04732
Jolibrain retweeted
Our NeurIPS paper is published on arXiv.
In this paper, we propose a new optimizer ADOPT, which converges better than Adam in both theory and practice.
You can use ADOPT by just replacing one line in your code.
arxiv.org/abs/2411.02853
Jolibrain retweeted
Is pixel diffusion passé?
In 'Simpler Diffusion' (arxiv.org/abs/2410.19324) , we achieve 1.5 FID on ImageNet512, and SOTA on 128x128 and 256x256.
We ablated out a lot of complexity, making it truly 'simpler'. w/ @tejmensink @JonathanHeek @KayLamerigts @RuiqiGao @TimSalimans
ALT Pixel-space diffusion performance over the years.
Jolibrain retweeted
in order to **generate** images from an autoregressive transformer trained only for next-token prediction, you must use discrete tokens. there is some new work on autoregressive image generation without VQ (see arxiv.org/abs/2406.11838) that uses a new diffusion loss formulation.
Jolibrain retweeted
There exists a single increasing map between two distributions, it is the optimal transport, and it is defined from the cumulative functions. en.wikipedia.org/wiki/Histog…
Jolibrain retweeted
SAM & SAM-2 are great but depend on costly annotations.
Can we 'segment anything' without supervision?🤔
Yes! Check out UnSAM @NeurIPS24—an unsupervised segmenter that achieves SAM-level results! 🎉
Even better—UnSAM+ beats SAM with +6.7% AR & +3.9% AP, using just 1% labels!💪
Jolibrain retweeted
New theoretical work on diffusion objectives: arxiv.org/abs/2303.00848
We e.g. show that under a simple condition (monotonic weighting, satisfied by e.g. the v-prediction loss), diffusion objectives equal the ELBO with data augmentation, namely additive noise.
1/2
Jolibrain retweeted
Framing regression as a classification has been “dark knowledge” for some time. We wanted to shed some light on this phenomenon in deep RL:
Framing value-learning as a classification significantly improves performance and scalability in deep RL.
But... not all classification methods are built equally.
Paper: arxiv.org/abs/2403.03950
🧵👇
Jolibrain retweeted
Hourglass + Diffusion = ❤️
We introduce a new transformer backbone for diffusion models that can directly generate megapixel images without the need for multiple stages like latent diffusion.
Read here! → arxiv.org/abs/2401.11605
Project page → crowsonkb.github.io/hourglas…
Jolibrain retweeted
Excited to share AIM 🎯 - a set of large-scale vision models pre-trained solely using an autoregressive objective. We share the code & checkpoints of models up to 7B params, pre-trained for 1.2T patches (5B images) achieving 84% on ImageNet with a frozen trunk.
(1/n) 🧵
Jolibrain retweeted
Apple finally open-sourced σReparam.
stabilizes transformer training in PyTorch and JAX.
fantastic work by the whole team. and thanks @jramapuram for chasing this up!
github.com/apple/ml-sigma-re…
ALT Regularization of the spectral norm (via sigma-reparameterisation of Linear layers) 1) prevents any arising instability pushing the model too far away from the current solution, and 2) keeps the attention echanism useful, such that gradients are available for any required correction.
Jolibrain retweeted
Take Flow Lenia, add random inflow and outflow of matter + ecological niches, what happens ? 🧐🔬
💡Spontaneous self-organization of an evolution process ? Generating exploring "agents"?
Jolibrain retweeted
Bonjour @midjourney 👋
We're Heetch, probably not on your radar, but we, along with the 12.5 million residents of the French Banlieue, have just sent out thousands of postcards that you - we mean, your 11 employees - should receive soon.
Once these postcards land on your desks, reach out! We're all ears for a chat ☺️
Best regards from La Banlieue,
Heetch 🚘🚘
PS: Feel free to check the video for more details
Jolibrain retweeted
DINOv2+registers=♥️
We are releasing code and checkpoints for DINOv2 augmented with registers and a slightly better training recipe. No more of those pesky artifacts!
Simple one-liner, try it out:
dinov2_vitg14_reg = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitg14_reg')
Jolibrain retweeted
PØDA: Prompt-driven Zero-shot Domain Adaptation
M. Fahes @tuan_hung_vu @abursuc P. Pérez R. de Charette
tl;dr: we adapt a model trained on a source domain to a novel target domain using only a general prompt description of it #ICCV2023
pdf: arxiv.org/abs/2212.03241
Jolibrain retweeted
Our paper "Spontaneous symmetry breaking in generative diffusion models" was accepted at @NeurIPSConf 2023!
We found that the generative capabilities of diffusion models are the result of a phase transition!
Preprint: arxiv.org/abs/2305.19693
Code: github.com/gabrielraya/symme…
Jolibrain retweeted
📢Delighted to share Recurrent Interface Network (RIN), a new model/architecture that improves performance of state-of-the-art UNet-based image and video diffusion models (on pixels) while achieving 10X efficiency gain.
arxiv.org/abs/2212.11972
Joint work w @ajabri, @fleet_dj
