Pinned Tweet
Our tech report for Zamba-7B-v1 is out. We manage to come close to Llama 3 8B, Mistral 7B and others' level of performance, with only 1T tokens, with faster inference and less memory usage at a fixed context length. Read up to learn about our not-so-secret sauce!
Zyphra is dropping the tech report for Zamba-7B, along with:
- Model weights (phase 1 and final annealed) at huggingface.co/Zyphra
- Inference/generation code (both pure PyTorch and HuggingFace) at github.com/Zyphra/Zamba-torc… and github.com/huggingface/trans…
Tech report: arxiv.org/abs/2405.16712
Intermediate checkpoints and improved phase 1 dataset (+dataset paper!) coming soon.
Intermediate optimizer states available on request.
Some highlights:
Adam Ibrahim retweeted
Another #ICML2025 paper!
Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
TLDR: Predicting language model performance with scale on multiple choice question-answer (MCQA) benchmarks is made difficult b/c ...
1/3
Adam Ibrahim retweeted
Excited to announce our paper ⬇️ was selected as an **Outstanding** paper at @TiFA_ICML2024 🔥🔥🔥
What did the paper show?
Let's try to summarize the paper in a single tweet!!
1/3
❤️🔥❤️🔥Excited to share our new paper ❤️🔥❤️🔥
**Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?**
w/ @haileysch__ @BrandoHablando @gabemukobi @varunrmadan @herbiebradley @ai_phd @BlancheMinerva @sanmikoyejo
arxiv.org/abs/2406.04391
1/N
Adam Ibrahim retweeted
❤️🔥❤️🔥Excited to share our new paper ❤️🔥❤️🔥
**Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?**
w/ @haileysch__ @BrandoHablando @gabemukobi @varunrmadan @herbiebradley @ai_phd @BlancheMinerva @sanmikoyejo
arxiv.org/abs/2406.04391
1/N
Worth noting that we're working with @huggingface to release the model over the next week. Stay tuned !
Adam Ibrahim retweeted
Look at our preprint on Continual Learning for increasing the scalability of LLMs pretraining. A great piece of work led by @ai_phd @benjamintherien and @kshitijkgupta
🔥
Interested in seamlessly updating your #LLM on new datasets to avoid wasting previous efforts & compute, all while maintaining performance on past data? Excited to present Simple and Scalable Strategies to Continually Pre-train Large Language Models! 🧵arxiv.org/abs/2403.08763 1/N
Here is the full paper of the continual pretraining project I have been working on last year. I encourage you to check it out if you pretrain LLMs (in particular, I recommend to start with takeaways in Section 2 and the Table of Contents at the start of the appendix).
Interested in seamlessly updating your #LLM on new datasets to avoid wasting previous efforts & compute, all while maintaining performance on past data? Excited to present Simple and Scalable Strategies to Continually Pre-train Large Language Models! 🧵arxiv.org/abs/2403.08763 1/N
Adam Ibrahim retweeted
Simple and Scalable Strategies to Continually Pre-train Large Language Models
Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start the process over again once new data becomes available. A much more efficient solution is to continually
Adam Ibrahim retweeted
Mila presents Simple and Scalable Strategies to Continually Pre-train Large Language Models
Shows efficient updates to LLMs using simple strategies, achieving re-training results with less compute
arxiv.org/abs/2403.08763
Adam Ibrahim retweeted
State-space models (SSMs) like Mamba and mixture-of-experts (MoE) models like Mixtral both seek to reduce the computational cost to train/infer compared to transformers, while maintaining generation quality.
Learn more in our paper: zyphra.com/blackmamba
Looking forward to see you at the #NeurIPS2023 #NeurIPS23 ENLSP workshop (rooms 206-207), where we'll have a poster about this work at 16:15 !
Adam Ibrahim retweeted
Hi-NOLIN Hindi model will be presented by our @NolanoOrg team (@imtejas13 @_AyushKaushal) and collaborators from our CERC-AAI team (@kshitijkgupta @benjamintherien @ai_phd) at the #NeurIPS2023 this Fri, at this workshop: sites.google.com/mila.quebec…
Adam Ibrahim retweeted
(1/8) The great success of diffusion models such as Stable Diffusion, DALLE & Emu, have raised questions about the use of synthetic data for classification. Our work, "Feedback-guided Data Synthesis for Imbalanced Classification," addresses this question: arxiv.org/abs/2310.00158
Adam Ibrahim retweeted
Rarely been so excited about a paper. Our model has a quality level higher than Stable Diffusion 2.1 at a fraction (less than 12%) of the training cost, less than 20% of the carbon footprint, and it is twice as fast at inference too! That's what I call a leap forward.
Würstchen is a high-fidelity text2image model working at a fraction of the compute needed for StableDiffusion achieving similar/better results.
Now the preprint to v2 is out. Thanks @M_L_Richter, @pabloppp, @dome_271, @chrisjpal for the great collab!
teapawt.org/dome_271/status/170199…
1 Ever wondered how to keep pretraining your LLM as new datasets continue to become available, instead of pretraining from scratch every time, wasting prior effort and compute ? A thread 🧵
15 This will allow you to leverage having already trained on many tokens, with our results suggesting that it will beat training a new model from scratch on the new dataset.
16 See the arXiv link arxiv.org/abs/2308.04014 for a bit more results, and stay tuned for the next steps, starring among others larger models, replay, comparisons with retraining on both datasets mixed together (for those with the budget), evals, and longer sequences of tasks !
