Tejas Pote retweeted
Deleting ~0.1% of weights from an LLM can be competitive with full-rank and LoRA fine-tuning! 🚀
But how do you know which weights to delete? Check out our paper in the below thread which details the method and will be presented at @NeurIPSConf 2026.
#machinelearning #LLMs #ai
Even the simplest weight edit, a deletion, can teach an LLM new skills!
Meet Deletion Tuning (#NeurIPS2026): just zeroing <0.1% of weights matches full fine-tuning's accuracy at scale. It's thousands of times smaller and avoids forgetting, a step toward continual learning. 🧵👇
Tejas Pote retweeted
📄 Paper: arxiv.org/abs/2611.10866
🌐 Project page: agenticlearning.ai/deleting-…
💻 Code: github.com/agentic-learning-…
🤗 Trained masks: huggingface.co/Jacklu0831/de…
So grateful to have co-led this with @yangzb159 and worked with @mikelasby, @Tejas_915, @yanii, and @mengyer at NYU's @agentic_ai_lab and @UCalgaryML!
This was a fun discovery, and it still surprises us that something as simple as deleting weights just worked on everything we tried, across model sizes and RL, SFT, and SDFT. See you at NeurIPS!
Tejas Pote retweeted
Olá, ICLR 2026 in Rio de Janeiro! 🇧🇷
Excited to present our paper, “REAP the Experts: Why Pruning Prevails for One-Shot MoE compression” this Friday from 10:30-13:00 at Pavilion 3 (P3-#701).
REAP is a one-shot MoE pruning algorithm that scales to 1T params. Details 🧵👇
Tejas Pote retweeted
📢 Submissions are OPEN for the Weight Space Symmetry Workshop @icmlconf!
⏰ Deadline extended → April 30 (23:59 AOE)
Consider submitting any work related to weight symmetries: optimization, model merging, weight space learning, and so on!
#ICML2026 #weightsymmetry2026
Tejas Pote retweeted
Need to compress SMoE models without sacrificing their generative abilities? Is expert merging or pruning the better approach?
I’m thrilled to share our latest paper, “REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression”, which tackles these questions. 🧵 (1/9)
Tejas Pote retweeted
[1/2] 🧵
Presenting our work on Understanding Normalization Layers for Sparse Training today at the HiLD Workshop, ICML 2025!
If you're curious about how BatchNorm & LayerNorm impact sparse networks, drop by our poster!
Tejas Pote retweeted
I am excited to share that I will present two of my works at #ICML2025 workshops!
If you are interested in AI security and XAI, feel free to come by and chat - I'm looking forward to all feedbacks and discussions!
Tejas Pote retweeted
@UCalgaryML will be at #ICML2025 in Vancouver🇨🇦 next week: our lab has 6 different works being presented by 5 students across both the main conference & 6 workshops!
We are also recruiting fully-funded ML PhD/MSc students🧑🎓If attending, reach out to me on Whova app to meet 📅 – at Vancouver, British Columbia
ALT The Calgary Machine Learning Lab, will have 6 different works being presented by 5 students across the International Conference on Machine Learning (ICML) 2025 workshops (HiLD, ES-FoMo III, DIG-BUGS, WiML, MusiML) and main conference! Please see our full conference schedule on the lab website for more details.
Tejas Pote retweeted
🎉 Excited to share our latest work has been accepted at #ICML2025.
We explain how Lottery Ticket masks can be extended to arbitrary random initializations by leveraging permutations of weight symmetries achieving strong generalization.
Check out the details: arxiv.org/abs/2505.05143
See you all in Vancouver!😃
1/10 🧵
🔍Can weight symmetry provide insights into sparse training and the Lottery Ticket Hypothesis?
🧐We dive deep into this question in our latest paper, "Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry", accepted at #ICML2025
Tejas Pote retweeted
Excited to share our work with the community. Thanks @CohereForAI for hosting!
Mark your calendars for January 17th when @JainRohan16 presents "Winning Tickets from Random Initialization: Aligning Masks for Sparse Training"
Special thanks to @Sree_Harsha_N and @aniervs
for organizing this event 🥳
Learn more: cohere.com/events/cohere-for…
Tejas Pote retweeted
Our workshop proposal on sparsity in LLMs, including quantization, MoEs, etc. was accepted at @iclr_conf 2025! 🎉
openreview.net/forum?id=AqpD…
Collaboration with @TianlongChen4 @utkuevci @BerivanISIK @Shiwei_Liu66 @adnan_ahmad1306 Alexandra Novak and @PandaAshwinee
Tejas Pote retweeted
✨Our new @unireps paper tries to answer why the Lottery Ticket Hypothesis (LTH) fails to work for different random inits through the lens of weight-space symmetry.
We improve the transferability of LTH masks to new random inits leveraging weight symmetries.
🧵(1/6)
Tejas Pote retweeted
Knowledge #distillation is a widely used model compression method. We explore the nuanced impact of temperature on distilled models' #fairness ⚖️. Our findings show distilled students are less fair than their teachers at typical temps, but can be more fair in some instances…🧵👇
