Chandar Lab

We got these & other insights from @dicemasz main MSc work, which focused on evaluating LLMs’ limitations for communicating risk in natural language — already out on preprint if you want to learn more!! See: arxiv.org/abs/2607.03882

From small open source models, to commercial giants, and across different domain contexts, LLMs tend to exhibit a narrow vocabulary and consistently violate their own previously established criteria when describing how likely something is to occur.

Turns out that many contemporary LLMs still struggle to explain probabilities and uncertainty in plain natural language 👀

We went from basic python commands all the way to training and optimizing their very first supervised ML model! All in a month! It’s becoming increasingly important to introduce young students to computer science and algorithmic thinking, as AI continues to transform the world.

For the third year in a row, we held our annual Highschool Internship for recent graduates from across Montreal! 🎓🎉💻 Ten bright students underwent a month-long summer school and got introduced & coached on the basics of computer science and programming!

One might suspect this diversity comes from flattening the next-token distribution, destroying overall confidence. We ruled this out via PALOMA perplexity evaluation. General language fit is preserved or improved, though we observe a small trade-off in math reasoning (GSM8K).

Beyond math: these gains transfer to open-ended natural language generation. Our soft-target configuration significantly broadens stochastic support for requests like "name a random city" and improves semantic spread on NoveltyBench.

Across 12 models spanning four families, both methods substantially improve structured-sampling fidelity, generalizing both to entirely new mathematical distributions and to unfamiliar parameter configurations of the distributions they did see during training.

To do this, we train LMs on synthetic prompts needing samples from exact mathematical distributions. We compare two strategies: soft-target (mapping target distributions into token space and optimizing via KL divergence) and hard-target (training directly on sampled completions).

New paper from the lab: Ask a base LLM to pick a random weekday, and it might output "Wednesday" 80% of the time. We show that this can be fixed and that probabilistic calibration is actually a trainable capability.

Sugar-Shack with our lab!! 🍁🪵 Saying bye bye to a long, harsh winter and hello to sunny days!!☀️🤩

🔥 This was the first work to show that you can successfully use adaptive gradient optimizers for lifelong learning and still beat SGD!!! (i.e. RMSProp < SGD < TAG-RMSProp) 🔥 Across benchmarks, TAG also improved final accuracy over baselines like ER and A-GEM.

TAG tracks gradient traces from previously learned tasks and estimates task similarity: ⬇️ Lower α = related tasks= transfer encouraged ⬆️ Higher α = conflicting tasks = reduce interference