Beyza Ermiş (@beyzaermis) profile picture
Joined April 2010
Really glad I got to be part of this one. Huge credit to @TheyCallMeMr_ for leading the work, and grateful to the whole team!
1/ RLVR has driven big gains in math and code because many outputs admit reliable automatic checks: an answer matches the expected result, or a program passes tests. But many real tasks are not like that. Code can be functionally correct but qualitatively terrible or a response may satisfy 4 syntactic constraints but fail 1 semantic constraint.
2
12
1,681
So happy this is out in EACL Findings! 🥳 Loved being able to contribute a bit, big congrats to @_joestacey_ and all the authors! Really nice practical takeaways on robustness for closed-source LLM fine-tuning.
Super excited to have my last PhD paper about NLI robustness published at EACL Findings😍 We investigate how to make closed-source LLMs more robust after fine-tuning. Here are the paper highlights 🧵
2
4
19
2,497
Proud to share this work led by @oliverjbolton 🎉 We introduce SimMerge, a practical way to make model merging more reliable at scale. Paper and results in the thread 👇
🧩In modern LLM development, we often end up with many specialized checkpoints. Merging into one model is attractive, but the results depend a lot on the selected merge method & order.  Our paper introduces SimMerge: a simple way to choose the merge configuration automatically.
10
28
5,410
Sadly, this echoes what many authors and ACs have been feeling. The peer-review system is under serious strain. We can't keep pretending it's fine.
I am an AC for ICLR 2026. One of the papers in my batch was just withdrawn. The authors wrote a brief response, explaining why the reviewers failed at their job. I agree with most of their comments. The authors gave up. They are fed up. Just like many of us. I understand. We pretend the emperor has clothes, but he is naked. Here is the final part of their withdrawal notice. I took the liberty to make it public, to highlight that what we are doing with AI conference reviews these last few years is, basically, madness. --- Comment: We thank the reviewers for their time. However, upon reading the reviews for our paper, it became immediately apparent that the four "reject" ratings are not based on good-faith academic disagreement, but on a critical failure to read the submitted paper. The reviews are rife with demonstrably false claims that are directly contradicted by the text. The core justifications for rejection rely on asserting that key components are "missing" when they are explicitly detailed in the manuscript. Some specific examples are (and many are even fake claims). Claim: Harder tasks like GSM8K are missing. Fact: GSM8K results are in many tables, like Table 2 (Section 4.2) and Appendix G. Claim: The method does not use per-layer ranks. Fact: This is the entire point of our method. The reviewer clearly mistook our method for the baselines. (Section 2, Table 1). Claim: The GP kernel is not specified. Fact: It is specified in Appendix E (Table 6). Claim: There is no ablation of the method's three stages. Fact: Section 4.4 ("Ablation Study") and Appendix J are dedicated to this. Reviewers have a fundamental responsibility to read and evaluate the work they are assigned. The nature of these errors is so fundamental, so systemic in overlooking explicit content, that it goes far beyond what "limited time" or "oversight" can explain. This work has gone through several rounds of revision over the last year. In earlier submissions, the paper usually received borderline or weak-accept scores. Numerous signs strongly suggest that some reviewers are relying entirely on AI tools to automatically generate peer reviews, rather than fulfilling their fundamental responsibility of personally reading and evaluating manuscripts. We strongly protest this. This is a gross disrespect to the authors. It is a flagrant desecration of the reviewer's sacred duty. It fundamentally undermines the integrity of the entire peer-review process. Given that the reviews are not based on the actual content of our paper, we have decided to withdraw the submission. We leave this comment so that future readers of the OpenReview page are aware that the items described as "missing" are already present in the submitted manuscript. These negative reviews for this submission are factually unsound and do not reflect the content of the paper. We cannot and will not accept an assessment that is not based on the work we actually submitted.
13
1,574
I’m looking forward to working with the next cohort. Don’t miss the deadline!
Applications are now open for the next cohort of the Cohere Labs Scholars Program! 🌟 This is your chance to collaborate with some of the brightest minds in AI & chart new courses in ML research. Let's change the spaces breakthroughs happen. Apply by Aug 29.
3
8
107
9,159
I’m very excited to be co-organizing this @NeurIPSConf workshop on LLM evaluations! Evaluating LLMs is a complex and evolving challenge. With this workshop, we hope to bring together diverse perspectives to make real progress. See the details below:
We are happy to announce our @NeurIPSConf workshop on LLM evaluations! Mastering LLM evaluation is no longer optional -- it's fundamental to building reliable models. We'll tackle the field's most pressing evaluation challenges. For details: sites.google.com/corp/view/l…. 1/3
1
14
52
6,571
Happy to have contributed to this work led by @_joestacey_, where we explore improving the robustness of closed-source LLMs fine-tuned on NLI under realistic data constraints.
We have a new paper up on arXiv! 🥳🪇 The paper tries to improve the robustness of closed-source LLMs fine-tuned on NLI, assuming a realistic training budget of 10k training examples. Here's a 60 second rundown of what we found!
4
6
1,738
Our new Aya Vision report is out! 🚀 Big congratulations to @TheyCallMeMr_, @YiyangNan, and the fantastic team at @Cohere_Labs @cohere! 🎉 📄 Technical report: arxiv.org/abs/2505.08751 🛠️ Models and datasets: huggingface.co/collections/C…
How do we build multimodal systems that work effectively across the globe? 🌍 Today we release the Aya Vision Technical Report, the detailed recipe behind Aya Vision models, unifying state-of-the-art multilingual capabilities in multimodal and text tasks across 23 languages!
9
21
2,886
When a benchmark becomes the standard, it naturally becomes a target. The real challenge is keeping it robust — which is exactly what we explore in our latest work: 📄 arxiv.org/pdf/2504.20879
It is critical for scientific integrity that we trust our measure of progress. The @lmarena_ai has become the go-to evaluation for AI progress. Our release today demonstrates the difficulty in maintaining fair evaluations on @lmarena_ai, despite best intentions.
6
36
2,021
So happy to be a part of this! 🌎🌍🌏
Is MMLU Western-centric? 🤔 As part of our cross-institutional work: 🥢 We conduct a large-scale cultural bias study on MMLU 🔍 Examine how cultural sensitivity impacts multilingual evaluations 🌍 Release Global-MMLU: a benchmark with MMLU translations in 42 languages
1
3
19
2,204
Have you had a chance to check out Aya Expanse on @huggingface Space yet? 👀
We're thrilled to see that Aya Expanse is trending as one of the @huggingface spaces of the week! 🔥 Check out the Aya Expanse Hugging Face Space to chat, speak, and visualize with Aya in 23 languages. ✍️🗣️🎨huggingface.co/spaces/Cohere…
1
3
14
2,707
I can't imagine a better way to kick off my X journey than by sharing the 🌿Aya Expanse 🌿, a state-of-the-art multilingual model family that spans 23 languages. To learn how Aya Expanse is built: huggingface.co/blog/aya-expa…
Introducing ✨Aya Expanse ✨ – an open-weights state-of-art family of models to help close the language gap with AI. Aya Expanse is both global and local. Driven by a multi-year commitment to multilingual research. cohere.com/research/aya
5
8
1
45
9,442
It was a lot of work, but I enjoyed every moment of collaborating with the brilliant people at @CohereForAI, @cohere, and the amazing research community 💙
1
11
201