@llm4eval

The Third Workshop on Large Language Models for Evaluation in IR (LLM4IREval@SIGIR2025)

 Padua, Italy
llm4eval.github.io/
Joined February 2024
  • Tweets44
  • Following62
  • Followers82
9Media
LLMs are reshaping how we evaluate ranked lists, conversations, and even simulate users. At #LLM4Eval @ #SIGIR2025, we’re spotlighting 3 special themes: 🧠 Beyond query-doc 🗣️ Simulating users 🧪 Synthesizing evaluation data 👇 Curious? Read on & submit by April 23! #LLMs #IR
LLM4Eval Workshop@llm4eval17 Mar 2025
📢 Call for Papers: LLM4EVAL @ SIGIR '25 - The 3rd Workshop on Large Language Models for Evaluation in IR! @SIGIRConf 🗓️ Submission: April 23, 2025 (AoE) 🔗 Full CFP: llm4eval.github.io/SIGIR2025… 🌐 Workshop: llm4eval.github.io/SIGIR2025…

Call For Paper

Call For Paper

llm4eval.github.io
1
9
2
14
2,601
Chirag Shah is this morning giving a keynote on: From Relevance to Reality: Scaling Human-Centered Evaluation in the LLM Era
2
5
293
You are welcome to join us at Giotto Platea for the LLM4Eval workshop #SIGIR2025 .
1
3
166
The deadline for submitting your LLM evaluation paper is tomorrow. Looking forward to interesting contributions!
LLM4Eval Workshop@llm4eval23 Apr 2025
📢 Deadline extended! Submit to #LLM4Eval @ #SIGIR2025 by April 30 (AoE) We welcome work on LLM-based evaluation, including: 🧠 beyond query-doc 🗣️ simulating users 🧪 synthetic eval data 🔗 llm4eval.github.io/SIGIR2025… #CFP #IR #LLMs #SIGIR2025

Call For Paper

Call For Paper

llm4eval.github.io
3
140
📢 Deadline extended! Submit to #LLM4Eval @ #SIGIR2025 by April 30 (AoE) We welcome work on LLM-based evaluation, including: 🧠 beyond query-doc 🗣️ simulating users 🧪 synthetic eval data 🔗 llm4eval.github.io/SIGIR2025… #CFP #IR #LLMs #SIGIR2025

Call For Paper

Call For Paper

llm4eval.github.io
LLM4Eval Workshop@llm4eval9 Apr 2025
LLMs are reshaping how we evaluate ranked lists, conversations, and even simulate users. At #LLM4Eval @ #SIGIR2025, we’re spotlighting 3 special themes: 🧠 Beyond query-doc 🗣️ Simulating users 🧪 Synthesizing evaluation data 👇 Curious? Read on & submit by April 23! #LLMs #IR
1
7
1
9
952
cc @cnsiro @srahmanidashti @maliannejadi @nickcr @claclarke @guglielm0f @UnderdogGeek @pt_ir @emine_yilm
1
1
249
We will host multiple breakout discussions to pick your brains 🧠 on the future challenges and research on this topic. Let's shape the LLMEval research agenda. Interested? Already have thoughts and ideas? Then, submit your extended abstracts or just join us in the workshop.
LLM4Eval Workshop@llm4eval17 Mar 2025
📢 Call for Papers: LLM4EVAL @ SIGIR '25 - The 3rd Workshop on Large Language Models for Evaluation in IR! @SIGIRConf 🗓️ Submission: April 23, 2025 (AoE) 🔗 Full CFP: llm4eval.github.io/SIGIR2025… 🌐 Workshop: llm4eval.github.io/SIGIR2025…

Call For Paper

Call For Paper

llm4eval.github.io
1
4
15
893
🧵 Theme 2: 🗣️ Simulating users: What if LLMs could act as realistic user proxies? We are looking for papers on simulating user behavior, preferences, context, or personas for evaluation.
1
2
3
464
🧵 Theme 3: 🧪 Synthesizing evaluation data: When judgments are scarce, can we generate evaluation data with LLMs? We want approaches for creating evaluation datasets in low-resource or privacy-sensitive scenarios, where collecting real data is hard, expensive, or impossible.
2
3
509
📢 Call for Papers: LLM4EVAL @ SIGIR '25 - The 3rd Workshop on Large Language Models for Evaluation in IR! @SIGIRConf 🗓️ Submission: April 23, 2025 (AoE) 🔗 Full CFP: llm4eval.github.io/SIGIR2025… 🌐 Workshop: llm4eval.github.io/SIGIR2025…

Call For Paper

Call For Paper

llm4eval.github.io
1
7
3
11
3,558
In this series, we move BEYOND search relevance to tackle: ✨ Personalized search evaluation ✨ Conversational interfaces ✨ Holistic assessment of integrated experiences ✨ Bias & fairness in LLM evaluations
1
3
2
314
Organized by @cnsiro, @srahmanidashti , @maliannejadi, @nickcr , @claclarke, @guglielm0f , @UnderdogGeek , @pt_ir , and @emine_yilm #SIGIR, #IR, #Evaluation, #LLM4Eval
2
6
473
In today's 2nd Workshop on #LLMs for Evaluation in #InformationRetrieval (#LLM4Eval) at #WSDM2025 (09:15 CET), @edgarmeij, our Head of #AI Platforms Engineering, will deliver a keynote on "Synthetic Evaluations & #GenAI Application Development" bloom.bg/4hyocEi #AI
3
11
820
The workshop will focus on models, techniques, data collections, and methodologies for information retrieval evaluation in the era of LLMs. #WSDM2025 #IR #Evaluation #Performance #Workshop #NLP #SIGIR
2
3
429
Organiser: @srahmanidashti, @cnsiro, @maliannejadi, @nickcr, @claclarke, @guglielm0f, @UnderdogGeek, @pt_ir , and @emine_yilm
1
1
2
331
The call for papers is now open for the second workshop of #LLM4Eval: 💡 See CFP: llm4eval.github.io/WSDM2025/… 🗓 Submission deadline: January 20, 2025
1
3
3
190
🔊 The second workshop on Large Language Models for Evaluation in IR (#LLM4Eval), co-located with #WSDM2025, is now accepting submissions! If your work explores or leverages LLM capabilities for evaluation, we encourage you to submit and share your research.
1
6
25
1,644
📢 Thrilled to announce that, "SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval" is publicly available. w/ @wangxieric, @emine_yilm, @nick_craswell, @UnderdogGeek, @pt_ir Check it out: arxiv.org/abs/2408.16312

SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval

Large-scale test collections play a crucial role in Information Retrieval (IR) research. However, according to the Cranfield paradigm and the research into publicly available datasets, the...

arxiv.org
2
14
49
4,388
LLM4Eval Workshop retweeted
Guglielmo Faggioli@guglielm0f18 Jul 2024
Great panel discussion with @ian_soboroff, @lauradietz99, @claclarke, Michael Ekstrand moderated by @UnderdogGeek at @llm4eval. #SIGIR24
6
31
871
Load more