
🌴Heading to #EMNLP2024! Presenting AmbigNLG with @ayaniwa1213 on Tuesday at 4pm (Riverfront Hall).
Paper: aclanthology.org/2024.emnlp-…
Data: github.com/megagonlabs/ambig…
You can also stop by our @MegagonLabs sponsor booth or DM me to chat about full-time and internship opportunities :)
Key Findings:
📊 Info amount affects LLM performance more than context length.
🏆 Llama-3.1-405B excels with short contexts, while GPT-4o dominates in longer ones.
📍 Llama-3.1-405B handles info position well; GPT-4o is more sensitive to it. 5/
1
58
To build HoloBench, we used the NL2SQL dataset: defining control factors🛠️, extracting subtables⚗️, verbalizing tables into contexts✏️, obtaining ‘gold answers’ via SQL queries⚙️, and evaluating LLMs’ holistic reasoning with natural language queries and generated context🦙. 4/
1
73
Introducing 🫧 HoloBench—a new benchmark for measuring LLMs' reasoning capabilities across massive document collections!
- RAG models fetch info well but struggle with multi-doc reasoning.
- HoloBench evaluates how LLMs synthesize and aggregate info to solve complex tasks.
🧵 1/
🔔New Preprint🔔
Introducing 🕊XATU🕊: 1st text editing benchmark with fine-grained instruction! 📝
No more coarse-grained instruction👋 – XATU offers fine-grained instructions to accurately measure the true editing capability of LLMs!🦙📈
📚: arxiv.org/abs/2309.11063
#NLProc
🥳 Our paper "Zero-Shot Triplet Extraction by Template Infilling" has been accepted at #AACL2023! We've reformulated the triplet extraction task into a template infilling task, making the unknown relation known 🔄
#NLProc
📚: arxiv.org/abs/2212.10708
👩💻: github.com/megagonlabs/zett
LLMs excel at summary evaluation but struggle with high costs and missing key info in long docs.
Introducing Extract-then-Evaluate! It reduces costs 💸⬇️, aligns with human evaluations 🤝🎯, and doesn't miss crucial info in the middle 📄🔍.
📚: arxiv.org/abs/2309.07382
#NLProc








