Hayate Iso (@iso_map) profile picture
Santa Clara, CA
Joined September 2023
🌴Heading to #EMNLP2024! Presenting AmbigNLG with @ayaniwa1213 on Tuesday at 4pm (Riverfront Hall). Paper: aclanthology.org/2024.emnlp-… Data: github.com/megagonlabs/ambig… You can also stop by our @MegagonLabs sponsor booth or DM me to chat about full-time and internship opportunities :)
4
1
28
7,878
Key Findings: 📊 Info amount affects LLM performance more than context length. 🏆 Llama-3.1-405B excels with short contexts, while GPT-4o dominates in longer ones. 📍 Llama-3.1-405B handles info position well; GPT-4o is more sensitive to it. 5/
1
58
To build HoloBench, we used the NL2SQL dataset: defining control factors🛠️, extracting subtables⚗️, verbalizing tables into contexts✏️, obtaining ‘gold answers’ via SQL queries⚙️, and evaluating LLMs’ holistic reasoning with natural language queries and generated context🦙. 4/
1
73
Introducing 🫧 HoloBench—a new benchmark for measuring LLMs' reasoning capabilities across massive document collections! - RAG models fetch info well but struggle with multi-doc reasoning. - HoloBench evaluates how LLMs synthesize and aggregate info to solve complex tasks. 🧵 1/
1
1
3
723
🔔New Preprint🔔 Introducing 🕊XATU🕊: 1st text editing benchmark with fine-grained instruction! 📝 No more coarse-grained instruction👋 – XATU offers fine-grained instructions to accurately measure the true editing capability of LLMs!🦙📈 📚: arxiv.org/abs/2309.11063 #NLProc
1
1
1
8
1,762
🥳 Our paper "Zero-Shot Triplet Extraction by Template Infilling" has been accepted at #AACL2023! We've reformulated the triplet extraction task into a template infilling task, making the unknown relation known 🔄 #NLProc 📚: arxiv.org/abs/2212.10708 👩‍💻: github.com/megagonlabs/zett
1
2
10
599
LLMs excel at summary evaluation but struggle with high costs and missing key info in long docs. Introducing Extract-then-Evaluate! It reduces costs 💸⬇️, aligns with human evaluations 🤝🎯, and doesn't miss crucial info in the middle 📄🔍. 📚: arxiv.org/abs/2309.07382 #NLProc
2
4
2
11
3,139