Maya Varma (@mayavarma23) profile picture
Stanford, CA
Joined October 2020
Photo by Maya Varma (@mayavarma23)Photo by Maya Varma (@mayavarma23)
(1/4) Excited to share RaVL, which is appearing this week at #NeurIPS2024! RaVL discovers and mitigates spurious correlations in fine-tuned vision-language models (VLMs). 📄 Paper: arxiv.org/abs/2411.04097 💻 GitHub: github.com/Stanford-AIMI/RaV…
1
5
1
18
4,037
Maya Varma retweeted
SMMILE An Expert-Driven Benchmark for Multimodal Medical In-Context Learning
1
10
3
40
21,942
🚨New preprint! 🚨In-context learning (ICL) is the intriguing ability of LLMs to learn to solve tasks purely from context w/o parameter updates. For multimodal LLMs (MLLMs), ICL is poorly understood, especially in the medical domain where doctors would often face few relevant prior cases, prior imaging studies etc. But can MLLMs actually learn from context in the medical domain? 🤔 To answer this question, we introduce SMMILE— an expert-curated benchmark for multimodal ICL in the medical domain! TL;DR: 💡MLLMs perform surprisingly poorly at multimodal ICL in the medical domain! --> ICL performance sometimes actually drops—some models perform worse than random baselines. Models have still a way to go here! Previously, many ICL evaluations leveraged random, potentially irrelevant examples -> with the help of clinical experts, here we carefully craft explicit task demonstrations to more robustly probe the ICL ability. 📊 The benchmark: -> 111 expert-curated problems (517 ICL problems) -> 6 medical specialties, 13 imaging modalities -> SMMILE++ variant with 1,038 permuted problems -> Comprehensive evaluation of 15 state-of-the-art MLLMs Check it out: 🔗 Page: smmile-benchmark.github.io/ 📜 Paper: arxiv.org/abs/2506.21355 💻 Code: github.com/eth-medical-ai-la… 📁 Dataset: huggingface.co/smmile Great collab with stellar colleagues @mrieff_02* @mayavarma23* (co-first) @IAMJBDEL# (co-last) and more!
2
20
84
8,083
Maya Varma retweeted
💥 We unveil our paper accepted at the #ACL2025 Main Conference: Automated Structured Report Generation Let's revisit automated radiology report generation for CXR. Free-form reports make it hard for AI systems to learn accurate generation, and even harder to evaluate. 🧵👇 @StanfordAIMI @hopprai
1
6
1
4
1,043
🧵 What if AI could learn from millions of unlabeled radiology images and reports—and then flexibly adapt to new clinical tasks? In a new comprehensive review in @radiology_rsna, we dive into how foundation models (FMs) are set to revolutionize radiology! @AIMI_Stanford (1/6) 👇
3
9
1
62
6,429
1/ Updates on our improved open-source CheXagent with new transparent benchmarks! We ran a new reader study mimicking real workflows: Radiology residents drafted reports that attendings reviewed/edited. Results from 8 rads show major efficiency gains. Key findings: 👇
2
7
2
26
4,076
(3/4) We evaluate RaVL by introducing a large-scale evaluation framework with 654 fine-tuned VLMs annotated with ground-truth spurious correlations. We also show that RaVL can surface novel spurious correlations in off-the-shelf VLMs, both in the general and medical domains!
1
203
Maya Varma retweeted
So proud of the release of the GREEN metric. See what you can do when you merge medical AI research and open-sourceness. 26,000 downloads on 🤗 Hugging Face and counting. 🟥 EMNLP proceedings: aclanthology.org/2024.findin… 🤗 Dataset: huggingface.co/datasets/Stan… 🤗 Models: huggingface.co/collections/S… 🤗 Space: huggingface.co/spaces/Stanfo… 🐍 GitHub: github.com/Stanford-AIMI/GRE… 📄 Project Page: stanford-aimi.github.io/gree… 🧵 👇
3
10
29
4,727
🧙 Excited to introduce Merlin, a vision language foundation model for 3D computed tomography 🐈‍⬛🩻 Trained to understand 3D abdominal CT scans using supervision from: 💾 Structured electronic health records (1.8+ million codes) 🗒️ Natural language radiology reports (6+ million tokens) Paper: arxiv.org/abs/2406.06512 🧵 1/10
3
36
13
112
38,880
Five years ago, thanks to the leadership of @mattlungrenMD, @stanfordAIMI released the CheXpert images: 223K JPG CXRs with labels for 14 conditions. CheXpert has been cited >6000 times, mostly related to development of supervised learning methods. Much has changed since then.🧵
3
33
4
127
23,989
Maya Varma retweeted
🧵[1/3]🚀Exciting news from AIMI! We're leading the charge in #ClinicalTextGeneration with two pivotal tasks at #BioNLP2024 & #ACL2024: 🩺 Radiology Report Generation, enhancing clinical communication via AI-generated X-ray reports, & 📄 Discharge Me!, streamlining EHR notes to combat clinician burnout. Details at stanford-aimi.github.io/RRG2… & stanford-aimi.github.io/disc…
1
4
2
14
3,351
Ran out of OpenAI credits?💰 We present a prompting strategy that enables open-source and off-the-shelf GPT-J-6B to outperform *few shot* GPT3-175B on 15 popular language benchmarks! 🚀 Paper and code: 📜 arxiv.org/abs/2210.02441 💻​ github.com/HazyResearch/ama_…
8
105
12
636
Maya Varma retweeted
New preprint alert! 📣 How do we produce transferable and robust representations with supervised contrastive learning? We need *geometric spread* and an inductive bias towards *latent subclass clustering* in representation space. 📜 arxiv.org/abs/2204.07596 👇 (1/n)
2
54
1
239
New preprint alert! 📣 How do we improve long-tailed performance of entity retrieval? We use a supervised contrastive loss to *geometrically encode entity types* in representation space w/ bi-encoders. Check out our paper on TABi! 📜 arxiv.org/abs/2204.08173 Details👇 (1/n)
2
25
3
94
Excited to share our paper “Cross-Domain Data Integration for Named Entity Disambiguation in Biomedical Text,” which appeared this week at #EMNLP2021! Paper: arxiv.org/abs/2110.08228 Code: github.com/HazyResearch/medi… Data: huggingface.co/datasets/mvar…
2
12
1
27
(5/6) We utilize our integration scheme to augment structural resources and generate a large pretraining biomedical NED dataset (available at huggingface.co/datasets/mvar…).
2
(6/6) A model pretrained on this dataset and injected with structural knowledge improves disambiguation of rare entities by between 2.5 and 57 accuracy points across two benchmark datasets!