🛡️ Forget Forgetting! We solve the problem of catastrophic forgetting. When an LLM loses its original spoken language skills after learning a new task. Our Multitasking Fine-Tuning strategy (mixing DGS & spoken language data, OpenOrca) successfully mitigates this data shift.🧵5/n

🚀 Our Core Contribution: We introduced new fine-tuning strategies to build the first text-based and multimodal LLMs capable of SLP. This includes: ✅ Video-Based Input capabilities via fine-tuned LLaVA (V2T task). ✅ Text-Based Models fine-tuned on LLaMA3. 🧵4/n

Unlike other LLMs, our models are trained and tested on 6 additional SLP tasks: 🌿(G2T) DGS to German 🌿(T2G) German to DGS 🌿(V2T) DGS Videos to German 🌿(I-G2T) Intensified DGS to German 🌿(T2I-G) German to Intensified DGS 🌿(G2E) DGS to English 🧵 3/n

🏃‍♀️💨While waiting for SignGemma to become available... You can check out our first text-based and multimodal LLMs capable of Sign Language Processing (SLP) called SignAlignLM! #SignLanguage #LLM #AIAccessibility @aclmeeting 📜Paper: aclanthology.org/2025.findin… 🧵1/n

We present a taxonomy of different friction movements, and show that positive friction, 1⃣Extends Dialogue Acts 2⃣Helps Model User Mental States 3⃣Helps Accomplish User Goals

We hypothesize that systems can improve goal alignment, modeling of user mental states, and task success by deliberately slowing down conversations in strategic moments to ask questions, reveal assumptions, or pause.

‼️ Ever wish LLMs would just... slow down for a second? In our latest work, "Better Slow than Sorry: Introducing Positive Friction for Reliable Dialogue Systems", we delve into how strategic delays can enhance dialogue systems. Paper Website: merterm.github.io/positive-f…

One last thing: we also observe that LLMs can represent our eye gaze depending on our reasoning!! 😱 🧵 8/9

We also introduce a new visualization tool ⚒️ so that researchers can better see how much our eyes focus on an object in the image. 🧵7/9

We introduce a new gaze pattern ranking algorithm 💻 to better extract eye gaze movements according to the salient entities in the image and the discourse goals. 🧵6/9