Letian Peng (@LetianPeng) header
Letian Peng (@LetianPeng) profile picture

ボクは旅に出よう!

San Diego, CA
Joined October 2022
Photo by Letian Peng (@LetianPeng)Photo by Letian Peng (@LetianPeng)Photo by Letian Peng (@LetianPeng)Photo by Letian Peng (@LetianPeng)
Letian Peng retweeted
Can LLMs discover scientific laws? Not even GPT-6 can uncover half of our hidden laws! • A good fit is not always a valid law. • Models can recall known laws but struggle to discover new ones. • Models can generate a good law but don't recognize it.
3
6
4
30
31,807
Letian Peng retweeted
🍫 CocoaBench v1.0 is out! CocoaBench is a benchmark for unified digital agents, built around open-world tasks that require composing 💻 coding, 👀 vision, 🌐 search. Since our first research preview last December, we have expanded the benchmark substantially with community contributed tasks, and spent months testing and refining the tasks, evaluations, and agent runs. Some takeaways: • Even the best agent system reaches only 45.1% on CocoaBench v1.0. • Coding agents like Codex are already surprisingly strong on general tasks beyond software engineering. • Stronger agents tend to push more of the work into code. • Open source models still lag behind leading frontier models on these general tasks. 👇More on the website and in the paper #AI #Agents #LLM #Benchmark #CocoaBench
🍫 CocoaBench is calling for contributions from the community! Join us and help shape how next-generation agents are evaluated and built🚀✨ #LLM #AI #Agent #CocoaBench More details in the threads 👇
2
34
3
80
12,438
Letian Peng retweeted
🍫 CocoaBench is calling for contributions from the community! Join us and help shape how next-generation agents are evaluated and built🚀✨ #LLM #AI #Agent #CocoaBench More details in the threads 👇
What makes us humans general agents? 🤔 It’s probably not that we’ve mastered every app or UI. New tools appear constantly, yet we become productive quickly. What really transfers are our cognitive abilities: how we perceive, reason, and use memory. 🧠 Introducing CocoaBench, an evaluation framework for general agents with compositional cognitive abilities. 👉 Features 1⃣Complex, realistic tasks🧩 • Human-crafted tasks that are long-horizon, understandable, and span diverse scenarios and domains • Assume only a small set of general tools (browser, terminal, file system; no per-task APIs) • Challenging for existing agent systems. ChatGPT Agent reaches only 44% success rate. • Check out our example tasks on the website (link in reply) and see if you can solve them. They’re fun 🙂 2⃣Covering diverse cognitive abilities 🧠 CocoaBench covers different choices in the following dimensions of cognitive architecture: • Perception: how the agent gathers and preprocesses information from websites, terminal outputs, files, and images • Reasoning : planning, deductive/inductive/abductive inference, and both symbolic & visual reasoning skills • Memory : managing working memory in the long horizon, saving and loading procedural memory (skills), etc. 3⃣CocoaAgent framework🛠️ • Seamless integration with the AIO Sandbox for isolated browser / terminal / file operations • Easy to plug in any models or your own custom agent • Flexible evaluation functions This is just our first release, and we’re actively expanding CocoaBench with more tasks, analyses, and agents. Excited to see what the community builds on top of it! Project website + more details in the thread👇
3
14
1
32
15,866
Letian Peng retweeted
🔭 Towards Extending Open dLLMs to 131k Tokens dLLMs behave differently from AutoRegressive models—they lack attention sinks, making long-context extension tricky. A few simple tweaks go a long way!! ✍️blog albertge.notion.site/longdll… 💻code github.com/lbertge/longdllm
5
48
6
201
21,330
🚀 New Preprint Alert: LLMs can be radically efficient when their pre-training objectives are tailored for specific paradigms! 🌱💡 Introducing Cuckoo 🐦: a 0.3B LM specialized for Information Extraction (IE), pre-trained solely on extractive patterns of language modeling. 🔓 Cuckoo unlocks the scaling potential of IE models by shifting from vocabulary prediction to token tagging - enabling lightweight, task-specific performance that scales effortlessly with trillions of LLM's pre-training data. 🔍 Key Insights: 🪶 Tiny Yet Mighty: 96% fewer parameters and 20X faster than standard 8B LLMs. 🎯 Paradigm-Tuned: Tags tokens instead of predicting them, aligning perfectly with IE tasks. 🚀 Free-Riding Evolution: Cuckoo leverages LLM advancements without extra cost. 🔥 Performance: - Outperforms an 8B LLaMA on few-shot IE tasks (entities, relations, machine reading comprehension (MRC), & instruction-following). - Beats IE pre-training on human-labeled web links, GPT-4-generated data, & MRC datasets. 📄 Paper: arxiv.org/pdf/2502.11275 🛠️ Code: github.com/KomeijiForce/Cuck… Time to let your Cuckoo fly! 🐦💥 #LLM #InformationExtraction #NLP
5
1
11
1,617
The paper addresses the challenge of understanding how LLMs generalize knowledge, especially in compositional tasks. LLMs struggle with basic knowledge composition, like reverse or transition inference. This paper proposes that linear correlations exist between related knowledge within LLMs. It suggests that a linear transformation can map next token prediction logits from one knowledge prompt to another, mirroring human knowledge composition. ----- 📌 LLMs implicitly use linear transformations for knowledge composition. This linearity, observed in logit space, explains both generalization and hallucination as a function of transformation precision. 📌 The resilient linear correlation, even after fine-tuning, suggests a fundamental architectural bias in LLMs. This bias, rooted in vocabulary representations, predetermines how knowledge is composed and generalized. 📌 The linear transformation matrix W acts as a knowledge composition operator. Its weights directly reflect real-world correlations and its precision governs the accuracy of knowledge transfer across related prompts. ---------- Methods Explored in this Paper 🔧: → The authors fit a linear transformation (W, b) between the next token prediction logits of related prompts. For example, they examined the relationship between "X lives in the city of" and "X lives in the country of". → They sampled numerous output logits from prompts with various inputs to fit this transformation. → Pearson correlation coefficients were used to evaluate the linear relationship between different knowledge types. → The precision of the linear transformation W was analyzed by checking if its weights align with real-world knowledge pairs, using Hit@Top-N metrics. ----- Key Insights 💡: → A linear transformation can approximate the relationship between next token prediction logits for related knowledge. → This linear correlation is resilient to large-scale fine-tuning of LLMs. → The linear transformation's weights often mirror real-world knowledge relationships. High weights are assigned to correct knowledge pairs like (Paris, France) in the City→Country example. → When the linear transformation is precise, it can enable compositional generalization. → However, an imprecise linear transformation can lead to compositional hallucination, where learning one knowledge incorrectly generalizes to related knowledge. → Vocabulary representations are crucial for forming these linear correlations and enabling knowledge composition. ----- Results 📊: → High Pearson correlation coefficients were observed for related knowledge pairs like City→Country (0.89) and Math operations (0.93 for X+1→X+2). → Lower correlation was found for less related pairs like CEO→Company (0.55) and cross-language pairs. → In City→Country knowledge composition, W precision reached 42% for top-1 influenced cities and 67% for top-1 influencing countries using Hit@Top-1 metric. → Generalization success was significant (53.70% for City→Country) only when both correlation intensity and W precision were high.
1
3
27
3,636
✨My New ICLR2025 pub: Some knowledge is memorized by LLMs🤖, but cannot be sampled by even prompting 1,000,000 times😵‍💫 - we provide a simple way to find it. Here's why and how. Why a prompt cannot hold all potential answers? Because answer (next token) representations are grouped in clusters. One example is a character cluster (A, B, C, ...). Whenever the LLM wants to predict one of those characters (e.g., What is a white and black animal?➡️P➡️Panda🐼), it cannot avoid predicting other characters (e.g., What is a white and black animal?➡️Q, V, X, ...➡️❌) with high probability (rank). On the other hand, some correct answers in other clusters are lost because a cluster dominates the probability. We find that LLMs memorize out-of-cluster knowledge even if it's almost impossible to sample it from the initial prompt. We get such knowledge by prompting "What is a white and black animal other than Panda?" This eliminates the probability of "P" (and the character cluster) and navigates the LLM to search in another cluster for the answer. The navigation successfully finds the correct knowledge that is missed by the initial prompt. This phenomenon also reminds us of recent efforts to sample reasoning paths for reinforcement learning. Sometimes, we fail to sample any correct thinking path from the initial prompt. Maybe we need a smarter navigation strategy to explore all potential thoughts hidden by the spurious correlation between vocabularies. ➡️Dive deeper into our work, which includes the application of navigation to math reasoning and data synthesis: openreview.net/forum?id=Vipc… ➡️View the spurious correlation: huggingface.co/datasets/Kome… ➡️Code: github.com/KomeijiForce/KeyN…
1
5
8
977
New preprint: Do you know LLMs 🤖 learn knowledge composition in their parameters? We attribute a certain kind of compositional generalization to the linear correlation between answers from LLMs for certain prompts. That is to say, if LLM believes X lives in the city of Paris🗼, then LLM will also believe X lives in the country of France 🇫🇷. The correlation is also resilient to gradients, thus updating X’s residential city to NYC 🗽 will also update X’s residential country to USA 🇺🇸. However, some correlations learned by LLM are imprecise. For example, Indianapolis 🏙️ is correlated with India 🇮🇳 rather than the USA 🇺🇸 (Ground-truth). We find that learning that X lives in Indianapolis 🏙️ will cause hallucinations as if X lived in India 🇮🇳. ➡️ Dive deeper into the connections of correlation, generalization, and hallucination: arxiv.org/abs/2502.04520 ➡️ Code: github.com/KomeijiForce/LinC…
6
16
1,213
We're thrilled to announce that 4 groundbreaking papers by our brilliant students have been accepted to #ICLR2025! 🎉 Focused on core representations in #LLM, our research covers both their limitations and applications. Join us in Singapore 🇸🇬 for these exciting insights! 🚀
3
1
21
1,303
Thrilled to share that our brilliant team at #SDLab has published 6 papers (4 Main, 2 Findings) at #EMNLP2024! From incubating small models by LLMs to addressing risks in LLM benchmarking, our research pushes the boundaries of #LLM and #NLP. Huge congrats to the team! 🚀💡🌴☀🐊
5
1
14
1,299
🎇With great pleasure, we announce that 🛌InBedder has been accepted by the main conference of #ACL2024!🎇 🛌InBedder is a great choice to customize the semantics in your text embeddings! Code: github.com/zhang-yu-wei/InBe… Paper: arxiv.org/abs/2402.09642
As a boss, you want to cluster topics that people care about but your embedder does it on opinions? To analyze your texts smartly, check our new instruction-following embedder: InBedder! ⭐Star our repo to support us: github.com/zhang-yu-wei/InBe… 📄Paper: arxiv.org/abs/2402.09642
2
7
953
As a boss, you want to cluster topics that people care about but your embedder does it on opinions? To analyze your texts smartly, check our new instruction-following embedder: InBedder! ⭐Star our repo to support us: github.com/zhang-yu-wei/InBe… 📄Paper: arxiv.org/abs/2402.09642
1
5
2
9
2,642
Great thanks to the talent authors that contribute their intelligence to InBedder! @shangjingbo @YuweiZh49446108 @zlwang_cs @zihanwangki
175