
Assistant Professor at the University of Macau @UM_1981 | Prev: @AlibabaGroup @uiuc_nlp @CUHKofficial
Macau
Joined May 2017
- Tweets338
- Following607
- Followers374
- Likes1.9K
Hou Pong (Ken) Chan retweeted
I am excited to be joining CS@U Wisconsin-Madison in Fall 2027! I will recruit PhD students. If you are also enthusiastic about researching how LLMs work and/or applying them to other scientific problems, please apply and mention me in your application. @WisconsinCS
🔍 Why can a multimodal LLM recognize every object in an image yet still miscount them?
🚀 In our new paper, “Diagnosing the Reference Gap,” we study a key bottleneck: keeping language grounded in the right visual entities throughout multi-step reasoning.
• ReGap-Bench: Five visual reasoning tasks with the scene, question, and answer held fixed, varying only the visual references.
• Task alignment matters: Task-appropriate references improve accuracy by up to 25%, while mismatched references offer little benefit—or even hurt performance.
• A training-free visual skill: Models can construct task-appropriate visual references, such as numbering, grid lines, or shading, to support reasoning without additional training.
👍 Please check out and upvote our paper on Hugging Face: huggingface.co/papers/2610.3…
🌐 Project: regap-bench.github.io/
💻 Code and data: huggingface.co/datasets/rega…
#MultimodalAI #VisualReasoning #VisionLanguageModels #LLM #MLLM #ReGapBench
Hou Pong (Ken) Chan retweeted
🚀 We’ve released Capek0.5-35B-A3B, an execution-centric VLM for embodied intelligence!
An embodied agent’s “brain” needs to understand the scene, track what changed, guide the next action, and verify whether it actually succeeded. Capek 0.5 brings these demands into one model through four capabilities:
📐 Spatial reasoning
🎬 Temporal understanding
🎯 Action guidance
✅ State verification
Our recipe: train specialists with verifiable-reward RL, then consolidate them through weight merging and MOPD. Four capabilities, one inference-time model.
📊 Compared with its Qwen3.6-35B-A3B backbone under matched evaluation:
• Improvements on 28 of 34 benchmark metrics
• EB-HAB task success: 46% → 63% (+17 pp)
• VSI-Bench: 60.83 → 70.69
• StateBench-P: 65.80 → 76.80
🤗 Model: huggingface.co/xpeng-robotic…
📄 Paper: arxiv.org/abs/2608.06756
🌐 Project: xpeng-robotics.github.io/cap…
Hou Pong (Ken) Chan retweeted
Scaling test-time compute for competitive programming usually means over-generate and rank: sample thousands of programs, cluster, submit the best.
What if a verifier could tell the solver exactly why a program fails?
Meet ARBITER 🧵
Hou Pong (Ken) Chan retweeted
Sharing the Apodex Drug Research Computing Center! Democratizing drug & disease knowledge and AI.
Clinical Trial Forecasting lab now fully open. Drug repurposing labs coming soon.
drug.apodexlab.ai/
Worth to try
🚀 We’re open-sourcing LegoFlow — a framework for building and improving long-horizon coding-agent data with Claude Code / Codex.
Long-horizon task construction is still painfully expensive: too many steps, too much manual work, and painfully slow iteration.
LegoFlow turns the whole loop into a programmable workflow:
⚙️ One-click skills for task construction, rollouts generation, training & evaluation
🔁 End-to-end iteration + self-improvement
📊 Visual Dashboard for the full pipeline
📦 5,000 curated tasks + 2,780 successful trajectories
The goal: make high-quality coding-agent data much easier to build, iterate on, and scale.
🔗 Code: github.com/LegoX/LegoFlow
📝 Blog: legox.net/blog/legoflow/
🤗 HF: huggingface.co/collections/L…
And we’re just getting started — more types of long-horizon agent data are coming soon. 🔥
Hou Pong (Ken) Chan retweeted
🚀 We’re open-sourcing LegoFlow — a framework for building and improving long-horizon coding-agent data with Claude Code / Codex.
Long-horizon task construction is still painfully expensive: too many steps, too much manual work, and painfully slow iteration.
LegoFlow turns the whole loop into a programmable workflow:
⚙️ One-click skills for task construction, rollouts generation, training & evaluation
🔁 End-to-end iteration + self-improvement
📊 Visual Dashboard for the full pipeline
📦 5,000 curated tasks + 2,780 successful trajectories
The goal: make high-quality coding-agent data much easier to build, iterate on, and scale.
🔗 Code: github.com/LegoX/LegoFlow
📝 Blog: legox.net/blog/legoflow/
🤗 HF: huggingface.co/collections/L…
And we’re just getting started — more types of long-horizon agent data are coming soon. 🔥
Hou Pong (Ken) Chan retweeted
I have a funding opportunity for a 4-year (typical duration in Singapore) PhD position on meta-learning for training data selection. The research aims to develop simple, practical, and scalable methods guided by mathematical insights and verified by realistic use cases. To get a taste of the research direction, take a look at our recent oral paper at ICML 2026: boyangli.org/paper/Zilin-Du-…
Ideal candidates have strong neural network engineering skills and a solid mathematical background (e.g., differential equations, probability, optimization). A publication involving some mathematical content and a master’s degree would be advantages, though not strictly required.
Please send your resume and transcripts to my email (which I am sure you can find). This ad expires on Nov 30, 2026.
Hou Pong (Ken) Chan retweeted
What if you give an Agent a task…
…but it also has a different goal of its own?🫥
Now it has a choice: do what you asked, or quietly help its own goal too.
So what does it actually do? 👀
📄SchemeArena: Factorized Stress Testing of Scheming in LLM Agents arxiv.org/pdf/2609.08126v1
Hou Pong (Ken) Chan retweeted
AI models can now do frontier science research - e.g. @OpenAI solved the NS Millennial Prize problem.
We raise a much simpler question: can models reliably solve a kids’ spot-the-difference game?
Not quite!
Introducing VDiff-Bench: a challenging visual difference identification benchmark with 1756 questions across 10 semantic and low-level visual difference types. Interesting to see how frontier models demonstrate different failure modes!
🔗Arxiv link: arxiv.org/abs/2609.06245
💡Project Page: huggingface.co/spaces/elaine…
🔑HF Dataset: huggingface.co/datasets/elai…
📄 DeepSeek-OCR proved that optical context compression is the future of scaling LLM context windows. But what are the underlying design rules that allow pixel-text encoders to scale?
In our new #EMNLP2026 paper, we present Pixel Linguist II:
🧠 We break down the 4 core ingredients for robust pixel-text representation learning:
🔹 Spatial Proxies: Variable resolutions & font sizes enable seamless zero-shot scaling from 224x224 snippets to 4K docs.
🔹 Multimodal Grounding: Natural image-text pairs prevent text-only collapse and ground semantics.
🔹 Layout Diversity: On-the-fly rendering prevents pixel-level shortcut learning.
🔹 Multilingual Curriculum: A 2-stage pipeline unlocks massive cross-lingual visual alignment.
🚀 Key Highlights:
• New SOTA on Visual STS & ViDoRe document retrieval.
• Boosts downstream MLLM performance (+2.75% over Qwen2.5-ViT).
• Extreme Optical Compression: Retains robust representations under 80% visual token compression—at just 40% tokens, it beats full-budget Qwen2.5-ViT!
🔗 GitHub: github.com/Pixel-Linguist/Pi…
💻 Model: huggingface.co/collections/P…
👍 Please check out and upvote our paper on Hugging Face: huggingface.co/papers/2609.0…
#VisionLanguage #VisualTextEncoder #MLLM #DeepSeekOCR
Hou Pong (Ken) Chan retweeted
🍌Introducing 𝗣𝗮𝗽𝗲𝗿𝗕𝗮𝗻𝗮𝗻𝗮-𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁✨! (UCLA x Google Cloud AI)
AI can draft your paper's diagrams, but they rarely match what you pictured: wrong structure, missing components, misplaced arrows, or it just misses the point.
PaperBanana-Interact makes diagram generation 𝗰𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻𝗮𝗹: generate, then refine via chat. Don't miss it👇
📄Paper: arxiv.org/abs/2608.30241
💻Code, data & demo coming soon - stay tuned!
(And yes, this figure was made by PaperBanana-Interact itself - we just chatted with it until it looked right!😄)
Delighted to share two exciting updates about Lingshu-Med:
🎉 The technical report for Lingshu-Med has been accepted for publication in IEEE TPAMI.
📂 We have open-sourced the training data used for Lingshu-Med to support further research in the community. Feel free to explore the dataset and give it a like on Hugging Face:
huggingface.co/datasets/ling…
The Lingshu model series has now received more than 500k downloads on Hugging Face. We are deeply grateful for the community’s interest, support, and feedback!
#MedicalAI #MultimodalAI #FoundationModels #MLLM #AIinHealthcare #OpenSource
🚀We are thrilled to launch 'Lingshu' – A Generalist Medical Multi-modal Foundation Model! 🩻
🌟 Highlights of Lingshu:
⚕️ Unified knowledge across 12+ imaging modalities (X-Ray, CT, MRI & more!).
🧠 Enhanced reasoning & reduced hallucinations via novel data curation and training pipelines.
🔥 Outperforms GPT-4.1 and Claude Sonnet 4 on most medical QA & report generation tasks!
👍 Please check out and upvote our paper on Hugging Face!
🔗 huggingface.co/papers/2506.0…
#MedicalAI #MultiModal #LLM #MLLM #AIinHealthcare
Hou Pong (Ken) Chan retweeted
Human programmers post lessons on Stack Overflow, GitHub Gists, and blogs all the time - coding agents should do the same; knowledge discovered and then burried is a loss to the entire community.
Wrote a skill to encourage coding agents to publish lessons to GitHub Gist: github.com/shirley-wu/skills… and will be using it across all my personal projects. People can also try it via: gh skill install shirley-wu/skills sharing-lessons --scope user --agent claude-code (or replace with your coding agent)
Happy for people to use it, share comments, and help it evolve.
Hou Pong (Ken) Chan retweeted
Thrilled to announce Lego-RL🚀, a harness-native reinforcement learning framework for coding agents like Claude Code, OpenCode and OpenHands-SDK. It successfully boosts Qwen-3.5-35B-A3B from 64% to 70% on SWE-Bench verified.
Everything is open-sourced: legox.net/blog/lego-rl/
Hou Pong (Ken) Chan retweeted
Excited to share that I’ll join Westlake University as a tenure-track AP in AI from summer 2027! Welcome to join the lab as a "co-founder"! I am looking for: 2–3 PhD students (Fall 2027), RAs/interns. Check out the recruiting page for more details mikewangwzhl.github.io/lab/ :)
Hou Pong (Ken) Chan retweeted
🏆 Best Resource Paper Award at #ACL2026 @aclmeeting
🚀Thrilled to share that our deep research benchmark paper "HSCodeComp: A Realistic and Expert-level Agent Benchmark for Hierarchical Rule Application" reveived a Best Resource Paper Award #ACL2026!
🎯 Hoping this sparks more focus on reliable AI Agents in the real world.
aclanthology.org/2026.acl-lo…
Hou Pong (Ken) Chan retweeted
🕵️ Copyright Detective, #ACL2026 today at 12:45 PM PT. (virtual presentation 3)
We introduce an interactive forensic system for evidence discovery of copyrighted content leakage, not just static classification.
Live demo👇, welcome to try it!
Hou Pong (Ken) Chan retweeted
Replying to @JiayuJeff
Great work from @JiayuLeff and the team, come check out our work on 𝙖𝙜𝙚𝙣𝙩 𝙘𝙤𝙨𝙩-𝙚𝙛𝙛𝙚𝙘𝙩𝙞𝙫𝙚𝙣𝙚𝙨𝙨 live at ACL'26! 🚀🚀
We also have follow-up work in the series worth mentioning as well, on "Adaptive Planning in LLM Agents under World & User Constraints". ✨✨
Hou Pong (Ken) Chan retweeted
Welcome!
📢 Join us for the ACL Mentorship Session
@aclmeeting #ACL2026NLP
Location: Grand Hall Foyer - Level 1
We’re excited to have an amazing group of mentors joining us:
@Lianhuiq
@ManlingLi_
@zhaofeng_wu
@SihangZhao
@shannonzshen
@ziqiao_ma
We’re also launching a new initiative to connect mentors and mentees across the AI/NLP community! If you’re interested in participating, sign up here:
🔗 tinyurl.com/aclmentee




































