phd in cs @northwesternu agents to real world grounding × context × self-evolution
Joined March 2024
- Tweets156
- Following316
- Followers230
- Likes588
Pinned Tweet
🤖 Reusing skills is one possible way for embodied agents to generalize. Manipulation varies but shares a few skills: wiping a table or a window is one skill. Many works give agents skills to plan with. But where do these skills, and their data, come from?
📼 Today, VLMs mostly label videos one at a time, and nothing carries over. People don't learn that way. As experience streams in, we spot skills we know, add new ones, and reuse them later. This streaming setting matters, but it has received little attention.
🧠 VLMs may become the brains of embodied agents. So we ask: from streaming experience, can they find reusable skills and keep one consistent skill library?
🧵 Video2Skill: From Streaming Experience to Reusable Embodied Skills
Why it matters:
📈 It can label much more data for training agents.
🔁 It is planning in reverse. If a model can't find skills in what it has seen, how can it plan with them for something new?
📄 Paper: arxiv.org/abs/2609.36691
🌐 Project: andyzworks.github.io/video2s…
🤗 Data: huggingface.co/datasets/Ster…
What on earth is mixed into this training data for GPT6-Astra🤣
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Jianshu Zhang retweeted
Realtime-Bench ⏱️
Key findings:
- Best model (GPT-6-Astra): 36%.
- Fastest model (Jev, answers in under 1s): 8%.
- More reasoning helps only when the clock is paused.
- With a real clock, slow thinking costs the win.
- Even with time frozen, many tasks stay unsolved.
Play with it at realtimeagents.org!
Great work led by @JingyuTang621 @jiaxin_pei and team 👇
Jianshu Zhang retweeted
Spent years watching models ace a single image and completely whiff on anything that moves or changes angle. Today we’re fixing that.
Try Glimpse now, our first visual thinking model with competitive frontier performance compared to Astra on key public visual reasoning benchmarks.
Jianshu Zhang retweeted
🚀 Excited to share 3DCodeVerse: Exploring 3D Worlds as Code!
What if we could generate entire 3D worlds through executable code—fully editable, verifiable, and reusable?
We introduce 3DCodeVerse, a research ecosystem for scalable 3D generation with coding agents:
📦 2.03M code pairs, 269K+ executable 3D programs, 128K+ CAD objects, 119K+ procedural shaders, 7 languages/runtimes, and 4 generation tracks !!
🤖 3DCodeVerse Harness: A multi-agent framework that plans, generates, executes, verifies, and iteratively refines 3D programs through spatial measurements and visual feedback—all without human intervention during generation. As a proof of concept, we demonstrate the effectiveness of our data and harness:
📈 Fine-tuning Qwen3.8-27B with SFT + DPO improves execution rates from 51.9% → 92.9% on 3DCodeBench.
🏆 Our harness achieves 90% human preference over Claude Design using the same Claude Opus 5.5 backbone.
We hope 3DCodeVerse provides a foundation for scaling 3D generation through code!
📄 arXiv：arxiv.org/pdf/2610.22341
🌐 Project Page：3dcodeverse.com
💻 GitHub：github.com/gaoypeng/3dcodeve… 🤗 Dataset：huggingface.co/datasets/ilab…
#LLM #3D #Coding
How do robots know they're making progress? 🤖
We're actively updating Awesome-Progress-Models, curating papers on progress reward modeling for Embodied AI & robot learning.
⭐ Star to follow updates; paper suggestions & PRs welcome!
github.com/andyzworks/Awesom…
How is this any different from the NBA getting into sports betting? Strongly opposed.
Prediction market for ICLR: acceptodds.com/?kind=ICLR+20…
yes, this is what we get in 2026 : )
Jianshu Zhang retweeted
iPhone AGI has been achieved 🔥🔥
An agent running entirely on my iPhone, with no computer connected or in the loop at any point, able to control absolutely anything on my iPhone.
✨ iPhone computer use ✨
Everything after the first message is an agent tapping and typing.
Make multimodal great again?! Real-time UI is everything. How are they pulling that off so fast?
Jianshu Zhang retweeted
Surprisingly, embodied data is now scaling fast: teleop, UMI, human video.
Yet embodied models aren't scaling with it.
The bottleneck now is accurate labeling 🧑💻
Our new work turns long human & robot videos into fine-grained actions, grounded to objects, in an open yet consistent vocabulary. Raw footage → reusable skills.
🤖 Reusing skills is one possible way for embodied agents to generalize. Manipulation varies but shares a few skills: wiping a table or a window is one skill. Many works give agents skills to plan with. But where do these skills, and their data, come from?
📼 Today, VLMs mostly label videos one at a time, and nothing carries over. People don't learn that way. As experience streams in, we spot skills we know, add new ones, and reuse them later. This streaming setting matters, but it has received little attention.
🧠 VLMs may become the brains of embodied agents. So we ask: from streaming experience, can they find reusable skills and keep one consistent skill library?
🧵 Video2Skill: From Streaming Experience to Reusable Embodied Skills
Why it matters:
📈 It can label much more data for training agents.
🔁 It is planning in reverse. If a model can't find skills in what it has seen, how can it plan with them for something new?
📄 Paper: arxiv.org/abs/2609.36691
🌐 Project: andyzworks.github.io/video2s…
🤗 Data: huggingface.co/datasets/Ster…
Jianshu Zhang retweeted
Long context doesn’t solve long-horizon reasoning if the model doesn’t know what to look for.
The challenge is not simply remembering more history — it’s recovering the right history at the right time. That distinction seems increasingly important as embodied agents take on longer and longer tasks.
🚨 VLAs struggle on long, context-dependent tasks, so dense progress, a score at every step, matters for them.
🤔 Can progress reward models label dense progress for these context-dependent tasks?
🧭 Our work shows why they fail: not because they are blind, but because they get lost in the context. Given the right context, the same progress reward models are reliable again.
🧵 So we propose ProgressCompass, a training-free agentic system: off-the-shelf progress models + VLMs, guiding progress estimation in long tasks.
📄 ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context
🌐 andyzworks.github.io/progres…
Thanks for sharing! Yeah VLAs now have their “memory” components, we argue that embodied Progress Reward Models also need that for dense progress reward annotation✅
🚨 VLAs struggle on long, context-dependent tasks, so dense progress, a score at every step, matters for them.
🤔 Can progress reward models label dense progress for these context-dependent tasks?
🧭 Our work shows why they fail: not because they are blind, but because they get lost in the context. Given the right context, the same progress reward models are reliable again.
🧵 So we propose ProgressCompass, a training-free agentic system: off-the-shelf progress models + VLMs, guiding progress estimation in long tasks.
📄 ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context
🌐 andyzworks.github.io/progres…
📊 Results. Wrapped in the loop, the same frozen RoboMeter goes from MAE 25.4 → 9.3 (−63%) and rank agreement 0.53 → 0.93 (+76%). It is the best method on every context form, ahead of every frozen PRM and of R²VLM.
🛡️ It stays robust when the video stops early, does more than asked, or follows an unrelated task.
⚡ Parallelizing the loop, so that no component sits idle, cuts wall-clock time by 65.6%.
💡 Takeaway: a PRM doesn't need to interpret the whole history itself. It needs to be told where in the task it is. For long tasks, the right context matters more than more history.
📝 Limits: we study robot manipulation so far, and the loop relies on the VLMs reading the scene correctly.
👥 Joint work with Keliang Wu (co-first), Chengxuan Qian (@chengxuanqian), Xiyuan Yang, @zhangce1203, Ariel Tian, Anbang Liu, Haoran Lu, and Han Liu (Northwestern, UCSB, UIUC, CMU).
🌐 andyzworks.github.io/progres…
🎬 2-min walkthrough below.
Jianshu Zhang retweeted
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:
Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better:
Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better:
Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better:
Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work!
In summary:
- As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.
- Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
