
HKUST Natural Language Processing Research #NLProc
Joined October 2024
- Tweets31
- Following113
- Followers286
- Likes60
Pinned Tweet
Great work! @ZhaoweiWang4
2
784
HKUST NLP retweeted
Check out our new work on multimodal long-context!
HKUST NLP retweeted
How far are we from Artificial General Intelligenceโand what might follow as Artificial Superintelligence? IJCAI Closing Panel opened by James Kwok, #IJCAI2025 Programe Chair.
share.google/DbB2Ky20bCc3B0Qโฆ
#Montreal #AI
HKUST NLP retweeted
๐ Congrats to our students on the new #EMNLP2025 paper acceptances! 1โฃ LLM Natural-Formal Hybrid Reasoning arxiv.org/abs/2505.23703 2โฃ Text-Instructed Image Editing on Medical Domain arxiv.org/abs/2506.01921 3โฃ Knowledge Boundary Aware Multi-Compositional Reasoning arxiv.org/abs/2504.21773 4โฃ End-to-End Optimized Multimodal RewardRAG
HKUST NLP retweeted
Actually, we implemented this kind of capability in our AdaCtrl paper three months ago by injecting difficult-aware tags (i.e., easy, hard, adaptive) to trigger different reasoning behaviors of LLMs.
Paper: arxiv.org/pdf/2505.18822
Updates to ChatGPT:
You can now choose between โAutoโ, โFastโ, and โThinkingโ for GPT-5. Most users will want Auto, but the additional control will be useful for some people.
Rate limits are now 3,000 messages/week with GPT-5 Thinking, and then extra capacity on GPT-5 Thinking mini after that limit. Context limit for GPT-5 Thinking is 196k tokens. We may have to update rate limits over time depending on usage.
4o is back in the model picker for all paid users by default. If we ever do deprecate it, we will give plenty of notice. Paid users also now have a โShow additional modelsโ toggle in ChatGPT web settings which will add models like o3, 4.1, and GPT-5 Thinking mini. 4.5 is only available to Pro usersโit costs a lot of GPUs.
We are working on an update to GPT-5โs personality which should feel warmer than the current personality but not as annoying (to most users) as GPT-4o. However, one learning for us from the past few days is we really just need to get to a world with more per-user customization of model personality.
HKUST NLP retweeted
Excited to present our work, MMBoundary, at #ACL2025!
Come chat with us at our poster session!
๐ Hall 4/5, Session 12: Poster Session 4
๐๏ธ Wednesday, July 30
โฐ 11:00-12:30
HKUST NLP retweeted
If you are at ACL, please talk to our members of @HKUSTKnowComp and @hkustNLP big family
HKUST NLP retweeted
๐คฏ Multimodal LLMs can be confidently wrong. A single early mistake in perception can lead to a completely incorrect answer.
๐Introducing our work, MMBoundary, a new framework to make MLLMs aware of their own knowledge boundaries! ๐งต
Paper:arxiv.org/abs/2505.23224
#ACL2025
HKUST NLP retweeted
Find us at the poster booth โจWed 7/30 11am Hall 4/5 โจ
[1/n] "๐๐ข๐ต๐ค๐ฉ๐ช๐ฏ๐จ ๐ค๐ถ๐ฆ๐ด ๐ง๐ฐ๐ณ ๐ช๐ฅ๐ฆ๐ฏ๐ต๐ช๐ค๐ข๐ญ ๐ฐ๐ฃ๐ซ๐ฆ๐ค๐ต๐ด, ๐ฅ๐ช๐ด๐ต๐ช๐ฏ๐ค๐ต ๐ข๐ต๐ต๐ณ๐ช๐ฃ๐ถ๐ต๐ฆ๐ด ๐ง๐ฐ๐ณ ๐ถ๐ฏ๐ช๐ฒ๐ถ๐ฆ ๐ฐ๐ฏ๐ฆ๐ด." Such ๐๐ง๐ค๐จ๐จ-๐๐ค๐ฃ๐ฉ๐๐ญ๐ฉ ๐ซ๐๐จ๐ช๐๐ก ๐ง๐๐๐จ๐ค๐ฃ๐๐ฃ๐ is extremely simple and straightforward for the human cognitive process, but ๐พ๐๐ถ๐๐ฒ ๐ฐ๐ต๐ฎ๐น๐น๐ฒ๐ป๐ด๐ถ๐ป๐ด ๐ณ๐ผ๐ฟ ๐ฐ๐๐ฟ๐ฟ๐ฒ๐ป๐ ๐น๐ฎ๐ฟ๐ด๐ฒ ๐๐ถ๐๐ถ๐ผ๐ป ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น๐ (๐๐ฉ๐๐ ๐), especially across multiple images and videos โผ๏ธ
๐๐๐ฎ, and ๐๐ค๐ฌ ๐๐๐ฃ ๐ฌ๐ ๐๐ข๐ฅ๐ง๐ค๐ซ๐ upon this major shortcoming in future research? ๐
๐ฅ๐ Check out our latest release, โ๐ฉ๐๐ ๐ฎ -๐๐ฒ๐ป๐ฐ๐ต: ๐ ๐๐น๐ผ๐๐ฒ๐ฟ ๐๐ผ๐ผ๐ธ ๐ฎ๐ ๐๐ผ๐ ๐ช๐ฒ๐น๐น ๐ฉ๐๐ ๐ ๐๐บ๐ฝ๐น๐ถ๐ฐ๐ถ๐๐น๐ ๐๐ถ๐ป๐ธ ๐๐
๐ฝ๐น๐ถ๐ฐ๐ถ๐ ๐ ๐ฎ๐๐ฐ๐ต๐ถ๐ป๐ด ๐ฉ๐ถ๐๐๐ฎ๐น ๐๐๐ฒ๐โ! ๐๐ฅ
๐ Project page: vlm2-bench.github.io/
๐ Paper: arxiv.org/abs/2502.12084
๐ปย Github: github.com/vlm2-bench/VLM2-Bโฆ
๐ค Huggingface: huggingface.co/datasets/Sterโฆ
---
Work led by our amazing team of students Jianshu, Dongyu, and Renjie at The Hong Kong University of Science and Technology. Find us at the poster booth โจWed 7/30 11am Hall 4/5 โจ
HKUST NLP retweeted
[1/n] "๐๐ข๐ต๐ค๐ฉ๐ช๐ฏ๐จ ๐ค๐ถ๐ฆ๐ด ๐ง๐ฐ๐ณ ๐ช๐ฅ๐ฆ๐ฏ๐ต๐ช๐ค๐ข๐ญ ๐ฐ๐ฃ๐ซ๐ฆ๐ค๐ต๐ด, ๐ฅ๐ช๐ด๐ต๐ช๐ฏ๐ค๐ต ๐ข๐ต๐ต๐ณ๐ช๐ฃ๐ถ๐ต๐ฆ๐ด ๐ง๐ฐ๐ณ ๐ถ๐ฏ๐ช๐ฒ๐ถ๐ฆ ๐ฐ๐ฏ๐ฆ๐ด." Such ๐๐ง๐ค๐จ๐จ-๐๐ค๐ฃ๐ฉ๐๐ญ๐ฉ ๐ซ๐๐จ๐ช๐๐ก ๐ง๐๐๐จ๐ค๐ฃ๐๐ฃ๐ is extremely simple and straightforward for the human cognitive process, but ๐พ๐๐ถ๐๐ฒ ๐ฐ๐ต๐ฎ๐น๐น๐ฒ๐ป๐ด๐ถ๐ป๐ด ๐ณ๐ผ๐ฟ ๐ฐ๐๐ฟ๐ฟ๐ฒ๐ป๐ ๐น๐ฎ๐ฟ๐ด๐ฒ ๐๐ถ๐๐ถ๐ผ๐ป ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น๐ (๐๐ฉ๐๐ ๐), especially across multiple images and videos โผ๏ธ
๐๐๐ฎ, and ๐๐ค๐ฌ ๐๐๐ฃ ๐ฌ๐ ๐๐ข๐ฅ๐ง๐ค๐ซ๐ upon this major shortcoming in future research? ๐
๐ฅ๐ Check out our latest release, โ๐ฉ๐๐ ๐ฎ -๐๐ฒ๐ป๐ฐ๐ต: ๐ ๐๐น๐ผ๐๐ฒ๐ฟ ๐๐ผ๐ผ๐ธ ๐ฎ๐ ๐๐ผ๐ ๐ช๐ฒ๐น๐น ๐ฉ๐๐ ๐ ๐๐บ๐ฝ๐น๐ถ๐ฐ๐ถ๐๐น๐ ๐๐ถ๐ป๐ธ ๐๐
๐ฝ๐น๐ถ๐ฐ๐ถ๐ ๐ ๐ฎ๐๐ฐ๐ต๐ถ๐ป๐ด ๐ฉ๐ถ๐๐๐ฎ๐น ๐๐๐ฒ๐โ! ๐๐ฅ
๐ Project page: vlm2-bench.github.io/
๐ Paper: arxiv.org/abs/2502.12084
๐ปย Github: github.com/vlm2-bench/VLM2-Bโฆ
๐ค Huggingface: huggingface.co/datasets/Sterโฆ
---
Work led by our amazing team of students Jianshu, Dongyu, and Renjie at The Hong Kong University of Science and Technology. Find us at the poster booth โจWed 7/30 11am Hall 4/5 โจ
HKUST NLP retweeted
๐ข Join us for the ACL Mentorship Session @aclmeeting #ACL2025NLP #NLProc
โข Session Link: mentorship.aclweb.org/scheduโฆ
โข Ask Questions: tinyurl.com/y2v2j462
Mentors:
โข @May_F1_ (@hkust)
โข @d_aumiller (@cohere)
โข @vernadankers (@Mila_Quebec)
โข @ziqiao_ma (@UMichCSE)
โข @WeijiaShi2 (@uwnlp)
โข @ZhijingJin (@UofTCompSci)
โข @OanaIgnatRo (@SantaClaraUniv)
HKUST NLP retweeted
#COLM2025 Our work has been accepted to COLM 2025๐
Looking forward to discussing Scalable Oversight and Synthetic Data with old and new friends in Montrรฉal @COLM_conf ๏ผ
๐ Data to pre-train LLMs on are reaching critical bottleneck. ๐ฟ๐ค๐๐จ ๐ข๐ค๐๐๐ก-๐๐๐ฃ๐๐ง๐๐ฉ๐๐ ๐จ๐ฎ๐ฃ๐ฉ๐๐๐ฉ๐๐ ๐๐๐ฉ๐ ๐ฌ๐ค๐ง๐ ๐จ๐๐ข๐๐ก๐๐ง๐ก๐ฎ ๐ฌ๐๐ก๐ก ๐๐ค๐ง ๐จ๐๐๐ก๐๐ฃ๐ ๐ฅ๐ง๐-๐ฉ๐ง๐๐๐ฃ๐๐ฃ๐ ๐๐ช๐ง๐ฉ๐๐๐ง? Let's dive into the "๐ฆ๐ฐ๐ฎ๐น๐ถ๐ป๐ด ๐๐ฎ๐๐ ๐ผ๐ณ ๐ฆ๐๐ป๐๐ต๐ฒ๐๐ถ๐ฐ ๐๐ฎ๐๐ฎ ๐ณ๐ผ๐ฟ ๐๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐ ๐ผ๐ฑ๐ฒ๐น๐" together!
Several key findings:
โญ๏ธ ๐๐ฟ๐ฎ๐ฝ๐ต-๐ฏ๐ฎ๐๐ฒ๐ฑ ๐ฒ๐
๐๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป & ๐ฟ๐ฒ๐ฐ๐ผ๐บ๐ฏ๐ถ๐ป๐ฎ๐๐ถ๐ผ๐ป of concepts across docs transforms pre-training corpora into ๐ฅ๐ช๐ท๐ฆ๐ณ๐ด๐ฆ, ๐ฉ๐ช๐จ๐ฉ-๐ฒ๐ถ๐ข๐ญ๐ช๐ต๐บ ๐ด๐บ๐ฏ๐ต๐ฉ๐ฆ๐ต๐ช๐ค ๐ฅ๐ข๐ต๐ข๐ด๐ฆ๐ต๐ด;
โญ๏ธ๐ฃ๐ฟ๐ฒ๐๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด ๐๐ถ๐๐ต ๐๐๐ป๐๐ต๐ฒ๐๐ถ๐๐ฒ๐ฑ ๐ฑ๐ฎ๐๐ฎ reliably follows ๐ณ๐ฆ๐ค๐ต๐ช๐ง๐ช๐ฆ๐ฅ ๐ด๐ค๐ข๐ญ๐ช๐ฏ๐จ ๐ญ๐ข๐ธ across various model sizes;
โญ๏ธ ๐๐ฎ๐ฟ๐ด๐ฒ๐ฟ ๐บ๐ผ๐ฑ๐ฒ๐น๐ approach optimal performance with ๐๐๐ฌ๐๐ง ๐ฉ๐ง๐๐๐ฃ๐๐ฃ๐ ๐ฉ๐ค๐ ๐๐ฃ๐จ;
โญ๏ธ ๐ฟ๐๐ข๐๐ฃ๐๐จ๐๐๐ฃ๐ ๐ง๐๐ฉ๐ช๐ง๐ฃ๐จ start to kick in near 300B tokens.
๐Check preprint for further details~
๐ arxiv.org/html/2503.19551v2
HKUST NLP retweeted
Excited to share our new survey on the reasoning paradigm shift from "Think with Text" to "Think with Image"! ๐ง ๐ผ๏ธ
Our work offers a roadmap for more powerful & aligned AI. ๐
๐ Paper: arxiv.org/pdf/2506.23918
โญ GitHub (400+๐): github.com/zhaochen0110/Awesโฆ
HKUST NLP retweeted
Thrilled to share a major milestone: the culmination of a 15-month project, ATLASโa new benchmark in event graphs and conceptualization! This journey began with Probase in 2012, evolved through ASER (2019), AbstractATOMIC (2022), and AbsPyramid (2023), and now realizes a decade-long dream of building a web-scale knowledge graph integrating entities, events, and a conceptualization framework. Itโs also a tribute to Marvin Minskyโs K-Lines Theory.
I still vividly recall 2012 when I worked with Haixun, including intern Fangting Xia, spent months exploring event semantic primitives and conceptualization, only to pivot due to limitations. Fast forward to the era of large language models, and weโve finally brought this vision to life!
Someone argued that knowledge graphs are obsolete in the age of large models. We disagreeโitโs about having high-quality graphs that scale with these models. Thatโs why we invested HK$1.5M to process Wikipedia, academic abstracts, and 3% of web data from Dolma, creating a 1B-node graphโthe best we could achieve.
While entities cover 70% of factual queries, adding event semantic primitives boosts coverage to 90%. This is why text decomposition is gaining traction. Our extracted knowledge graph excels on complex factual queries like HotpotQA, MMLU, and FELM, making it the largest open-source GraphRAG dataset available.
This work is the result of 10+ students and nearly 10 collaborators pouring their hearts into it. Iโm deeply grateful for their contributions to this meaningful milestone, and I hope theyโve gained as much from the journey as I have.
Big shoutout to anyone curious about pushing this further! Parsing the entire webโs data would cost ~HK$40M, and Iโd love to collaborate with bold thinkers to make it happen.
Paper: arxiv.org/abs/2505.23628
Code & Resources: github.com/HKUST-KnowComp/Auโฆ
Project: pypi.org/project/atlas-rag/
Feedback and critiques are welcome! Letโs keep advancing the field together. #KnowledgeGraphs #ATLAS #GraphRAG
HKUST NLP retweeted
We studied both rule-based and model-based verifiers and found that each has unique limitations. Rule-based verifiers are often unreliable, even in math, and are unavailable in many domains.
Model-based verifiers can be easily hacked. In our paper, we construct simple adversarial patterns to successfully crack several open model-based verifiers.
๐งInterestingly, as the community shifts toward generative verifiers, we found them to be far more vulnerable than discriminative ones. This suggests that discriminative verifiers might be more robust against reward hacking in RLVR.
๐ Are Verifiers Trustworthy in RLVR?
Our paper, Pitfalls of Rule- and Model-based Verifiers, exposes the critical flaws in reinforcement learning verification for mathematical reasoning.
๐ Key findings:
1๏ธโฃ Rule-based verifiers miss correct answers, especially when presented in different formats.
2๏ธโฃ Model-based verifiers are susceptible to reward hacking, undermining RL outcomes.
3๏ธโฃ A probe study reveals that most model-based verifiers, particularly generative ones (like those using chain-of-thought reasoning), are highly vulnerable to adversarial attacks.
๐Paper: arxiv.org/abs/2505.22203
๐ปCode & Model & Dataset: github.com/hkust-nlp/RL-Veriโฆ
Special thanks to my amazing collaborators @AndrewZeng17, Xingshan Zeng, and Qi Zhu, and to our advisor @junxian_he! ๐ฅฐ
HKUST NLP retweeted
Great to see the wonderful series of work that @WangCarrey has been leading at UIUC. We also had a fun collaboration recently together with my incoming PhD student Shijue. Check out our latest release
๐๐ฅ๐ข๐๐ต๐ณ๐ญ: ๐๐ฅ๐ข๐ฑ๐ต๐ช๐ท๐ฆ ๐ข๐ฏ๐ฅ ๐๐ฐ๐ฏ๐ต๐ณ๐ฐ๐ญ๐ญ๐ข๐ฃ๐ญ๐ฆ ๐๐ฆ๐ข๐ด๐ฐ๐ฏ๐ช๐ฏ๐จ ๐ท๐ช๐ข ๐๐ช๐ง๐ง๐ช๐ค๐ถ๐ญ๐ต๐บ-๐๐ธ๐ข๐ณ๐ฆ ๐๐ถ๐ฅ๐จ๐ฆ๐ต๐ช๐ฏ๐จ ๐ฅ๐ฅ๐ฅ
.
.
.
arxiv.org/pdf/2505.18822
Time flies. My visiting at UIUC is coming to an end. Iโm sincerely grateful to Prof. Heng Ji @hengjinlp for the invaluable opportunity to join such a warm, collaborative, and world-leading research group. Iโve learned so much from Prof. Heng and lots of amazing students here @qiancheng1231 @xiusi_chen @emrecanacikgoz @BowenJin13 @zhenhailongW @Yuji_Zhang_NLP @Glaciohound. This experience will always be a defining chapter of my Ph.D. life, filled with friendships, collaborations, and unforgettable moments. Looking back, we truly did something together for the world โ from SMART to ToolRL, OTC, RM-R1, ModelingAgent, and DecisionFlow, with even more impactful work on the way. Iโll always be cheering for BlenderLab and looking forward to whatโs next!
HKUST NLP retweeted
๐ We release MMLongBench: Benchmark for evaluating long-context VLMs.
๐ 13,331 examples across 5 tasks:
โ Visual RAG
โ Many-shot ICL
โ Needle-in-a-haystack
โ VL Summarization
โ Long-document VQA
๐ Lengths: 8 / 16 / 32 / 64 / 128K
๐ Benchmarking both thoroughly & effectively!
HKUST NLP retweeted
๐ Our new survey "From Automation to Autonomy: LLMs in Scientific Discovery" is live!
๐ Paper: arxiv.org/pdf/2505.13259
โญ Awesome Github Repo: github.com/HKUST-KnowComp/Awโฆ
#LLMs #ScientificDiscovery #AI #NLP #Agent #AIforScience @HKUSTKnowComp @hkustNLP
(cont'd)๐









































