Maybe an PSY Researcher | PSY PhD | @RiceUniversity | Quant | Tech in selection | https://teapawt.org/t.co/xHsSft7LP6 (also check paper here :)
Joined October 2023
- Tweets19
- Following51
- Followers29
- Likes84
Pengda Wang retweeted
AI evaluation is entering an interactive benchmark era.
Across tool-use agents, web/OS benchmarks, multi-agent systems, and reliability evaluations, interaction is becoming central to how modern AI systems are tested.
But the field risks adding interaction faster than it develops the scientific principles for evaluating interaction.
Our position:
Interactive evaluation is not just longer tasks, tool use, or multi-turn interaction.
It requires a design science for mapping trajectories to valid evaluative claims.
📄 arxiv.org/abs/2605.17829
💻 github.com/keyangds/interact…
Pengda Wang retweeted
⚠️ Fine-tuning LLMs can easily compromise their safety alignment, even with seemingly harmless data!
🚀 We propose ✨GR-SAP✨ to preserve safety alignment with model-synthesized data during fine-tuning.
Check our new preprint!
📄 arxiv.org/abs/2603.10243
I believe one reason is that current personality scales lack nuance and are often detached from the context of actual tasks. We need more nuanced personality assessments. And
@annazou1103
started her PhD at Northeastern this fall! Talk to her at the conference!
🤔 What are we actually measuring when we give LLMs personality questionnaires? Are those results valid?
🔍 Key finding: In a study with 500 participants, we found that LLM's self-reported personalities barely correlate with how humans perceive them during interaction.
🎯 Implication: Personality evaluation should be grounded in interaction. We need behavior-based, interactive evaluation that capture how chatbots actually interact with users.
🎉 The paper has been accepted to COLM 2025!
🧩 Join us at our poster session on Oct 9 (11:00 AM - 1:00 PM) in Room 710, Poster#35. Come chat with us!
📰 News story: sciencenews.org/article/ai-c…
📄 Full paper: arxiv.org/pdf/2412.00207
#CoLM2025 #AI #LLM #Chatbots #HCI #Psychometrics #Evaluation
3
113
Pengda Wang retweeted
How can we interpret LLM's response to psychometric questionnaires? 🧐In our #COLM paper, we questioned the validity of such use and argue that the evaluation should be grounded in interaction. @annazou1103 started her PhD at Northeastern this fall! Talk to her at the conference!
🤔 What are we actually measuring when we give LLMs personality questionnaires? Are those results valid?
🔍 Key finding: In a study with 500 participants, we found that LLM's self-reported personalities barely correlate with how humans perceive them during interaction.
🎯 Implication: Personality evaluation should be grounded in interaction. We need behavior-based, interactive evaluation that capture how chatbots actually interact with users.
🎉 The paper has been accepted to COLM 2025!
🧩 Join us at our poster session on Oct 9 (11:00 AM - 1:00 PM) in Room 710, Poster#35. Come chat with us!
📰 News story: sciencenews.org/article/ai-c…
📄 Full paper: arxiv.org/pdf/2412.00207
#CoLM2025 #AI #LLM #Chatbots #HCI #Psychometrics #Evaluation
Pengda Wang retweeted
📢I'm recruiting a postdoc at Rice University! Looking for someone with a PhD in CS (or a related field) and expertise in LLMs and Trustworthy & Responsible AI.
I'll be at #COLM2025 from 10/8-10/10. Happy to chat in person!
Feel free to email or DM me if you're interested.😀
This is a science of science research project, and we are actively collecting feedback.
If you have any suggestions or ideas, feel free to reach out to any of our team members :)
🌉How does #AI research (mis)engage with psychology?
In this survey, we analyze citation patterns across 1k+ #LLM papers to uncover how #psychology is integrated, highlighting popular & underexplored theories/frameworks and identifying common misuses.
🔗arxiv.org/abs/2507.22847
Pengda Wang retweeted
My student @RuidiChang will present her first work "SAFR: Neuron Redistribution for Interpretability" at #NAACL2025✨🥳
📍Hall 3, Session B: Oral/Poster 1, Wednesday 4/30/2025, 11:00-12:30
Come and say hi~👋
🚀 Thrilled to announce SAFR is here! #NAACL2025
Superposition is powerful — but it buries interpretability. We control that!
🧠 Neurons often mix too many features (superposition) — making models a black box.
🎯 SAFR strategically redistributes neurons:
· 🧩 Monosemantic for important tokens
· 🔗 Polysemantic where interactions matter
🔥 SAFR controls how neurons represent features — without sacrificing accuracy.
💬 Excited to present this work at #NAACL2025 Findings — come chat with us!
Can AI replace human respondents in psych research? 🤖
We compared #LLM-generated and human responses using BFI-2 & HEXACO scales. AI mimics broad patterns but misses subtle human nuances at the item level. 📊
Validation is key!
osf.io/preprints/osf/rwy9b
#AI #Psychometrics
I know that for many researchers, this is not even an important problem.
But as a potential tool, it has indeed attracted the attention of a lot of SSs.
And do we really know LLMs well? The training corpus (exact sources) for the models tested has never been disclosed.
174
Pengda Wang retweeted
🚨PhD Opportunities!🚨
I'm recruiting 1-2 #PhD students for Fall 2025 in #IOPsych @RiceSocSci @RiceUniversity! 🏅
Interested in #personnelselection, #personalityscience, #AI, & #psychometrics? Join our SMART Lab! 💻👥
Deadline: Dec 1, 2024.
More info: tinyurl.com/SMART-phd-ads
Pengda Wang retweeted
Our paper, "Will the Real Linda Please Stand up...to Large Language Models? Examining the Representativeness Heuristic in LLMs" was just selected as an oral spotlight presentation (top 2%) at @COLM_conf.
arxiv.org/abs/2404.01461
Congratulations to @PanDwww, @ZilinXiao2, and @hanjie_chen!
Wanna know more about performance ratings? Check this out!
Invest in rater training, embrace tech, and unlock REAL employee potential.
#FutureOfWork
On putting the horse (raters and criteria) before the cart (variance components in ratings) | Industrial and Organizational Psychology | Cambridge Core - bit.ly/4cHSOkg
Congratulations to @PanDwww and Hwayeon Meyong!
2
3
298
Pengda Wang retweeted
Super fun project on investigating the prediction behavior of LLMs from a psychological perspective🧠
🚨GPT-4 can fall into a cognitive trap and make mistakes even when it has the correct knowledge, like us humans🤯
✅A simple hint helps
Check out our paper for more details👇
Will the Real Linda Please Stand up...to Large Language Models? Examining the Representativeness Heuristic in LLMs
arxiv.org/abs/2404.01461
@PanDwww
@ZilinXiao2
@hanjie_chen
Pengda Wang retweeted
Leveraging Synthetic Data to Advance Organizational Science - goo.gl/scholar/FmzJLS #ScholarAlerts
Pengda Wang retweeted
Congratulations to @PanDwww for leading this paper, in collaboration with Zilin Xiao and @hanjie_chen, and accepted at @COLM_conf:
Will the Real Linda Please Stand Up...to Large Language Models? Examining the Representativeness Heuristic in LLMs
arxiv.org/abs/2404.01461
Thanks to @COLM_conf for their constructive review process in support!