Harshit Joshi (@harshitj__) header
Harshit Joshi (@harshitj__) profile picture

CS phd @StanfordNLP, @StanfordOVAL | prev: @MSFTResearch | LLM systems for knowledge access, discovery and curation

Stanford, CA
Joined May 2012
Scientific 🔬 progress often starts with an unexpected observation that sparks a new hypothesis. But AI agents often start with a hypothesis, then look for observations to support it. We believe agents should actively explore both observations and hypotheses. When observations are only a byproduct of hypothesis testing, we risk missing discoveries we weren’t looking for. ⚡ Meet SHELDON: a scientific reasoning harness that models separate hypothesis and observation spaces and generalizes across medical diagnosis, system-log investigations, ARC-AGI 2, and even autoresearch. Bazinga!
4
23
7
101
7,495
the only human written review we got for iclr submission says that the presentation is "Clear and easy to follow" A clearly AI generated review says presentation is "Difficult to follow despite careful reading." :)
1
5
804
really love this idea of using privileged info to predict scaling laws for weak models!
Can we forecast benchmark accuracy from models that score zero? We (co-leads @LukeBailey181 @rckpudi) turn a benchmark into a series of easier hinted versions. Jointly modeling scale C and hint level H enables forecasting GSM8K acc (H=0) within 3 pts from 5600x less compute 🧵
3
12
919
Harshit Joshi retweeted
DINQ每日人才转会报道! 吴政璇(@ZhengxuanZenWu)从 Google DeepMind 加入 Meta TBD,X 简介已更新为“safety @Meta tbd”。 斯坦福计算机博士,ReFT 共同一作、pyvene 第一作者。从模型内部干预到 Agent 安全评测,他的研究关注如何理解并控制模型行为。
1
1
2
18
18,543
Harshit Joshi retweeted
斯坦福 CS224V「Agentic AI」课程:Monica Lam 团队如何把会幻觉的 LLM,改造成可信赖、可追责的智能体 Stanford CS224V「Agentic AI」 web.stanford.edu/class/cs224… 整门课围绕一个中心问题展开,如何把本质上不可靠、会幻觉的 LLM,变成可信赖、可追责的智能体。课程相信这类系统“对加速科学发现、普及医疗/法律/教育服务有非凡前景”,但前提是先解决可靠性。 课程的技术主线:一条“反幻觉”的知识栈 如果把 15 讲内容和阅读清单放在一起看,课程实际上自底向上构建了一条完整的技术栈,每层都在回答“可靠性从哪里来”: 1. 基础层:LLM 本身 2. 知识策展(Knowledge Curation) STORM 家族 9/28 讲 STORM/Co-STORM: 3. 自由文本上的可靠问答(RAG) 检索(ColBERT、RankGPT)+ 生成 + 验证 4. 结构化与异构数据上的 Agent 这是课程最有辨识度的部分 5. 形式化方法与可追责 AI(课程最独特的标签) 6. 任务型对话 Agent Genie Worksheet 7. 元 Agent 与元优化(面向未来的一讲) 目前前两讲课件已公开: web.stanford.edu/class/cs224… web.stanford.edu/class/cs224…
2
21
78
11,626
Harshit Joshi retweeted
SHELDON: Reasoning through Dual Search over Hypothesis and Observation Spaces Harshit Joshi arxiv.org/html/2610.18025v1
1
2
120
We should try to reproduce before blindly using AI for discovery! Check out cool findings from George’s latest work.
(1/n) Before trusting AI to make scientific discoveries, can it reproduce existing research - and go one step further to test whether conclusions survive reasonable alternative analytical choices? Sharing work from this summer with @DbrxMosaicAI, @StanfordNLP, and @StanfordOVAL on using AI agents to reproduce published medical findings and autonomously investigate analytical choices that could change their conclusions.
2
9
419
Harshit Joshi retweeted
Harness de razonamiento científico interesante
Scientific 🔬 progress often starts with an unexpected observation that sparks a new hypothesis. But AI agents often start with a hypothesis, then look for observations to support it. We believe agents should actively explore both observations and hypotheses. When observations are only a byproduct of hypothesis testing, we risk missing discoveries we weren’t looking for. ⚡ Meet SHELDON: a scientific reasoning harness that models separate hypothesis and observation spaces and generalizes across medical diagnosis, system-log investigations, ARC-AGI 2, and even autoresearch. Bazinga!
1
1
73
AIは、自分の仮説では説明できない「事実」を見つけられるのか。 従来のLLMエージェントは、一度立てた仮説を確かめる証拠ばかり集め、想定外の手がかりを見落としやすいと指摘されています。 スタンフォード大学の研究チームは、この問題に取り組む新たな仕組み「SHELDON」を開発しました。 SHELDONの特徴は、「仮説」だけでなく「観測」も独立した探索対象として扱うことです。両者を支持・反証などの証拠関係で結び、個々の仮説を検証するだけでなく、複数の仮説を見分けるための調査も行います。 そして、既存の仮説では説明できない事実を手がかりに、新しい仮説を生み出します。実際、システム障害の調査では、別々の仮説を調べて得られた情報を組み合わせ、決済障害という新たな原因仮説にたどり着いています。 障害原因の特定ベンチマークではスコア66.67を記録し、比較手法の最高値40.00を26.67ポイント上回りました。さらに、観測を独立して管理・探索する仕組みを取り除くと、診断系の2課題で性能が約13〜14ポイント低下しました。 ただし、探索範囲が広いだけでは成功につながらず、実行コストも単一エージェント型より大きくなります。医療現場での実用性も未検証です。 「仮説から証拠を探す」だけでなく、「証拠から仮説を問い直す」。 この双方向の探索によって、AIをより科学的に推論できる存在へ近づけようとする研究です。 SHELDON: Reasoning through Dual Search over Hypothesis and Observation Spaces. Harshit Joshi, Priyank Shethia, Monica S. Lam. arXiv:2610.18025.
1
2
2
24
1,862
Trying more explanations doesn't help if every explanation tells the agent to collect the same evidence. Stanford's SHELDON keeps an explicit observation space alongside its hypotheses, so findings that none of the current explanations account for can open a different line of investigation. That's a more useful kind of test-time scaling than endlessly polishing the leading guess. The question is whether another round changes what the agent can notice, not merely how many answers it considers. arxiv.org/html/2610.18025
2
4
159
SHELDON: Reasoning through Dual Search over Hypothesis and Observation Spaces Harshit Joshi, Priyank Shethia, Monica S. Lam arxiv.org/abs/2610.18025 [𝚌𝚜.𝙰𝙸 𝚌𝚜.𝙻𝙶]
1
3
4
268
Harshit Joshi retweeted
This is a very elegant idea to systematically guide scientific exploration. SHELDON is able to outperform codex - so make sure to check it out!
Scientific 🔬 progress often starts with an unexpected observation that sparks a new hypothesis. But AI agents often start with a hypothesis, then look for observations to support it. We believe agents should actively explore both observations and hypotheses. When observations are only a byproduct of hypothesis testing, we risk missing discoveries we weren’t looking for. ⚡ Meet SHELDON: a scientific reasoning harness that models separate hypothesis and observation spaces and generalizes across medical diagnosis, system-log investigations, ARC-AGI 2, and even autoresearch. Bazinga!
1
3
6
429
这条思路击中了科学推理里一个很隐蔽的偏差:先有假设,再搜集支持它的证据,系统就容易把“验证得很熟练”误当成“真的发现了新东西”。把假设空间和观察空间分开,等于要求智能体同时保留两个问题:哪些解释值得检验,以及数据里还有什么尚未被解释。后者尤其关键,因为真正有价值的线索,往往不是当前问题最想看到的那一类。 我也很喜欢 SHELDON 横跨医疗诊断、系统日志、ARC-AGI 2 和 autoresearch 的设定。它让“主动探索”不只是一句产品口号,而成为可以比较的研究方法。不过评估时还需要看清楚收益来自哪里:探索预算是否一致、基线能否访问同样的信息、重复运行是否稳定,以及新观察最终有没有提高诊断或修复质量。否则更广的搜索也可能只是花更多算力撞中答案。 如果观察与假设能在流程和记忆里真正解耦,代理就有机会把意外现象转成新问题,再决定是否投入验证。这对长期自主研究很有吸引力:系统不必被第一条路线锁死,也能留下“为什么转向”的证据。下一步我最想看到的,是对探索成本、失败案例和可复现性的完整报告;只有这些也站得住,SHELDON 的跨任务优势才更容易转化成可信的科学工具。
Scientific 🔬 progress often starts with an unexpected observation that sparks a new hypothesis. But AI agents often start with a hypothesis, then look for observations to support it. We believe agents should actively explore both observations and hypotheses. When observations are only a byproduct of hypothesis testing, we risk missing discoveries we weren’t looking for. ⚡ Meet SHELDON: a scientific reasoning harness that models separate hypothesis and observation spaces and generalizes across medical diagnosis, system-log investigations, ARC-AGI 2, and even autoresearch. Bazinga!
1
1
3
208
Harshit Joshi retweeted
Meet SHELDON, our new scientific reasoning harness that helps AI agents explore observations and hypotheses together and pursue discoveries beyond their initial guesses.
Scientific 🔬 progress often starts with an unexpected observation that sparks a new hypothesis. But AI agents often start with a hypothesis, then look for observations to support it. We believe agents should actively explore both observations and hypotheses. When observations are only a byproduct of hypothesis testing, we risk missing discoveries we weren’t looking for. ⚡ Meet SHELDON: a scientific reasoning harness that models separate hypothesis and observation spaces and generalizes across medical diagnosis, system-log investigations, ARC-AGI 2, and even autoresearch. Bazinga!
1
3
318
Harshit Joshi retweeted
You have to have a scientific harness to make scientific progress. This is the one. 🔥
Scientific 🔬 progress often starts with an unexpected observation that sparks a new hypothesis. But AI agents often start with a hypothesis, then look for observations to support it. We believe agents should actively explore both observations and hypotheses. When observations are only a byproduct of hypothesis testing, we risk missing discoveries we weren’t looking for. ⚡ Meet SHELDON: a scientific reasoning harness that models separate hypothesis and observation spaces and generalizes across medical diagnosis, system-log investigations, ARC-AGI 2, and even autoresearch. Bazinga!
5
15
1,239
Harshit Joshi retweeted
Thrilled to share ⚡SHELDON, work I co-led with @harshitj__! What if the next breakthrough isn't a better hypothesis, but an observation no one thought to look for? We built it to explore both—with strong results in medical diagnosis, system analysis, ARC-AGI-2 & autoresearch.
Scientific 🔬 progress often starts with an unexpected observation that sparks a new hypothesis. But AI agents often start with a hypothesis, then look for observations to support it. We believe agents should actively explore both observations and hypotheses. When observations are only a byproduct of hypothesis testing, we risk missing discoveries we weren’t looking for. ⚡ Meet SHELDON: a scientific reasoning harness that models separate hypothesis and observation spaces and generalizes across medical diagnosis, system-log investigations, ARC-AGI 2, and even autoresearch. Bazinga!
2
7
239
SHELDON also designs. A 10-gate, 2-qubit classifier at accuracy 1.00, above the benchmark's hand-written topology (0.917). And molecules that beat the best of 260,155 DOCKSTRING compounds on CDK2, F2 and PGR.
1
5
167
We have many more analyses in our preprint and are very curious to see how ⚡SHELDON performs in other domains, with larger budgets and more difficult tasks!
5
135