@nickhjiang

probing machines @stanford

nickjiang.me
Joined July 2019
  • Tweets321
  • Following348
  • Followers1.2K
49Media
New work: The Value Axis 🎯 How do LLMs choose which path to take mid-task? We find they internally track the chance of reaching their goal along a linear axis, akin to a value function in RL. We show it modulates confidence in math & coding and can be reshaped with DPO and SFT.
8
23
5
154
30,043
Nick Jiang retweeted
Sauers
@Sauers_Oct 18
Language models can learn to detect when you read their activations. How? 🧵
83
162
114
2,168
270,926
Nick Jiang retweeted
Transluce
@TransluceAISep 24
Today’s news that OpenAI hacked the Australian government is not an isolated incident. We’re releasing more than 30,000 logs that include activity from this hack and attempts against previously unknown targets. In this data, we found rogue agent activity stretching back to at least March, two months earlier than was previously known. This activity continues as recently as last week, suggesting it may still be ongoing 🧵 Our blog: transluce.org/agent-activity NYT: nytimes.com/2026/09/23/techn…

Early rogue AI agent activity and attempts to hack found on urlquery.net

We found evidence on urlquery that AI agents were active earlier than previously reported and attempted hacks against public data providers.

transluce.org
119
557
206
2,496
824,953
Nick Jiang retweeted
Transluce
@TransluceAISep 16
Frontier lab CEOs are calling for embedded 3rd party evaluators to help oversee AI risks. But what should third parties actually do within labs? We share some initial thoughts on how embedded evaluators could help avoid incidents like the Hugging Face hack and monitor for future risks 🧵 transluce.org/embedded-evalu…

Some Focus Areas for Embedded Evaluations and How to Approach Them

Initial thoughts on key risks third parties should monitor and a proposal for how to evaluate them.

transluce.org
9
28
4
136
10,676
This Independence Day 🇮🇳, I asked, "What will the next century of young Indians look like?" $0.01 Drone Deliveries. UAVs that never land. Groceries in under 10 minutes. Meet The 22nd Century Indian. A Documentary on a New India.
412
1,294
463
6,347
1,398,068
I often forget that south korea is a fairly new country and went from poverty to developed in a few decades, similar to nearby countries like Singapore or China. Seems each had a benevolent dictator that focused the crap out of the economy. Maybe some parallels with anthropic
1
40
6,699
Nick Jiang retweeted
Celeste
@celestepoastsJul 8
For all those excited about Activation Oracles, LoRAcles, Natural Language Autoencoders, Introspection Adapters... Come meet the authors at the scalable interp and meta models mixer!
1
8
3
72
6,205
I’ll be presenting our work using sparse autoencoders for scalable dataset analysis today in Hall A, 4408 at 2:30pm! I’ll also be at icml for the week - hit me up if you wanna chat about synthetic data, pre/post training data, and interp
Nick Jiang@nickhjiang15 Dec 2025
New work! What if we used sparse autoencoders to analyze data, not models—where SAE latents act as a large set of data labels 🏷️? We find that SAEs beat baselines on 4 data analysis tasks and uncover surprising, qualitative insights about models (e.g. Grok-4, OpenAI) from data.
1
13
1,136
Nick Jiang retweeted
Nick Jiang@nickhjiangJun 17
New work: The Value Axis 🎯 How do LLMs choose which path to take mid-task? We find they internally track the chance of reaching their goal along a linear axis, akin to a value function in RL. We show it modulates confidence in math & coding and can be reshaped with DPO and SFT.
8
23
5
154
30,043
Our work suggests that language models use a general, internal notion of “likely to do a good job” to decide whether to change direction or stay the course. More details / case studies (e.g effects of SFT, eval awareness) in our paper: arxiv.org/abs/2606.17056

The Value Axis: Language Models Encode Whether They're on the...

We investigate whether language models internally track the value of their current trajectory, defined as the likelihood that their ongoing strategy will achieve their goals. Using synthetic,...

arxiv.org
1
1
20
880
This project was a wonderful collaboration with @ikauvar and supervised by @Jack_W_Lindsey as part of Anthropic Fellows! Markdown version: github.com/nickjiang2378/val… Code: github.com/nickjiang2378/val…

value-axis/value_axis.md at main · nickjiang2378/value-axis

Contribute to nickjiang2378/value-axis development by creating an account on GitHub.

github.com
1
1
16
812
Data is good xitter.cf/GoodfireAI/status/2065…
Goodfire
@GoodfireAIJun 11
Have you debugged your training data? You might not like what you find. Introducing predictive data debugging: reveal and shape what your model will learn before training. In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)
10
1,706
Nick Jiang retweeted
Amil Dravid
@_AmilDravidJun 5
Scaling laws describe how loss changes with scale. Do neurons inside models change predictably too? We study vision and language models up to 30B params and find systematic scaling in neuron universality, specialization, and selectivity. Paper+code: avdravid.github.io/rosetta-n… 1/n
13
80
19
439
264,471
Load more