customer @ saizeriya non-technical member of staff former digital nomad, human data / rl
Suginami-ku, Tokyo
Joined September 2024
- Tweets1.3K
- Following641
- Followers120
- Likes18.5K
Pinned Tweet
People say Jev is dead but I don't think so
I ran all 14 decision models on OpenRouter against 1273 questions in MedQA
Jev is still near the top on accuracy & latency with a balanced cost
Jyan Z. retweeted
peter thiel is the only one who understands that the whole point of getting rich is getting people to take your essays seriously
Jyan Z. retweeted
CI pro tip:
GitHub Actions (and Blacksmith etc.) round every job UP to the nearest minute
... so a 10 second job still costs a full minute!
If you’ve got a bunch of short jobs, you’re potentially better off combining them into one and parallelizing within
Jyan Z. retweeted
not to start anything, but models can get >60% of verifiers correct yet still score zero % because of the binary scoring. increasingly suspicious of scoring methodologies on benchmarks, especially when it isn’t presented up front
Jyan Z. retweeted
Two robot arms, two interlocked claw pieces: get them apart without breaking either. 11 of 13 models we've run have failed it.
GPT-6 Astra: apart in 53 min, $14.36.
Kimi K3.1 Pro: still locked after 101 min, $13.45.
Jyan Z. retweeted
I just built a *functioning* React app by describing what I wanted to GPT-3.
I'm still in awe.
Jyan Z. retweeted
We asked two isolated copies of the same AI to exchange a secret word with only a robot and a camera.
They succeeded by using air-writing, Morse code, flag semaphore and a 5-bit binary alphabet, without agreeing on a protocol beforehand.
Jyan Z. retweeted
I hate sf halloween, what do you mean you’re a human in the loop and a data labeler
Jyan Z. retweeted
So we do a few things, including:
1. We don't use monitors to directly reward / filter / etc. examples during training time.
2. If we do notice some consistent misalignment issue, e.g., a type of reward hack, we try to fix things via more general interventions like making a RL reward model more robust to certain outputs.
3. We try to have monitors as much as possible be action-only and not see CoT if we do not need them to be.