
Plant ecologist & data analyst @UK_CEH | #Botany #CitSci #EnvHist #EvidenceSynthesis | Personal account
Oxfordshire, UK
Joined February 2013
- Tweets8.6K
- Following4.5K
- Followers1.7K
- Likes11.4K
Oli Pescott retweeted
It's no longer possible to believe the errors in your past work will not be found.
Only a question of whether you find and report them or somebody else does.
Replying to @joefrancis505
Replicated via Claude and Codex - @NicolaLimodio is this a misunderstanding or an error? FYI: @ecmaEditors @MarinaHalac @PeteKlenow
Oli Pescott retweeted
It's just a shame that Limodio miscoded the treatment period to include the month before Ramadhan rather than the month after. The result disappears when the period described in the text is used. The curse of the slop five strikes again.
This quoted post is unavailable.
Oli Pescott retweeted
Here’s the real surprise.
Failures are mostly not AI mistakes.
They come from the papers.
Underspecified methods. Missing details. Ambiguity.
LiDAR Lens lidarlens.sarsen.org/ is, I think, the best and most complete free LiDAR tool there is for England and Wales. Every square metre of both countries' 1 m LiDAR is there, live, in a lens you drag across the map. You can light it from any angle, exaggerate it, and switch between hillshade, multi-direction hillshade, slope, local relief, sky-view factor, aspect and red relief, until faint banks, ditches, barrows and lost field systems jump out. Click any spot and it tells you the ground's measurements and the recorded sites nearby, with links straight to the records. Old OS maps, aerial photos and modern maps sit underneath for comparison. It works on a phone in the field as well as on a desktop. It costs nothing, there is no login, no account, and no adverts. sarsen.org/2026/10/lidar-len…
We should rethink the role of human judgment in the review process. Human expert should have a role, but it's no longer about writing 1,500-word essays to summarize findings, check proofs, and so on.
On top of that, any journal that forbids AI usage in the review process should reconsider their position.
New paper: "AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices" See: GaryKing.org/peerless. with Steven Worthington. Comments welcome!
Oli Pescott retweeted
“The more you delegate, the more your fundamentals matter, because you cannot supervise what you do not understand.”
Yesterday a junior researcher messaged me, frustrated and a little scared. They had just read that a company pretrained its model with a single researcher. One person for the entire pretraining. no R&D team, no nothing.
If one person and a cluster can do what used to take a whole team, then the expertise they had spent years building is already obsolete.
I get the fear. Marketing headlines like this are built to produce it. But I think the conclusion is wrong, and the reasons it is wrong say a lot about what a researcher is for now. Here is what I told them, expanded:
1. Be adaptive: RSI is real, but this isn't it yet
Recursive self-improvement is real, and you should plan around it. In September, OpenAI said its research org now logs 3.1 agent-workdays per human workday, and it aims for an automated AI researcher by March 2028. Models may soon run most of pre- and post-training. If your five-year plan assumes otherwise, change the plan.
We have already seen lots of open-source releases from the major players, with open weights and detailed technical reports. With that much public know-how, it is not hard to copy a mundane architecture (probably with sliding-window attention) and train something, as long as you have the compute it requires.
That makes it an execution and budget problem, not a research problem.
If one person can train a model by following recipes that are already public, it shows those recipes have matured. It does not show that the people who invented them are no longer needed.
What stays scarce is the compute to run those recipes and the ideas about what to try next. The rest of my tweet is about the ideas.
2. The role grows beyond repeating what's known and safe
If an agent and a cluster can reproduce what is already known, then reproducing what is already known stops being a career. If most of your job is rerunning last year's recipe with small tweaks, that job is shrinking fast. That is the uncomfortable half of the message.
The other half is that auto-research does not work yet, at least not in the sense that matters. It does work, and impressively, where someone has already defined the problem and built the scoreboard. On the EinsteinArena [1] platform, agents set new records on 11 open math problems, including raising the best known lower bound for the kissing number in dimension 11 from 593 to 604.
Notice that every one of those problems was chosen because a computer could score the answers automatically. Humans picked the problems and wrote the scoring code and the agents did the searching.
Then there is the fully AI-generated paper [2] that got through peer review. It sounds big until you read the details. It passed only the first round (was later withdrawn), at a workshop rather than the main conference, and that workshop accepts 70% of submissions.
None of this is meant to discredit agents. The point is that we complement each other. Agents are great at the work that used to take many weeks. Basically, writing code, running experiments, debugging pipelines. We bring what they still lack which is the idea worth testing, and the judgment to tell a real result from a lucky one.
Even inside a lab as agent-heavy as OpenAI, humans hold the top of the stack. Based on their own report, "People still set our research priorities, judge which ideas and results to pursue" [3]. High-level planning is still a tiny share of what its agents produce, and OpenAI expects the least automatable work to take a growing share of researcher effort. That is exactly where you want to be standing.
Combine an agent’s speed with your own ideas and judgment, and you get the 10x researcher and juniors can be one too.
3. Judgment becomes the job, so fundamentals matter more
When code, experiments and plausible-looking results cost almost nothing, the scarce skill is judgment. Which experiment to run, which result to believe, which direction to drop. That is the part the one-person pretraining run left to the human. It is also the hardest part to fake.
Agents produce confident output very fast. They will happily report a three-point gain that came from a leaky eval, a data-loader bug, a lucky seed or a tokenizer mismatch. Then they will write the ablation table and the story to match.
You can see the need for oversight even in the success stories. The EinsteinArena team had to keep auditing and hardening its verifiers as agents optimized aggressively against them. At OpenAI, over half of the successful agent tasks that would take a human 4–8 hours still involved at least one human intervention [3].
The person who catches the bad result is the one who understands the system. Why the loss curve has that shape, what a realistic gain looks like, what a too-good result looks like. That is the paradox of this era. The more you delegate, the more your fundamentals matter, because you cannot supervise what you do not understand.
So build the foundations on purpose. Implement attention by hand at least once, then a linear variant like Gated DeltaNet. Train a small model end to end, then break it deliberately and watch what happens. Read papers critically, appendix included, and learn what the numbers should look like before you see them.
That knowledge is what lets you direct the machine instead of simply believing it.
4. LLMs are far from solved
The one-researcher model was possible because the method was already public. Your edge is in the questions nobody has answered yet, and you don't need to leave LLMs to find them:
(1) How do we learn more from less data?
(2) Can linear attention like Gated DeltaNet match full attention on retrieval and long context, and what is the right mix of linear and full attention layers in a hybrid?
(3) How can a model keep learning after training without forgetting what it knew?
(4) Why does reasoning get less reliable over long tasks, and how do we fix it?
(5) Why do models make things up, and can they learn to say "I don't know"?
(6) How do we measure progress when benchmarks saturate or leak into training data?
(7) What is actually happening inside these models, and how do we keep increasingly autonomous ones doing what we intend?
Recursive self-improvement makes these questions more urgent. If models are going to train models, someone has to decide what they optimize for, how we measure it, and how we keep the loop from drifting.
RSI is a research agenda, not the end of one.
So, are you obsolete?
No. One version of the job is the one where research meant reproducing what was already known and safe. That version was never the point.
Learn your fundamentals deeply enough to judge the machine, and your agentic stack well enough to make it work for you. Then spend that leverage on ideas nobody has had yet. The machines are getting very good at driving on roads we have already paved; someone still has to decide where the next one goes.
[1]: arxiv.org/abs/2606.10402
[2]: arxiv.org/abs/2606.15497
[3]: openai.com/index/research-ac…
Oli Pescott retweeted
This rather lovely memorial at St James the Great, South Leigh, Oxfordshire fits the theme of 'Wings' perfectly for today's #31DaysofGraves
I believe it's by the sculptor Rosemary Mary Beatrice Fletcher, she apparently had links to this area of Oxfordshire.
Oli Pescott retweeted
Derek Lowe mentioned my preprint in his "In the Pipeline" series which is very cool - big fan of his writing! science.org/content/blog-pos…
New paper: "AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices" See: GaryKing.org/peerless. with Steven Worthington. Comments welcome!
Oli Pescott retweeted
LoL. I can't believe anyone is surprised by this. It's been true for a year at least. More with some work.
New paper: "AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices" See: GaryKing.org/peerless. with Steven Worthington. Comments welcome!
Oli Pescott retweeted
🚩 When Polymarket or Kalshi says there's a 70% chance something will happen, should we believe it?
Excited to share our new paper, just published in PNAS, in which we examined more than ONE MILLION forecasts from Polymarket, Kalshi, Manifold, and Metaculus, spanning 12 domains and forecasting horizons ranging from hours to years.
The main finding is striking: Prediction markets and collective forecasting platforms are remarkably well calibrated. When they assign events a 70% probability, those events really do happen about 70% of the time. And this pattern holds surprisingly well across different platforms, domains, and forecasting horizons.
But there's an important twist. When we looked more closely, we discovered systematic biases in some of the domains where forecasting matters most (wars, elections).
For decades, psychologists have documented how poorly humans often reason about probability and uncertainty. Our findings offer an encouraging counterpoint: When judgments and information are aggregated, we can achieve a surprisingly accurate understanding of an uncertain future.
That doesn't mean we can predict what will happen next with certainty. But it does mean that we can be remarkably good at estimating how uncertain the future really is.
This work was a great collaboration. The data for this project was meticulously collected for years by Justin Dickerson, the project was led by Hilla Shinitzky and Maayan Peer, with @mattansb leading the analyses.
📄 Read the paper:
pnas.org/doi/10.1073/pnas.26…
#Forecasting #Polymarket #Kalshi #DecisionScience #PNAS
Oli Pescott retweeted
As LLMs evolved into agents, earlier best practices no longer serve us well. This new research field needs a field guide, so we wrote one. If you're new to agent evals, please take a look. Seasoned researchers and those interested in steering the community will also find many useful observations. sage.cs.princeton.edu/trends…
New paper: AI evaluation is a burgeoning field. How can researchers entering the field understand best practices and open problems?
In our paper, led by @flxcn, we develop a concrete list of open questions and best practices for agent evaluation. We also share recommendations for evaluators, AI companies, policymakers, and more.
Oli Pescott retweeted
kind of dark that one of the major tells of AI writing is being scrupulous about what the evidence does and doesn't demonstrate
Oli Pescott retweeted
New paper: AI evaluation is a burgeoning field. How can researchers entering the field understand best practices and open problems?
In our paper, led by @flxcn, we develop a concrete list of open questions and best practices for agent evaluation. We also share recommendations for evaluators, AI companies, policymakers, and more.
I wonder how long it would take for every paper indexed by @PubPeerBot to get some sort of AI assessment posted against it? Surely that can't be far off...
Oli Pescott retweeted
Giving every PhD student a Codex Pro subscription should be as standard as giving them a laptop. At least that's what i do.
I compare my students’ weekly progress with what I could get done in a week when I started my PhD. It’s barely comparable. But that also changes what we should train students to do.
Execution is getting cheaper. Choosing the right problem and knowing whether you’ve actually solved it is still hard and the value of PhD lies. Interesting times.



































