Seeing this study being widely re-shared with the take away that now AI >= human in peer review, in every dimension. People need to tone it down a little... This is primarily GPT evaluating Claude's comments. To me, the more useful signal is the 3 past editors of unnamed journals rating human vs Claude reviews, but this is only related to just 10 papers, it's not clear the editors have specific expertise in the paper topics or methods, and it reads as though the editors looked at the reviews but not the actual papers. I know we will need AI in the loop in some fashion to manage the paper deluge. I also realize that AI is already widely used in peer review and it's better to step in to manage / govern that process than let people do whatever they want. In my experience, AI certainly surfaces useful / good comments, particularly on methods, but many of the comments can be pedantic. Many models also still have terrible judgment about the other aspects that determine a study's value, e.g., whether the question is important. We're not yet at the point of ditching the human. Note: I'm pushing back here on the interpretations people are sharing when circulating this work, not the discussion in the paper itself.

As LLMs make things easier, we must raise our expectations on what researchers are expected to produce. This is particularly true for writing: clear writing was rarely a focus in scientific publication, and it's only got worse due to LLMs. We're trying to reverse this trend.

Data doesn't speak for itself. Reasonable analyses of the same data can lead to very different conclusions—yet almost all papers ignore this researcher freedom. We introduce Agentic Bootstrap, using diverse persona agents to emulate analysts, and m-value to measure robustness🧵

GPT reviewer "scores above each paper's top-rated human reviewer" but AI review agents "overlap far more than humans do...and exhibit 16 recurring weaknesses humans do not share..." Results "position current AI reviewers as complements to, not substitutes for, human reviewers."