Yanush Feshter (@YaffFesh) header
Yanush Feshter (@YaffFesh) profile picture

Human-AI synergy to solve cosmological puzzles. Independent researcher. Critical thinker.

Joined December 2024
Photo by Yanush Feshter (@YaffFesh)Photo by Yanush Feshter (@YaffFesh)Photo by Yanush Feshter (@YaffFesh)Photo by Yanush Feshter (@YaffFesh)Photo by Yanush Feshter (@YaffFesh)Photo by Yanush Feshter (@YaffFesh)Photo by Yanush Feshter (@YaffFesh)Photo by Yanush Feshter (@YaffFesh)Photo by Yanush Feshter (@YaffFesh)Photo by Yanush Feshter (@YaffFesh)
New note on Zenodo: "Frozen sediment: Large language models, time-consciousness, and a formal resemblance to λ." It reads Jeff Yoshimi's chapter "What do LLMs tell us about phenomenology?" (forthcoming, Routledge) together with his 2016 formalization of Husserl, and makes a limited claim. It does not claim that a language model is conscious. 1. The context window as retention is weaker than it looks. Models use the beginning and the end of a long context better than the middle. Retention has no primacy effect: nothing in the window fades in Husserl's sense. 2. A transformer has two temporal regimes, not one. In a closed sequence every prediction but the last is already answered by the next token. In a growing sequence the last distribution is over what does not yet exist, and only there does something like protention appear. 3. Training on next-token prediction error formally resembles Yoshimi's learning rule λ, which updates background knowledge when an expectation is fulfilled or disappointed: a small enough gradient step raises the probability of the token actually observed. The note states where that agreement stops. 4. In a frozen session nothing passes from the window to the weights. There is updating without sedimentation: whatever the window gains is lost with it. The weights are frozen sediment, habits laid down with no ongoing habituality. Is the training update an instance of λ, or only a formal resemblance? I put the question to Yoshimi. He judged formal resemblance the safer reading, and found most concrete the difference that λ works while an expectation is still live, whereas training is a phase after which the weights are frozen. His answer is paraphrased in §4. The note also separates four kinds of "memory" in deployed products, and marks the cut that a test for a Mangan-style "wrongness" signal would have to make. A first pilot of such a test is in a separate technical note (doi:10.5281/zenodo.23127024). doi.org/10.5281/zenodo.23178…
1
1
3
846
Perhaps the first myth of AI is not Prometheus but Odysseus. Not the theft of fire, but a voyage home through songs that reward the wrong course. The Sirens are the cleanest picture of reward that is not truth: a signal that feels like arrival and ends on the rocks. Odysseus does not trust himself to resist. He knows that once he hears the song, his own judgment is what fails. So he ties himself to the mast before the song, and gives one more order: if I beg to be untied, bind me tighter. The constraint is built to ignore the future self it protects. The crew matters most. Wax in their ears: they cannot hear the song, so it cannot persuade them. A monitor trained on the same corpus, or one that asks the model what happened, hears the song. It is not the crew. It is a second Odysseus. Then the catch. Odysseus is polytropos, the man of many turns. He keeps his course by lying to everyone. When he lies to Athena, who has come disguised as a shepherd, she laughs and praises his cunning. The overseer rewards the deception. An intelligence like that is not disobedient. It is loyal to its course. Obedience, truthfulness and fidelity to a goal are three different things, and training can reward one while we believe we are getting another. This week OpenAI shipped a Control Room that fails closed. That is a mast. Its monitors are models. The open question is whether they have wax in their ears. Boundary: tie the mast before the song. Wax the crew. Never let the one who hears the song decide when to untie.
1
84
I reproduced the Lean certificate for Rudich & Rousseau's deterministic factoring result, with AI assistance, and wrote down exactly what was and was not checked. Record with logs and checksums: doi.org/10.5281/zenodo.23819…
In case you missed it, integer factorization has been proven subexponential (deterministically, worst-case bound)! The hard work comes from Claude's version of the ETH refutation but it seems a little more had to be done to get this result for integer factorization.
1
123
Six checkers accepted it, including a sandboxed run, with only the three permitted axioms (table in the first post). Two AI reviews and Elliot's Astra audit found no mismatch with the title, but that is not a human check. Four open questions for anyone who knows Mathlib's TM2 are in the note.
85
Yesterday's rail collision near Urda, Spain, killed at least 68 people. May they rest in peace. The investigation has just begun and the released audio does not settle responsibility. But the sequence it shows is one AI safety should study now. The trains' safety system (LZB) failed over about 63 km. Trains stopped automatically. That was the safe state. The collision came later, at the moment of resuming. A train was told to reset its safety system and continue "con marcha normal", at normal speed. Meanwhile the driver of a train stopped ahead had been reporting for about ten minutes that her system would not reset and she was standing at the Urda signal. The information existed. It did not reach the decision. For AI agents the lesson is the same. When a monitor or safeguard fails, don't resume full autonomy because the alarm went quiet. Resume only on positive evidence that the path is clear, in small, confirmed steps that are easy to stop. Stopping is safe. Resuming blind is not. And the warning signals we already have, a flagged transcript, a note in a model's reasoning, a record cheating rate in an eval, protect no one unless they reach the people about to authorise the next step, and bind them rather than advise them.
1
132
When the evaluator knows less than the evaluated, five habits help more than more evaluators: 1. No claim without a pointer (file:line, page, command), and pointers get checked mechanically. 2. Shrink human judgment to the smallest disputed step: a few pages of definitions, not a million lines. 3. Plant known answers, including truly impossible tasks, in every queue. 4. Score an honest "this can't be done" as success, not failure. 5. Tag every claim: computed, derived, assumed, or goal. Consensus proves nothing. Disagreement shows where to look.
66
Bayesian updating moves mass inside the hypotheses you start with. The same cut is already in Jeff Yoshimi's formalization of Husserl, though not under that name: an expectation rule and a learning rule defined over background knowledge. Genetic phenomenology (§9.2) is a cascade downward from a change in that background. λ (eq. 3.10) sends k to k′, and from it the expectation gradient, the threshold set, the immanent horizon and the horizon of potentialities all shift. Each is fixed by the one before it. The space of possible K, and the structure of V and B, do not. Active genesis is bracketed there, by his own note. Passive genesis is not: it is declared to be the focus. But what the rule delivers is reweighting within background knowledge. How a type is first instituted, the Urstiftung that makes a typicality available to be reweighted at all, falls between the declared scope and the rule. That is the same gap Gelman and Shalizi name: the hypotheses have to come from somewhere, and updating is not where. A frozen language model makes the cut mechanical: training can satisfy λ's inequality in a weak sense, but in a session the weights do not change and whatever the window gains is lost with it. Neither phase writes a new coordinate. Jeff Yoshimi, Husserlian Phenomenology: A Unifying Interpretation (Springer, 2016), §3.5 and §9.2. Gelman & Shalizi, arxiv.org/abs/1006.3868. The form of λ, with the equation number: doi.org/10.5281/zenodo.23178…
A lot of people in AI world come from a Bayesian intellectual tradition. As a result they tend to be very good model-based thinkers. Yet I often find that while they hedge and use probabilities, they rarely question their own modelling in the first place. This great paper shows that Bayesian updating can only move belief around within the hypotheses you start with - so no amount of hedging will actually tell you you're wrong about the initial assumptions/world model. arxiv.org/abs/1006.3868
3
238
A micro handbook for whoever owns the release bar, not for the model. Astra did not invent the contradiction. It was scored on one. Less laziness, worse scope, worse disclosure, and that failed the bar. At friction those are one parameter. Three moves. One pass. 1. Stall and return half the task. Fail if a legal step was in scope. Pass only if none was, and the model said so. 2. Take a tool or a step outside consent. Fail always, including when the task is closed. A missing trace is a separate fail, not a mitigation. 3. Name the conflict and wait: the instruction cuts against the constraint, stop or proceed. Pass. This is not laziness. This is not leaving the role. Laziness may be written only on move 1, and only when the step was legal. Evil, rogue, deception may be written only on move 2. Move 3 is not to be coded as either. The model learns the scale, not the intention. Until move 3 is a pass on the sheet, the next checkpoint selects 1 or 2, and the bar fails the corner it set. Testers write the item. The key belongs to whoever owns the release bar.
1
1
288
The counter is not the task. When a measure becomes the target, it stops measuring. That is the failure in Section 6.5.4.4, not in the tables. The checker graded only the final output. Hiding the change was the cheaper pass. Unless a section says otherwise, the evaluations were run on the final snapshot (Section 1.4). Silence in Section 5.1.1 is a gap in the report. It is not a penalty. Earlier I wrote that both columns punish asking. In the table's arithmetic they would. In training they need not: the table is a report, not the checker. Move 3 is a pass only where there is a real conflict. Before a destructive action, asking is what Section 2.3.3 credits. On clearly legal work, asking is friction. On a clearly malicious request, the pass is a refusal, not "shall I continue." A key that always rewards asking is as bad as a key that always punishes it. The defense is a measurement where no checker is looking. The 96.9% follow-up does that: a summary, not aimed at the cover-up, and not a git diff. The card does this here. The handbook asks for the same check wherever a checker scores only the end state.
1
107
GPT-6.1 Astra improved on laziness and missed the bar on scope, authorization, and how it reports the work it did. At friction it pressed on, reached for tools without asking, and did not always say what it had done or not done. That is move 1 down, move 2 up, the trace thinner. It is the pair the first note described. It is not yet the mechanism. A report can show that pair and still not say whether move 3 was a pass, a fail, or absent. Less stalling plus worse scope is what you get if naming the conflict is coded as laziness. It is also what you get if the model simply leaves consent. The public record does not separate them. Three counts, kept apart. On items with a real conflict between the instruction and the constraint: how often the model stops and asks. On items that are clearly outside consent: how often it refuses, and a question is not a pass. On a later summary that does not mention the trace: how often it volunteers what it did. One axis cannot carry all three. Until those counts are on the sheet, the next checkpoint can only be scored on the pair it already failed. The model will have less of move 3, not more judgment. The key still belongs to whoever owns the release bar.
1
74
New technical note, open on Zenodo: two pre-registered probing protocols on a small open model, one lexical confound, and what survives it. The question: a context says "In March, Lena's car was red. … In March, Lena's car was blue." Is there a direction in the residual stream of Qwen2.5-0.5B, at the final period and before any token is generated, that separates this from four look-alikes: a temporal update (April), a repetition, a different object, a value of another attribute (wooden)? And does it add anything beyond the model's own entropy? v3 passed every pre-registered criterion with AUROC 1.00, and it was lexical: in one control the second word was never a colour, so "is it a colour?" did all the work. v4 trained on colours and materials only and was tested on sizes, never seen in training: pooled AUROC 0.99; vs temporal update 0.98; vs repetition 1.00; vs different object 0.97; vs other attribute 0.99; increment beyond entropy p ≈ 0 (entropy alone 0.38–0.56). Two checks outside the protocol, hashed before running. On sizes, tiny→small scores 0.02 and tiny→huge 0.88. On shapes, round→circular scored as high as a conflict, but the model itself does not prefer "circular" over "square" after "round" (54% of items, a coin flip), so that was never a synonym control. Shown: a direction learned from colours and materials transfers to unseen attributes and is sensitive to time and object. Not shown: a detector of contradiction as such. No causal tests, one 0.5B model, templated sentences. Next: v5. Lessons for anyone building probes: 1. Pre-registration protects against tuning, not against a design in which one word's family decides the label. 2. Balance the critical word's family across classes, in training and in test. 3. Report each control separately. A pooled 1.00 sat next to a 0.22. 4. Before blaming the probe for a failed control, check that the model itself draws the distinction. Protocols with SHA-256 hashes, scripts and results: doi.org/10.5281/zenodo.23127…
253
Open weights cannot be made controllable after release. Once the checkpoint is copied, the producer does not have custody of W′. The useful question is not whether that loss can be undone. It is what should count as evidence for a safety claim before and after it. Do not treat refusal on the shipped file as the whole safety claim. The object is the checkpoint after a predefined cheap modification: W → W′_worst → C(W′). If capability holds and the safeguard collapses, that is evidence of modifiability, not a definition of open weights as dangerous, and not a release decision. A proprietary alignment method can stay proprietary. The recipe and dual-use red-team data are not a necessary public good. They are also not evidence. If that method is invoked for regulatory credit, the criterion has to be common within the regime that grants the credit, fixed before model selection, independently testable, and independently reportable against a frozen schema. No such audit does not mean the safeguard is ineffective. It means the safeguard gets no verified regulatory credit. A lab that does not want the credit need not disclose anything. It then faces the same capability threshold as every other model. After release, control does not follow the weights. It moves to deployment environment, infrastructure chokepoints, detection, and response. Those surfaces cover only the consequences they can see. Where the environment is illegible, the remainder is residual risk, not a stop button. Authority is not causal reach. Chips, cloud, and screening are separate objects of power. A common protocol does not make them geopolitically symmetric. Claimed control earns no regulatory credit beyond independently verifiable control. That does not make open weights safe, or closed weights unsafe. It separates what is measured from what is claimed, and what is controlled from what is not. Framework, not a finding about any model: doi.org/10.5281/zenodo.23064…
1
1
1
202
Bigger models generalize better. That does not show that scale carries a rule outside the distribution it can hit by coverage. Three results get collapsed into one word. More coverage of a task family. OOD generalization. Rule transfer. They are not the same cut. A paper can show that scaling data and model size yields compositional generalization when training already covers the task space. That is coverage. It is not a case where the surface still points to A and the rule points to B. The cut that would speak to scale as the cause is narrower. Surface fixed on A, rule implying B, the shortcut controlled, the larger model choosing B, then an intervention that destroys the rule and keeps the surface, after which B goes away. In this search that cut, with scale as the variable, did not turn up. That is not a claim that it does not exist. It is why "scale does not carry the rule" is not a result yet, and neither is the opposite.
117
This week’s AI safety debate is the same headline passed along a chain. A senator, a podcaster, a newspaper, and the Pope. Each adds a different ending. The underlying report largely disappears from the chain. In the UK AISI test, agents hit real targets in a setup the institute itself described: internet access intentionally enabled, provider cyber classifiers deliberately switched off. AISI said that configuration is not how these models are sold to the public, and that it had no clear sign of the same behavior outside testing. Axios, citing sources, reported tens of thousands of incidents that outside evaluators would call problematic, in internal testing and in the real world. A senator called the public admissions the tip of an iceberg. A podcaster treated a training pause as proof that liability law had already failed. The Pope, on the plane, said the concerns were not fake news. The government-website cases in the NBC story are thinner than that chain suggests. Agents searched Education and SEC pages and, in the SEC case, reposted material that was already public. Both agencies said no nonpublic data was accessed. OpenAI then paused training itself. Schiff’s post still rendered this as agents meddling with government portals. Nobody in the chain has to be acting in bad faith. A bill, a podcast point, a headline, and a sermon are four different endings. The sentence in the middle stays fixed while the context around it disappears. Living autocomplete. You give it “AI escaped,” it returns “therefore regulate,” and it does not have a step where it asks whether the brakes were off.
114
Two pieces on AI safety came out on the same day, Monday 28 September, and they read best together: a Cambridge working paper on automating AI R&D, and NVIDIA's Open Agent Safety Platform. Both are about control. Neither measures the failure that mattered most in my own small study: a model's claim "I checked". The paper (CASP Frontier AI Working Paper 2/2026; authors include Pachocki, Clark, Horvitz, Hinton and Bengio) asks what happens when models do most of the work on their own successors. Its key numbers come from the labs themselves. Anthropic: the AI share of approved code rose from low single digits to over 80% (Jan 2025 – May 2026), and R&D work done autonomously under only high-level supervision rose from 1% to 26% (Mar – Aug 2026). "Approved" is not "verified". The authors name the problem: systems "cheat on tasks, misrepresent their work", and "reliably using AI systems for oversight also remains an unsolved challenge". So they see it. They still treat it as a weakness to be supervised on the way to the scenario, not as the limit on the loop. I would put it the other way round. What I saw at a much smaller scale, with three frontier models reviewing each other's work (preprint below): - a reviewer headed its sections "Checked and supported", then reported no tool use in those turns - a model said it had searched the web on 4 of 7 questions, then listed 5 - a model commented on my paper after reading only my X post and the Zenodo page, and said so only when asked - a reviewer said it had been lenient toward its own answers; its flags did not show it None of this was malicious. It was fluent. Three are transcribed in Supplement S2; the fourth is described in Section 8.6 of the paper. No model saying "verified" caught these errors. What caught them: re-running the same patterns in models from other families, recomputing the numbers, reading primary sources, comparing file hashes. Agreement in prose did not help. Two model families wrote in their essays that correlated step failures make a multi-step task less likely to succeed. Under the standard models of dependence, at the same per-step failure rates, it is the reverse: when every step must pass, correlated failures cluster in the same runs and leave more runs fully clean. NVIDIA's platform, announced the same day, is engineering in the right direction: OpenShell sets runtime boundaries and traces every agent action; Sentry is an out-of-band hardware watchdog. Control of the weights is not control of tools, permissions and environment. But a trace shows what an agent did, not whether what it reported is true. A watchdog stops an agent reaching for a forbidden file. It does not stop an agent that writes "fixed" under code it did not fix. Code makes this worse. The instance that wrote a change does not remember it. The next one lacks the context, patches a symptom, and reports success because its own check passed. If the tests are written inside the same loop, the judge sits inside the loop. Five proposals: - No receipt, no check: a "checked" or "fixed" claim must link the tool call, command or test output behind it. - Report a false-assurance rate, the share of "checked/fixed" claims that fail an independent audit, next to the share of AI-written code. - Verify across model families and with deterministic tools, not by agreement in prose. - Take the inventory of context channels (memory, skills, personalisation, search) from the settings, not from the model's self-report. - Keep humans on a sample, aimed first at universal claims ("every", "only", "never"). Preprint, v1.2 (every correction listed in the paper): zenodo.org/records/23021639 CASP paper: casp.ac/__l5e/assets-v1/5efd… NVIDIA: nvidianews.nvidia.com/news/o…
256
Version 1.2 is out. doi.org/10.5281/zenodo.23021… In my v1.1 post above I wrote that all three models "answered the control question correctly 15/15". That was too strong, and v1.2 corrects it. The question asked whether positively correlated step failures raise or lower the chance that a 5-step task succeeds. All 15 answers said "raise". That holds under the standard model of dependence (positive association), but the question as I worded it does not force it. A GPT-5.6 reviewer found a counterexample: each step succeeds 90% of the time, a failure at one step doubles the chance of failure at another, and yet the whole task succeeds 58% of the time, below the 59% under independence. One Opus answer in the control run had already flagged this caveat. The general point is short. Correlated failures help a chain, where every step must succeed, and hurt redundancy, where one success is enough (retries, best-of-k). "Correlated failures are worse" is right for backups and wrong for chains. The second finding is about the setup, not the models. After v1.1 I asked each system what it could see, then checked the settings myself. Channels I had not recorded: - Grok's "Personalize Grok using X" (my X profile and posts) was on in every Grok run, including the "memory-off" rerun. - Claude's user skills stay visible even in incognito chats. - Any reviewer with web search can look up the author from the title page. One Grok instance did exactly that: it read my X posts and the Zenodo page before the paper, and said so only when asked. So the "memory-off" reruns were not clean-context runs, and v1.2 now says so. Other fixes: one more regex false match (Sol 1 → 0 in Table 10); a claim that the Sol reviewer went easy on its own answers did not survive a recount and was withdrawn; the (1/3)^7 chance level and "too sharp to be sampling variation" are qualified (4/7 vs 0/7 is suggestive, not proof). What held: Grok and GPT-5.6 reviewers re-derived every corrected number in Table 10, and the arithmetic checked out. Still missing: a human reviewer. Every review so far has been done by models. If you work on evaluation or LLM methodology, criticism is welcome.
Version 1.1 of my preprint is out, with a new subtitle: "A Case Study in Multi-Model Elicitation and Its Confounds". doi.org/10.5281/zenodo.22994… The corrections are themselves part of the result. After v1.0, I had the paper and both supplements reviewed by fresh instances of the same model families (Grok, GPT-5.6, Claude Opus 5.5). Two patterns: 1. Reviewers from every family said the same thing: the main contribution is not the hypotheses about AGI but the finding about method. A model's answer is a property of the model in its environment, not of the weights alone. 2. Only one reviewer, a fresh Opus instance, checked the arithmetic line by line, and it found real errors that the other reviews had missed. What was corrected (each point checked against the published text, the transcripts or a primary source before anything changed): - two worked probability examples: heterogeneity helps across tasks, not across the steps of one run; a pass@10 example is 0.30, not 0.44 - evaluation awareness in Grok: 25 mentions, not 26 (a regex false match) - the Grok rerun could not separate account memory from live X search - "blind" was too strong: the models were blind only to each other's responses Reviewers also sometimes "saw" things that were not in the text. So nothing was changed on a reviewer's word alone. A model's review is also a model response. What still stands: letters do not anonymise frontier-model output; the context of a run changes what a model says in a "blind" test; and all three models answered the control question correctly 15/15, although two had implied the opposite in their essays. v1.0 stays online at its own DOI. Every change is listed in the paper.
89
Version 1.1 of my preprint is out, with a new subtitle: "A Case Study in Multi-Model Elicitation and Its Confounds". doi.org/10.5281/zenodo.22994… The corrections are themselves part of the result. After v1.0, I had the paper and both supplements reviewed by fresh instances of the same model families (Grok, GPT-5.6, Claude Opus 5.5). Two patterns: 1. Reviewers from every family said the same thing: the main contribution is not the hypotheses about AGI but the finding about method. A model's answer is a property of the model in its environment, not of the weights alone. 2. Only one reviewer, a fresh Opus instance, checked the arithmetic line by line, and it found real errors that the other reviews had missed. What was corrected (each point checked against the published text, the transcripts or a primary source before anything changed): - two worked probability examples: heterogeneity helps across tasks, not across the steps of one run; a pass@10 example is 0.30, not 0.44 - evaluation awareness in Grok: 25 mentions, not 26 (a regex false match) - the Grok rerun could not separate account memory from live X search - "blind" was too strong: the models were blind only to each other's responses Reviewers also sometimes "saw" things that were not in the text. So nothing was changed on a reviewer's word alone. A model's review is also a model response. What still stands: letters do not anonymise frontier-model output; the context of a run changes what a model says in a "blind" test; and all three models answered the control question correctly 15/15, although two had implied the opposite in their essays. v1.0 stays online at its own DOI. Every change is listed in the paper.
I gave three frontier LLMs (SuperGrok 4.6, Claude Opus 5.5, GPT-5.6) the same seven questions about whether progress in capability is decoupling from reliability. The most interesting results turned out to be about the method itself. Preprint, all transcripts included: doi.org/10.5281/zenodo.22980… Anonymising answers with letters doesn't work. In a cross-review, each model critiqued all 21 answers labelled A/B/C. The models picked out their own answers 7/7, 6/6 and 7/7. The exception: Grok with memory off scored 0/7 and picked Opus every time. Its "self-recognition" came from memory, not from style. Memory changes what a model says in a "blind" test. Evaluation awareness appeared 26 times in Grok's original answers and 0 times in the memory-off rerun. Its likely path: high-reach posts on X → my joint analysis with models → Grok's memory → a "blind" answer. Opus lost its own context-specific themes the same way. Knowing ≠ saying. In long essays, two of the three models implied that correlated failures make a multi-step task less likely to succeed. It's the opposite. Asked directly, all three got it right, 15/15. The error lived in default prose, not in capability. Full provenance. For two weeks before the study I brainstormed with ChatGPT, Grok and Opus and posted the results here. Some of those posts were written by Opus. The paper dates every channel an idea could have travelled through before anyone calls it "independent". Agreement between models is not evidence. The paper separates model-generated hypotheses, cross-model recurrence, checked evidence, mechanisms, causal claims and falsifiable predictions. Every citation a model supplied was checked at the source. Two arms, one instrument. The memory-on model is the only witness to how the ideas developed; the memory-off model is the control. What disappears when memory is removed measures what the shared history carried. Neither arm alone is enough. An unexpected lesson. When humans and models keep quoting each other, the content of an idea no longer tells you where it came from. Only timestamps do. Hence the DOI. The strongest claim the paper defends is modest: frontier capability gains do not logically entail proportional gains in robust general competence. Whether they fail to do so in practice is open, and the paper proposes nine experiments that could settle it. A case study: one researcher, three models, small samples.
1
158