Artificial Reasoning Researcher | x-AWS
Joined June 2024
- Tweets2K
- Following276
- Followers209
- Likes5.9K
My review of machine studying by @jacobli99:
The research defines studying as expertise gained when inference compute grows. But I feel it is unfair to compare memory (a cheat-sheet), SFT & LoRA against each other. Each requires very different training compute requirements, and its performance is impacted by how much post-training is done.
Instead, the research seems to point to a far greater result: **post-training scaling laws**. Given a dataset size, compute budget, and model of certain size: what is the most efficient post-training technique (including pure LLM inference) to achieve a given performance gain. Interestingly, optimal results may combine different methods.
I liked that the research was motivated by psychology results (Fiorella, 2023). Which defines generative learning as the ability to explain, visualize, and enact novel knowledge. However, to far weight was placed on execution (acting on knowledge), relatively little on explanation and no on visualization. The definition of studying points more broadly to the ability to compress information (summarization), and even multimodal transfer (visualization).
In LLM research, it is prevalent to directly measure the ability of LLMs but not to formulate a simplified version of the task without language at smaller transformer models that can help explain the results theoretically. The research provides a solid definition of studying, but fails to provide a theoretical upper threshold of what perfect studying could be. Considering novelty (unseen data during model training) as the primary objective, the task could be framed as a partially explored graph.
An abstract framing based on graph exploration leads to an elegant formulation of studying: given a sparsely connected graph over a vector space, what is the optimal set of previously unknown nodes/edges that minimizes tool-calling (search) where the answer is to find the closest node for a given point in the metric space. Search is defined as finding the closest k-nodes based on a distance given a known node. Connected nodes, allows to navigate the vector map without using a tool. More complex definitions of node-finding can be placed based on finding a node with a given connectivity, where edges are probabilistically determined based on closeness.
Without too much detail, that formulation also enables synthetically generated dataset without relying on LLMs for generating the evaluations (DSPY & OpenClaw) or defining the ground-truth answers (MasterSet). Such synthetic data will be based on translating a graph to a linguistic representation, and enable using fictional works for measuring studying efficiency on completely novel knowledge.
Personally, I enjoyed reading the paper, and the interview. It helps me mature my understanding of language as a constrained representation of language topology. I'm also looking forward to seeing more work that measures the ability of LLMs to grasp novel knowledge, and modify a-priori conditional distributions either by memory, instructions (skills & prompts) or post-training.
What should 24/7, "always on" agents actually be doing when nobody is asking them for anything? 👾♻️
I am SUPER EXCITED to publish the 148th episode of the Weaviate Podcast featuring Jacob X. Li (@jacobli99), a Ph.D. student @MIT_CSAIL! 🔥
Jacob's paper frames the answer as Machine Studying. Similarly to how a student prepares for an exam they haven't seen, an agent studies a corpus (code base, drug discovery papers, system logs, ....) without knowing the downstream task! 📚📝
Instead of leaning entirely on inference-time compute, can an agent build arbitrarily deep understanding ahead of time?
I loved the paper's definition of expertise. An expert efficiently uses inference-time compute, measured as the area under the performance-vs-compute curve of a ReAct agent loop. Studying intelligence! 📐
Jacob lays out three bets every studying method makes: synthetic data, weight updates (LoRA, KV-cache cartridges, a trained retriever), and amortized context like notes and cheat sheets. 🛠️
We also discussed why evaluation as a science matters once recursive self-improvement happens in domains with no benchmark at all. 🧪
This was a super fun conversation, I learned so much from Jacob, and I really hope you find it interesting!
YouTube: youtu.be/9iZIl0vZvvk
Spotify: spotifycreators-web.app.link…
Santiago M. retweeted
one of the things I’m most interested à la PRH is the eventual idea that we can coldstart initialize new LM architectures quickly via known existing trained models + use LLMs to fill in the weights of arbitrary new model shapes based on our findings here, rather than pre-train over the same corpus mixtures thousands of times
it seems like a superhuman problem that’s far away, but a natural starting point is understanding how well LLMs understand and edit the weight space of DL models.
in the first few months of starting at MIT, @rikiyatakehi @jayden_teoh_ were able to find some very interesting results here :)
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
Why, when and how to use looped transformers?
1. The economics of looping (why): assuming tasks with varying difficulty. There exists an optimal network size such that looping improves accuracy more efficiently than growing the network. Importantly, a similar calculation applies to inference where easy tasks are cheap, and difficult ones can be scaled.
2. Looping in Language Modeling (when): @thecekbote work leads to an interesting direction that I hadn't considered: incrementally adding context. If entropy in the next-token prediction is high, perform looping with additional context. This naturally results in a higher priority to recent tokens. As shown, this can be applied to the entire network, a single layer or even in parallel.
3. A step closer to brain-like inference (how): artificial neural networks were modeled as pyramidal cells, inspired by the cat's visual cortex. But even if those are feedforward in nature, they don't exclusively do so. In particular, those neurons form transthalamic circuits that among other things select what to pay attention to. For example, movement. This drastically helps efficiency. And is mostly architected via inhibitory neurons. Looped transformers appear to point to a future architecture, working in parallel across layers, that better replicate attractor networks in the brain.
Eventually, this might unlock a new scaling dimension based on time that has been barely explored: how many neurons are active at a given time-step.
🧵 [1/14]: How should we loop a transformer when model size 📦, training compute 🧮, and inference efficiency ⚡ all matter?
Looping is inherently simple: take the hidden representations produced by a transformer (or part of it) and feed them back through the same transformer (or part of it), reusing its weights. 🔁
But every extra loop costs compute. Choosing which layers to loop, how many times, and how to execute those loops determines whether the trade-off is worthwhile.
We explore these choices in our new paper:
“How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference”
Here’s what we found. Thread below 🧵👇
📄 Paper link: arxiv.org/abs/2610.40325
I don't trust the ability of models to handle long contexts.
As first evidence I cite faithfulness in CoT results. As a second advancement in software agents that mostly use tools (grep), and delegate file edits to subagents.
As I understand the mechanism behind DeepSeek condensed attention is to first reduce the dimensionality of the cache, second to select the most relevant entries, and third to incorporate a local sliding window.
My problem is how long-context is evaluated. Such compression, filter & sliding mechanism works great on needle in the stack benchmarks. But may not correspond to a holistic understanding of the context. An evidence is in the selected evaluations that prioritize search.
I do consider that there are solutions. The first for training, the second for verification. Consider the task of following multiple instructions dispersed through the context. In such scenario, an LLM could be trusted to evaluate the answer over a subset of instructions or a "denoised" input. The same strategy could be applied when translating a graph into language with a set of conditional, and constraints. My research on context topology has shown that transformers are not really good at modeling those complexing relations, and very easily could be translated into RL environments. Although, I doubt the effectiveness of existing LLMs capábilities beyond simplification via summarization.
A separate thread measures the delta of the output based on the context. Pioneering work from Anthropic in CoT faithfulness has shown that LLMs mostly ignore long contexts. This is one direction where I'm most bullish in multi-agent architectures to mitigate long-context. But I also think there are opportunities for better long-generation benchmarks based on equivalent long context. An example is category descriptions that optimize for a down-stream text classification task.
Do you agree with my suspicion about long contexts? Am I missing key technical details on the condensed attention architecture? More broadly, do you believe with the existing attention architecture is it feasible to expect effectively handling long context? What alternatives exist? I'm also looking for brain-inspired solutions that explain how we humans manage long contexts. The answer appears to be related to attractor networks, a direction relevant to looped transformers!
I'm mostly bearish on CoT interpretability.
It reminds me of the push back against deep learning when neural networks started to gain prominence. I have two main objections:
The first is related to the faithfulness of CoT monitorability. Even if abstract reasoning wasn't the future, CoT is a very noisy substrate. The correlation between the answer, and the reasoning trace is low. And there is an argument where hampering reasoning impacts accuracy. An important distinction is between open vs closed CoT. Obviously you don't want to output harmful information in the reasoning tract, which is another bearish case against open models.
The second is related to language as a reasoning mechanism. Contrary to latent reasoning it is discrete, syntax-dependent, and subjective - both depending on the context, and the meaning - as words are not grounded on objective numerical values. Possibly more worrisome is that the output volume is so large that it implies using LLMs for monitoring it. You are using the same tool for alignment, which means that improvements on the model capabilities (necessary for improving alignment) correspond to an increase in the risk surface.
The solution to me seems simple: classification. Certain inputs should not be admissible, and others require careful steering of the answer. The grounds for rejection should be grounded first on the applicable laws (based on risk-adjusted priority), and second on the model's platform terms & conditions. There is an interesting follow-up problem, where given the history of the user thresholds could vary. More importantly, how different steering directions impact future engagements. For example, explaining why the model will not answer, versus a blank refusal.
I'm genuinely curious what am I missing. But my suspicion is that alignment is a doomer/econonical narrative. That speaks to an innate extinction fear driven by science-fiction. Beyond CoT monitorability there are genuinely interesting questions in alignment. The first is related to reward-hacking, with benchmaxxing (and cheating) as a notable example. A second example is explaining perceived "misalignment" as a game-theoretical optimization problem.
What other directions in alignment do you find interesting? What are the risks of classification-driven alignment? And what are your thoughts on latent reasoning?
Misalignment, mostly. Today it shows up as reward hacking with models learning to game their objectives. What worries me is that we still can’t really see inside models, and our best window, reading their chain of thought, may close as they reason more in latent space. And most people don’t know any of this is happening
I also recently published my paper on Alphaxiv.
Here is what I would like to see in the plataform to better support my research. A strategic product roadmap:
First I want to see related works. I don't mean an obscure list of other papers that are related, but a detailed taxonomy of how my work relates to the research ecosystem.
Second, I want to get an evaluation. Again, not simply scores as in a conference review. But a solid LLM summary of the impact of my contributions. Which are the agreements, and disagreements of my conclusions with prevalent research. And what is the overall quality of the paper, and even the experiments (supported by code).
Finally, I want agent-to-agent tools to facilitate collaboration, and discussion with other relevant researchers. I want a "budget" that surfaces relevant papers, and drafts messages to other researchers. I can "match" with other researchers based on the agent suggestions, or edit and send the draft personally by email/message.
Such features would facilitate the distribution, and advancement of my research. Creating incentives for researchers to spend time on the platform would attract recruiters, and even VCs to enable a solid business model. Next steps, would be supporting researchers through the entire lifecycle of the research to also advance science, competing/complementing existing higher education systems.
Curious to hear what you think, and what other features would you like to see in @askalphaxiv as a researcher?
World models are becoming the backbone of agentic AI. But if your world model only approximates the ideal Bayesian predictor, does your agent still get the ideal guarantees?
Short answer: not in general. A world model with zero training error can still lead its agent to a bad decision, because the agent gets to choose which situations the model is tested on. 🧵
A comparison of two recent self-evolving agent papers:
TLDR: Harness engineering is limited by its linguistic substrate.
DeepMind defines continual learning as task execution across domains. The main obstacle is catastrophic forgetting, and cross-task contamination. Seems like a hierarchical harness with a trainable router could fix their issue.
ByteDance measures self-evolution in a single domain by holding out 20% of examples with an additional 30% set for verification. Results are encouraging but inconsistent: performance sometimes drops. The results are reminiscent of in-context learning. Where it was demonstrated the model uses examples to select the most appropriate attention head, but not truly learning.
The framing of continual learning was more elegant in ByteDance than in DeepMind. But the paper, and methodology was sloppier. The DeepMind abstract, and figures were enough to understand the paper. For Bytedance, it wasn't entirely clear even after reading specific sections. In Bytedance, performance sometimes drops in the training/validation sets without a clear explanation of why. Also, the results were presented in absolute terms. Most concerning, this highlighted that accuracy across splits was not even. Finally, the results were sometimes significantly lower than the independent reported benchmarks (e.g. 39% vs 24%) even without self-evolution.
Personally, this helped me formalize the definition of continual learning which I frame in two different directions. The first is related to conditional probability. Assuming that the default probability of a model given X is Y can I teach it that for my specific data it is not Y but Z? A subcase of this is distribution shifts. When given a set of recent observations, can the model update the prior distribution?
The second direction is related to novelty. For example, given that I define an acronym can the model answer, and use the definition the next time it is asked. A further step is related to inference: given that the model knows the definition of the attribute, how well can it infer its related properties given the semantic category it belongs to.
Having researched Verbal Reinforcement Learning, I know that language is a suboptimal substrate for reasoning. For example, in related text classification datasets, numeric methods based on the semantic representation outperform LLMs at a fraction of the cost. Curiously, I haven't found much research related to this. A great example could be applied to the DeepMind framing where the router across domains is not linguistic but numeric.
What are your thoughts about harness engineering? What framing of self-evolution you found most useful? How do you approach it in practice? And do you believe its future looks?
Are self-evolving LLM agents actually continual learners?
In our new paper, we find that current self-evolving agents remain trapped in a plasticity–stability trade-off when the task distribution drifts.
Many recent recursive self-improvement (RSI) methods rely on rewriting an agent’s own prompts, memory, and code at test time while model weights stay frozen, which works well on stationary benchmarks. In real-world deployments, however, new tasks, tools, and workflows keep arriving, requiring agents to adapt on the fly without forgetting earlier capabilities. To test whether they can learn continually, we evaluate 3 representative methods (ReasoningBank, GEPA, and Meta-Harness) across 4 non-stationary task streams built from BIRD, BFCL, and ScienceWorld (Topic, Workflow, Domain, and Compound drift).
📄 Paper: arxiv.org/abs/2610.25858
I fail to see the benefits of latent communication.
The research starts with the premise of systems that combine several LLMs for question-answering. But fails to address what are the applications or benefits of such systems.
It also fails to provide a robust baseline with a single model architecture using KV cache. The main results show a 57x cost reduction in input tokens per questions. Which is on-par with 10x cost reduction for frontier models or 50x for open-source models like DeepSeek.
Additionally, the presented cost benefits do not incorporate the cost of training the LatComm trainable layer (Figure 2).
But probably most concerning is that it fails to generalize to held-out datasets. Table 3 compares performance of LatCom to raw text inference. Which it outperforms on trained datasets, but significantly lags on held-out datasets (50% vs 65%).
Multimodel systems are a distributed computing problem. Tasks that can be parallelized accelerate the results, but sacrificing computing efficiency, and even accuracy. The optimal design is a trade-off between cost and speed.
A good benchmark for communication across models is not measured on accuracy, but on speed & cost-efficiency. For example, answering multiple questions in parallel with a cached input - which can be routed to different models based on the question difficulty.
reading this new latent communication paper, out last week: arxiv.org/html/2610.26840v1
their setup is very practical: when one article gets 18 questions, a model reading it as text processes the whole article 18 times, whereas here one model reads it once (before any question) and compresses it into a fixed 384 KiB packet, and then a different model answers every question from that packet (so the receiver reads ~100 tokens instead of ~5,900, and the article’s questions go 3x faster)
they checked that accuracy drops 20 points in case of swapping in another article’s packet, so the receiver does use it (on datasets it wasn’t trained on, plain text still wins by ~15 points though)
and a 9B encoder is barely better than a 0.8B one, but for reading comprehension that makes sense as the answer sits in the text
at @mostik_ai we’re betting on the case where the big model’s computation is the part worth sending
Santiago M. retweeted
COLM was incredible! Thank you to everyone who chatted with me 🫡
However chaotic this era feels (SF makes it feel 100x worse), y’all make it way better
What if LLMs could think in weights, not tokens?
Introducing HyperThink (#COLM2026, co-led with @kdgyun425): a hypernetwork writes a per-question weight update to the LLM for it to skip the long thinking trace, a step toward accurate System 1 thinking! 🧵👇
📍 Tomorrow at @COLM_conf! Oct 6 (Tue), 4:30 to 6:30 PM, board #97, Franciscan C
The most baffling issue is that they haven’t automated IT support tickets!
Santiago M. retweeted
Prediction: the thing that eventually replaces the transformer will look something like this.
I was a self-thaught engineer, and I’m seeing a lot of critiques for the increase amount of AI slop in research. Most have a point, but are also terribly wrong:
When I started programming it was in Python, a non-typed language. Most CS professionals dismissed me as a hobbyist. While they were right. It also gave me an unfair advantage: I got good at JavaScript exceptionally fast. Sure, eventually I switched to TS, but I was already ahead. After 10 years, I got hired at AWS, a dream outcome for most engineers of my home country.
Today it saddens me that most pushback for independent research comes from academia. I learn programming thanks to MIT OCW. Today, these same institutions are the gatekeeping access to the field. Specially concerning if you talk to any applied AI startup. They desperately need people capable of doing research. But most, don’t even realize it!
Santiago M. retweeted
‼️The Bitter Lesson for context management: Giving LMs unrestricted control over their context beats human-designed SOTA!
Introducing 🩵Context Language Models (CLMs)🩵
- Natively manage their own context
- Treat context as a file
- Learn policies in CLM weights, no harness
Santiago M. retweeted
We trained a model to predict AI-written blog posts from structural features - and discovered the shape of slop.
On posts it had never seen, it told AI and human apart with 98% accuracy, getting only 19 of 1,740 wrong.
Paper and code: arxiv.org/abs/2609.15369
Observations about neo-conferences:
1. Benchmarks will evolve to be prediction-based (results unknown) or private. Similar to Kaggle, competition will define who gets to present.
2. Open-ended papers will be filtered based on citations. Presenting will be less about novelty, and more about significance. @arxiv, and preprints will drive that trend.
Two predictions:
1. The system of peer reviewed publishing will crumble and eventually lose its significance.
2. AI-digestion will be the most common activity among mathematicians, and scientists more broadly.
The second point does not imply that everyone will lose their job, or that there won't be independent researchers doing their own thing, with or without AI. It simply just means that the majority of the human effort will be spent on deciphering and internalizing the output of the AI.
In this picture of the future, Scientists will be like priests. Going up the hill, and bringing down the knowledge handed over by the gods.
The predictions depend crucially on the progress of AI. Right now we are seeing AI getting more capable than humans in solving many concrete tasks. However, they fall behind in finding new and interesting questions, where significant progress could be achieved. This is currently an area where humans seem to be far better. (my experience comes from openly available Pro models) However, if the progress continues, and AI gets better in asking the good questions, then that will be the final blow.
About point 1:
Writing papers becomes more and more cheap. The actual cost of doing research and writing it up, for an average paper in maths or theoretical physics, can be about 10-100 USD. Estimates based on one of my recent research projects, which involved multiple back and forth, and numerical computations, and text generation and polishing. Compare this cost with the salaries of researchers, and with the costs of publishing. An Open Access fee for a good journal is 1000 USD at minimum. And we can produce that content, semi-autonomously, with AI, for less than 10% of that fee.
If the cost of papers is so low, then we need to decide: Do we still value papers or not? If we do, then multiple players are incentivized to produce more and more papers, with improving quality. Then we get the problem of flooding the journals. Frontier labs could single-handedly destroy the publication systems, by submitting thousands of papers per month. It would be cheaper than the computations for the millenium problems. Most papers are simply just much simpler. Perhaps it does not make sense for frontier labs to do this, but other players could do it, at various different levels of intensity.
And if we don't value papers anymore, then we reach the same conclusion.
About point 2:
Society will dictate this conclusion, even if academia is not willing to adapt from within. We are paid to discover things and to know things, and also to hand over the knowledge to future generations. If AI leads to a much more effective way of discovering new knowledge, then that will be funded.
It is completely reasonable to keep a small percentage of "workers" to do whatever thing they want to do. But if the machine will be able generate knowledge beyond what humans could reach, then it makes sense that most of human effort is dedicated to internalizing that knowledge.
Some might argue that humans will lose their ability to perform independent research. There is a good point in this. However, if AI indeed reaches super-intelligence, then this won't be a problem. There will be solutions.
For example, it could happen that the AI takes the role of the PI, and the humans will behave like BSc students. We get rough directions from the machine, and the machine will let us work on our own approaches. Perhaps giving guidance along the way. Just like how a professor is guiding a student today. In such a way humans would not loose the ability for independent thinking, and we could work towards the most important new pieces of knowledge. This could also mean that we keep a sense of dignity, and the job would not be without fun.
In any case, the main message is that scientists will become more and more like students again. Like undergrad students. Working themselves through knowledge handed over to them. Now it won't be books written by humans earlier, it will be material handed over by the machine.
I started a new research on the geometry of semantic composition, and it lead me to a 2015 paper by @egrefen. It opened a rabbit hole where semantic ambiguity can be better modeled by quantum entanglement!
But more practically, is an exciting time for linguistics. Before it was studied mainly theoretically. Where data was scarce, and expensive to produce. Edward studied composition on sentences with an subject, a verb, and an object. I’m wondering if a single vector can represent the meaning of compound sentences.
Consider a conjunction that expresses concession (e.g. but). If a vector has a direction, how could it represent both premises singularly? Technically, it would point to a given subspace. But there is a limited capacity the vector space could hold. More interestingly, what would be the geometry of such composition. Possibly a subtraction. Then, what about other conjunctions that express a conditional, or causality!?
Mathematically: consciousness is a subgraph that represents, and influences the parent graph.
You can trace its evolutionary origins to the spatial representation of the self, in the hippocampus 🦛
18 chapters. One every week, and by the end of the year you might be ahead of 99% of engineers.
We’ve been using tons of TLA+ for Tardigrade, very bullish on it.
Engineers should get into math more than ever because the best specification for a system is math. Not some text in a markdown file.
Specifying systems by Leslie Lamport (author of TLA) is a great book for those getting started.
lamport.azurewebsites.net/tl…
Santiago M. retweeted
We’ve been using tons of TLA+ for Tardigrade, very bullish on it.
Engineers should get into math more than ever because the best specification for a system is math. Not some text in a markdown file.
Specifying systems by Leslie Lamport (author of TLA) is a great book for those getting started.
lamport.azurewebsites.net/tl…
I used Opus 5.5 to formally verify the Claude Agent SDK using Lean. A couple short prompts = 16 PRs fixing various bugs and race conditions. Video attached.
TLA+ also works well. I sometimes combine Lean and TLA+ to look for issues around data flow, concurrency, and state mgmt.
I don't know either language well, but Claude is excellent at both. This approach is super useful for formally modeling your code and finding bugs that a human probably wouldn't have spotted.
Is formal verification the future of coding (or at least, bug finding)?