I forgot my mantra
Charlotte
Joined June 2008
- Tweets574
- Following1.2K
- Followers230
- Likes1.5K
Kyle Little retweeted
Modern "memory" benchmarks are saturated since they measure recall of facts -- in the coding harness / bash loop era, recall is solved, while continual learning in token space remains unsolved.
FrontierMemory is a new benchmark designed to actually measure self-improvement through memory.
It's built on a simple premise - if AI-generated memory is *good*, it should actually improve downstream task performance. An agent with good memory should be more efficient (lower cost), or it should be more capable (higher score) than memory-less agents.
When you measure memory this way, it turns out there's a very wide spread in frontier model capabilities. Opus 5.5 and GLM-5.3 for example are *significantly* better at generated useful memories than GPT-6.1 Sol.
FrontierMemory can measure overall harnesses as well: and it also turns out that different implementations of AI dreaming vary wildly in performance. For example, it may surprise you that @PrimeIntellect Agent's memory system works significantly better than Hermes Agent's.
FrontierMemory is a continuous benchmark - we will be adding more tasks and revising them over time, and will be adding extremely long-horizon RSI-style tasks in our next release.
If you're building an AI system with memory, please reach out / DM! Likewise if you're building models or harnesses and would like to submit.
My Sunday night token burning project before reset: letta-durable. A pi-durable version of a local letta agent. The harness has a MemFS-inspired memory and skills repo, runs on pi-durable, and has 3 clients: pi, ACP, and Pi-pocket. Fable damn near one-shotted it.
Burn your tokens on your wildest ideas, folks.
Ok this is seriously cool
fully functional opencode 2.0 inside of 𝕏
x-opencode.exe.xyz/play
This is a tough space to get alpha out of, in my opinion. There is so much noise in our sessions, an infinite number of variables in your context, and then trying to tie specific evidence in your traces to subsequent outcomes is exponentially more difficult.
I don’t have the answers. But I have been preserving my traces since at least the beginning of this year. Maybe it will come in handy one day, but also maybe not.
poteto is right. I hadn’t thought of typed and lint rules this way before, but I suppose it is indeed a form of memory. Wisdom here.
something that keeps bothering me is how some people see slop as something that is happening to them, like they're some kind of helpless bystander powerless to do anything about it
excuse me but you are a software engineer, you literally learned how to make sand think and make computers do your bidding?! and now you can't do anything about it? please
the 2 things you should know to cut through all the noise:
1. you need to give your agent a way to verify its own work (i just posted about this so won't rehash)
2. your codebase is memory
the 2nd is not new either, we have always had lint rules, compilers with good diagnostics and strong type systems, LSPs, etc to help us write better code. there is no difference when agents write the code instead, you just need more of them, and stronger ones.
forget rules and skills. you can add those later.
yes, your codebase is a form of memory to agents! agents love extending what already exists. so every nit and every annoyance you've ever seen, every mistake and bad pattern you observe from agents, needs to turn into a constraint, or the agents will just keep repeating those same mistakes like a virus in your code. then you keep observing and repeating this process until your codebase is locked down enough that even agents with little context can one shot a high quality PR.
an even better direction is to refactor/rewrite your code such that agent anti-patterns become categorically impossible. use type systems to your advantage, use libraries/frameworks that reduce the architectural search space, or heck, create your own framework like we did for grok bot and design high quality primitives and well thought out APIs
Happy to see @t3dotcodes supporting custom ACP providers. I connected my letta agent up and now I have a top notch GUI and mobile app from T3, plus the brains and persistent state of my letta agent wherever I go. It's a very nice app.
I think they will eventually rug OAuth once they get enough platforms on Sign In with ChatGPT. Once that happens they can gatekeep by deciding who gets access to their auth solution.
My favorite feature of @t3dotcodes is being able to resume or create a new project anywhere on my machine remotely. No fuss.
Claude Code, by contrast, wants you to start the session in the specific directory before you walk away from your machine. Like a caveman.
I'm pretty sure a solid 10% of my day is spent logging into sites, applications, password managers, and juggling OTP over sms, Google Authenticator, or other. There's got to be a better way.
I need an agent-friendly and open source PIM, so I stood up an instance of @uno_pim yesterday. I built a cli last night for it using @EffectTS_ v4 (link to repo is below). My agents seem to like the cli, and today i'm wiring it in to my enrichment workflow.
Here's the human and agent-friendly cli I built for @uno_pim : github.com/klittle32/upim. Install it with npm install @ kylelittle/upim
Yes. This is why usage doesn’t count against your quota. They want to lock you in or at least get you more immersed in their ecosystem.
The labs know that models will eventually become commodities, even more so than they are already. Long term, they want you in their ecosystem as a customer and to build a moat with that.
Concerning: OpenAI Dots have a black box memory system. You can’t view, modify, or delete individual memories.
Implications:
- you can’t inspect/mine exactly what a Dot has learned about you or your company
- if it learns the wrong thing, you can’t directly correct its memory
- the only way to delete that context is to delete the Dot
Tough dot to swallow for companies.
Ok this sounds convenient
One more launch today: the new MCP Events spec is now supported in ChatGPT 🎉
MCP servers can now push real-time updates (messages, comments, status changes, whatever) to ChatGPT, triggering automations for your users 🤯
It’s easy to add to your MCP server, too! Just point Codex at the docs and let it rip:
developers.openai.com/plugin…
