𝗖𝘂𝘁𝘁𝗶𝗻𝗴 𝗮𝗻 𝗔𝗜 𝗯𝗶𝗹𝗹 𝘂𝘀𝘂𝗮𝗹𝗹𝘆 𝘀𝘁𝗮𝗿𝘁𝘀 𝘄𝗶𝘁𝗵 𝘁𝗼𝗸𝗲𝗻 𝗰𝗮𝗽𝘀 𝗮𝗻𝗱 𝗰𝗵𝗲𝗮𝗽𝗲𝗿 𝗺𝗼𝗱𝗲𝗹𝘀, 𝘄𝗵𝗶𝗰𝗵 𝗹𝗼𝘄𝗲𝗿𝘀 𝘁𝗵𝗲 𝗽𝗿𝗼𝗱𝘂𝗰𝘁 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗮𝗹𝗼𝗻𝗴 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲 𝘀𝗽𝗲𝗻𝗱. There is a different place to look at first. In most AI applications a large share of the cost comes from the work the system has already done and is paying for again: ➡️ The same question arrives in different words, so an exact-match cache never sees it and the model answers it from scratch every time ➡️ The agent drops context between turns and sessions, so it re-retrieves what it had and asks the user for what it was already told ➡️ All of that state, cache entries, session history, memory vectors, features, has to stay fast, and memory is the most expensive thing in the rack None of those are model problems. They are caching, memory and storage decisions, which is the part of an AI system that gets the least design attention and ramps up the spend significantly. I have been reading through how Redis approaches this. They offer three mechanisms that tackle the above problems in a single platform: ✅ Semantic caching: embed the incoming prompt, search prior prompts by meaning, and serve the stored response when it is close enough. Redis reports cache hits up to 15x faster and token spend down by up to 70% ✅ Agent memory: live session state with a TTL, plus facts extracted out of conversations into long-term memory as vectors, with the extraction running in the background instead of in the request (hot) path ✅ Flex: one database moving both keys and values between RAM and NVMe SSD storage tiers. You can adjust RAM ratio 10% and 50% per workload. Redis states this costs up to 80% less per gigabyte than running all-RAM, with no code change The part I find interesting is that memory architecture is usually treated as a quality feature, something that makes the agent feel coherent. It is also a cost lever - an agent that forgets incurs the price twice for the same context. 👉 Here is their cost optimisation guide: fandf.co/3TVKDxt Huge shoutout to @Redisinc for partnering on this post! What share of your production traffic is a repeat of a question you have already answered? I rarely run into a team that is actually measuring this! 👇