Last night Railway, our hosting provider, told us someone had read its customers' environment variables, some of them decrypted. Then it paused every deploy. We run AI agents for several small businesses there. Here's what the night looked like, and what held.
Same day: a PHP bug rated 10 out of 10, a Node security release, a compromised Python package. We'd built the Node fix into our image that afternoon. It's still sitting there, because nothing can deploy right now.
What was in those variables: our own platform keys (model providers, payments, email, integrations) and each instance's access tokens. What wasn't: the keys a business connects for its own tools. By design, those live encrypted inside its own instance.
Each business gets its own instance with its own tokens. The agent runtime isn't reachable from the internet, and our services talk over a private network. We chose this from day one. It cost us more complexity and more build time, and last night it paid for itself.
Then the work. More than a dozen providers, 24 services. Every key created, checked and saved without ever printing it. We found one public door we no longer needed and closed it. Model usage and payments looked normal the whole night.
The annoying part: you can't finish. New keys only go live on a deploy, and deploys are paused. Revoke the old ones now and live services go down. So it's all staged, old keys still on, and we're waiting for the green light.
It still took us a few hours, and we knew where every key lived. A small business, or a "we can build our own tool" app, that wired its own keys into its own agents would have to start by finding them, in the middle of the night, with deploys frozen.
If you're on @Railway too: create new keys now, keep the old ones on until deploys come back, then redeploy and revoke. And check what can still reach your database from the internet.
#Railway #infosec #cybersecurity #AIagents #buildinpublic
Oct 22, 2026 · 12:49 AM UTC
