Last night Railway, our hosting provider, told us someone had read its customers' environment variables, some of them decrypted. Then it paused every deploy. We run AI agents for several small businesses there. Here's what the night looked like, and what held.
Same day: a PHP bug rated 10 out of 10, a Node security release, a compromised Python package. We'd built the Node fix into our image that afternoon. It's still sitting there, because nothing can deploy right now.
What was in those variables: our own platform keys (model providers, payments, email, integrations) and each instance's access tokens. What wasn't: the keys a business connects for its own tools. By design, those live encrypted inside its own instance.
Each business gets its own instance with its own tokens. The agent runtime isn't reachable from the internet, and our services talk over a private network. We chose this from day one. It cost us more complexity and more build time, and last night it paid for itself.
Then the work. More than a dozen providers, 24 services. Every key created, checked and saved without ever printing it. We found one public door we no longer needed and closed it. Model usage and payments looked normal the whole night.
It still took us a few hours, and we knew where every key lived. A small business, or a "we can build our own tool" app, that wired its own keys into its own agents would have to start by finding them, in the middle of the night, with deploys frozen.
Oct 22, 2026 · 12:45 AM UTC
It's one of the things businesses value us for most at Orchestra, and it rarely shows. Building an agent gets easier every month. Knowing where every key lives and changing them all overnight without breaking the work is what pays them back many times over.
If you're on @Railway too: create new keys now, keep the old ones on until deploys come back, then redeploy and revoke. And check what can still reach your database from the internet.
#Railway #infosec #cybersecurity #AIagents #buildinpublic