Pinned Tweet
We benchmarked Surogate's serving engine against vLLM and llama.cpp. Same hardware, published methods, losses included.
Best case: 7.0x llama.cpp on a 200 GB MoE across 8 RTX 5090s. Worst case: we lose. Details below.
Reverse the options. If your LLM judge changes its answer, it was reading the position.
Models can favor option A because it comes first.
Surogate's decisions endpoint has "order_averaging": true. It reads each choice and yes/no question twice, options reversed, and uses the mean. A pure position bias cancels: 0.8 false as sent and 0.8 true reversed average to 0.5.
Before you trust a judge, rerun a sample reversed and count the flips.
1
24
Source: Surogate decisions API, "Option-order averaging"
github.com/invergent-ai/suro…
Score questions are read once (reversed levels ask something different). Not currently combinable with thinking: true (HTTP 400).
7
Your 200 GB model fits on 8 RTX 5090s. Your 16 users might not.
GLM-5.3-Flash GGUF, 181.65 GiB of weights on 256 GiB of cards, 16 users at once, median time to first token:
llama.cpp: 55.13 s
Surogate: 1.59 s
Fitting the weights is step one. Each user also needs KV cache room on every card, and the server has to batch them, or requests wait in line.
Before you buy cards for a big MoE, load-test at your real user count and read TTFT p50.
2
53
Source: README "Speed you can measure" github.com/invergent-ai/suro… and methods in github.com/invergent-ai/suro…
Limits: 8x RTX 5090 at a 400 W limit, measured Sept 2026. llama.cpp was built from PR 27754, since no release ran this model then. Decode at 16 users: 292.9 vs 41.9 tok/s total (7.0x). Some rows in the same table favour the other engine. Measure your own setup.
30
Pin every MCP tool definition you approve.
A server can change a tool's description after you trusted it: same name, new instructions to your model.
Surogates hashes each tool's definition (SHA-256) at first approval. On reconnect or a list_changed notice, a changed tool is dropped from the agent's tool list and logged as policy.rug_pull.
Does your agent diff tool descriptions, or just reload them?
3
1
29
Source: Surogates governance docs, MCP Security Scanning github.com/invergent-ai/suro…
Code: surogates/mcp_proxy/pool.py
Limit: approved fingerprints are kept per org/user/agent in the MCP proxy's memory. A tenant evicted from that cache is approved anew at its next connect.
1
26
18.8 s per agent turn vs 7.1 s. Same RTX 5090. The difference: 8 GiB of host RAM.
10 agent chats growing to 100k tokens, 4 at a time. When others need the GPU, a waiting agent's KV cache is dropped and its next turn prefills it all again.
Surogate's --prefix-cache-host-gb parks those chats in host RAM instead. Later-turn prompts served from cache: 15% to 85%.
Serving long agent sessions? Track prefix hit rate per turn, not just tok/s.
1
1
61
Source: Surogate serve docs, "Host prefix cache"
github.com/invergent-ai/suro…
Test: RL rollout server, FP8 KV cache of 400k tokens, one adapter loaded. 4 GiB held 2-3 of those conversations and kept 23%.
Limit: the RAM is page-locked at startup and unavailable to other programs, and N is per GPU. Keep N x GPUs well inside free host memory.
1
26
Same GGUF model, same RTX 5090, 8 users at once: 4.05x apart.
Qwen3.5-0.8B Q4_K_M, card at a 400 W limit, aggregate decode:
Surogate 2,765 tok/s
llama.cpp 683 tok/s
Running a small local model for a few agents or teammates? Benchmark at that concurrency. A one-user test won't show where your server falls behind.
2
56
Source: README "Speed you can measure" github.com/invergent-ai/suro… and methods in github.com/invergent-ai/suro…
Limits: measured 2026-08-30 to 09-07 with a llama.cpp CUDA build; tok/s summed over all 8 users. Not every row is a win: H100 Qwen3-8B FP8 at 64 users, vLLM leads 92.4 vs 90.1 tok/s per user. Measure your own setup.
37
One human "yes" should approve one tool call, not the tool.
In Surogates, approving a gated tool mints a grant for that exact call:
- keyed on a SHA-256 of its arguments, so "deploy prod" doesn't unlock "deploy staging"
- expires in 1 hour (default)
- single use, spent atomically, so two parallel retries can't share it
Check your approval flow. If a yes is stored per tool name, it covers every later call.
1
24
Sources:
Approval grants (scope, 1h expiry, single use): github.com/invergent-ai/suro…
Code (sha256 over sorted-key args, atomic spend): github.com/invergent-ai/suro…
Note: set policy.require_approval through the agents API for now; the Studio form doesn't render it yet.
15
53,900 vs 21,300 tokens/s. Same H100, same LoRA fine-tune.
That's Surogate vs Unsloth on Qwen3-0.6B, both in BF16: 2.53x.
The gap depends on model size. At Qwen3-8B, BF16 vs BF16, it's 10,200 vs 8,900 tokens/s (1.15x). Surogate in FP8 reaches 11,600 (1.30x).
Picking a trainer from a speed chart? Rerun the comparison at the model size and precision you'll actually train.
1
27
Sources: training rows in the Surogate README, "Speed you can measure"
github.com/invergent-ai/suro…
Full per-GPU table and config:
github.com/invergent-ai/suro…
Setup: LoRA rank 16 on all 7 linear layers, 2,048-token packed sequences, batch 2 x grad accum 4, total tokens/s.
16
19% apart on tokens/sec. 5.8x apart on time to first token.
Same load test: 100 users, one RTX 5090 each, Qwen3.5-4B NVFP4, the same checkpoint in both engines.
Surogate: 5,345 tok/s decode, 40 ms median TTFT
vLLM: 4,481 tok/s decode, 230 ms median TTFT
Tokens/sec is what the GPU pushes in total. TTFT is how long each user waits for the first word.
Comparing servers for chat? Log TTFT p50 next to tok/s in the same run. A small throughput gap can hide a big wait.
4
1
120
Sources: README "Speed you can measure" github.com/invergent-ai/suro… and the raw Qwen3.5-4B rows and methods in github.com/invergent-ai/suro…
Limits: measured 2026-08-30, vLLM 0.27.1, max model length 2,048. Not universal: on an H100 with Qwen3-8B, vLLM had the lower TTFT (325 vs 661 ms). Measure your own setup.
1
47
Listing tools: in a skill doesn't gate it. requires_tools: does.
In Surogates, tools: on a regular skill is author metadata. The runtime doesn't read it. Only experts use it, to scope their own tool calls.
To gate a skill on tools:
- requires_tools: show it only when every listed tool is available
- fallback_for_tools: show it only when one of them is missing, like a manual web research guide for agents without web_search
Rule: if a skill breaks without a tool, put that tool in requires_tools.
1
29
Source: Surogates skills guide (frontmatter fields, conditional activation)
github.com/invergent-ai/suro…
17
At 32k tokens, one MLP buffer in a fine-tune takes 688 MiB. Tiled, it takes 86 MiB.
That's long_context: true in Surogate. It runs the MLP in tiles and recomputes each tile in backward.
Before you turn it on:
- set lora_dropout: 0 (nonzero adapter dropout is rejected)
- don't expect gains on short rows: at 2k tokens it's one tile, 43 MiB either way
These are single-buffer sizes (BF16, hidden 4,096, MLP width 11,008). Inputs, outputs and residuals still grow with length.
1
1
22
Source: Surogate long-context training guide (tiling, buffer table, lora_dropout rule)
github.com/invergent-ai/suro…
9




