rogue

GLM 5.2 is ~5x cheaper than the frontier models and matched them on many evals. So we tried swapping it into one of our production pipelines... it failed 68% of real requests vs. 16% for Sonnet, at ~8x the cost per success. Is it a model weakness? Yes, a targeted one. GLM has basically closed the gap on reasoning and coding. Where it still lags is strict structured output, especially cross-field invariants, rules like: “If field A is ‘unspecified,’ field B must be null.” Frontier models (GPT/Claude) are noticeably better at holding these implicit multi-field constraints under structured-output pressure - it's a subtle, heavily-RLHF'd skill. Two mechanisms compound: 1. Types and enums Frontier providers use constrained decoding. At each token, the sampler is masked so the model literally cannot emit an invalid type or enum value. Validity is guaranteed. Open-weight routes often skip this, so you get "false" as a string instead of a boolean. 2. Cross-field rules and length caps No JSON-schema grammar can enforce “if unspecified then date is null” or “max 200 chars.” The model has to hold those by reasoning, and frontier lab models are just better at it. GLM reliably gets neither on a picky schema. Prose demos and benchmarks hide this. You only see it when you test on your real, strict schemas. Sometimes the cheap model isn't cheap.