Using LLMs (Claude) to spin up a single-use testing harness is super easy. Prepare the dataset, tell it the metric and ask for an HTML report, done.
Got from $0.8/1k to $0.0ish/1k with ~10% less accuracy. I may be able to ensemble cheaper models keeping lower costs.

ALT Table showing a model each row and columns being the accuracy % on different datasets