Denshin / Blog / AI
Choosing an LLM: a decision framework, not a leaderboard
Public leaderboards are a weak signal for your specific workload. A practical framework for choosing a model: the dimensions that actually decide it, a weighted scorecard, a two week bake off that fits in a sprint, and an honest look at whether multi-provider support is worth its complexity.
Denshin Engineering · Engineering Team · 27 August 2026 · 7 min read
Someone sends the leaderboard link, a name is at the top, and that becomes the decision. It is a bad way to choose, not because the numbers are fake but because they answer a different question from yours. This is the process we use instead: the dimensions that actually decide a model choice, a weighted scorecard you can fill in with your own numbers, and a two week bake off that ends with evidence rather than opinion.
Why public leaderboards are a weak signal for your workload
A leaderboard measures average performance on a fixed set of tasks that are almost certainly not your tasks, scored in a way that is almost certainly not how you judge a good answer. It is useful for a rough sense of which generation a model belongs to and useless for deciding whether it will handle your support tickets, your invoices or your legal clauses. Treat it as a shortlist filter, never as the decision.
Three specific problems. Benchmark contamination: public test sets leak into training data over time, so scores drift upward without capability moving with them, and a model that has seen the answers looks better than it is. Task mismatch: your job may be extracting fields from messy PDFs in Indian English, which no public suite measures. Averaging: a single headline score hides the one dimension you care about, so a model that is stronger overall can still be worse at the specific thing your product does all day. The same reading discipline applies to launch posts, which is why we treat model announcements and rumours as input to a test, not a conclusion.
The dimensions that actually decide it
Task fit, measured on your own eval set
This dominates everything else. Build a set of cases from real traffic, including the ugly ones, with graders that reflect your definition of correct. A hundred well chosen cases will tell you more than every public benchmark combined. If you have no eval set, that is the first piece of work, not an optional extra, and evals that catch real failures covers how to build one that is worth trusting.
Latency and streaming behaviour
Measure time to first token and tokens per second separately, at your prompt length, from your region, at the hour your users are actually awake. For a chat surface, time to first token drives perceived speed. For a batch job, total throughput is what matters and first token time is irrelevant. Also check how the model behaves under load and whether latency is stable or spiky, because a good median with a bad tail is a support problem.
Context window and effective use of it
The advertised window is a ceiling, not a promise of quality. What matters is whether accuracy holds when the window is full, whether instructions early in a long prompt still get followed, and how cost and latency scale as you fill it. Test at the length you actually intend to use, not at the maximum.
Structured output and tool calling reliability
If the model feeds another system, the question is not whether it can produce JSON but how often it produces valid JSON matching your schema, over hundreds of runs, including on inputs designed to confuse it. Measure schema violation rate, tool selection accuracy and argument correctness. This is usually where a model that looked equal on prose turns out not to be, and it is the practical heart of structured outputs and tool calling.
Cost at your real token profile
Per token prices are only half the arithmetic. What decides your bill is your actual input to output ratio, how much of the prompt is cacheable and reused, how many retries you make, and whether the model is verbose enough to inflate output tokens on every call. A cheaper model that needs two attempts is not cheaper. Compute cost per successful task, not cost per thousand tokens, and check current pricing on the vendor's own page when you do it, because published prices change and any figure quoted in an article ages badly.
Rate limits, quotas and regional availability
Find out your starting limits, how they are raised, and how quickly. Check which regions serve the model and whether that satisfies your data residency requirements. A model that is unavailable in the region you must run in is not a candidate, however good it scores.
Data handling and retention terms
Read the actual terms for training use, retention periods, subprocessors and regional processing. If you serve regulated clients, you need the commitment in the contract, not in a marketing FAQ. As of writing in August 2026 these terms vary by vendor and by plan, and they get revised, so verify them at decision time.
Provider stability and deprecation policy
How much notice does the vendor give before retiring a model, and what is their track record on honouring it? Is there a status page with real history? How stable have identifiers and defaults been? This dimension is invisible in a bake off and expensive later.
A weighted scorecard
Write the weights down before you see any results, because deciding what matters after the numbers arrive is how a team talks itself into a preference. Weights are yours to set, the ones below are a starting point for a customer facing feature with tool calls.
| Dimension | Example weight | How to measure it |
|---|
| Task fit on your eval set | 35 | Pass rate and grader scores on real cases |
| Structured output and tool reliability | 15 | Schema violation rate over hundreds of runs |
| Latency | 15 | Time to first token and p95 total, from your region |
| Cost per successful task | 15 | Real token profile including retries |
| Data terms and residency | 10 | Pass or fail against your contractual needs |
| Limits and availability | 5 | Starting quota, path to increase, regions |
| Provider stability | 5 | Deprecation notice period and track record |
Score each candidate one to five per dimension, multiply, total. The number is not the decision, it is a way to make the trade off explicit and to notice when a preference is being smuggled in under a rounding.
A two week bake off that fits in a real sprint
- Days one and two: agree the weights, write down the decision rule, and shortlist two or three candidates. More than three wastes a week.
- Days three to five: assemble or refresh the eval set from real traffic, including the failure cases that prompted this exercise. Anonymise anything sensitive.
- Days six and seven: put each candidate behind your provider interface so the only difference between runs is config, not code.
- Days eight to ten: run the evals, capture latency percentiles from your own infrastructure, and record token usage per case rather than in aggregate.
- Days eleven and twelve: shadow the leading candidate against live traffic without showing users its output, and read fifty responses by hand. Numbers miss tone, formatting and refusal behaviour.
- Days thirteen and fourteen: fill the scorecard, write a one page decision record with the date and the version tested, and ship behind a flag.
Keep prompts constant across candidates for the first pass to compare like with like, then allow a short tuning round for the finalists, because a model that needs a different prompt style is not disqualified by that alone.
Multi-provider: insurance with a running cost
Supporting a second provider protects you against an outage, a price change, a deprecation and a capability regression. It also costs you a second prompt to maintain, a second set of quirks in structured output, a second bill and a doubled eval matrix, and it tends to pull your prompts toward the lowest common denominator.
The compromise we prefer: build the seam, use one provider. Put every call behind your own interface, keep model choice in config, keep the eval set runnable against any candidate, and validate that a second provider works before you need it. That is a day of work and it buys most of the benefit. Full active redundancy is worth the running cost only when downtime has a contractual price, and the same reasoning that decides when a smaller model wins often means routing by task within a single provider first, which is cheaper than adding a second one.
What to do next
If you are choosing a model this week, do not open the leaderboard. Write down your weights, collect thirty real cases, and run two candidates. You will have a defensible answer in days, with an artefact you can rerun the next time something launches. Record the date and the model versions you tested, because the answer has a shelf life and the next decision should start from evidence rather than memory. It is also worth checking the cost side against your traffic before committing, along the lines of adding AI without setting money on fire.
If you would rather have someone run the bake off with you and leave the eval harness behind, talk to Denshin. We build these into products on AWS every month, and the harness outlives the decision.
Tags: LLM, Model Selection, AI Evals, AI Strategy, Engineering Leadership
All posts · Work with Denshin