Denshin / Blog / AI
Small language models vs frontier models: when smaller wins
Routing every call to the biggest model is an expensive default. Small models now handle classification, extraction, routing and short summarisation well, while frontier models still earn their price on long-horizon reasoning and agentic work. How to build a cascade that escalates only when it needs to.
Denshin Engineering · Engineering Team · 27 August 2026 · 7 min read
Most teams pick one model and route everything to it. Usually the biggest one, because it was the one that worked in the prototype. That is a reasonable way to start and an expensive way to run, because a large share of the calls in a typical product are small, well-defined jobs that a much cheaper model handles just as well. This post is about routing work to the right-sized model: what small models genuinely do well now, where frontier models still earn their price, and how to build a cascade that escalates only when it needs to.
What counts as a small model
A small language model, in practice, is any model priced and sized well below the current flagship tier: the compact hosted models every major vendor offers alongside their frontier line, and the open-weight families you can run yourself. The label is relative and it moves, because this year's small model is roughly as capable as a flagship from a couple of generations ago.
That drift is the whole opportunity. Work that genuinely needed a frontier model eighteen months ago may not need one today, and nobody re-checks. The check costs an afternoon.
What small models are genuinely good at
The pattern is consistent: small models do well when the task is narrow, the input is bounded, and success is checkable. Specifically, they are usually sufficient for
- Classification and routing. Is this support ticket about billing, delivery or a bug? Which of these eight tools should handle the request? Constrained label sets are close to the ideal small-model task.
- Extraction into a schema. Pulling dates, amounts, names and line items out of a document into a fixed shape, especially when you enforce the shape with structured outputs rather than hoping for valid JSON.
- Summarising short text. A ticket thread, a meeting note, a product description. Long documents with cross-references are a different problem.
- Structured tagging and normalisation. Mapping messy free text onto your taxonomy, deduplicating variants, standardising addresses and job titles.
- Rewriting to a template. Tone adjustment, translation of short strings, turning bullet points into a paragraph with a fixed structure.
What these share is that a wrong answer is detectable. You can validate a label against an enum, a schema against a parser, an amount against a total. That checkability is what makes a cascade possible.
Where frontier models still earn their price
The gap has narrowed on the tasks above and has not closed at all on others. Keep the expensive model for
- Long-horizon reasoning. Problems where an early wrong step invalidates everything after it, and where the model has to notice its own mistake and back out.
- Agentic tool use. Choosing among many tools over many turns, recovering from a failed call, deciding when to stop. This is the area where the capability difference is most visible, as covered in AI agent architecture.
- Ambiguous requirements. Requests where the right move is to ask a question, or to notice that the user's stated request conflicts with their obvious intent.
- Code across many files. Holding a change consistent across a schema, an API handler, a shared type package and a front end is exactly where smaller models start producing plausible code that does not compile.
- Anything customer-visible and unscripted. Free-form conversation where a subtly wrong tone or a fabricated detail costs you more than the token savings.
A decision table
| Signal | Lean small | Lean frontier |
|---|
| Output shape | Fixed schema or label set | Free-form prose or code |
| Verifiability | Machine-checkable | Needs human judgement |
| Steps | One shot | Many turns, tools, backtracking |
| Input length | Short and bounded | Long, cross-referenced |
| Volume | High, per-call cost dominates | Low, per-call cost is noise |
| Cost of a wrong answer | Retry or escalate cheaply | Real damage, hard to detect |
Building a cascade
The practical structure is a cascade: try cheap, verify, escalate on failure. It is not complicated, and it is where most of the savings live.
- Call the small model first with a tight prompt and a schema.
- Validate the result mechanically. Did it parse? Is the label in the allowed set? Do the extracted line items sum to the stated total? Does the cited passage actually exist in the source? Prefer checks like these over asking a model how confident it is, because self-reported confidence is weakly calibrated and you are paying for a second opinion from the same source.
- Escalate on failure to the frontier model with the same input, and record that it happened.
- Watch the escalation rate. This is the number that tells you whether the cascade is working. If almost nothing escalates, try pushing more traffic down. If most calls escalate, you have added latency and cost for nothing, so route that task class straight to the big model and stop pretending.
Two warnings. Do not build a cascade around a task with no cheap verifier, because then your escalation trigger is guesswork. And avoid running both models on every request to compare them in production, which is a common accident and costs more than never having started.
A worked example: ticket triage
Take an inbound support queue. The first step is a label: billing, delivery, bug report or other. A small model with the four labels in the prompt and an enum-constrained output handles that, and validation is trivial because anything outside the enum is a failure. The second step, drafting a reply that quotes the right policy, is retrieval plus generation and is customer-visible, so it goes to the better model. The third step, deciding whether the reply is safe to send without a human, is a judgement call and should not be a model decision at all until you have data on it.
One feature, three different answers. That is the normal shape once you stop treating "which model do we use" as a single project-level decision and start treating it as a per-call one.
Latency and user experience, not just cost
Cost gets the attention, but latency is often the better reason to go small. A classification step that runs while the user is typing, an autocomplete, a validation hint on a form: these have a budget measured in a few hundred milliseconds, and a smaller model is structurally faster to first token. If the interaction is synchronous and in the user's way, treat speed as a product requirement and pick the model that meets it.
The inverse also holds. For work that runs in a queue, where nobody is watching, latency is nearly free and you should optimise for quality per rupee instead. Sorting your AI calls into "user is waiting" and "nobody is waiting" is a five-minute exercise that changes several model choices at once.
Measure quality per rupee with your own eval set
Public leaderboards tell you almost nothing about your task. They are aggregate scores on academic benchmarks, they are gamed to varying degrees, and they do not include your documents, your taxonomy or your users' phrasing. A small model that is mid-table overall can be the best available choice for your extraction job.
Do this instead. Take fifty real inputs. Write down the correct output for each, including the awkward ones. Run every candidate model against them, and record three columns: accuracy on your definition of correct, cost per call at current published pricing, and time to first token from your region. Then compute what you actually care about, which is quality per rupee at an acceptable latency. Building that set is the same work described in how to evaluate AI agents, and it pays for itself on the first routing decision. The wider decision framework is in choosing an LLM: a decision framework, not a leaderboard.
Two hedges worth stating plainly. Model prices and tiers change often, so re-run the cost column rather than trusting a figure from any blog post, this one included, as of writing in August 2026. And if you are considering running an open-weight small model on your own infrastructure, remember the comparison is against a hosted API's all-in price, not just GPU hours, which is the subject of open-weight models: when self-hosting makes sense.
What to do next
If you already have an AI feature in production, this is a one-afternoon exercise with a real payoff.
- List your model calls and sort them by monthly volume. The savings are concentrated at the top of that list.
- For the highest-volume call, ask whether the output is machine-checkable. If it is, it is a cascade candidate today.
- Build a fifty-case eval set for that one call and test the smallest model you have access to.
- Ship the cascade behind a flag, log the escalation rate, and leave the flag in place.
This complements rather than replaces the broader cost discipline in adding AI to your product without setting money on fire. If your AI bill is growing faster than your usage and you want a second opinion on where it is going, talk to us.
Tags: Small Language Models, LLM, AI Cost Optimization, Model Selection, AI Engineering
All posts · Work with Denshin