Choosing a model for the task: quality, speed, cost

Model choice is often made once, by reputation: “take the strongest one, sort it out later.” Later never comes. Six months on, the product has ten features on one model, and nobody knows which of them actually needs it.

A model is a trade-off between three things, and you can’t max out all three. The job is to figure out what matters for this feature.

Three axes

Quality. How well the model handles your task on your data. Not on a public leaderboard, but here: does it classify your requests correctly, understand your terminology, stick to your format.

Latency. How long a person waits. It has two parts worth measuring separately:

time to first token, which matters when the answer is streamed;
total response time, which matters when you need the whole result to fill a form or make a decision in code.

Cost. The price of one call times how often the feature is called. A feature triggered on every keystroke and one run once a week are different problems, even with the same prompt.

Set priorities per feature

For each feature, decide which of the three comes first:

suggestions while typing, autocomplete, sorting: speed. An answer after three seconds is worse than no answer;
contract review, complex analysis, code generation: quality. People will wait, and mistakes are expensive;
background processing at volume, like classification, field extraction or labeling: cost and throughput. Nobody’s waiting, but there are many calls.

Done this way, it’s clear that one model for everything is wrong for at least some features.

Start with the smaller tier

A sensible order is bottom up. Take the lightest model that could plausibly do the job and run your eval set on it: twenty to fifty real examples with criteria for a good answer. If it falls short, move up a tier. Stop where the result is good enough.

Why not top down? Because it’s easy to get used to the larger model’s output and treat any difference as a regression. Starting from the bottom, you see what each tier adds and decide whether it’s worth it.

Before moving up, try a better prompt: add a couple of examples, tighten the format, split the task into two steps. That often closes the gap between tiers.

Routing instead of one model

You don’t have to pick one model per feature. Patterns that work:

by difficulty: simple requests go to a small model, hard ones to a larger one. Difficulty can be judged by a rule (length, document type) or by the small model itself;
with escalation: the small model answers first, the answer is checked, and if the check fails the request goes to the larger model;
by stage: a small model selects and prepares the data, a larger one writes the final answer.

Test any such setup with the same eval set. Routing that misjudges difficulty can end up worse than just using the larger model everywhere.

Measure in production

The eval set tells you whether a model is good enough. Production tells you how it actually performs. For every call, record the model, time to first token, total time, token counts and the outcome: did the person accept the result, retry or edit it. After a couple of weeks you’ll see where a model has headroom and where it’s at its limit.

Don’t lock yourself in

Models get updated, new tiers appear, old ones are retired. To make switching a config change rather than a rewrite:

keep model calls behind one thin layer in your code, not scattered across the project;
store the model name in configuration, not in each feature’s code;
have a fallback, another model or provider, for when the primary is unavailable;
when a new model comes out, run it through the same eval set before switching.

The upshot

Choosing well means finding a sufficient model for each feature, not the best one: quality on your data, latency people will tolerate and cost at real call volumes. That gets settled by an eval set and measurements, not by debate or a leaderboard.

SUBXCAT can compare models on your actual tasks and set up routing so that switching models takes minutes.