How to think about choosing an AI model

A model’s aggregate benchmark rank and its fitness for a specific task are different properties. A model evaluated across dozens of scenarios may rank first overall and fourth on the specific thing your application does. Task fitness depends on which trade-offs a specific application requires.

Capability is task-specific

Benchmark rankings aggregate performance across many different scenarios. The HELM framework, developed at Stanford’s Center for Research on Foundation Models, evaluates models across accuracy, fairness, calibration, and efficiency, deliberately exposing trade-offs rather than collapsing them into a single score (Liang et al., 2022). A model that leads on general reasoning may trail on structured extraction or multi-step arithmetic. Aggregate rankings and per-task performance are different measurements of the same model.

Context length is a hard constraint

Every model processes a fixed window of text: the combined length of input and output in a single call. Advertised context lengths and effective context lengths differ. Research on multi-document question answering found that retrieval accuracy peaks when relevant content appears near the start or end of a context window and degrades significantly for content in the middle, even within the model’s stated token limit (Liu et al., 2023). For tasks involving large documents or long conversation histories, this gap between advertised and effective context length is a constraint distinct from the raw token count.

Latency and throughput pull against capability

Large models are slow. For interactive tasks (a live conversation, a real-time suggestion as someone types), response latency directly affects whether the system feels usable. Batch workflows, where outputs are processed offline, can absorb the wait. The same task can justify different model tiers depending on whether the user is watching the clock.

Cost scales with volume

Per-token pricing is negligible at low volume and dominant at high volume. The most accurate frontier models are also the most expensive, so the cost of a task scales with how much accuracy it actually requires rather than how much is theoretically available. For high-frequency, narrow tasks, a smaller model that holds most of the capability gap often makes the cost difference decisive.

Modality is a gate, not a dial

Text-only models cannot process images or audio. Multimodal models can, but tend to be larger and more expensive per call. If the task involves diagrams or tables embedded in documents, modality is a hard constraint that precedes any comparison on capability or cost.

Privacy and where the model runs

Using a cloud API means your inputs leave your infrastructure. For regulated industries or workflows where sensitive content cannot leave your systems, that may be disqualifying regardless of capability. Open-weight models, those with publicly released parameters that can be run locally, trade operational overhead for that control.

The non-obvious point: defaulting to the most capable model is a specific choice

Selecting the largest, most capable model carries distinct trade-offs: higher cost and slower responses than many tasks require. On narrow tasks with clear success criteria, the accuracy differential over smaller models may be minimal; on tasks requiring broad knowledge or complex reasoning, scale tends to matter more. Understanding what a language model is is the underlying concept; which one to use for a given task is the applied question.

Sources

• Liang, P. et al. Holistic Evaluation of Language Models. arXiv:2211.09110, 16 Nov 2022.
• Liu, N. F. et al. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172, 2023.