Choose Coding Models by Cost per Verified Task

Quick answer: Compare the total cost of a fixed workload with the number of results that meet a predefined acceptance rule. Include unsuccessful attempts, escalation, verification, and human review. Report completion rate and latency alongside that cost. A cheap or local model is a useful worker when its outputs can be checked economically and its failures have a bounded path to resolution.

A model can be inexpensive per request and expensive to use. It may need several attempts, leave ambiguous changes for a reviewer, or fail after consuming most of the task budget. A more expensive request can also be wasteful when ordinary software could perform the operation directly.

Model selection starts with the task and its acceptance rule. Price becomes meaningful once you know what counts as a useful result.

The examples and accounting worksheet below are illustrative. They are not measurements of a provider, model, or local runtime, and they do not establish a ranking.

Define what “verified” means for this task

Verification has different strengths. Matching a trusted canonical result can settle a bounded transformation. Passing a finite test suite provides evidence about tested behavior. A persuasive explanation provides something else again.

Consider a fictional migration that renames a configuration key while preserving its value and every unrelated field. Its acceptance checks can inspect the parsed before-and-after objects. By comparison, a proposed architectural simplification may require human judgment about maintainability and future constraints.

Task	Useful acceptance evidence	Remaining limitation
Rename a specified configuration key	Parse both versions; compare required and unchanged values	The migration rule itself may be wrong
Implement a parser behavior	Representative cases, boundary cases, regression checks	Uncovered inputs can still fail
Summarize a design discussion	Grounded claims, preserved disagreements, reviewer assessment	Automated scoring may miss misleading omissions
Propose an architecture	Requirements, trade-offs, review against actual constraints	No simple mechanical correctness test

JSON Schema can enforce required properties and restrictions on additional properties. Those checks are useful for output shape; they do not establish that a migration preserved the intended meaning. JSON Schema object documentation.

The article on typed AI decisions explores a related boundary: software can consume a structured answer while still needing to decide whether that answer warrants action.

Check the checker before comparing workers

A gate that accepts bad output makes a weak worker look economical. A gate that rejects good output creates unnecessary retries and makes a capable worker look expensive.

For the configuration example, challenge the proposed verifier with deliberately incorrect candidates: a missing field, a changed value, a renamed unrelated field, and an unexpected extra key. Also include valid edge cases. These are suggestions for your evaluation design, not cases run for this article.

Mutation testing offers one established way to investigate whether tests detect deliberately introduced code changes. Stryker describes running tests against such mutations and identifying those that survive. That can expose weaknesses in a test suite; it does not turn the suite into a proof of correctness. Stryker's mutation testing explanation.

Keep the acceptance rule under separate control from the worker being evaluated. A model should not improve its score by deleting a failing case or weakening the comparison. Review changes to the evaluator as carefully as changes to the task.

For tasks requiring judgment, calibrate automated scores against human assessments. OpenAI's evaluation guidance recommends task-specific evaluation and human feedback to check automated scoring. This is general methodological guidance, not evidence that a particular evaluator is accurate. OpenAI evaluation best practices.

Count every route to an accepted result

Choose a fixed set of tasks and record the entire path each one takes:

Initial attempt → verification → bounded retry or escalation → final assessment

Count all attempts, including those that produce no accepted result. Deduplicate successes by task identity so three successful retries of one task cannot become three completions.

A useful accounting equation is:

Cost per verified task =
  total attempt, tool, verification, escalation, and review cost
  divided by the number of distinct accepted tasks

Keep costs in compatible units. If you include review time as money, state the hourly rate and allocation method. Also report raw reviewer minutes so a disputed rate does not hide the operational burden.

Here is a hypothetical comparison on the same 100 tasks. Cost units are invented accounting units, not current prices.

Cost or outcome	Configuration A	Configuration B
Initial model attempts	30	100
Retries and escalation	60	15
Tools and verification	40	35
Human review	120	50
Total cost units	250	200
Distinct accepted tasks	80	90
Cost per accepted task	3.13	2.22
Completion rate	80%	90%

Configuration A has the cheaper initial attempts. Under these invented inputs, B has the lower total cost per accepted task. Different retry behavior, review time, or task selection could reverse the result.

Always show completion rate. A worker that cheaply completes only the easiest cases may look excellent when the unresolved workload disappears from the report. If no task passes, report zero completions and an undefined cost per completion.

Include the costs that token totals miss

Record tool charges, infrastructure, and the work needed to operate the system. A local model may avoid a hosted inference charge while still consuming hardware capacity, energy, setup time, and maintenance. Treat those as explicit assumptions instead of assigning local inference an automatic zero cost.

Separate one-time engineering from recurring execution. Building a verifier may be worthwhile across thousands of similar items and uneconomical for one unusual task. State the expected reuse count when amortizing that work.

Measure elapsed time separately from cost. Parallel workers may reduce latency while increasing total resource use. Extra review can delay delivery even when the model finishes quickly.

Also record model round trips and tool calls. Repeatedly returning a large conversation to the model can affect resource use. Batching already-known independent reads is a reasonable optimization candidate; dependent decisions and conflicting writes require their own ordering. Do not infer billed savings from logical token counts alone.

Use a bounded comparison worksheet

Before comparing configurations, fill in:

• Workload: task identities, source revisions, and selection method.
• Acceptance: evaluator version, required checks, and review criteria.
• Configuration: model, settings, tool access, and relevant environment.
• Budget: maximum attempts, elapsed time, and external spend.
• Outcomes: accepted, rejected, blocked, and unresolved tasks.
• Accounting: complete run cost, review minutes, and latency distribution.

Keep the workload and acceptance policy comparable. Report results by task family as well as in aggregate. Repeated measurements may be needed where model variability or a small sample makes the apparent difference uncertain.

Tie retained results to the actual source using a verification receipt. Select a cheaper worker when the complete measured process supports it, and retain an escalation path for work its verifier cannot settle. The useful comparison is the cost and coverage of that whole process.