Model choice is an operating decision.
What separating the role from the model taught us about quality, cost and changing a system without losing its purpose.
Keep the job stable. Evaluate the model against the work.
The largest model is not a business strategy.
Consider a support inbox. One task sorts routine requests into known categories; another prepares a response to an unusual complaint. The useful model choice depends on how well each task is performed, how long a person spends correcting it and how quickly the team needs the result.
As we built model configuration and usage visibility, one design principle became central: the role should describe the work, while model selection should be an explicit operating choice. A change in model should not silently change who owns the task, what information it can use or when a person must review the result.
Flexibility also needs boundaries. Choose among the models approved for your workspace. Processing requirements and organisational policy constrain the choices before quality and cost comparisons begin.
Take this with youAsk “which approved model does this job well?” before asking which model is newest.
One role, an explicit model decision
Keep the role
The job, tools, reviewer and expected output remain the starting point for evaluation.
Under the hood
Separate the agent definition from model selection so a model change does not become an accidental rewrite of the business workflow.
Apply policy
Choose within the models and processing arrangements approved for the environment.
Under the hood
Choose from the models approved for the workspace. Include the business requirements for processing, quality and cost in the selection.
Compare results
Try routine cases, exceptions and missing information. Review quality alongside latency and cost.
Under the hood
Hold the task set constant. Include tool behavior, retries and structured output in the comparison, not just text quality.
Verify the change
Confirm which model actually handled the work, then check the result with the role owner.
Under the hood
Record the selected model alongside the task results. Keep the comparison repeatable so the role owner can understand and revisit a change.
Separate configuration from execution.
The Command Centre gives model selection a clear home alongside the AI teammate and its usage. Teams can choose an approved model for the role without burying that choice inside the business instructions.
For an operations team, that means comparing request classification, response drafting and exception handling against the same standard. The role owner can see which choice delivers a useful result with an acceptable amount of review.
Use model usage and estimated cost alongside the reviewed outputs. A fast response that needs extensive editing may cost the team more than a slower response that is ready to use.
Under the hood +
Compare models using a consistent task set, expected output format and review criteria. Include response time, tool use, retries and editing effort so the evaluation reflects the complete business workflow.
Compare the cost of a usable result.
Token rates are only part of an operating cost. A cheaper response can create more retries or more review work. A more capable model may still be unnecessary for a well-defined sorting task. The comparison needs to include what happens after the first response.
For the inbox example, compare the same set of requests before and after a change. Count incorrect routing, missed escalations and replies that need substantial editing. Include the review time alongside model and tool costs. That gives the team a reason to choose a model beyond its headline token rate.
The lesson is to make model changes deliberate and recoverable. Keep the role definition clear, choose within approved boundaries, verify the effective configuration and check the outcome. Flexibility becomes useful when a team can explain why it made the choice.
Take this with youQuality, review effort, latency and total cost belong in the same decision.
What should we measure when comparing AI models?
Use the same tasks and review criteria for each model. Compare accepted results, incorrect routing, missed escalations, response time, retries, editing effort and operating cost.