Artificial Analysis has introduced Optima, a platform built around a practical question in AI adoption: which model works best for a specific job, not just for a public leaderboard. The platform lets users create custom AI benchmarks from their own material, then compare leading current models on quality, cost per task, and time per task.
The idea responds to a persistent weakness in AI benchmarking. General-purpose tests can be useful for broad comparisons, but they often do not answer whether a model fits a real workflow, a particular writing standard, a proprietary dataset, or an agentic task with business constraints.
How Optima turns a use case into a benchmark
Optima gives users multiple ways to define what they want to test. Those who already have evaluation assets can upload existing datasets from their own files or from Hugging Face. The platform can also use AI agent traces from Arize, Braintrust, or Langfuse.
Developers have another route. According to Artificial Analysis, they can install a skill that gathers information from their coding environment and past sessions. That gives Optima more context about the kinds of tasks the benchmark should reflect.
Users who do not have structured benchmark data can start with a description instead. They can explain the intended use case and provide sample inputs and outputs. Optima then proposes test inputs, evaluation criteria, and example tasks, which users can review and adjust before the benchmark is run.
This matters because a useful benchmark is not just a list of prompts. It needs to represent the work a model is expected to perform. If the goal is to choose a model for a finance agent, a legal writing task, or image analysis on proprietary data, the benchmark has to resemble those scenarios closely enough to produce a meaningful comparison.
Two ways to score model performance
Optima supports two evaluation methods. The first is rubric-based evaluation, where outputs are judged against objective criteria. This approach fits cases where the user can define what a good answer must contain, avoid, or satisfy.
The second method is pairwise comparison. Artificial Analysis also uses this approach in benchmarks such as GDPval-AA and AA-Briefcase. In Optima, users evaluate a sample of response pairs and choose the preferred answer. The platform then uses those preferences to produce a ranking across the full test dataset.
These options reflect two different ways teams often judge AI output. Sometimes performance can be checked against clear requirements. In other cases, the better answer is easier to recognize through comparison, especially for subjective or style-sensitive tasks.
Why cost and speed are central metrics
Optima does not limit the comparison to model quality. It also tracks cost per task and time per task as separate dimensions. That lets users see whether a higher-quality model is worth the added expense or slower processing time.
This is especially important for agentic applications. Token price alone can be misleading. A model that looks cheaper at the token level can become more expensive if it needs repeated attempts, fails more often, or creates outputs that require additional cleanup.
Cost per completed task is therefore a more useful measure for many workflows. It connects the model choice to the actual work being done, rather than to a raw pricing table.
Early testers used Optima for several kinds of evaluations, according to Artificial Analysis. Some built finance and accounting agent benchmarks to identify which model could cut costs by a factor of ten without major quality loss. Others tested which model best matched the writing style of lawyers or most accurately identified elements in a proprietary image dataset.
The platform’s pricing also reflects usage. Artificial Analysis says Optima charges only the actual token costs of the models used, with no markup. Rubric-based evaluations cost $0.125 per criterion per model, while pairwise evaluations cost $0.375 per comparison. At benchmark creation, each benchmark run, and each evaluation round, Optima holds a balance based on an estimate, then bills according to actual usage and evaluation costs incurred.
The benchmark problem Optima is trying to narrow
Optima is aimed at a known problem: public AI benchmarks can fail to predict performance in specific real-world settings. The source article points to an analysis by Epoch AI showing that results can shift based on implementation details that are often not disclosed. Prompt wording and temperature settings changed scores for the same model, and in agentic benchmarks like SWE-bench, changing the scaffold accounted for up to 15 percentage points of difference.
A broader study examining 445 benchmark papers from leading AI conferences found wider methodological problems. Nearly all had weaknesses in at least one area, including unclear definitions, unrepresentative samples, and missing statistical validation. Only about 10 percent used complete real-world tasks that reflected actual application scenarios.
Those findings help explain why custom benchmarking has appeal. A company trying to choose an AI model does not only need to know which system performs well on a public test. It needs to know which one handles its own inputs, constraints, formats, and quality expectations.
Custom benchmarks still need careful design
Optima can make benchmarks more relevant to a user’s actual work, but it does not remove the need for sound methodology. A custom benchmark can still be weak if the target capability is vague, the test cases are not representative, or the evaluation process is poorly implemented.
The same applies to business value. Even cost per task and time per task do not fully explain what an AI output is worth. A workflow can be cheap and fast while still producing results that require heavy rework or do not add much value to the process they support.
That makes Optima useful as a model-selection tool, but not a shortcut around judgment. Its strength is in moving AI evaluation closer to the work users actually need done. The remaining challenge is making sure the benchmark itself is clear, representative, and tied to outcomes that matter.