AI agents can be tested on cloud operations by asking them to carry out the kinds of tasks DevOps teams face: repair services, remove resources, restrict access, roll out updates, and decommission projects. CloudOpsBench is an open-source benchmark built to measure how well they manage that work. Its first version contains 45 tasks on AWS, and even the strongest models in the reported results left many tasks unsolved.
A benchmark built around practical cloud work
Cloud operations involve more than knowing what a service does. A task may require an agent to identify which resources matter, take the requested action, and determine whether the work is finished. CloudOpsBench puts these demands into a set of test tasks rather than assessing capability through general questions.
The task list covers several kinds of operational changes. Agents may need to fix services, remove resources, change access rights, deploy updates, or shut down projects. Together, these examples test whether an agent can respond to instructions that affect real cloud infrastructure.
Version 1.0 focuses on AWS and was built by experienced DevOps engineers. The benchmark runs tasks identically in the real AWS cloud and in an emulator. That gives the project two settings for assessing the same work, while keeping the tasks themselves consistent.
Reasoning is the main reported weakness
The top results show that success remains limited. Claude Fable 5.1 solved 60 percent of the tasks, while GPT-6 Astra solved 53 percent. Those figures mean that neither model completed every task in the benchmark.
The team led by Ang Chen at the University of Michigan attributes most failures to reasoning rather than a lack of knowledge. About 80 percent of failures came from agents misjudging how broad a task was, failing to check all relevant services, or stopping too early. In other words, an agent may know useful facts about cloud systems and still fail to work through the whole assignment.
These failure patterns point to a practical challenge for cloud automation: interpreting the scope of an instruction and following through matter as much as choosing an individual operation. An agent that checks only part of the relevant infrastructure may miss work the task requires. An agent that stops before verifying completion may leave the task unfinished.
Some errors can affect infrastructure
Roughly 20 percent of errors involved wrong actions. The article notes that some of those actions could have irreversible consequences for infrastructure. That makes the benchmark's distinction between reasoning failures and incorrect actions consequential: mistakes in an operational setting may change the systems being managed.
The benchmark does not suggest that a strong score alone guarantees safe or reliable cloud operations. Its results instead show how much room remains for improvement on these tasks, including the need to understand what a request covers, inspect relevant services, and avoid acting incorrectly. For organizations considering AI agents in DevOps, those are capabilities to assess alongside task completion.
What CloudOpsBench makes possible
Because the tasks run in both AWS and an emulator, CloudOpsBench offers a way to assess agent performance across matching environments. The project is open source under the Apache 2.0 license, and a detailed failure analysis is available. That analysis can help readers examine the types of errors behind the overall results, rather than treating the score as the only measure.
Extensions for GCP, Azure, and OCI are planned. If added, they would broaden the benchmark beyond its current AWS focus. For now, CloudOpsBench provides a concrete starting point for examining how AI agents handle cloud operations, and its early results put task reasoning and follow-through at the center of that assessment.