A new AI benchmark is putting pressure on a familiar assumption: that the strongest models are already ready to take over complex knowledge work. The AA-Briefcase benchmark from Artificial Analysis suggests the reality is still much less settled.
The test uses multi-week knowledge work projects built from thousands of fragmented source files, including Slack threads, emails, meeting transcripts, and large data exports. Even the top performer, Claude Fable 5, fully solves just 3 percent of tasks.
What AA-Briefcase Tests
AA-Briefcase is designed around work that looks less like a clean prompt and more like a real office problem. Instead of asking a model to answer from a tidy document, the benchmark gives it many scattered materials and expects it to reconstruct the task from partial evidence.
That matters because knowledge work often depends on context. A useful answer may require finding the right email, comparing it with a meeting transcript, checking a Slack thread, and connecting those details to a large data export. The difficulty is not only producing text. It is knowing what matters, where it is, and how the pieces fit together.
The source files in the benchmark are described as fragmented. That makes the test harder than a simple retrieval task. A model may see many files that look relevant, but still miss the one detail that changes the outcome.
The Best Result Is Still Limited
Claude Fable 5 records the highest rubric pass rate in the benchmark. But the headline result is less about victory and more about the remaining gap: it nails all criteria on just 3 percent of tasks.
That figure is important because the benchmark is not only asking whether an answer sounds plausible. It is checking whether the model satisfies the full set of requirements. In real knowledge work, partial success can still be failure if a missing detail changes the recommendation, the analysis, or the next action.
The weakness is broad, not isolated to one difficult corner of the test. On 31 out of 91 tasks, no model clears 50 percent. That means many tasks remain hard across the field, even when stronger systems are included.
Why Better Models Fail Differently
The benchmark also points to a shift in error patterns as models improve. Weaker models tend to fail in obvious ways. They miss relevant files, break down during basic execution, or produce outputs that are not usable.
Stronger models can look more capable because they satisfy the visible parts of the task. Their failures are quieter. They may answer the main question, follow the most obvious instructions, and still overlook details that require combining information from several sources.
That is a harder problem for teams using AI at work. An unusable output is easy to reject. A polished but incomplete answer can be more difficult to catch, especially when the missing requirement is buried across Slack threads, emails, meeting transcripts, and large data exports.
For workplace AI, this points to a practical lesson. The challenge is not only whether a model can write well or summarize well. It is whether it can reliably track requirements across messy source material and avoid losing important details along the way.
Cost Creates Another Tradeoff
AA-Briefcase also highlights a significant price gap between models. Per-task costs span more than 800x, from about $0.04 for DeepSeek V4 Flash to over $31 for Claude Fable 5.
That range creates a direct tradeoff for anyone trying to apply AI to knowledge work. The model with the strongest benchmark result may also be much more expensive per task. A cheaper model may cost little to run, but the benchmark suggests weaker systems can struggle with execution and output quality.
The numbers do not support a simple conclusion that every organization should use the most expensive model or the cheapest one. Instead, they show why evaluation matters. If a task is complex, fragmented, and detail-sensitive, low cost alone may not be enough. If a task is routine or tolerant of review, cost may matter more.
What This Means For AI At Work
The AA-Briefcase result is a reminder that real knowledge work is not just a language problem. It is also a coordination problem, a retrieval problem, and a reasoning problem across messy evidence.
For readers tracking AI progress, the benchmark offers a more grounded way to think about capability. Models can be impressive and still unreliable on projects that require sustained attention to scattered information. A system can lead a benchmark while still fully solving only a small share of tasks.
The main implication is straightforward: AI models are improving, but the hardest workplace tasks still demand caution. When the work depends on fragmented files and hidden requirements, human review remains central. The risk is not only that a model fails loudly. It is that it gets close enough to seem finished while leaving important criteria unmet.