Puzzles have long been a useful way to test artificial intelligence. They compress reasoning, memory, perception, and attention into small tasks where success or failure is easy to see.
The latest lesson is not that AI models are simply good or bad at puzzles. It is that their strengths are uneven. Some tests show rapid progress, while others expose limits that humans may notice almost immediately.
Games have always shaped AI testing
The connection between AI and games goes back to the early history of the field. The term “machine learning” was popularized in a 1959 article by the IBM computer scientist Arthur Samuel about an algorithm that learned to play checkers.
Since then, puzzles and games have become recurring test beds for machine intelligence. Chess and the Chinese board game Go are among the best-known examples, but the same principle applies to crosswords, logic puzzles, visual grids, and riddles.
These tests matter because they separate surface fluency from actual problem solving. A model may produce a confident answer, but a puzzle can reveal whether it has tracked the rules, understood the structure, or merely recognized a familiar pattern.
Fast progress does not remove brittle reasoning
AI performance on puzzles can improve quickly. In late 2024, a team of scientists from Columbia University showed that even the best models could solve only 18% of the infamous New York Times Connections puzzles. By early 2025, some models could solve them near perfectly every time.
That kind of improvement is striking, but it does not mean models reason like people across the board. The same systems can still stumble when a familiar puzzle is changed in a small but important way.
This is where memory becomes both an advantage and a weakness. Frontier LLMs have been trained on a huge amount of information, which helps them with trivia and familiar formats. But when a question resembles something in training data, a model may lean on a memorized response and miss the altered details.
A 2024 study by researchers from Google and the University of Illinois Urbana-Champaign tested this issue with variations of Knights and Knaves puzzles. In that classic format, some characters always tell the truth and others always lie. The task is to determine who belongs to which group based on what they say.
The source also points to SimpleBench, where questions resemble harder problems that models may have seen before. Humans can often spot the trick, while even top-tier models may overcomplicate the question or answer the wrong problem.
Vision remains a major weakness
Spatial reasoning is one of the clearest areas where humans can still have a large advantage. Mental rotation problems ask whether shapes are the same object viewed from different angles. People with strong spatial skills, including architects and mechanical engineers, can often manipulate such objects mentally.
Language models may accept visual inputs, but the source says they still fail badly on these tasks. That gap is important because it suggests that describing an image is not the same as understanding a three-dimensional object.
Visual weakness is not limited to 3D puzzles. Two-dimensional abstract reasoning can also be difficult. ARC-AGI, one of the best-known puzzle-based benchmarks, asks solvers to infer a general rule from example grids and apply it to a new grid.
Models do better on ARC puzzles when the grid is given as a string of numbers that encodes cell colors rather than as an image. That detail matters. It implies that models may sometimes handle symbolic representations more effectively than direct visual patterns.
Research cited in the source suggests that when models get ARC-AGI puzzles right, they may rely on complex rules that do not generalize well. Humans, by contrast, tend to use simpler visual concepts. Even so, models have improved on ARC-AGI over the past year, though some puzzles still defeat them.
Humans have blind spots too
The story is not one-sided. Some puzzle suites are designed to catch human intuition rather than machine weakness. In these tests, people often give a fast answer that feels obvious, while models may respond more deliberately.
The source describes a lightning round built around this idea. Some questions exploit errors in intuitive math. Others are phrased to suggest an answer that collapses when the wording is read carefully.
This is a useful reminder: human cognition and machine cognition fail in different ways. People may jump too quickly. Models may memorize, pattern-match, or miss physical and visual structure. A good benchmark does not just ask whether a system can answer correctly; it asks what kind of mistake appears when it fails.
Complexity changes the result
Some failures appear only when a puzzle becomes large enough. A study from researchers at Apple found that LLMs could solve simple Tower of Hanoi problems and river-crossing puzzles. But when the number of disks or people reached six and higher, the models began to falter.
Another study, from researchers at the University of Washington, Stanford University, and the Allen Institute for AI, found a similar struggle with logic grid puzzles. These puzzles require solvers to infer attributes of individuals from a list of clues until only one solution remains.
The interpretation is still contested. The Apple paper went viral, but commentators questioned whether the results show a special limitation in LLM reasoning or simply the familiar effect of errors accumulating as complexity increases.
That distinction matters for how people should use AI systems. A model that handles a small logic task may not scale cleanly to a larger version of the same task. The challenge may be less about knowing the rule and more about maintaining consistency as the number of moving parts grows.
Puzzles do not provide a complete measure of intelligence. But they offer a clear window into how AI models process language, images, rules, memory, and complexity. For now, they show a technology that is advancing quickly while still leaving visible gaps for careful human reasoning to exploit.