Artificial intelligence can write software, analyse documents, solve mathematical problems and produce remarkably convincing arguments. Yet a deceptively simple question remains surprisingly difficult to answer.
Does an AI system actually understand how the world around it works?
Researchers from the Basis Research Institute and collaborators including researchers at MIT, Harvard, Cambridge and Cornell have developed a new way of investigating that question. Their framework, called WorldTest, is designed to examine whether an AI can explore an unfamiliar environment, work out its underlying rules and then use what it has learnt when circumstances change.
The results reveal a significant gap between the capabilities of today’s frontier AI models and those of humans.
What Is a World Model?
Humans constantly build mental models of the world without consciously thinking about it.
Imagine walking into an unfamiliar kitchen. You open cupboards, try switches and observe what happens. Within a relatively short period, you begin constructing an understanding of where things are and how the environment works.
More importantly, you can use that understanding when something changes.
This ability to build a flexible, predictive understanding of an environment is what researchers describe as a world model.
Modern AI systems can be remarkably good at recognising patterns. WorldTest asks a different question: can they actively discover the rules governing an unfamiliar environment and then apply that knowledge to a new problem?
Putting Humans Against Frontier AI
The researchers created AutumnBench, which contains 43 interactive grid-world environments and 129 tasks covering prediction, planning and detecting changes in how an environment behaves.
They then compared 517 human participants with three leading AI models: Anthropic’s Claude 4 Sonnet, Google’s Gemini 2.5 Pro and OpenAI’s o3.
The humans performed substantially better.
But the difference in scores was only part of the story. What was particularly interesting was how humans and machines attempted to understand the environments.
Humans experimented.
They reset environments, repeated actions and deliberately changed variables to work out what caused particular outcomes. When new evidence contradicted their assumptions, they were more likely to revise what they believed was happening.
The AI models behaved differently.
Researchers found that they often failed to explore strategically and could continue relying on an initial interpretation even when subsequent evidence suggested that interpretation was wrong.
The Scientific Method in Miniature
There is something fascinating about this distinction because the human behaviour resembles a miniature version of the scientific method.
We observe something.
We develop a hypothesis about why it happened.
We change something.
We observe what happens next.
If the evidence contradicts our original explanation, we revise the explanation.
WorldTest suggests that today’s frontier AI systems remain considerably weaker than humans at aspects of this process in unfamiliar interactive environments.
Giving the models additional computational resources helped in some environments, but not consistently across all of them.
That suggests the challenge may not simply disappear by allowing an AI to “think” for longer.
Pattern Recognition Is Not the Same as Understanding
This distinction matters because AI systems increasingly appear intelligent in ways that are deeply persuasive to humans.
A language model can explain physics, write a business strategy or discuss philosophy with extraordinary fluency. That fluency makes it tempting to assume there must be an equally sophisticated understanding underneath.
WorldTest offers a useful reminder that performance and understanding are not necessarily the same thing.
The researchers are not claiming that AI merely memorises everything it encounters, nor does this benchmark settle the philosophical question of whether machines can ever genuinely “understand”.
What the experiments show is more specific.
Current frontier models still have considerable room to improve when they must independently explore an unfamiliar environment, infer its dynamics, test their assumptions and flexibly update their beliefs.
Why This Matters Beyond a Benchmark
The implications become more important as artificial intelligence moves beyond chatbots and into robotics, autonomous systems and AI agents operating in the physical and digital world.
A chatbot can sometimes recover from misunderstanding a question because the user can correct it.
An autonomous system operating machinery, navigating a warehouse or making decisions inside an unfamiliar environment faces a different problem.
The world changes.
Equipment behaves unexpectedly. Rules change. Information is incomplete. Previous assumptions stop being true.
An intelligent system therefore needs more than the ability to recognise patterns it has encountered before. It needs to determine when its existing understanding of the world is wrong and work out what has changed.
Humans perform this kind of adaptation remarkably naturally.
WorldTest suggests that machines still have some distance to travel.
The Bigger Picture
Artificial intelligence has made extraordinary progress by learning patterns from enormous quantities of human-generated information.
The next frontier may be different.
Instead of asking whether an AI can produce the correct answer, researchers increasingly want to understand whether it can discover how an unfamiliar world works.
That requires exploration, experimentation, uncertainty and, perhaps most importantly, the ability to admit that an earlier assumption was wrong.
WorldTest therefore exposes something more interesting than another leaderboard showing humans beating machines.
It highlights one of the remaining differences between recognising patterns and constructing a flexible understanding of the world.
AI has become remarkably good at answering questions about worlds we describe to it. The next challenge is whether it can enter an unfamiliar world and work out the rules for itself.