Defining Intelligence for Machines: Formal Definitions and Measures
Formal definitions of machine intelligence range from behavioral tests like the Turing Test to mathematical measures such as universal intelligence. Chollet's skill-acquisition efficiency underlies the ARC-AGI benchmark, which emphasizes generalization. The Levels of AGI framework classifies systems by performance and generality.
The difficulty of defining intelligence
The question of what constitutes intelligence has no single agreed answer. In artificial intelligence, the term is used widely, and informal definitions vary considerably. Legg and Hutter's collection of definitions, which gathers more than 70 examples from experts, shows that even specialists do not share a common formulation. Legg and Hutter's collection of definitions
The lack of a precise definition becomes especially problematic when evaluating machines that behave very differently from humans. A definition that works for human cognition may not apply to artificial systems. Without a rigorous measure, progress in AI cannot be compared or tracked reliably.
The Turing Test as a behavioral criterion
Alan Turing's 1950 proposal, known as the Imitation Game, offers a behavioral criterion. An interrogator converses with a hidden person and a machine, and tries to tell which is which. The Stanford Encyclopedia of Philosophy's entry on the Turing Test explains that Turing presented the game as a way to make the question of whether machines can think precise. The Turing Test entry at Stanford
The test judges only outward behavior, not internal reasoning. It has been criticized for being too easy or too narrow, but it established a baseline for thinking about machine intelligence.
Formalization by Legg and Hutter
Legg and Hutter extracted essential features from many informal definitions and expressed them mathematically. Their paper on universal intelligence defines a measure that considers a machine's expected performance across all possible environments, weighted by complexity. Legg and Hutter's universal intelligence paper
This measure is theoretical rather than practical, but it unifies ideas from algorithmic information theory and optimal learning agents. It emphasizes the capacity to adapt and learn in diverse situations, rather than performance on a single task.
Chollet's skill-acquisition efficiency and ARC-AGI
François Chollet argued that skill at a fixed task does not equal intelligence, because extensive prior knowledge or training data can mask a system's generalization power. He proposed defining intelligence as skill-acquisition efficiency, the rate at which a learner turns experience and priors into new skills. Chollet's On the Measure of Intelligence
The Abstraction and Reasoning Corpus for AGI (ARC-AGI) implements this definition by using core knowledge priors that are innate or acquired very early in development. As the ARC Prize Foundation describes, ARC-AGI tasks are easy for humans but difficult for AI, isolating the ability to generalize from limited examples. ARC Prize's description of ARC-AGI
Levels of AGI as a measurement framework
A recent proposal from researchers at Google DeepMind introduces Levels of AGI, classifying systems by depth (performance) and breadth (generality). The paper also considers autonomy and interaction paradigms for deployment. Levels of AGI paper
This framework provides a common language for comparing models and measuring progress. It recognizes that intelligence is multidimensional, and that a single numeric score would ignore important aspects such as adaptability and autonomy.
| Approach | Basis of definition | Primary measure | Associated benchmark |
|---|---|---|---|
| Turing Test | Behavioral imitation | Ability to fool an interrogator | None specified |
| Universal Intelligence (Legg-Hutter) | Mathematical formalization | Expected performance over all environments | Theoretical, not practical |
| Skill-acquisition efficiency (Chollet) | Learning rate with limited priors and experience | Efficiency of acquiring new skills | ARC-AGI |
| Levels of AGI | Depth and breadth of capabilities | Performance and generality levels | Proposed future benchmarks |
Frequently asked questions
What is the Turing Test?
The Turing Test, originally the Imitation Game, asks an interrogator to distinguish a hidden person from a machine through conversation. It is a behavioral test for machine thinking.
How does Legg and Hutter define universal intelligence?
They define it as expected performance across all possible environments, weighted by complexity, based on algorithmic information theory.
Why does Chollet argue that task skill is not intelligence?
Because skill can be bought with unlimited priors or training data, which masks the system's own generalization ability. Intelligence is better measured as skill-acquisition efficiency.
What is ARC-AGI?
ARC-AGI is a benchmark measuring skill-acquisition efficiency with core knowledge priors. It contains tasks easy for humans but hard for AI, isolating generalization.
What are the Levels of AGI?
Levels of AGI is a framework that classifies AI systems by depth (performance) and breadth (generality) of capabilities, also considering autonomy and deployment.