🌐 中文 English
This article aims to discuss some of the more well-known AI intelligence testing schemes currently.
Introduction
The father of computer science, Alan Turing, published a landmark paper in the field of artificial intelligence in 1950: Computing Machinery and Intelligence. In it, he posed a revolutionary question: Can machines think?, and outlined a testing scheme, which we now know as the Turing test. However, with the advent of the AI wave, whether the Turing test remains effective has become a highly debated topic.
The Turing Test
The setup of the Turing test is typically as follows: a human judge engages in a conversation with both a human and a machine. The judge’s task is to determine which is the machine and which is the human. If the judge cannot correctly distinguish them, or the rate of misjudgment is very high, then the machine is considered to have passed the Turing test.
The advantage of this test is that it cleverly transforms the complex and difficult-to-judge question of whether a machine can possess consciousness, emotion, or thinking ability into the question of whether it can successfully imitate a human. Therefore, many people refer to it as an imitation game.
Limitations of the Turing Test
As an imitation game based on behavior, the Turing test inevitably becomes a benchmark for judging and understanding things from a behaviorist perspective, but it inevitably conflicts with cognitivism. Cognitivists argue that external behavioral performance cannot substitute for the more important internal cognition. Merely based on external behavior, one cannot determine that a machine possesses intelligence. They famously refuted it with the Chinese Room argument.
Furthermore, factors such as the number of judges, the threshold for judge misjudgment, the type of questions, testing time and environment, and the background and ability of the judges all lack precise definitions, which can affect the results of the Turing test. Especially, whether the judges use AI in their daily lives has a significant impact on the experimental results.
Moreover, the Turing test focuses on how to imitate humans; in other words, how to deceive humans. Deliberately answering incorrectly or answering “I don’t know” to questions with this goal in mind is inherently meaningless because the purpose of artificial intelligence should be to enhance productivity and liberate people from heavy physical labor.
The Chinese Room
Among the experiments opposing the Turing test, the most famous is John Searle’s Chinese room experiment.
Searle imagines himself locked in a sealed room, not understanding Chinese, but having a detailed rulebook at hand. Someone slips a piece of paper with a Chinese question through a window. He only needs to follow the rules in the manual to combine these characters into an appropriate answer and then pass the answer out.
The Chinese room experiment highlights several key limitations of the Turing test:
- Syntax vs. Semantics: The Turing test result relies on external behavior (syntactically correct output), while the Chinese room experiment emphasizes the importance of semantic understanding—merely generating correct answers is not equivalent to true understanding.
- Imitation, Not Intelligence: If the machine’s goal is merely to imitate human behavior, then it might be good at pretending but not necessarily possess cognitive ability. For example, modern large language models can simulate conversation, but their understanding is questioned.
- Lack of Consciousness and Cognition: Searle points out that even if a system passes the Turing test, it might still lack “consciousness” or “cognition” because it is merely executing rules, not thinking.
Although many people agree or disagree with John Searle, his experiment indeed provides new lines of thought for testing the internal cognition of machines.
LLMs
Those who understand the principles of large language models (LLMs) know that they are essentially a black-box experiment, answering human questions based on pattern statistics and statistical prediction. Their hallucinatory outputs themselves indicate a lack of true knowledge understanding and reasoning ability. Therefore, in the article published in Nature last year, ChatGPT broke the Turing test — the race is on for new ways to assess AI (link), although it stated that ChatGPT-4 successfully passed the Turing test, people remain skeptical about whether it possesses intelligence.
Test Set Method
Because the Turing test has obvious limitations in the above aspects, people have thought of using test sets to measure AI intelligence. This involves using different test sets containing various types of questions, testing their accuracy, scoring them, and quantifying the model. This method assesses AI capabilities from different angles by constructing diverse test sets.
However, this approach also has clear drawbacks. First, a model’s problem-solving ability does not fully equal its generalization ability. Second, a model being strong in certain specific areas does not mean it has universality. Additionally, the model’s training data affects its score—the more it is trained in a certain area, the more accurate its answers. These reasons make the results difficult to be representative.
ARC-AGI Test
In 2019, François Chollet proposed ARC (Abstraction and Reasoning Corpus) to evaluate the capabilities of Artificial General Intelligence. ARC is a novel testing method focusing on measuring a machine’s abstract reasoning and inductive abilities.
Example image:
ARC-AGI Github Link
The AI must find the pattern from the first three sets of data that it should return the red segment different from the other red segments to correctly return the corresponding image’s red segment. Because it requires the AI to judge the internal abstract logic of the images, this method demands the AI to have a certain level of generalization ability, making it a more representative testing scheme for judging whether an AI possesses intelligence.
Conclusion
The testing of artificial intelligence, from the early Turing test to modern test set evaluations, and then to the ARC-AGI testing method, has been continuously exploring how to scientifically measure the true capabilities of machine intelligence. Future intelligence assessment methods need to better balance a model’s generalization ability, cognitive depth, and adaptability to truly reveal AI’s potential in complex environments and propel it towards genuine artificial general intelligence.
