Twenty Years Beyond the Turing Test: Moving Beyond the Human Judges Too

Abstract

In the last twenty years the Turing test has been left further behind by new developments in artificial intelligence. At the same time, however, these developments have revived some key elements of the Turing test: imitation and adversarialness. On the one hand, many generative models, such as generative adversarial networks (GAN), build imitators under an adversarial setting that strongly resembles the Turing test (with the judge being a learnt discriminative model). The term "Turing learning" has been used for this kind of setting. On the other hand, AI benchmarks are suffering an adversarial situation too, with a 'challenge-solve-and-replace' evaluation dynamics whenever human performance is 'imitated'. The particular AI community rushes to replace the old benchmark by a more challenging benchmark, one for which human performance would still be beyond AI. These two phenomena related to the Turing test are sufficiently distinctive, important and general for a detailed analysis. This is the main goal of this paper. After recognising the abyss that appears beyond superhuman performance, we build on Turing learning to identify two different evaluation schemas: Turing testing and adversarial testing. We revisit some of the key questions surrounding the Turing test, such as 'understanding', commonsense reasoning and extracting meaning from the world, and explore how the new testing paradigms should work to unmask the limitations of current and future AI. Finally, we discuss how behavioural similarity metrics could be used to create taxonomies for artificial and natural intelligence. Both testing schemas should complete a transition in which humans should give way to machines -not only as references to be imitated but also as judges-when pursuing and measuring machine intelligence.

A Study of Technological Intentionality in C++ and Generative Adversarial Model: Phenomenological and Postphenomenological Perspectives

“…For example, the system does not just have a pre-defined picture of a specific apple, but it creates a set of all pictures of apples. Usually, the Generative Adversarial Model consists of two networks-a generator and discriminator-which permanently interact one with the other during the learning process (Hernández-Orallo, 2020). The generator has to produce realistic images while the discriminator has to specify if the image is real or not.…”

Direct Human-AI Comparison in the Animal-AI Environment

“…The Animal-AI Environment is clearly well-placed in serving as an o.o.d. AI benchmark (see Chollet, 2019 ; Crosby, 2020 ; Hernández-Orallo, 2020 ). However, what benefit does human testing offer?…”

“…Douglas Heaven reports that many developers, engineers, and scientists see DNNs as “fundamentally brittle” ( Heaven, 2019 ). They are exceptional at solving test problems taken from the same distribution as their training data but perform poorly and unpredictably when faced with slightly different problems ( Crosby, 2020 ; Geirhos et al, 2020 ; Hernández-Orallo, 2020 ). Geirhos et al (2020) argue that DNNs suffer from several issues that give rise to this phenomenon, wherein they appear to be intelligently solving tasks when they are actually only finessing solutions via a number of “shortcuts” (the so-called Clever Hans Effect of AI ; Sebeok and Rosenthal, 1981 ; Sturm, 2014 ; Hernández-Orallo, 2019 , 2020 ).…”

Direct Human-AI Comparison in the Animal-AI Environment

“…The Animal-AI Environment is clearly well-placed in serving as an o.o.d. AI benchmark (see Chollet, 2019;Hernández-Orallo, 2020). However, what benefit does human testing offer?…”

“…Douglas Heaven reports that many developers, engineers, and scientists see DNNs as "fundamentally brittle" (2019). They are exceptional at solving test problems taken from the same distribution as their training data but perform poorly and unpredictably when faced with slightly different problems (Geirhos et al 2020;Hernández-Orallo, 2020;. Geirhos et al (2020) argue that DNNs suffer from several issues that give rise to this phenomenon, wherein they appear to be intelligently solving tasks when they are actually only finessing solutions via a number of 'shortcuts' (the socalled Clever Hans Effect of AI; Sebeok and Rosenthal, 1981;Sturm, 2014;Hernández-Orallo, 2019.…”