Comparing Humans and AI Agents

Abstract

Abstract. Comparing humans and machines is one important source of information about both machine and human strengths and limitations. Most of these comparisons and competitions are performed in rather specific tasks such as calculus, speech recognition, translation, games, etc. The information conveyed by these experiments is limited, since it portrays that machines are much better than humans at some domains and worse at others. In fact, CAPTCHAs exploit this fact. However, there have only been a few proposals of general intelligence tests in the last two decades, and, to our knowledge, just a couple of implementations and evaluations. In this paper, we implement one of the most recent test proposals, devise an interface for humans and use it to compare the intelligence of humans and Q-learning, a popular reinforcement learning algorithm. The results are highly informative in many ways, raising many questions on the use of a (universal) distribution of environments, on the role of measuring knowledge acquisition, and other issues, such as speed, duration of the test, scalability, etc.

On Measuring Social Intelligence: Experiments on Competition and Cooperation

“…The first evaluations using these tests [7] show that they work well at evaluating very different agents (humans and RL algorithms), but they do not properly reflect their supposed difference in intelligence. Many possible explanations are suggested in [7], with incremental knowledge acquisition and social intelligence being two of the abilities which this test is not giving enough importance.…”

“…Some preliminary results of this evaluation [7] show that the setting is able to compare and evaluate different kinds of agents, but it fails at placing them on the same scale, since humans usually get similar scores to those of other relatively simple agents. One possible explanation for these results is that it is virtually impossible to find other agents in the test, so social intelligence is not measured.…”

“…Many possible explanations are suggested in [7], with incremental knowledge acquisition and social intelligence being two of the abilities which this test is not giving enough importance.…”

“…Environments are composed of a space of cells (a graph of nodes) and the patterns for Good and Evil (a simplified adaptation of [7]). Once an environment has been constructed, evaluation is performed in the following way.…”

“…Good and Evil cannot share a cell with other agents. This re-introduces some degree of reactivity (with respect to the prototype in [7]), even in the single agent case.…”