Fast and flexible: Human program induction in abstract reasoning tasks

Abstract

The Abstraction and Reasoning Corpus (ARC) is a challenging program induction dataset that was recently proposed by Chollet ( 2019). Here, we report the first set of results collected from a behavioral study of humans solving a subset of tasks from ARC (40 out of 1000). Although this subset of tasks contains considerable variation, our results showed that humans were able to infer the underlying program and generate the correct test output for a novel test input example, with an average of 80% of tasks solved per participant, and with 65% of tasks being solved by more than 80% of participants. Additionally, we find interesting patterns of behavioral consistency and variability within the action sequences during the generation process, the natural language descriptions to describe the transformations for each task, and the errors people made. Our findings suggest that people can quickly and reliably determine the relevant features and properties of a task to compose a correct solution. Future modeling work could incorporate these findings, potentially by connecting the natural language descriptions we collected here to the underlying semantics of ARC.

Communicating Natural Programs to Humans and Machines

“…The high frequency of framing tags in the LARC corpus suggests that humans carefully establish context and resolve uncertainty over which programmatic concepts may be relevant: natural programs spend roughly a third of the times identifying which library function to run, and how to parse the current task. This corroborates a key claim in [11], where they state that unlike a typical interpreter that understands only a handful of programmatic concepts, the human interpreter contains a multitude of concepts useful in solving ARC tasks. Thus, if we were to build a rich system capable of solving complex tasks such as ARC with a multitude of concepts, it might be challenging to refer to these concepts via function names in a way similar to computer programs.…”

“…If the describer fails the verification task, the submitted natural program is deemed incorrect and discarded. Since we are primarily interested in communicating rather than solving ARC tasks (as opposed to [11]), each describer was shown all previous verified descriptions for a task, allowing the describer to focus on constructing an informative natural program. This results in generations of descriptions, forming a chain of improving natural programs written by a group of humans in collaboration.…”

“…In LARC, a describer describes the rule present in the examples so that a different person, a builder, can generate the correct output using the description alone. from a recent Kaggle competition 2 find that the best AI systems solve at most 20% of the tasks, while a study benchmarking human performance found that most humans easily solve over 80% [11]. 3 How can we build intelligent systems that achieve human-level performance on challenging and structured domains (like ARC), with or without additional human guidance?…”

Solving Bongard Problems With a Visual Language and Pragmatic Constraints

“…In contrast, in the recent Abstraction and Reasoning Corpus, the visual stimuli are small, abstract pixel images and, therefore, the domain is also well-defined, but much richer (Chollet, 2019). It thus offers a way to probe a similarly wide variety of visual and abstract concepts as BPs (Johnson, Vong, Lake, & Gureckis, 2021).…”

Generalized Planning for the Abstraction and Reasoning Corpus

“…One of these tasks, the Abstraction and Reasoning Corpus (ARC) introduced by Chollet (2019), remains an open challenge. ARC tasks are challenging for machines because they require object recognition, abstract reasoning, and procedural analogies (Johnson et al 2021;Acquaviva et al 2022). ARC comprises 1000 unique tasks where each task consists of a small set (typically three) of input-output image pairs for training, and generally one or occasionally multiple test pairs for evaluation (Figure 1).…”