Synthesizing theories of human language with Bayesian program induction

Abstract

Automated, data-driven construction and evaluation of scientific models and theories is a long-standing challenge in artificial intelligence. We present a framework for algorithmically synthesizing models of a basic part of human language: morpho-phonology, the system that builds word forms from sounds. We integrate Bayesian inference with program synthesis and representations inspired by linguistic theory and cognitive models of learning and discovery. Across 70 datasets from 58 diverse languages, our system synthesizes human-interpretable models for core aspects of each language’s morpho-phonology, sometimes approaching models posited by human linguists. Joint inference across all 70 data sets automatically synthesizes a meta-model encoding interpretable cross-language typological tendencies. Finally, the same algorithm captures few-shot learning dynamics, acquiring new morphophonological rules from just one or a few examples. These results suggest routes to more powerful machine-enabled discovery of interpretable models in linguistics and other scientific domains.

Neurosymbolic Models for Computer Graphics

“…While neurosymbolic representations are always structured, not all structure‐aware representations fall under our definition of a “symbolic program”.As such, our framework does not capture some approaches that represent visual data as sequences of discrete codes [YLM*22], or those that simply combine primitives or [TSG*17] learned parts [PKGF21]. In later sections, we discuss methods for visual reconstruction tasks that use learning methods to produce symbolic representations that, when executed, generate visual outputs that reconstruct the input [TLS*19, ENP*19, ERSLT18, JWR22].…”

“… Neurosymbolic models produce visual data via a combination of symbolic programs and machine learning. From left to right: outputs of CAD programs written by a neural network [XWL*22]; inferring a 2D drawing program that reproduces an input hand‐drawn diagram [ERSLT18]; procedural material programs (i.e. node graphs) generated by a neural network [GHS*22].…”

Inductive Logic Programming At 30: A New Introduction

“…Besides ILP, inductive program synthesis has been studied in many areas of ML, including deep learning (Balog et al, 2017;Ellis et al, 2018Ellis et al, , 2019. The main advantages of neural approaches are that they can handle noisy BK, as illustrated by ∂ILP, and can harness tremendous computational power (Ellis et al, 2019).…”

“…45. Inductive program synthesis is often called program induction (Lin et al, 2014;Lake et al, 2015;Cropper, 2017;Ellis et al, 2018), programming by example (Lieberman, 2001), and inductive programming (Gulwani et al, 2015), amongst many other names.…”

“…Although too much irrelevant BK is detrimental to learning performance (Srinivasan et al, 1995(Srinivasan et al, , 2003, there is almost no work in ILP on trying to identify relevant BK. One emerging technique is to train a neural network to score how relevant programs are in the BK and to then only use BK with the highest score to learn programs (Balog et al, 2017;Ellis et al, 2018). However, the empirical efficacy of this approach has yet to be clearly demonstrated.…”