Phy-Q: A Testbed for Physical Reasoning

Abstract

Humans are well-versed in reasoning about the behaviors of physical objects and choosing actions accordingly to accomplish tasks, while it remains a major challenge for AI. To facilitate research addressing this problem, we propose a new testbed that requires an agent to reason about physical scenarios and take an action appropriately. Inspired by the physical knowledge acquired in infancy and the capabilities required for robots to operate in real-world environments, we identify 15 essential physical scenarios. For each scenario, we create a wide variety of distinct task templates, and we ensure all the task templates within the same scenario can be solved by using one specific strategic physical rule. By having such a design, we evaluate two distinct levels of generalization, namely the local generalization and the broad generalization. We conduct an extensive evaluation with human players, learning agents with varying input types and architectures, and heuristic agents with different strategies. Inspired by how human IQ is calculated, we define the physical reasoning quotient (Phy-Q score) that reflects the physical reasoning intelligence of an agent. Our evaluation shows that 1) all agents are far below human performance, and 2) learning agents, even with good local generalization ability, struggle to learn the underlying physical reasoning rules and fail to generalize broadly. We encourage the development of intelligent agents that can reach the human level Phy-Q score. Website: https://github.com/phy-q/benchmark

The anticipatory paradigm

“…2021; Kejriwal and Thomas 2021; Xue et al. 2022) usually solve this problem by incorporating human‐intuited knowledge or search constraints, at the cost of limiting creativity. The conditions above ‐ namely high‐impact relevance, noticeability, and controllability ‐ may serve as domain‐independent heuristics to filter out nuisance novelties.…”

“…2023), Monopoly (Kejriwal and Thomas 2021), Science Birds (Xue et al. 2022), CartPole8 and VizDoom8. Some of these approaches, such as OpenMIND (Musliner et al.…”

“…It is important to note that a difficulty of domainindependent novelty generators is generating useful novelties, or in the AT case, "novelties that are worth attempting to manage risk for." Domain-specific generators (Balloch et al 2022;Goel et al 2021;Kejriwal and Thomas 2021;Xue et al 2022) usually solve this problem by incorporating human-intuited knowledge or search constraints, at the cost of limiting creativity. The conditions above -namely high-impact relevance, noticeability, and controllabilitymay serve as domain-independent heuristics to filter out nuisance novelties.…”

“…Recently, novelty adaptive agents (Goel et al 2022;Loyall et al 2022;Musliner et al 2021;Piotrowski et al 2023) have shown promising results dealing with novelty in domains such as Minecraft (Goss et al 2023), Monopoly (Kejriwal and Thomas 2021), Science Birds (Xue et al 2022), CartPole 8 and VizDoom 8 . Some of these approaches, such as OpenMIND (Musliner et al 2021), demonstrate novel AT with domain-independent model modification and goal-reasoning heuristics in response to novelty.…”

Case Evaluation of Structural Strength Improvement of Lateritic Soil Using Selected Binders

“…Quite a lot of other investigators [16][17][18][19] described soil improvement performed using sisal fibre, jute fibre as well as cone fibre, with encouraging outcomes. The utilization of fibres for diverse engineering uses has been described in literatures [11,[20][21][22][23][24][25][26].…”