People often believe that orderly structures were created by agents. We examine the cognitive basis of this tendency, and its malleability, focusing on orderly musical sounds as a case of interest. If simple associations mediate the link between order and agents, then detection of orderly stimuli should lead people to infer the presence of agents, independent of other factors. If causal reasoning mediates the link, then an alternative physical explanation for order should explain away its link with agents. We presented participants with orderly or disorderly outcomes in the form of musical sound sequences, and manipulated whether there was an alternative physical explanation for each outcome. Participants saw a scene involving a staircase-like xylophone, where a ball rolling downhill would produce an orderly descending scale. This provided an alternative physical explanation for how this orderly outcome could be generated. We found that when this scene was shown, then occluded, people did not infer the presence of an agent after hearing the orderly descending scale. In contrast, people expected an agent after hearing a scrambled sequence of tones, which could not be produced by gravity in this context (Exp. 1). When we subtly changed the scene by shuffling the xylophone’s bars, so gravity would not create an orderly outcome, participants readily inferred the presence of an agent from the same orderly stimulus. When we controlled for the presence of an alternative explanation, and manipulated orderliness, orderliness had no effect (Exp. 2). Across both experiments, alternative causal explanations flipped participants’ judgments such that disorder, and not order, led people to infer the presence of an agent. These findings show that the link between order and agents is mediated by causal reasoning, and can be weakened by understanding an alternative mechanism by which order could arise.