Percussionists inadvertently use visual information to strategically manipulate audience perception of note duration. Videos of long (L) and short (S) notes performed by a world-renowned percussionist were separated into visual (Lv, Sv) and auditory (La, Sa) components. Visual components contained only the gesture used to perform the note, auditory components the acoustic note itself. Audio and visual components were then crossed to create realistic musical stimuli. Participants were informed of the mismatch, and asked to rate note duration of these audio-visual pairs based on sound alone. Ratings varied based on visual (Lv versus Sv), but not auditory (La versus Sa) components. Therefore while longer gestures do not make longer notes, longer gestures make longer sounding notes through the integration of sensory information. This finding contradicts previous research showing that audition dominates temporal tasks such as duration judgment.

Here, we demonstrate that "moving to the beat" can improve the perception of timing, providing an intriguing explanation as to why we often move when listening to music. In the first experiment, participants heard a series of isochronous beats and identified whether the timing of a final tone after a short silence was consistent with the timing of the preceding sequence. On half of the trials, participants tapped along with the beat, and on half of the trials, they listened without moving. When the final tone occurred later than expected, performance in the movement condition was significantly better than performance in the no-movement condition. Two additional experiments illustrate that this improved performance is due to improved timekeeping, rather than to a shift in strategy. This work contributes to a growing literature on sensorimotor integration by demonstrating body movement's objective improvement in timekeeping, complementing previous explorations involving subjective tasks.

Schutz and Lipscomb (2007) reported an audiovisual illusion in which the length of the gesture used to produce a sound altered the perception of that sound's duration. This contradicts the widely accepted claim that the auditory system generally dominates temporal tasks because of its superior temporal acuity. Here, in the first of 4 experiments, we show that impact gestures influence duration ratings of percussive but not sustained sounds. In the 2nd, we show that the illusion is present even if the percussive sound occurs up to 700 ms after the visible impact, but disappears if the percussive sound precedes the visible impact. In the 3rd experiment, we show that only the motion after the visible impact influences perceived tone duration. The 4th experiment (replacing the impact gestures with the written text long and short) suggests that the phenomenon is not due to response bias. Given that visual influence in this paradigm is dependent on the presence of an ecologically plausible audiovisual relationship, we conclude that cross-modal causality plays a key role in governing the integration of sensory information.

In this paper we offer a theory of cross-modal objects. To begin, we discuss two kinds of linkages between vision and audition. The first is a duality. The the visual system detects and identifies surfaces; the auditory system detects and identifies sources. Surfaces are illuminated by sources of light; sound is reflected off surfaces. However, the visual system discounts sources and the auditory system discounts surfaces. These and similar considerations lead to the Theory of Indispensable Attributes that states the conditions for the formation of gestalts in the two modalities. The second linkage involves the formation of audiovisual objects, integrated cross-modal experiences. We describe research that reveals the role of cross-modal causality in the formation of such objects. These experiments use the canonical example of a causal link between vision and audition: a visible impact that causes a percussive sound.[A fire is] a terrestrial event with flames and fuel. It is a source of four kinds of stimulation, since it gives off sound, odor, heat and light . . . . One can hear it, smell it, feel it, and see it, or get any combination of these detections, and thereby perceive a fire . . . . For this event, the four kinds of stimulus information and the four perceptual systems are equivalent.

In many ways, the structure of music resembles that of language, including the acoustic cues used to communicate emotion. In speech, sadness is imparted through a combination of low fundamental frequency, dark timbre, and a slow rate of articulation. As the acoustic properties of the xylophone are not conducive to mimicking these cues, it seems to follow that composers would avoid attempts to write "sad" music for it. We investigated this idea by comparing the repertoire of the xylophone with that of the marimba -a similar instrument whose acoustic structure permits a greater variety of timbres, pitch heights, and tone durations. An analysis of repertoire drawn from the Percussive Arts Society database of recital programs reveals that 60% of the tonal marimba examples surveyed were written in minor (nominally "sad") keys. In contrast, a parallel analysis of xylophone literature found minor keys used in only 6% of the examples surveyed. Further investigation revealed that the only examples of minor-key xylophone compositions included in this survey are in fact typically performed on the marimba. The avoidance of minor-key works on xylophone by both composers and performers is consistent with the idea that instruments restricted to producing tones with short durations, bright timbres, and high pitch heights are unable to mimic the speech cues used to convey sadness and/or depression.

Many studies demonstrate that musicians exhibit superior timing abilities compared to nonmusicians. Here, we investigated how specific musical expertise can mediate the relationship between movement and timing perception. In the present study, a group of highly trained percussionists (n = 33) and a group of non-percussionists (n = 33) were tested on their ability to detect temporal deviations of a tone presented after an isochronous sequence. Participants either tapped along with the sequence using a drumstick (movement condition) or listened without tapping (no-movement condition). Although both groups performed significantly better when moving than when listening alone, percussionists gained a greater benefit from tapping when detecting the smallest probe tone delays compared to non-percussionists. This complements both the musical expertise and timing perception literature by demonstrating that percussionists with high levels of training may further capitalize on the benefits of sensorimotor interactions. Surprisingly, percussionists and non-percussionists performed no differently when listening alone, in contrast to other studies examining the role of training in timing abilities. This raises interesting questions about the degree to which percussionists' known expertise in timing may interact with their use of motion when judging rhythmic precision.

Music is inherently an auditory art form, rooted in sound, and generally analyzed in terms of its acoustic properties. However, as the process of hearing is affected by seeing, visual information does in fact play an important role in the musical experience. Vision influences many aspects of music -from evaluations of performance quality and audience interest to the perception of loudness, timbre, and note duration. Moreover, it can be used to achieve musical goals that are in fact acoustically impossible. As such, understanding the benefits of embracing (and the costs of ignoring) vision's role is essential for all musicians. Furthermore, since music represents a pervasive and ubiquitous human practice, this topic serves as an ideal case study for understanding how auditory and visual information are integrated. Given that some musically-based studies have challenged and even contributed to updating psychological theories of sensory integration, this topic represents a rich area of research, relevant to musicians and psychologists alike. Why can't orchestral musicians wear t-shirts and flip-flops? The answer to these and other similar questions is that visual information in fact plays an important role in shaping the musical experience. In addition to increasing overall excitement and interest performers use it strategically, harnessing its communicative power to supplement and augment their acoustic output. Therefore rather than a distraction, it is actually a tool useful for musical communication.Although the role of visual information in music is receiving increasing attention by researchers (see Thompson, Graham, &Russo, 2005 andFinnäs, 2001), this work is rarely formally disseminated to or discussed by performing musicians. In part, this gulf stems from the fact that performers and researchers often work in different buildings (if not departments or even universities), rarely attend the same conferences, and almost never read one another's journals. This is not only unfortunate but also ironic, given the obvious importance of this topic to performing musicians (and educators). Therefore the goal of this article is to review research on the role of visual information in music in a manner helpful to performers and researchers alike.The first section will touch on three topics essential to understanding the role of visual information in music. The second reviews the growing literature on this topic organized by musical level. The third discusses questions raised from this survey with respect to the ability of certain instruments to benefit most from the use of visual information. The fourth and final section sets forth reasons for embracing vision as a useful, irreplaceable, and above all natural part of the musical experience. For the sake of clarity and easeof-use, this structure is outlined (with section numbers) in the table of contents below.

Recent work from our lab illustrates amplitude envelope’s crucial role in both perceptual (Schutz, 2009) and cognitive (Schutz & Stefanucci, 2010) processing. Consequently, we surveyed the amplitude envelopes of sounds used in Music Perception, categorizing them as either flat (i.e., trapezoidal shape), percussive (aka “damped” or “decaying”), other, or undefined. Curiously, the undefined category represented the largest percentage of sounds observed, with 35% lacking definition of this important property (approximately 27% were percussive, 27% flat, and 11% other). This omission of relevant information was not indicative of general inattention to methodological detail. Studies using tones with undefined amplitude envelopes generally defined other properties such as spectral structure (85%), duration (80%), and even model of headphones/speakers (65%) at high rates. Consequently, this targeted omission is intriguing, and suggests amplitude envelope is an area ripe for future research.