The sound of migration: exploring data sonification as a means of interpreting multivariate salmon movement datasets

Abstract

The migration of Pacific salmon is an important part of functioning freshwater ecosystems, but as populations have decreased and ecological conditions have changed, so have migration patterns. Understanding how the environment, and human impacts, change salmon migration behavior requires observing migration at small temporal and spatial scales across large geographic areas. Studying these detailed fish movements is particularly important for one threatened population of Chinook salmon in the Snake River of Idaho whose juvenile behavior may be rapidly evolving in response to dams and anthropogenic impacts. However, exploring movement data sets of large numbers of salmon can present challenges due to the difficulty of visualizing the multivariate, time-series datasets. Previous research indicates that sonification, representing data using sound, has the potential to enhance exploration of multivariate, time-series datasets. We developed sonifications of individual fish movements using a large dataset of salmon otolith microchemistry from Snake River Fall Chinook salmon. Otoliths, a balance and hearing organ in fish, provide a detailed chemical record of fish movements recorded in the tree-like rings they deposit each day the fish is alive. This data represents a scalable, multivariate dataset of salmon movement ideal for sonification. We tested independent listener responses to validate the effectiveness of the sonification tool and mapping methods. The sonifications were presented in a survey to untrained listeners to identify salmon movements with increasingly more fish, with and without visualizations. Our results showed that untrained listeners were most sensitive to transitions mapped to pitch and timbre. Accuracy results were non-intuitive; in aggregate, respondents clearly identified important transitions, but individual accuracy was low. This aggregate effect has potential implications for the use of sonification in the context of crowd-sourced data exploration. The addition of more fish, and visuals, to the sonification increased response time in identifying transitions.

Using Data Sonification to Overcome Science Literacy, Numeracy, and Visualization Barriers in Science Communication

“…As a mental exercise, think of a song you like, and speed it up or change the instruments it features. Because a sonification must play out over time, many sonification examples represent time series data, such as salmon migration patterns (Hegg et al, 2018), and brain wave fluctuations (Parvizi et al, 2018).…”

“…Adding visual monitoring to an auditory monitoring task has been shown to impair performance transiently, with performance returning to normal relatively quickly (e.g., ∼25 task trials) (Peres and Lane, 2005). However, in data sonification experiments, combining the modalities increased response time in listeners attempting to comprehend modeled ecological data, and the majority of listeners reported that the visuals were unhelpful or even detrimental to interpretation, competing for their attention (Hegg et al, 2018). Further research may identify the optimal ways to combine both methods for data interpretation and retention.…”

“…The nature of the data itself constrains possible sonic mappings: categorical variables (differences of kind) that do not change in time can only map to discrete parameter choices that do not vary during the sonification, such as instrumental timbre, while continuous variables (differences of scale) may translate into frequency or tempo, which may change continuously in time (Walker and Nees, 2011). Hegg et al's data sonification experiments found that participants were most sensitive to transitions in pitch and timbre, yielding the recommendation that the most important elements of the data should be mapped to these elements given their primacy (Hegg et al, 2018). Psychoacoustics literature can provide an empirical roadmapfor example, showing how we cue to pitch perception over space perception (Deutsch, 1975)-but a great deal of flexibility remains in the sonification process.…”

“…Some cases draw a clear relationship between data and sound: in a time series mapping of brain activity to pitch to hear seizures in epileptic patients, a simple power law relationship traces the microvolt amplitude of brain activity using pitch height (Parvizi et al, 2018). However, in instances where there is no clear variable to map to pitch, given how strongly we cue to this element of sound (Hegg et al, 2018), one might imagine a sonification composed of chord progressions, the structure of which is defined by higher-order moments in the data (e.g., slope, skewness, kurtosis) or weighted averages of multiple variables. While such mappings might provide a clear holistic picture of the shape of the scientific data, or draw attention to specific aspects, the relationship between the raw data and the audio becomes increasingly abstracted, potentially increasing the difficulty in conveying the science behind the sound.…”

“…This experience can be enriched by, but does not require, prior scientific expertise. Sonified data has also been theorized to require less time in training compared to visual data (Hegg et al, 2018). Data sonification's ability to convey a number of dimensions at once, as well as its potential to highlight local interactions between variables, makes it a powerful tool for data exploration for not only educators, but also scientific researchers.…”