Over the past several years we have developed a theoretical approach to auditory perception and attention, focusing on the pivotal role of temporal structure. The theory draws upon approaches from psychology, linguistics, music theory, neuroscience, and nonlinear dynamics. The objective of this research program is to experimentally evaluate current theoretical predictions, to further develop mathematical models that make predictions about auditory perception.
Our theory explains how people maintain a stable attentional focus over temporally extended events while flexibly adapting to transient temporal fluctuations (i.e. time-warping in speech, expressive timing in music). It provides mathematical models of dynamic structural representation, and it makes predictions about neural correlates of auditory representation, attention, and communication. Importantly, it applies to complex, temporally structured event sequences, explaining how people respond to the auditory complexity of the real world.
The theory makes the following assumptions and predictions: 1) The mental representation of an auditory event is self-organized, dynamic structure comprised of endogenous oscillations. The emergence of the structural representation is modeled as a pattern formation process whose the neural correlate is the formation of a spatiotemporal pattern of neural activity. 2) The primary function of this spatiotemporal structure is attentional: it enables anticipation of future events thus, targeting of perception, and coordination of action with exogenous events. 3) Stability and flexibility properties of attention arise through nonlinearities in the underlying pattern-forming dynamics. 4) Dynamic structural representations afford auditory communication. Transient stimulus fluctuations observed in both speech and musical performance (e.g. rate changes, intonation) are not noise, but rather communicate structural information, intention, and affect. These communicative gestures are recognized as deviations from temporal expectations embodied in the attentional structure.
Theoretical predictions are being tested using behavioral and neuroimaging techniques. The experimental methodology relies primarily upon manipulation of auditory event structure; our experiments investigate stimuli of varying complexity, from simple isochronous tone sequences to performed music. Behavioral experiments are used to assess predictions in four areas 1) formation and stability of structural representations, 2) real-time tracking of temporally structured sequences, 3) the role of rhythm in attentional selectivity, 4) the role of expectancy in auditory communication. Neurophysiological and neuroimaging techniques are measuring temporal and spatial aspects of neural function in auditory perception and attention, further evaluating theoretical predictions. Modifications and extensions to the theory are being developed, based on comparison of experimental results with predictions of computer simulations.
Funding: NSF SBR-9808446; NSF CAREER Award BCS-0094229; NIH 5 T32 MH 19116
Status: Funded projects complete; Followup projects underway
People: Ed Large; Heather Chapin; Summer Rankin; Ted Zanto; Dinesh Nair; Joel Snyder
Collaborators: Mari Jones; Caroline Palmer; Erin Hannon
The development of tractable mathematical models of gradient-frequency nonlinear resonator networks is critical to advanced computer simulation of human attention and perception. Conventional linear resonator models are computationally and analytically tractable, however tractability is achieved at the expense of capturing many significant features of attention and perception. Allocation of attention to complex event sequences displays significant nonlinearities, including phase transitions and higher-order resonances. The perception of acoustic events also shows significant nonlinearities including extreme perceptual sensitivity, high frequency resolution, and higher-order resonances. Modern theoretical models of attentional and perceptual phenomena have one thing in common: they are all nonlinear oscillators or networks of nonlinear oscillators responding to perceptual input. Thus a nonlinear time-frequency transformation software library will be developed for analysis of temporal structure across the various different time scales associated with human attention and perception. We are demonstrating the feasibility of signal analysis by gradient frequency nonlinear resonator networks. We will pursue advanced development of computer models of gradient-frequency nonlinear resonator networks. A software library for deployment in computer applications is being developed, tested and documented. Computer simulations of gradient-frequency nonlinear resonator networks will find application for enhanced signal processing in advanced applications (e.g. speech recognition, audio and music processing, cochlear implants), and for simulating and predicting human attentional allocation.
Status: Underway
Funding: AFOSR FA9550-07-C0095
People: Ed Large; Marc Velasco
Corporate sponsorship: Circular Logic, LLC
Music is considered to be among cultures' highest achievements, a high-level cognitive capacity, similar in many respects to language. Like language, musical systems vary across cultures with learning. However tonality – the structured, dynamic relationship among musical frequencies – is universal across musical cultures, and is also perceived by nonhuman primates. This leads to the hypothesis that fundamental principles of auditory processing underlie the perception of musical tonality. In particular, the properties of nonlinear resonance predict the features of tonality remarkably well. In this project, we explore the implications of a new theory of central auditory processing and development for the perception of music. Our hypothesis is that nonlinear resonance in the auditory system gives rise to the perception of tonality in music. In a forthcoming paper (Large, under review), we argue that 1) nonlinear resonance is fundamental to auditory processing, and 2) certain universal characteristics of nonlinear resonance predict highly structured responses to stimulation with tone sequences, and 3) these structured responses, or dynamic fields, predict the perception of tonality in music as a natural consequence of auditory processing. The objectives of the project are to create computer models of neural resonator networks, to study network responses to tones and tone sequences, to analyze the role of connectivity and learning in the models, to analyze tonal expectancy and sequence generation, and to simulate experimental results relating to the perception of tonality. In summary, this approach models tonal percepts as resonance relationships in a dynamic neural field. The theory explains the perception of stability and attraction and that have been widely discussed by music theorists and psychologists, and it addresses questions about musical consonance that date back to Pythagoras.
Status: Pilot stage, initial paper under review
Funding: J. William Fulbright Foreign Scholarship Board; NIH Proposal under revision
People: Ed Large; Marc Velasco
Collaborator: Ira Schwartz
Are music and language special, or do they share neural mechanisms? While some lesion studies seem to suggest that language and music are governed by separate processing modules, a growing body of research comparing these two auditory domains has found some evidence that they do have cognitive and neural resources in common. Large & Jones (1999), for example, in positing rhythmic attending claim that it is a general cognitive capacity. Similarities are specially striking when comparing musical melody to linguistic prosody. Prosody is a system of intonation, rhythm and lexical stress patterns that helps structure language at the word, sentence and discourse levels through variations in several acoustic parameters such as fundamental frequency, timing and intensity. The same acoustic features are also essential to music, where pitch, rhythm and accentuation combine to form melody. In song, the prosodic features of speech are combined with musical melody which itself may take precedence in determining the realization of metrical structure and rhythmic patterns, making song an excellent domain in which to study the complex relationship between music and language. In particular, language and music each have their own metrical structure, but these systems compete in song because only one metrical realization is possible (Lerdahl, 2001). Thus, song is the ideal medium for examining the key role of rhythm and meter in the relationship between speech and music in terms of timing and accentuation. The aim of this research is to exploit the relationship between linguistic and musical meter in song by testing the general hypothesis that good “alignment” of rhythmic patterns (prosodic and melodic accents) in sung language facilitates semantic integration. We are performing behavioral, EEG and fMRI studies. The results of this series of studies will elucidate the neural bases of song perception, and help define whether one central mechanism is responsible for processing rhythm in language and music.
Status: Underway
Funding: NIH 5 T32 MH 19116; American Association of University Women
People: Reyna Gordon; Ed Large
The objective of this study is to assess the improvement of stroke patients undergoing various levels of occupational therapy and music therapy. Patients are randomly assigned to treatment conditions lasting 4 weeks. The treatment conditions include varying amounts of occupational therapy, individual music therapy, and rhythmic auditory stimulation. Assessments are administered pre- & post-treatment that allow us to quantitatively measure improvement in motor skills, entrainment, cognitive skills, and activities of daily living (i.e. dressing, bathing, grooming). These outcome measures will be used to gain knowledge about the role of entrainment and music therapy in motor rehabilitation and it's impact when used in addition to, or concurrent with, occupational therapy. This research will improve therapy prescriptions for stroke and other brain damaged patients, as well as improve our understanding of neural recovery and the role of entrainment.
Status: Underway
Funding: Institute for Music and Neurologic Function
People: Summer Rankin; Ed Large
Collaborator: Dr. Concetta M. Tomaino
Transient stimulus fluctuations observed in both speech and musical performance (e.g. rate changes, intonation) are not noise, but rather communicate structural information, intention, and affect. These communicative gestures are recognized as deviations from temporal expectations embodied in the attentional structure. In this project, we investigate the ability of transient stimulus fluctuations to communicate affect to listeners. In a first study, we aimed to identify brain areas involved in responding to affect communicated by expressive piano performance. Our subjects listened to two versions of Chopin’s Etude in E major, Opus 10, No. 3. The first version was an expressive performance, recorded by a highly trained musician on a computer-monitored piano. Our control was a computer-generated, mechanical performance of the same composition. Data analysis revealed differential brain activation in the two listening conditions. The expressive performance elicited greater activation in anterior cingulate, right temporal pole, right inferior frontal gyri, inferior parietal lobe and superior temporal gyri, areas that have been associated with emotion, attention, speech perception. The mechanical performance elicited greater activation in cerebellum, parahippocampal gyrus, supplementary motor area and dorsolateral prefrontal cortex, areas primarily involved in motor and sequencing tasks. A second study is currently underway, in which participants listen to a performed piece of classical music and to the same piece with temporal and loudness variations averaged out. They rate their emotional response to each version of the piece on a 2-dimensional rating scale with valence and intensity representing the two dimensions. Comparisons will be made between the participants’ emotion ratings and data collected as they listen in the fMRI scanner.
Status: Underway
Funding: NSF CAREER Award BCS-0094229; FAU Newell Fellowship
People: Ed Large; Dinesh Nair; Heather Chapin
Recent research has revealed that the cochlea performs a type of active, nonlinear time-frequency transformation, using a network of locally coupled outer-hair cell oscillators, each tuned to a distinct eigenfrequency. Furthermore, our work in auditory brainstem physiology (e.g. Large & Crawford, 2002) suggests that the central auditory system may analyze sounds using nonlinear neural oscillators. Thus, our hypothesis is that temporal processing in the central auditory nervous system is based on nonlinear resonance. In other words, the central auditory system analyzes sounds using networks of nonlinear neural resonators. We are showing that universal properties of nonlinear resonance can be related to various phenomena in auditory psychophysics. This leads to a new theory of central auditory processing and development that makes predictions about auditory perception. The aims of this research project are to construct computational simulations of auditory perception and conduct mathematical analyses of the model, and to model normal auditory temporal processing using this theory. We also plan to conduct experiments to test basic theoretical predictions regarding normal auditory perception, such as pitch, loudness and timbre, as well as abnormal auditory percepts, such as tinnitus.
Status: Pilot stage, initial paper in preparation
Funding: J. William Fulbright Foreign Scholarship Board; NIH Proposal under revision
People: Ed Large; Marc Velasco
Collaborators: Steve McAdams, Ali Danesh
This project is based on our recent findings regarding high-frequency EEG and rhythm perception. It is guided by the following general hypotheses: 1) high-frequency bursts of cortical activity are the neural correlates of pulse and meter; 2) cortical bursting is observable in neuroelectric and neuromagnetic recordings as induced and evoked GBA; 3) cortical bursting is the neural mechanism of dynamic attentional allocation for acoustic sequences; and 4) during perception-action coordination, auditory and motor areas communicate via high-frequency bursts. The objectives of the project are to evaluate these hypotheses by using multiple brain imaging modalities (electroencephalography, magnetoencephalography, and functional magnetic resonance imaging) and advanced analysis tools (time-frequency analysis, source localization, and source coherence) to answer specific questions about the neural correlates of rhythm perception and production, and by using neurophysiologically inspired modeling (burst oscillators) to understand the underlying neurocomputational mechanisms of rhythmic expectancy. Specifically, brain imaging measurements will determine 1) how neural correlates of rhythmic expectancy depend on tempo; 2) whether neural correlates of rhythmic processing reflect inter-individual differences in perception of time; 3) how auditory and motor cortical areas interact during a sensory-motor timing task; 4) whether metric expectancy is an attention-dependent process; and 5) how cultural experience influences neural correlates of rhythmic expectancy. Neurocomputational modeling will determine: 1) whether burst oscillators can account for neural activity associated with rhythmic expectancy; 2) whether induced (non-phase locked to stimuli) and evoked (phase-locked to stimuli) oscillatory brain activity represent two different processes or two aspects of the same process; and 3) how cortical communication occurs between auditory and motor areas in a sensory-motor timing task.
Status: Pilot stage
Funding: Pending
People: Ed Large; Heather Chapin
Collaborators: Joel Snyder; Berhard Ross