Multimodal coordination: Pattern formation at a multisensory scale
One of the strength of the theory of dynamic patterns is its ability to capture the coordination of multiple components -each having a specific architecture and function- at a global level, without creating local theories for each configuration of those components. (see also: Jirsa, & Kelso, (2004). Integration and segregation of perceptual and motor behavior. In V.K. Jirsa, & J.A.S. Kelso (Eds.), Coordination Dynamics: Issues and Trends (Vol. 1. Springer Series in Understanding Complex Systems, pp. 243-259). Berlin : Springer-Verlag. for a review )
Theme 1: segregation/integration of multisensory events in the haptic, auditory and visual domain
An important factor in understanding synchronization of brain signals at a large (multisensory) scale is that neural signals -due to finite propagation times- require time to travel from one area to another resulting in disparate time delays in the communication between neural areas. We hypothesize that these time delays are a major determinant of the temporal properties underlying multimodal integration in the human brain.
We use a hybrid simultaneity/temporal-order judgement task on streams of bisensory events varying systematically the ISI of auditory/visual, visual/haptic or auditory/haptic stimuli. Our behavioral results replicate some well-established findings showing that, to be perceived as synchronous by the subjects, pairs of stimuli from different modalities have indeed to be presented with a certain delay.
Using fMRI, we identified the networks involved in multisensory processing. Our findings show that different distributed networks are activated when perceptually synchronous and asynchronous stimuli are processed. In particular, the superior colliculus is associated with the perception of synchrony and the inferior parietal cortex with the perception of asynchrony of auditory-visual signals. However, when there is no clear percept, these network components are disengaged and only a residual network comprised of sensory and frontal areas remains active. These results demonstrate that the processes of perceptual integration and segregation engage and disengage different brain subnetworks, but leave only a rudimentary network in place if no clear percept is formed.
Using a novel mode decomposition of the EEG, we demonstrated that over 90% of the measured scalp potential in response to multi-sensory stimulation may be attributed to the interaction of unisensory subsystems. The latency at which these interactions begin to take effect are identified to be in the vicinity of 30 ms suggesting that auditory-visual interactions occur remarkably early in time.
References:
Assisi , C.G., (2005). Theoretical and experimental studies of multisensory integration as a coupled dynamical system. PhD Dissertation, Program for Complex Systems and Brain Sciences, Florida Atlantic University .
Dhamala, M., Assisi , C.G., Jirsa, V.K., Steinberg, F.L., & Kelso, J.A.S. (submitted). Multisensory Integration for Timing Engages Different Brain Networks
Assisi , C.G., Dhamala, M., Jirsa, V.K., & Kelso, J.A.S. (2003). Multisensory integration in the human brain is parametrized by frequency and time delays. 4th International multisensory research forum, Hamilton , Ontario , Canada .
Dhamala, M. Assisi, C.G., Jirsa, V.K., & Kelso, J.A.S. (2003). Multisensory integration displayed in the human EEG and FMRI is parametrized by frequency and time delays. [CDROM] Program No. 912.14. 2003 Abstract Viewer/Itinerary Planner. Washington , DC : Society for Neuroscience, 2003.
Dhamala, M. Assisi, C.G., Jirsa, V.K., & Kelso, J.A.S. (2003). Multisensory integration displayed in the human EEG and FMRI is parametrized by frequency and time delays. Dynamical Neuroscience X, Orlando , Florida , USA .
Theme 2: integration of sound, touch and movement
Ubiquitous in everyday life, integration between the different senses provides a unified view of the world. However the interaction of different senses and movement is poorly understood.
In a behavioral study we investigated the dynamics of synchronization of rhythmic movements with sound and touch presented in anti-phase. We found that when the frequency of the stimuli is increased the synchronization between movement and stimuli is lost via a phase transition. This paradigm provides us a window into the outstanding problem of how the brain recruits and binds segregated areas for the production of adaptive behavior, and how this coherent unity is dissolved.
The same behavioral paradigm was applied to study multimodal integration and segregation using brain imaging methods. This paradigm allows us to explicitly parameterize how active movement, touch and auditory information become "bound" (and unbound) into a unitary perceptual-motor event. A considerable amount of evidence and theory suggests that transient, short-lived phase-coupled oscillations within and between specialized areas of the brain provide a mechanism for the neural integration required in multimodal integration. The idea is that these oscillations are coupled into a coherent network when people attend to a stimulus, perceive, remember, think and act.
The purpose of this project is specifically to use our multimodal paradigm to track the electroencephalographic correlates of the onset of binding of sound, touch and movement, and of its loss, as the frequency of stimuli is systematically varied.
References:
Lagarde, J., Kelso, J.A.S (in press) The binding of movement, sound and touch: Multimodal coordination dynamics. Experimental Brain Research.
Lagarde, J.F. & Kelso, J.A.S. (2004). Multimodal coordination dynamics: The binding of movement, sound, and touch. Journal of Sport and Exercise Psychology, 26, S12.
Lagarde, J., & Kelso, J.A.S. (2003). Multimodal coordination dynamics: The binding of active movement, sound and touch. [CDROM] Program No. 912.12. 2003 Abstract Viewer/Itinerary Planner. Washington , DC : Society for Neuroscience, 2003.
Center for Complex Systems and Brain Sciences - Florida Atlantic University
Charles E. Schmidt College of Science