Distinct Temporal Dynamics of Speech and Gesture Processing: Insights From Event-Related Potentials Across L1 and L2

Abstract

This study examined the neural and behavioral processing of speech and iconic gestures across L1-Turkish and L2-English when participants attended the speech or gesture channel. We recorded electroencephalogram activity in Experiment 1 and reaction times in Experiment 2 (24 participants in each) during a mismatch task where concurrent speech and gesture expressed either matching or mismatching information in relation to a preceding action. Participants were asked to detect whether the gesture (gesture-focused task) or the speech (speech-focused task) was related to the preceding action. Speech was presented in Turkish or English in separate blocks. In Experiment 1, we focused on N400 and N2 components as indices of late semantic processing and early sequential matching, respectively. In the gesture-focused task, our results demonstrated a gesture mismatch effect, which was evident in more negative N400 amplitudes for mismatching than matching gestures only in the context of simultaneous matching speech. In the speech-focused task, we observed the N2 effect, which was apparent in more negative N2 amplitudes for mismatching than matching speech, regardless of the simultaneous gesture. These dynamics were largely reflected in reaction times in Experiment 2. These results point to potentially distinct neural and temporal dynamics in processing speech versus gestures and suggest that speech processing might be instantiated earlier, whereas gestures recruit later stages of processing. Notably, we observed some differential patterns across L1-Turkish and L2-English, suggesting that speech and gesture processing may operate differently across languages. Our findings highlight a complex interplay between modality, modality focus, language, and neural processing of multimodal information.