Multimodal Language and Cognition

This PI group tries to understand cognitive and social pressures underlying language use, emergence and language learning in its multimodal and face to face context – that is the context languages are naturally are learned by children, evolved historically and are used mosty frequently.

In doing so it particularly investigates how adults and child speakers use and perceive hand gestures ( e.g, saying “drinking” with a drink gesture ) in a coordinated fashion with spoken utterances and what this reveals about their linguistic , cognitive and social processes . For example we try to unravel how speakers use gestures to disambiguate degraded speech and recruit them in first and second language learning. We also conduct research on how speakers process and integrate semantic information from speech and gesture in the brain recruiting different aspects of the language network and beyond and how they design their message for their addressees by modulating their speech and gesture such as in teaching contexts. We are also interested in how general non linguistic cognitive abilities such as memory, imagery and visual attention are related to the use and perception of multimodal aspects of language use in face to face interactions by adults and children.

Another focus of this PI group is on sign languages, the natural languages of deaf communities. Here we try to understand the role modality (visual vs. vocal) plays in shaping languages in general , cognitive and social biases that might play a role in the emergent patterns and structures in new sign languages ( in the wild or laboratory) and language acquisition patterns by deaf and hearing children and adults as well as in bimodal bilinguals ( hearing speakers who can sign and speak at the same time).

Our general mission therefore is to understand embodied and social foundations of language and cognition in multimodal and situated contexts in adults and children and provide new insights into understanding relations between language and other cognitive systems such as action-perception.

Please visit for further info.