Donders Institute for Brain, Cognition and Behaviour
Zoek in de site...

Dynamic Signs and Signals (Dynamos)

Dynamic Signs and Signals (Dynamos)

The Dynamos research program is about how communication emerges from (multiple) time-varying bodily systems that coordinate in productive ways so as to allow for novel, efficient, or more stable communicative behavior to emerge. An overarching goal in this research is identifying biological or and basic cognitive constituents and constraints on multimodal communication, in the hope to come to a more continuous understanding of communicative practices in humans. Continuous, in the sense of doing justice to the time-varying nature of communication. But, also in the sense of grounding human communicative practices in a wider basis of communicative practices shared with other animals.

Dynamos is aimed to develop quantitative approaches of bodily time-varying signs and signals (e.g., acoustic analyses, motion tracking methodology, as captured by time series analysis) which can be further related to semantic and pragmatic aspects which imbue signals with communicative potential. Dynamos combines a wide range of theoretical (embodied cognitive science, dynamical systems, ecological psychology, evolutionary biology, behavioral biologie) and methodological (e.g., time series analysis, computer vision, experimental psychology, data science) interests in its research outlined below.

A human movement approach to signifying gestures

Picture1Manual gestures are often defined as meaningful movements –movement forms that contain a message. The theoretical conundrum is how a movement can contain such content: What allows movements to become meaningful? In this line of research, a bottom-up approach is developed, where it is investigated whether gestures are meaningful in virtue of relating consistently with other gestures over time. Using tools from data science to understand the constellated patterning of gestural movements, this line of research aims to unravel the process of signification of movements, without going beyond movement itself. That is, by only analyzing the kinematic trajectory of gesture, can we gain insights into how gestures come to function communicatively?

Key publications:

Pouw, W., Dingemanse, M., Motamedi, Y., Özyürek, A. (2021). A systematic investigation of gesture kinematics in evolving manual languages in the lab. Cognitive Science. doi (open access): 10.1111/cogs.13014

Pouw, W., de Wit, J., Bögels, S., Rasenberg, M., Milivojevic, B., Ozyurek, A. (2021). Semantically related gestures move alike: Towards a distributional semantics of gesture kinematics. In: Duffy V.G. (eds) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Human Body, Motion and Behavior. HCII 2021. Lecture Notes in Computer Science, vol 12777. Springer, Cham. https://doi.org/10.1007/978-3-030-77817-0_20

Pouw, W., & Dixon, J. A. (2020). Gesture networks: Introducing dynamic time warping and network analysis for the kinematic study of gesture ensembles. Discourse Processes, 57(4), 301-319. doi: 10.1080/0163853X.2019.1678967

Communicative rhythms: Timing and multimodal/interpersonal coordination

Picture2Adults speak and move communicatively in such an unreflective way, that is easy to forget that composing a multimodal utterance requires exquisite timing. The same is true for many instances of animal communication. Firstly, it requires timing between bodily segments that have their own intrinsic rhythms, such as manual movements versus much more mobile speech/communication effectors. But it also requires timing between bodies, such as in conversations (turn-taking), music-making and dance. This line of research is focusing specifically on the temporal aspects of communication. For example, looking at timing between gesture and speech, the rhythms are present in the kinematics of gesture, as well as the timing of turn taking between persons in conversation. Again our goal is to search for stable communicative behavior through basic principles of coordination, such as by synchronizing to stabilized communication.

Key publications:

Burchardt, L. S., Norton, P., Behr, O., Scharff, C., & Knörnschild, M. (2019). General isochronous rhythm in echolocation calls and social vocalizations of the bat Saccopteryx bilineata. Royal Society Open Science, 6, 181076. doi:htps://doi.org/10.1098/rsos.181076

Pouw, W., Proksch, S., Drijvers, L., Gamba, M., Holler, J., Kello, C., Scheafer, R., Wiggins, G. (2021). Multilevel rhythms in multimodal communication. Philosophical Transactions of the Royal Society B: Biological Sciences, doi:

Pouw, W. & Dixon, J. A. (2019). Entrainment and modulation of gesture-speech synchrony under delayed auditory feedback. Cognitive Science, 43(3). doi: 10.1111/cogs.12721

Pouw, W., Holler, J. (2022). Timing in conversation is dynamically adjusted turn by turn: Evidence for lag-1 negatively autocorrelated turn taking times in telephone conversation. Cognition.

Gesture-speech biomechanics

Picture3 Gestures are often understood as visual communication. Yet we should not forget that moving our hands is a biomechanical process that interacts with the whole body, affecting processes that are mechanically important for producing speech. Specifically, the biomechanical relations between manual gestures and the respiratory-vocal aspects of speech are investigated.  We have found that interactions arise when manual movements accelerate or decelerate thereby producing physical impulses on the body that recruit counter-stabilizing muscle groups that also involve respiratory muscles. Wim Pouw is currently working under a VENI grant that is dedicated to further investigating this phenomenon.

Key publications:

Pouw, W., Paxton, A., Harrison, S. J., & Dixon, J. A. (2020). Reply to Ravignani and Kotz: Physical impulses from upper-limb movements impact the respiratory-vocal system. Proceedings of the National Academy of Sciences. doi: 10.1073/pnas.2015452117

Pouw, W., de Jonge-Hoekstra, L., Harrison, S., J., Paxton, A., & Dixon, J. A. (2020). Gesture-speech physics in fluent speech and rhythmic upper limb movements. Annals of the New York Academy of the Sciences. doi (open access): 10.1111/nyas.14532

Pouw, W. Harrison, S. J., Esteve-Gibert, N., Dixon, J.A. (2020). Energy flows in gesture-speech physics: The respiratory-vocal system and its coupling with hand gestures. The Journal of the Acoustical Society of America, 148, 1231-1247. doi: 10.1121/10.0001730

Multimodal signal processing

Picture4 Studying multimodal communication comes with unique challenges for quantitative analysis. Gestures are postures in movement through 3D space, which is wholly different in nature from the speech signal. In this line of research, we seek to innovate methods to capture the complexity of gesture as well as speech, and to utilize state-of-the art developments in computer science for analyzing body movements. Specifically, the aim is to develop methods that are reproducible and automatable and that are nevertheless tailored to be combined with more qualitative methods of inquiry. We think we should strive for a symbiosis between quantitative and qualitative methods if we want to make progress in the study of multimodal communication.

Pouw**, W., Trujillo**, J.P., & Dixon, J.A. (2020). The quantification of gesture-speech synchrony: An overview and validation of video-based motion tracking. Behavior Research Methods, 52, 723-740. Doi: 10.3758/s1342

Pouw, W., & Dixon, J. A. (2020). Gesture networks: Introducing dynamic time warping and network analysis for the kinematic study of gesture ensembles. Discourse Processes, 57(4), 301-319. doi: 10.1080/0163853X.2019.1678967

Pouw, W., Trujillo, J. P. (2019). Tutorial Gespin2019 - Using video-based motion tracking to quantify speech-gesture synchrony. doi: 10.17605/OSF.IO/RXB8J

Owoyele, B., Trujillo, J., De Melo, G., Pouw, W. (2022). Masked-Piper: Masking personal identities in visual recordings while preserving multimodal information. https://github.com/WimPouw/TowardsMultimodalOpenScience

Cwiek**, A., De Melo**, G., Edelman**, J., Owoyele**, B., Pouw**, W., Santuber**, J. Trujillo**, J.(2021). Envision Toolbox: Multimodal (Signal) Processing and Analysis in Communication. https://github.com/WimPouw/EnvisionBootcamp2021
(**Alphabetical order)


Back to:

Theme 1: Language and Communication

Research Group

Dynamic Signs and Signals

Research Fellow

Dr. Wim Pouw

Group members

PD's

dr. Lara Burchardt

Affiliated Members

Susanne Fuchs (ZAS Berlin)