SMASH

This projects aims to develop software called SMASH - a fully open-source tool that helps us better understand the composition of non-verbal communication aspects, like body language, facial expressions, and the melodic aspects of speech. It will combine an already developed masking tool that masks audio-visual data to protect people's privacy with analysis modules that inform about how speakers communicated beyond the words uttered, thereby serving as a general tool for data-archiving and behavioral analysis of audiovisual data in social sciences in humanities.

SMASH would take as input audio-visual recordings of (a) communicating individual(s) and a summary of their multimodal communication characteristics as output. For instance, it shows the number and diversity of hand gestures, changes in posture and facial expressions relevant to communication context, and how voice tone changes with other gestures. It would also capture several trouble moments, such as pauses and fillers in speech and summarize what is being discussed using automatic speech transcription and semantic analyses. By integrating these dimensions - body language, facial expressions, voice, and content - in an easily understandable way, SMASH allows a detailed zoomed-out view of the full gamut of communication modes. SMASH will also offer standard baselines based on already analyzed speaker databases. In this way, users of SMASH can conclude that a speaker “gestured more relative to a database of Dutch speakers” or “used more variation in their tone of voice, even when compared to a database of highly appreciated speakers”. The tool would also offer options for data export for customized analysis. We aim to make SMASH a versatile and easy-to-use tool for users without programming experience. Through initial testing and adaptation by social scientists, we aim for SMASH to become an indispensable tool in research and clinical contexts.