Communication generally takes place in social interactions, or between two (or more) people. However, in most previous research on communication we study one person speaking or listening, without including visual signals or interaction. In order to make studies of communication more representative of how we actually communicate in real life, we need to investigate people in social interactions, by letting them communicate while looking at each other.
Cordula Vesper’s work on social interaction
Scientist Dr. Cordula Vesper does exactly this. Originally from Germany, she works at the University of Aarhus in Denmark as an associate professor. She is on a continuous quest to find out how people behave in social interactions.
I visited Vesper in Aarhus to discuss how our research fields overlap and differ. In our conversation, we agreed that taking an interdisciplinary approach that combines more than one discipline is crucial. This approach promotes a broader view that can help bridge knowledge gaps by looking beyond one’s own expertise and understanding different concepts and ways to tackle questions through the lens of more than one discipline. For instance, Vesper approaches her research from multiple research disciplines such as Psychology, Cognitive Science, and Semiotics (the study of signs and symbols). This allows her to study behaviour and the cognitive mechanisms that underlie behaviour.
An example of Vesper’s work includes looking at how people communicate instructions for actions, such as moving objects, without using speech. In one of her experiments, the designated “leader” must communicate to a “follower” how to move some objects without talking. In interactions like this, even something as small and specific as hand-shape can make it easier for followers to understand the leader’s intention. This shows that visual signals communicate quite a lot of information, even in the absence of speech!
Like Vesper, I investigate communication in social interactions. One major difference between our approaches is that I focus on the body during speech. Specifically, I study whether facial signals contribute to communication in dialogue. I record and analyse spontaneous conversations to answer this question. I also use avatars (virtual characters) which allow me to control which visual signals are used and at which time, to test whether specific visual signals help differentiate questions from statements. For instance, are people better at recognizing a request for information if the avatar moves its eyebrows? So far, I’ve discovered that questions are detected more easily when eyebrows move, showing that visual signals of the face can help to signal specific intentions.
Future directions for this research
Vesper and I talked about the trade-off that comes with investigating communication using a natural scenario that is representative of how we actually communicate in real life: The more natural an experiment, the noisier and less controlled it is, and vice versa. It is challenging to design experiments that are precise and allow us to extract information with certainty but are not too artificial. The key is to find a good balance. Vesper uses markers to track body movement in her new motion-tracking laboratory instead of relying on automatic software. Vesper says these markers have to be placed on the body manually, which takes more time, but they track body movement more reliably. Similarly, in my own work, my avatars are of course virtual, but their movements are based on the natural behaviour of real people. This, again, gives us control over what we show participants, while also staying as close to real conditions as possible.
Future work using different approaches and methodologies will help us determine exactly when visual behaviour contributes to communication. We believe it would be fruitful to bring together different disciplines to focus on visual behaviour with and without speech in various scenarios with more or less control. It could be that how we move our body when we talk matters in certain contexts, but not in others. Vesper says that so far, communication has mostly been studied where people are trying to cooperate. She proposes research on competitive contexts like a football game or other competitive games where people are not always trying to cooperate. This would provide useful insight into whether moving our body during communication is a flexible strategy. Future research could also compare how people in different cultures use visual signals to communicate. In Mayan culture for example, eye contact is considered rude. So how does Mayan visual behaviour differ from ours in the west? In the end, this work could tell us how we can make social robots (see for example Sophia, a robot created for healthcare purposes) look and behave more like a real human.
Scientific readings
- Vesper, C., Morisseau, T., Knoblich, G., & Sperber, D. (2021). When is ostensive communication used for joint action? Cognitive Semiotics, 14(2), 101–129. https://doi.org/10.1515/cogsem-2021-2040
- Vesper, C., & Sevdalis, V. (2020). Informing, Coordinating, and Performing: A Perspective on Functions of Sensorimotor Communication. Frontiers in Human Neuroscience, 14. https://www.frontiersin.org/article/10.3389/fnhum.2020.00168
- Pezzulo, G., Donnarumma, F., Dindo, H., D’Ausilio, A., Konvalinka, I., & Castelfranchi, C. (2019). The body talks: Sensorimotor communication and its brain and kinematic signatures. Physics of Life Reviews, 28, 1–21. https://doi.org/10.1016/j.plrev.2018.06.014
Writer: Naomi Nota
Editor: Candice Frances
Dutch translation: Julia von der Fuhr
German translation: Natascha Roos
Final editing: Sophie Slaats
