Don’t Look at Me Like That: Integration of Gaze Direction and Facial Expression

Does eye contact always attract attention? Not in every case, as a research team at the University of Würzburg’s Institute of Psychology has recently shown. Why not? Because context matters.

The emotional expressions of faces influence how their gazes shape our attention.

In a crowd, people notice immediately when someone makes eye contact. In fact, it takes no more than a fraction of a second to register and process this eye contact.

What happens during eye contact from a psychological point of view? This is what interests Anne Böckler-Raettig, Professor at the Department of Psychology III at the University of Würzburg (JMU). Social cognition is one of the focal points of her research; she has headed the research group “More than meets the eye: Integration, influences and impairments of direct gaze processing” since 2017.

Now, together with her team and scientists from the U.S. and Canada, Böckler-Raettig has decoded new information about how we process gazes and facial expressions.

She presents the results of her studies in the Journal of Experimental Psychology.

Emotional expression shapes attention

“We were able to show that the emotional expressions of faces influence how their gazes shape our attention,” explains Dr. Christina Breil, the first author of the study. What this means in concrete terms is this: Faces showing joy, an emotion that expresses approach, grab attention when they look directly at the observer (i.e. when there is eye contact, which also signals approach). The same applies to an angry facial expressions because, from a psychological point of view, anger is also an approach-oriented emotion.

The situation is different with avoidance-oriented emotions such as disgust or fear. In these cases, it is averted gaze (hence, avoidant gaze) that attracts the observer’s attention more.

Eye contact draws attention with neutral faces

In a previous study, Böckler-Raettig and her team had already demonstrated that faces with a neutral expression attract the attention of the observer especially well when those faces look directly at them. The participants were presented with four images of a woman’s face on a computer screen. The only difference was that some faces looked at the test subjects, while others did not. In addition, all the faces had a small number 8 positioned on their forehead.

“After exactly 1.5 seconds we replaced the four eights with letters. One of them was either an S or an H,” explains Böckler-Raettig. The participants were asked to react as soon as these letters were visible on one of the four faces, by pressing an S on a keyboard when they saw an S, or an H when an H appeared. The reaction time was then the measure as an indication of the level of attention.

The results of this study showed that although faces can be ignored for this task, people recognize the relevant letters more quickly if they are presented on a face that is looking at them. Why? “Because eye contact attracts our attention,” says the psychologist.

Anger and joy signal approach

But how do different emotional facial expressions affect attention capture by direct gaze? An interesting hypothesis about facial expressions and gaze direction suggests that gazes and facial expressions have a particularly strong impact if they are congruent in terms of “approach” or “avoidance.”

“For example, a delighted face that looks at you is congruent in that sense because joy is an approach-oriented emotion and the direct gaze also expresses approach,” says Breil. Accordingly, a disgusted face that looks away is also congruent.

To investigate this effect, the team changed the design of the original study. In the new version, the faces changed from a neutral expression, when the eights were shown, to either an approach- or an avoidance-oriented emotional expression—in one series of experiments to anger or fear, in the other to joy or disgust.

With disgust, averted gaze attracts more attention

A total of 102 people participated in this study. The results are clear: If happy facial expressions follow neutral ones and the gaze is directed at the observer, the reactions are the fastest. The pattern is reversed if the facial expression changes to disgust. Then the reaction is faster if the gaze is averted.

To better understand these results, the team repeated this experiment and measured participants’ eye movements. “We find the same pattern in eye movements, too: Participants look faster and longer at faces that are happy and that are looking directly at them, and participants look faster at disgusted faces that look away,” says Böckler-Raettig.

Quick processing of expression and gaze direction

According to the psychologist, this pattern indicates that people can process and integrate the facial expression and gaze direction of faces very quickly and efficiently. In fact, this process of integration begins just 200 milliseconds or so after the stimulus is shown.

The results also suggest that it is not the case, as is often thought, that gazes are processed independently of the context and always have the same effect. Even when it comes to processes as basic as attracting attention with a gaze, the context plays a role.

Abstract

Don’t look at me like that: Integration of gaze direction and facial expression

Efficient decoding of facial expressions and gaze direction supports reactions to social environments. Although both cues are processed fast and accurately, when and how these cues are integrated is still debated.

We investigated the temporal integration of gaze and emotion cues. Participants responded to letters that were randomly presented on four faces. Two of these faces initially showed direct gaze, two showed averted gaze. Upon target presentation, two faces changed gaze direction (from averted to direct and vice versa).

Simultaneously, facial expressions changed from neutral to either an approach- or an avoidance-oriented emotion expression (Experiment 1a: angry/fearful; Experiment 1b: happy/disgusted).

Although angry and fearful expressions diminished any effects of gaze direction (Experiment 1a), a direct gaze advantage was found for happy and an averted gaze advantage for disgusted faces (Experiment 1b). This pattern is consistent with hypotheses suggesting a processing benefit when emotion expression and gaze information are congruent in terms of approach- or avoidance-orientation.

In Experiment 2, we tracked eye movements and, again, found evidence for an approach–avoidance-congruency advantage for happy and disgusted faces both in performance and gaze behavior.

Gaze behavior analyses suggested an integration of gaze and emotion information that was already visible from 300 ms after target onset.

The articles below were published in external links and do not represent HBL ideas or ownership.

https://neurosciencenews.com