By Trina Nguyen

Abstract:

Humans are naturally social beings, often basing our interactions around the emotions of others. Proper emotion identification can strengthen interpersonal relationships as well as enhance daily social interactions. Thus, it would be beneficial if there were methods to improve our emotion recognition. Although we take into account many signs such as body language and tone of voice, we heavily rely on facial expressions to identify emotion. Yet, in recent literature there has been a lack of emotion recognition training regarding a large forefront of the face – the eyes. I examined whether humans could train to enhance their emotion recognition ability based solely on the eye region of the face. Participants were either instructed to watch short video clips depicting distinct facial expressions for a week or were not asked to complete any activities. All participant emotion recognition abilities were measured before and after the weeklong period. Improvement in the experimental group scores suggests that eye-to-eye emotion identification can in fact be improved with training. Additionally, making a conscientious effort to watch and notate emotion from video clips can be an easy and effective method to improve emotion recognition.

            Keywords: emotion recognition, empathy, eye-to-eye emotion recognition, emotion training, emotion recognition ability, empathy experiment

Emotion recognition is an integral part of human social interaction. The skill of accurate emotion identification has a plethora of applications, from strengthening relationships to negotiating business deals. Therefore, training to improve our emotional recognition can in turn improve our everyday lives. Past studies regarding emotion recognition training have had positive outcomes, encouraging further exploration regarding new effective methods of training. Recently, researchers Matsumoto and Hwang (2011) have shown that the ability to perceive emotional microexpressions, facial expressions that last for less than a second, can be trained and retained, leading to improved social and communicative skills.

While Matsumoto and Hwang conducted successful training to improve recognition of microexpressions, there is significantly less data regarding emotional recognition training focused on human eyes. Researchers Wegryzn et. al (2017) pinpointed how observers rely heavily on the eye and mouth regions of the human face in order to successfully recognize emotions. Humans particularly rely on the eyes to determine basic emotions such as anger, sadness, surprise, and fear (Calvo et. al., 2018). This suggests that additional emphasis should be placed on the eyes when training to decipher emotions.

In this study, I explore whether identifying emotions based on the eyes of another person can be improved with training. Specifically, I explore whether training by watching short video clips with clear visibility of peoples’ faces and note-taking about the video clips can improve eye-to-eye emotion recognition. I hypothesize that this observing and note taking practice will be effective, leading to increased emotion recognition ability.

To quantify emotion recognition and improvement, I requested participants complete an online exam titled “Reading the Mind in the Eyes.” One group finished a week-long period of training by watching short video clips, while the other group did not do so. Score improvement in the experimental group suggests not only that emotion recognition can be improved with training, but also that observing and taking notes on video clips can be an effective training method. Hence, emotion recognition is a basic skill that can be improved upon with a simple activity that many can conduct in their own homes.

Methods:

There were eleven total participants in this study who were recruited via text message. During recruitment, I explained the steps of the experiment, but did not disclose the question of interest or my hypothesis to the potential participants so as to not introduce future biases while participants were completing activities. After an explanation of the steps and a guarantee of anonymity, the individuals replied with voluntary agreement to participate. All participants were young adults aged 20-21 attending various American universities. The participants were then separated into a control group in which no training was completed and an experimental group in which training was required. I used an online generator in order to randomly allocate the individuals to either the experimental or control group. The control group consisted of six individuals, two males and four females. The experimental group consisted of five individuals, one male and four females.

To begin, I asked all participants to complete an online social intelligence test titled “Reading the Mind in the Eyes” that contained thirty-six images of strangers’ eyes. Participants selected one emotion out of four options that best described the image seen. After each question, they were not told whether their selection was correct or incorrect as to not bias further selections. However, upon finishing, the test did allow participants to view their total scores. Following the completion of the test, I recorded the total of correctly selected emotion-image pairs out of 36 for each participant. This formed the base scores.

The control group was issued no further instructions after the initial exam to serve as a baseline of comparison for the experimental group. The experimental group underwent training by watching a short video clip without sound or subtitles from popular movies or television shows daily for a week. The video clips were administered each day via text message. The video clips I selected were all found on “YouTube”. Each video clip had obvious emotions of focus, such as sadness, anger, or happiness as well as clear visibility of the actors’ faces. A detailed list of which video clip played per day can be found in the appendix section.

I asked the participants to view the clips without sound in order to encourage emotional recognition based on the character’s eyes and not due to audio stimulation. In addition, I requested subtitles to be turned off so as to not distract the participants from focusing on the facial expressions of the characters. Participants were also required to make daily journal entries cataloging the characters and the emotions the characters expressed to verify that the activity was completed thoughtfully and to encourage more attention to the exercise. (Journal entries included in Appendix.)

A week after the completion of the initial “Reading the Mind in the Eyes” test, I had both the control and the experimental group retake the same original exam. Then, I recorded the new scores and calculated the difference between the second score and the base score for each participant. I averaged the difference in scores for the control group and the experimental group. The average differences were then compared.

Results:

When no training is completed, eye-to-eye emotion recognition ability does not improve. A compilation of the quiz scores of the control group can be found in Table One. Little score difference was observed between the first and second exam for the control group, suggesting that the participants’ emotion recognition ability had no change. The average base test score for the control group was 28.83/36 while the average score for the control group the following week was 28.5/36, a .33-point decrease (Table One). Only one participant (5A) in the group increased in score. However, the remainder of the participant scores in the control group either remained stagnant or decreased. This demonstrates that the eye-to-eye emotion recognition ability of the participants was not altered.

However, training to read emotions by watching short video clips does appear to increase eye-to-eye emotion recognition. The experimental group scores increased by an average of two points, suggesting that training does improve emotion recognition ability. The average base score for the experimental group was 27.6/36. After a week of observing and journaling about video clips with emotion present, the new average score was 29.6/36 (Figure Two). Interestingly, no participants in the experimental group decreased in score. All participants had increased scores except for one participant (3B) whose score remained the same. Because there was a marked difference in the average score changes between the control and experimental (-.33 vs. +2), there is evidence to show that eye-to-eye emotion recognition can indeed be improved with training, and that watching video clips and journaling is a valid method to do so.

Although the increase in the scores of the experimental group may be due to prior exposure/familiarity with the exam, this is unlikely because of the significant contrast between the control and experimental group results. All participants were required to take the exact same exam a total of two times. The control group, even with prior exposure, had a decrease in average score. Yet, the experimental group had a large gain in average score. However, participant 5A did increase in score despite being a part of the control group. Although familiarity may have played a role, there were many other factors such as testing environment, noise level, light level, etc. that may have affected the participant’s results. Because there are many varying factors, it is difficult to attribute the score increase to one or several specific factors. However, the control group’s overall average was lower, still supporting that training played a key role in score increase. Overall, this suggests that the increase in scores is attributed to the experimental group’s training and not simply familiarity with the test. Thus, training by watching short video clips can be an effective method to improve eye-to-eye emotion recognition.

Table One

Control Group Scores

Group One – Control
Participant	Gender	First Score	Final Score	Difference
1A	M	25	23	-2
2A	F	27	27	0
3A	M	28	27	-1
4A	F	29	29	0
5A	F	30	33	3
6A	F	34	32	-2
Average		28.83333333	28.5	-0.333333333

Table Two

Experimental Group Scores

Group Two – Experimental
Name	Gender	First Score	Final Score	Difference
1B	M	22	23	1
2B	F	24	27	3
3B	F	32	32	0
4B	F	33	36	3
5B	F	27	30	3
Average		27.6	29.6	2

Discussion and Conclusion:

Eye-to-eye emotion recognition ability can be improved with training. When the participants observed daily videos for a week while carefully noting the emotions they viewed on the actors’ faces, their scores improved by an average of +2. This improvement was much higher than that of the participants who did not watch any videos. The participants in the control group’s scores actually decreased by an average of .33. Thus, watching and journaling videos can serve as a simple method to improve emotion recognition ability. However, there were many factors such as the level of light, noise intensity, diet, variation in the time of day the exam was taken, etc. that may have interfered with participants’ performance on the quizzes. The participants took the quizzes in their own homes, so there was no method to control surrounding stimuli. There was also no method to verify whether or not the participant “cheated” by searching up images of emotions while taking the exam. In future experiments, it would be beneficial to have a set location for all the participants to take the exam at the same time with the same surroundings to ensure a similar test-taking environment for all and to deter possible cheating.

There were also many uncontrollable factors in regard to the training itself. Within the training group, it is likely that the videos were not watched at the same time every day because the participants were allowed to do the activity on their own time in their own homes. In future experiments, it would be beneficial to have participants maintain constancy when watching the videos and see if that further enhances emotion recognition. Additionally, as mentioned previously, light levels, noise intensity, and presence of others during training may have affected the training. There was no concrete method of determining whether the surrounding environment detracted or enhanced the training. Therefore, in future experiments, a set location can be determined to administer the training for the experimental group. Not to mention, the mental concentration of the participants during training likely varied. Although the journals were implemented to encourage thoughtful activity, there is no method of knowing how carefully each participant watched the video. Varying levels of effort can yield varying results. Thus, it may be interesting in future experiments to ask one group to take cursory notes, while requesting another group to write a detailed paragraph about the emotions they see in the video clips.

Notably, all participants were young college aged students, which is representative of a very small population. It is likely that training has different effects on different age groups. In addition, the sample size was quite small. It would be beneficial to have a larger sample size, so the results can be more indicative of a larger population.

Additionally, it would be remiss to not address that the “Reading the Mind in the Eyes” exam and the video clips were in English and examined by English speakers. Language likely influences expression and determination of emotions, thus the results may not be applicable to non-American cultures or non-English speakers. Future work could explore the differences of various languages on emotion recognition ability and the differences of various cultures on emotion recognition ability. Future work could also observe if different cultures have different ‘typical’ facial expressions for common emotions such as sadness, anger, or happiness.

Although not emphasized or analyzed, as present in the data, men typically scored lower than the women when taking the exam. It would be interesting in future experiments to see if there is a difference between the sexes for eye-to-eye emotion recognition and if training affects the ability of the sexes differently.

Nonetheless, eye-to-eye emotion recognition is a beneficial skill that can likely be enhanced if desired. Improved emotion-recognition ability can strengthen interpersonal relationships by forming stronger bonds with others as well as increased empathy with others. We interact with others quite frequently, in many cases, daily, therefore it would be beneficial if we could understand and empathize with each other to a greater extent. Increased levels of empathy can in turn lead to greater communication, less problematic interactions, and perhaps a happier lifestyle.

References

Calvo, M.G., Fernández-Martín, A., Gutiérrez-García, A. et al. (2018). Selective eye fixations on diagnostic face regions of dynamic emotional expressions: KDEF-dyn database. Sci Rep 8, 17039. https://doi.org/10.1038/s41598-018-35259-w

Matsumoto, D., Hwang, H.S. (2011). Evidence for training the ability to read microexpressions of emotion. Motiv Emot 35, 181–191. https://doi.org/10.1007/s11031-011-9212-2

Wegrzyn M., Vogt M., Kireclioglu B., Schneider J., Kissler J. (2017). Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS ONE 12(5): e0177239. https://doi.org/10.1371/ journal.pone.0177239

Appendix

Video clips:

Day One: The Bachelor – Hannah Ann Breaks up with Peter: https://www.youtube.com/watch?v=LOW6sFi1PAc

Day Two: The Notebook – Bird Scene: https://www.youtube.com/watch?v=d7_F5P5PygM

Day Three: Mean Girls – Meeting the Plastics: https://www.youtube.com/watch?v=re5veV2F7eY

Day Four:Anger Management – Rage on a Plane: https://www.youtube.com/watch?v=DzUc3Eqzzos

Day Five: Friends – Seven: https://www.youtube.com/watch?v=3NLVior-nLs

Day Six: Avengers Age of Ultron – Black Widow Flirting in a Bar https://www.youtube.com/watch?v=ycNVhd1IOzA

Day Seven: Hunger Games – Saying Goodbye: https://www.youtube.com/watch?v=KAal6-tvckw

Journal Entries

Participant 1B:

4/2/20:

4/3/20:

The Notebook (1/6) Movie CLIP – If You’re a Bird, I’m a Bird (2004) HD

• Man
  • Attracted and Cocky
  • Trying to act slick and cool
  • Fake hesitancy
• Woman
  • Playful and excited
  • Trying to be persuasive
  • They are kissing

4/4/20:

Mean Girls (1/10) Movie CLIP – Meeting the Plastics (2004) HD

• Blond Woman Middle
  • She seems a bit surprised
  • Seems like she is questioning the news questioning
• Blond Woman Left
  • Seems like she is questioning the news questioning
  • Seems like they are convincing her to do something that is not right
• Brunette Woman Right
  • Seems surprised
  • Seems like they are convincing her to do something that is not right
• Brunette Woman Alone
  • She is telling her something new
  • She seems excited
  • She seems nervous

4/5/20:

Anger Management (1/8) Movie CLIP – Rage on a Plane (2003) HD

• Adam Sandler
  • He is asking for something from the flight attendant, he looks like he wants it
  • He is put off and upset
  • He seems defensive after being accused of touching the lady
  • Nervous in court
• Flight Attendant Lady
  • She seemed nice and enthusiastic about fulfilling the request
  • She is having fun with her co-worker
  • She is upset and in accusational
• Air Marshal
  • In disbelief of the story he is telling
  • Doubting
  • Threatening towards adam

4/6/20:

Friends: Seven, Seven, Seven! (Clip) | TBS

• Man
  • Seems angry as he tries to explain stuff
  • Very intense in the way he tries to explain things to them
  • Maybe upset about something
  • Acting smarty with the blonde
  • Is intent on listening and curious
• Black Hair Woman
  • Teaching and Receptive
  • Explanatory
• Blonde Hair Woman
  • Chill and happy
  • Awkward reaction
  • Not buying into the advice

4/7/20:

Scarlett Johanson (Black Widow) Flirting with Mark Ruffalo (The Hulk)

• Black Widow
  • She seems like she is into him
  • She is talking in a flirty way and then dips
• Hulk
  • Seductive
  • Nervous
  • Uneasy
• Captain America
  • He is is kinda trying to be convincing

4/8/29

The Hunger Games (2/12) Movie CLIP – Saying Goodbye (2012) HD

• Prim
  • She devastated and sad
  • In distress and won’t let her go
• Mother
  • Serious and worried about Katniss
• Gayle
  • Giving advice and is trying to explain what to do
• Katniss
  • Sad but brave and reassuring

Participant 2B:

Video #1:

The girl looks distraught and upset and then her emotions turn to anger and frustration. At one point she looks irritated and confused. The man looks upset and guilty about something. The girl keeps looking up and licking her lips, while the guy won’t make eye contact with her. They both cry a lot.

Video#2-

The girl looks happy to be in the ocean and is trying to convince the guy to get in with her. He mouths the word no, but seems pleased with her. They are passionate and show affection. She seems turned on and he seems to be looking at her very lovingly.

Video #3

Lindsey Lohan’s character looks genuinely happy and shows some confusion. Regina George seems as if she’s sarcastically saying everything and is irritated and not genuinely happy. Gretchen wieners seems smiley and as if she’s trying to please Regina. The blonde one just looked confused and lost the whole time. Lindsey Lohan’s character also seems naive and innocent. Her smile is very pure and emits true happiness.

Video #4-

Adam Sandler looks content but grows more and more irritated with the flight attendant. The man next to him on the plane is a little suspicious and nervous he keeps looking back and forth between Adam Sandler and the flight attendant. After Adam Sandler touches the flight attendant she looks deeply disturbed and upset. The cop looks very intimidating and Adam Sandler looks scared. Everyone in the plane is curious as to what’s going on. Everyone else on the plane is terrified when the taser comes out.

Video #5-

The people alternate between looks of joy, happiness, suspicion, and confusion. The mans expression turns from distrust to surprise. It seems as if one of them is performing a magic trick.

Video #6-

Scarlet Johanson is obviously the one in charge and has the power in the scene. She keeps looking down as she speaks and won’t make eye contact with the man at first. Then she makes eye contact and he averts her gaze. He looks confused and in disbelief or denial about something. He looks as if he’s made a mistake but doesn’t know what his mistake is.

Video #7-

Prim looks sad and frightened and cry’s but Katniss tries to comfort her. She also seems scared but tries to be brave. Her and the boy exchange loving looks but both try to restrain themselves from showing their emotions. There is a somber tone.

Participant 3B:

Day 1: The Bachelor:

Peter: He looks really sad, regretful, and apologetic throughout the video. He seems anxious anticipating the girl’s reaction and worried that she might take it badly (?) and is trying hard to express himself

Hannah Ann: At first she seems like she’s smiling, like she doesn’t believe what the boy is saying but as time goes on it seems to set in more and she appears to be sad but disbelieving while also perhaps confused. At the end of the video she seems to accept what has happened and looks like she is trying to move on but is very sad.

Day 2: The notebook

Boy: looks at the girl admiringly and lovingly. He looks very happy and caring at her

Girl: looks flirtatiously at boy and seems carefree and blissful. Also very lovingly looks at boy but has a hint of mischief/joking manner in her expression

Observed emotions through facial expressions, noticing especially the eyes and mouths.

Day 3: Mean Girls

Regina: looks like she is superficially smiling and fake the whole time. Acts interested and friendly but has a disdainful expression at time. This is due to her lack of eye creasing when she smiles

The girls next to Regina look very attentive and eager to please, paying close attention and mirroring her expression

Cady: looks confused, cautious, and nervous. Seems afraid to say the wrong thing but eager to please. Looks sincere

Day 4:

Man: seems anxious as he tries to get the flight attendant’s attention, and perplexed/taken aback when she closes the curtain on him. He also seems slightly offended. When the people get mad at him he looks embarrassed and shocked, and then defensive, confused, and slightly intimidated

Flight attendant: looks annoyed and haughty. When her arm is grabbed she is angry and indignant

Police man (?): looks very angry

Judge and lawyer: look proud and resentful towards the man, like they think they are superior

Day 5: Friends

Man: looks upset and anxious about something. As woman 1 starts to talk, he seems perplexed and confused

Woman 1( black haired): looks assured and confident. Seems excited and emotional about what she is writing/chanting

Woman 2: looks skeptical and a little annoyed or over it after awhile. Also seems tired of it

Day 6:

Black widow: mysterious, appears disinterested but amused. Acts coy

Hulk: seems cautiously interested and vaguely intrigued. Also looks perplexed after captain America talks to him

Captain America: looks confident, friendly/pleasant, and knowing

Day 7: Hunger Games

Katniss: looks scared, anxious, and sad as well as stubborn/defiant

Girl: looks panicked and very sad

Boy: looks sad but accepting

Participant 4B:

I personally look at eyebrows, mouth, and body language for emotion. I think eyebrows become very expressive when angry or sad. The mouth is able to convey emotion through smiles or frowns. Closed body language such as crossed arms shows anger or anxiety, whereas more open stances seem relaxed.

Thursday, April 2

There is some unavoidable bias due to the title. Before watching the video, there is an expectation to see angry and sad emotions.

Girl looks disbelieving/skeptical at first

Boy is apologetic or upset

Girl is smug, but upset

Woman watching from small screen is disappointed

Main girl looks panicked/distraught

The characters watching themselves from the small screen look uninterested

Main girl is sad, but unsurprised

The boy came across as determined until the girl began crying, where he became ashamed

The girl then became accepting of the situation, whereas the boy seems sad

The girl is kind of mocking/taunting him, as if she is still sad but wants him to hurt instead

This worked, but now she can no longer hide from her face that she is still very upset

Girl watching herself from small screen finally shows emotion, looking sad/regretful

Friday, April 3

I’ve never seen The Notebook.

Girl is happy/excited

Boy is happy in a more subtle way, kind of nostalgic

Girl is flirtatious

The girl’s emotions are very open and exaggerated, whereas the boy is more subtle or guarded

Saturday, April 4

I have seen Mean Girls but will try to analyze emotions without bias. I can’t help but watch the video and hear the dialogue in my head though.

Regina looks determined and concerned

Cady looks happy, but confused

Regina is smug

Gretchen looks excited, but is then shot down by Regina

Karen looks confused the entire time, maybe concerned

Although Regina appears happy, it has underlying aggressive or accusatory tones due to the intensity of her stare (maybe biased by knowledge of movie dialogue)

Sunday, April 5

I have seen Anger Management before.

The old man (Buddy) and Dave (Adam Sandler) start out relaxed

The flight attendants seem carefree and happy

Dave starts to look uncomfortable or impatient

The main flight attendant seems condescending

The flight attendant is aghast/disgusted that Dave tapped her arm

The attendant becomes stern or angry

Dave seems apologetic

The man with the badge is very serious

Other passengers are shocked or concerned

Dave is becoming more frantic, where before he was calm

The man with the badge is becoming more aggressive in his anger, confrontational

Dave becomes very frustrated to a point of anger

Monday, April 6

I have never seen Friends.

The two girls are happy and interested/curious

The man seems angry, but not at the girls

The girls are sympathetic

The man seems doubtful

The girl with dark hair is acting earnestly and sincerely, whereas the blonde is teasing

The man seems confused or worried

Tuesday, April 7

I haven’t seen this movie, but the title of the video adds a bias towards flirty emotions.

The woman is uninterested by or wary of the man

The man is amused by whatever the woman is saying

The man is getting nervous as the woman speaks bluntly

The woman is now outright teasing (mocking) the man

The man is disappointed

The new man (Chris Evans) is smug

The original man (Mark Ruffalo) is either confused or intrigued by the other man’s statement

Wednesday, April 8

Wow, I watched The Hunger Games earlier today.

Prim is incredibly sad/distraught

Katniss wants to be strong, but both her and her mother are in shock

Katniss becomes very serious and stern

Prim is extremely panicked

Gale is being supportive, whereas Katniss starts to show fear for the first time

Participant 5B:

Day 2: Bachelor Video

• The bachelor appears to be sorry for something because he is crying
• Hannah Ann hands an engagement ring over and seems to be both upset and sad at the same time — they were engaged but are no longer
• Both are crying hysterically
• Hannah Ann has a look of disapproval and nods many times
• Peter closes his eyes and licks his lips, almost in disbelief and unable to do anything — he knows he has messed up
• After Hannah Ann walks away and Peter tries to talk to her again with sad eyes, you can tell she is done with him because she continually nods in anger and disapproval

Day 3: The Notebook

• Girl runs into the waves and throws hands up into the air (she appears really happy)
• The guy watches her in the water and nods, but this time he nods and smiles
• The girl runs out of the water and jumps up and down, trying to get some sort of approval from the guy
• The guy nods and she jumps into his arms after she gets the approval
• They talk to each other and appear to be in love

Day 4: Mean Girls

• The blonde girl in the center appears to be trying to prove herself to the girl sitting across from her
• Blonde girl raises her eyebrows multiple times, trying to question the new girl
• The brunette girl smiles and seems excited about something she is talking about, but then becomes sad and quiet after the blonde girl shuts her down
• The blonde girl sits back in her chair and keeps talking, belittling the other girl
• You can tell the blonde girl is the ring leader because all of the other girls look at her when she speaks

Day 5: Anger Management

• Flight attendants are laughing and talking to each other, ignoring the man on the plane trying to get their attention
• The man next to him is listening to a movie and is being obnoxious
• The guest is obviously very frustrated, but he gets the attention of a flight attendant
• The flight attendant takes this as anger and the man shakes his head that he wasn’t trying to be violent, but the flight attendant escalates the situation
• More people see the interaction and start ganging up on the passenger
• He ends up going to court against the flight attendants and it appears he loses; the attendants seem happy to get revenge

Day 6: Seven

• Guy looks like he messed up somehow and the two girls look like they feel bad for him
• He walks over to the couch and it seems like the girls are giving him advice
• I think this episode is about dating?
• The guy seems confused by what the girls are saying at times

Day 7: Black Widow

• The guy walks up to the bar and seems to be talking to the girl first
• But the bartender then seems interested in the guy because of her eye movement
• They both take a sip of a drink and maintain eye contact, so I think they are interested in each other
• Another man comes into the scene and he talks to the first guy
• He appears to be commenting on the guy’s interaction with the bartender and how well it went

Day 8: Hunger Games

• Katnis is leaving and says goodbye to her sister; it’s obvious that the sister is heartbroken
• She tries to tell her younger sister to be strong and that everything will be ok
• She then goes over to an older woman and tries to reassure her; they are both very upset and sad
• When the guy comes in he hugs her and they have some sort of relationship
• It ends with her leaving urgently