Nonverbal communication can be complex and difficult to interpret, yet we know it is highly important in human interactions. For years, HireVue has been interested in the many ways people communicate in the job interview context. By and large, we have found that data we get from the language used by candidates within our interview assessments are predictive of job performance and job-related competencies, with information around nonverbal behavior adding additional predictive value only in certain contexts.
Traditionally, much of the research in nonverbal communication through facial movement has been focused on the work of Carl-Herman Hjortsjö, Wallace Friesen, and Paul Eckman, who worked to develop the Facial Action Coding System (FACS), a taxonomy of facial muscle movements and their tie to emotion. Certain “action units” flex and combine to surface as familiar expressions people often interpret as “smile”, “anger”, etc. Eckman and others concluded that there is a small set of basic human emotions that can be accurately interpreted cross-culturally based on how they are displayed on the face.
However, more recently, much of this research has been called into question. Dr. Lisa Feldmann Barrett’s recent review of the literature in this space concludes:
- There are not just a small set of basic emotions.
- People/cultures express things differently, particular situations change how the same emotion is expressed, even by the same person.
- Emotion cannot be accurately inferred from facial movements.
Moving forward, there is a lot of interesting research to be conducted in this area. Humans convey information nonverbally, but how they do it can be quite complex. To further obscure reality, previous research has been focused on photos of facial configurations, often without context and overstated by actors.
Malcolm Gladwell’s recent bestseller Talking to Strangers is filled with interesting stories about how difficult it can be for humans to understand each other. Not only can we not reliably know how another person is feeling inside, but often we don’t know how we ourselves are feeling inside. Does a smile mean you are happy? Not necessarily. But that does not mean a smile is meaningless either. A smile may mean you are happy, it also may mean you are trying to look friendly, happy, or approachable to others. The behavior we exhibit to help others understand our intentions is a very important part of communication.
At HireVue, we build algorithms to find patterns of behavior that are predictive of job performance or job-related competencies. Pre-Hire assessments, including HireVue Assessments, have never been designed to determine emotional states, but rather quantifying aspects of candidate behavior.
By far, the most valuable data we can pull from a video interview is the language a candidate used. We then think of nonverbal communication in terms of what additional predictive power it may offer. It turns out there is a very big overlap in the language people use and their nonverbal behavior. The nonverbal data here consists of facial muscle movements and audio features, such as variation in tone.
The HireVue Data Science team trains machine learning algorithms to find differences in behavioral patterns between those who performed well on a job or exhibited a job-related competency and those who did not. In most cases, nonverbal behaviors are found to have predictive value and make a meaningful contribution to a candidate’s score. Simultaneously, they often do not have a lot of predictive value beyond language, meaning much of the information conveyed nonverbally is redundant. This is a very interesting result because it means that within a diverse dataset, nonverbal communication is a reflection of verbal communication in a fairly consistent way. Many of our algorithms do not see additional predictive power when nonverbal data is added to language data. As depicted in Figure 1, if we do not see significant additional predictive power from nonverbal data, we simply exclude those features from the process of training the final interview assessment model.
Figure 1 (top) The potential contribution to a candidate’s score of each feature class averaged over many deployed algorithms. This means that variation in facial muscle movements contribute 6% of the candidate’s score and so on. (bottom) The contribution to the predictive power of the algorithm. The contribution of nonverbal communication has very little incremental value, and therefore, it is not used in the final model.
Figure 2 shows the same breakdown, but with an algorithm that was built to predict success outcomes for a highly interactive service role. Many of the interview questions for these types of positions are scenario-based, i.e. role-play on the part of the candidate to respond to a difficult customer request. In contrast to most algorithms, nonverbal behaviors were seen to be significantly different in top performers in this role when contrasted with lower performers and this difference was not accounted for purely by the language used. In this particular case, facial expressions that are interpreted as warm in the western world were displayed more by top performers (i.e. smile), while expressions that are interpreted as negative (i.e. look of contempt) were much more common among those who did not perform well in this role. In this case, we would include nonverbal behaviors in the assessment as long as they do not cause significant bias, which is discussed in the next section.
Figure 2 Highly interactive customer-facing role. (top) The potential contribution to a candidate’s score of each feature class. This means that variation in facial muscle movements contributes to 14% of the candidate’s score. (bottom) The contribution to the predictive power of the algorithm. The contribution of nonverbal communication is far higher than it is for most roles (shown in Fig. 1).
Approximately 20% of HireVue interviews are scored by an algorithm, and the majority of those use only language for scoring. Companies are offered the mechanism and encouraged to clearly inform all candidates who are being scored by an algorithm and HireVue complies with all laws relating to consent and privacy in hiring.
There are cultural differences in both verbal and nonverbal communication. What makes someone successful in a role may include cultural behaviors that are not universal, but may be important nonetheless. A Manhattan bank teller might be considered rude in Cheyenne, Wyoming.
As with any machine learning algorithm, it is very important that the training data reflects the data that the algorithm will see “in-the-wild”. For this reason, it is important that training sets be diverse and an algorithm trained on one population in one part of the world not be applied to another population in another part of the world. It is also important that candidates’ predicted outcomes (scores from the algorithm) are only compared to the group of candidates that applied for the same role in the same geography. To cite the same example, the pool of potential bank tellers in Manhattan should not be compared to those in Cheyenne.
Bias is an important consideration for any algorithm. HireVue looks at several different bias/fairness metrics to ensure that different demographic groups are not treated in a significantly different way when subject to a given algorithm. If we find that a behavior is significantly correlated to a protected class, we will not use it. Protected class in this context always includes age, race, and gender, but when possible, also includes characteristics such as disability, veteran status, and so on, depending on the availability of data.
Both verbal and nonverbal behaviors are an important part of human interactions. Our data show that often the sentiments expressed through these two channels are aligned and nonverbal data does not add much to the predictive power of our algorithms. For some roles, however, nonverbal behaviors do offer additional value in understanding the differences between those who performed well in the job and those who did not. This is often seen in highly interactive roles where, for example, a calm tone-of-voice or demeanor is valued greatly.