Train, Validate, Re-train: How We Build HireVue Assessments Models

In my first blog post in this series, I shared the history behind HireVue Assessments, which use machine learning to reduce bias in decision-making for recruiters screening job applicants.. This approach solves the problems inherent in traditional multiple-choice pre-hire assessment tests, which are time-consuming and all too often a deal-breaker for the candidates who have the most options. HireVue has now built a system that aims to accurately predict job success from a structured video interview, making the screening process both faster and fairer.

A Quick Overview of how HireVue Assessments Work

The team at HireVue is composed of data scientists, industrial/organizational (IO) psychologists, and engineers. To create a custom HireVue Assessments model for a particular job role at a customer site, we use input gathered from a recorded video interview. To ensure that it’s effective at predicting success, our IO psychologists need to understand the job role in-depth — and most importantly, what it takes to perform well (or not) in that role. We use this understanding to train our model. When a candidate is assessed, HireVue Assessments provide a score that indicates how well that candidate is likely to perform in the role.

It’s important to note that HireVue Assessments don’t replace all person-to-person interviews, nor do they determine whom to hire. These scores provide decision support for recruiters, who then decide which candidates to move along to the person-to-person phases of the recruitment process.

Our Approach to Building Assessments Models

Our process for building models follows the best practices of industrial/organization psychology (the science behind traditional assessments) and machine learning:

• Job Analysis and Assessment Design: We analyze each job and identify the competencies required to be successful in the job. Then we design the interview questions that will be most useful in allowing candidates to demonstrate those competencies. This phase is where we identify the job performance measures that are critical to success in the role, such as sales quota attainment, customer satisfaction scores, call handle time, or annual performance review.
• Data Collection: Next we run a population through the assessment. This can either be existing employees or prospective job candidates. Each person that takes an assessment will also have a performance measure as a benchmark.
• Model Build: We train the machine-learning algorithms (also known as a predictive model) to predict the likelihood of success in the job based on data from the interview. Then the model is tested to ensure that it accurately predicts the job performance measure based on the assessment answers.
• Test for Bias: Each model is tested to make sure that it does not have adverse impact against specific groups (e.g. gender, race/ethnicity, or different age groups).
• Re-training: If the model is found to have any adverse impact or to contribute to bias, we look for any factors that are contributing to it and remove them from the model, which is then re-trained and tested before deployment. It’s possible to remove problematic factors and still maintain a highly accurate predictive capability because so many potential data points from the interview can be considered in the model.
• Implementation: We launch the assessment model and begin using it with real candidates. At this point, recruiters can see the list of candidates who have taken the video interview and assessment, play the recorded interviews, and see a score for each candidate. Higher scores show candidates who are likely to be most successful in the job.
• Ongoing Monitoring and improvements: HireVue periodically monitors the results to ensure that the model continues to predict the job performance metrics and is not creating any adverse impact against groups of candidates. Any factor found to contribute to bias is removed from the model. As the customer’s goals, objectives, or roles change, the model is updated to maintain predictive accuracy.

What Is the HireVue Algorithm Looking for in the Video Interview?

The HireVue Assessments model — the algorithm — is looking for the same things that you or I would notice if we were doing an interview. This includes which words are used, the type and meaning of statements made, and facial movements. Because there are variations between different cultures, we train the model using data featuring people from the same culture. See our Japanese assessments model press release for one example.

Our data scientists and IO psychologists are able to design HireVue Assessments algorithms to include only the data points proven to be predictive of job performance, and to leave out the things that don’t matter. This eliminates consideration of many characteristics that can be unintentionally distracting or influential to human evaluators.

It’s critical that companies creating artificial intelligence algorithms be committed to a code of ethics. At HireVue, we have been dedicated to doing good science in a methodical, fair, and ethical way since Day One, and you have our commitment that we will remain dedicated going forward.

In part three of this blog-post series, I address the issues and misconceptions about algorithmic bias in more detail.