As organizations across industries expand their need for talented software developers, recruiting teams require a scalable way to evaluate technical talent. One of the most effective, validated ways to do this is the technical assessment.
Technical assessments provide a way to evaluate a candidate’s ability to code, consistently and objectively. They are a crucial tool to ensure organizations hire candidates with the right skills to succeed.
In this article, I’ll examine the best practices behind world-class technical assessments, particularly those that expertly assess candidates from all learning backgrounds.
A common question we often receive when building the coding challenges that make up technical assessments is “Should hiring teams use simple challenges or complex challenges?”
Our recommended approach, which is light touch and high-value for screening candidates, is to use a relatively simple coding challenge in a specific programming language that you know is a requirement for the job.
Keep in mind that a “simple” coding challenge might look different depending on if developer candidates are experienced or entry level. Entry-level developers, because they need to have the basic skills and ability to learn on the job, should be evaluated with the simplest challenges. You don't need to send all your candidates down a wormhole into a three-hour programming task to ensure that they can be an effective coder.
A better approach is to use the coding challenge to confirm that they have some base competency in the relevant programming language. Using more complex challenges becomes a slippery slope because it can favor candidates that have a formal Computer Science education over those that are self-taught, have fantastic practical skills, or have stellar soft skills and the ability to ramp quickly.
A side note for hiring experienced developers: More experienced developers can have a more complex coding challenge and follow-up questions (more on this in consideration #3), but it shouldn’t take too long, or it may lead to dropout. Experienced candidates generally expect to have a conversation first, to understand what to look for and how they can prepare. An invitation to take a technical assessment should not be the first communication they receive. Depending on the scenario, it may be more beneficial to skip the technical assessments, and go straight to a live coding session with the technical hiring team.
If you’re using HireVue’s technical hiring solution, you can go beyond this. Using a follow-up recorded video response to a question turns the coding challenge into a conversation that gets candidates talking about:
These are far more interesting than optimizing an academically obscure algorithm as a way to confirm that they attended a formal computer science program (as opposed to solving problems in the real world).
You can also use video interview questions to evaluate candidates’ job-relevant soft skills. Situational judgment, scenario-based, and other behavioral-type questions yield insight into:
Most developers spend less than half their day coding. If you aren’t testing for soft skills like teamwork, communication, and collaboration, your technical assessment isn’t assessing for the reality of modern dev work.
Coding challenges and recorded interview questions are not the only type of question to use in an interview for an assessment of the candidate's domain of knowledge.
Additional scored, multiple-choice questions can validate candidates’ knowledge of frameworks, tool kits, or architectural topics that are relevant to the job. These typically are difficult to assess in a coding challenge, and would greatly lengthen the assessment if they were.
Combining multiple choice-type knowledge questions with validation of basic programming competency is a powerful combination that vets relevant knowledge. Knowledge questions provide a way to vet the high-level proficiency of both entry-level and experienced developers without requiring a significant time investment on their part.
Computer programmers are constantly solving a vast array of technical problems. Often, there are multitudes of solutions posted online for any given programming problem and no particular solution is the best for all potential applications of that code.
Unfortunately, this means that coding challenges and their respective solutions are frequently posted online. Technical hiring teams could be presented with a slate of the best cheaters, not the best developers.
Best-in-class technical assessments offer several ways to check the uniqueness and authenticity of a candidates’ code.
HireVue’s CodeVue technical assessment solution is built to empower technical recruiters and hiring teams and detect cheaters. We built it so non-technical recruiters could make informed, qualified screening decisions on technical candidates.
This is our approach:
One of the best ways to avoid cheating is to deter it before it happens. CodeVue offers several functionalities that empower hiring teams to deter cheaters:
CodeVue uses Measure of Software Similarity Scoring (“MOSS”) algorithm scoring to automatically flag candidates who likely cheated.
MOSS algorithm scoring is a widely recognized way to compare electronic documents and identify which ones have an unusual amount of similarity between them. CodeVue uses MOSS scoring to identify when a submission is similar to other candidate submissions for the same challenge. Although high code similarity is an indication of suspicious activity, coupling this with the code editor playback tools and follow up video questions are a good way to really understand if the candidate has cheated.
CodeVue also provides additional data points which can be used to gauge candidates’ thought processes.
If you want to take a deeper look at how a shortlist of candidates came to their solutions - and vet if they have cheated - the Code Editor Playback functionality allows you to play back a recording of the candidates coding session. You can watch a candidate’s keystrokes to see how they worked out the code and observe exactly how they developed their solution in the code editor. You can also speed up the playback up to 32x the actual speed, and skip through periods of inactivity, so you don’t have to sit through lengthy solutions as they are coded.
Code Editor Events are displayed as part of the playback to help you see key actions that the candidate took during their coding session:
Another method to identify potential cheaters is via Time Tracking. With time tracking, we capture how much time was taken for the candidate to produce their answer.
It's a pretty simple litmus test. Ask yourself when evaluating the candidate's submission: “Is this a realistic amount of time for them to have completed this on their own?” If a typical candidate solves the challenge in 15 minutes and one candidate submits a solution in under 5 minutes, there is a high probability they have cheated.
Another data point we alluded to previously: the ability to create Follow-Up Video Questions. Adding a question to the interview immediately following the coding challenge is a great way to get the candidate to talk about their solution. A three minute video response from the candidate is plenty of time to validate their understanding.
All of these tools are powerful ways to understand if a candidate’s solution is suspicious. And because you do not need to know code in order to identify suspicious activity, these resources allow non-technical recruiters to quickly disposition candidates who have cheated and flag suspicious activity before they pass the candidates over to the technical team for a deeper review.
Altogether, these functionalities empower recruiters to make qualified screening decisions on technical candidates. Rather than submit candidates to hiring managers based on a resume review and phone screen, recruiters gain deep insight into candidates’ coding proficiency. They can give the most qualified candidates an expedited, high-touch hiring experience, while freeing up time for managers to focus on the best candidates for their teams.
Scott Keller is HireVue's CodeVue Product Champion. As product champion for the CodeVue technical hiring solution, he consults Recruiters, Technical Hiring Managers, and HireVue’s Product Engineering Group on how to recognize and retain talented software developers. Prior to working at HireVue, Scott was Senior Architect and Consultant at Navitaire, where he worked with development teams across the globe on building custom back office solutions, earned data processing patents in 3 countries, and invented the Temporal Tables model of the ANSI-SQL 2011 standard. Find him on LinkedIn.