IBM WATSON’S APPROACH FOR HIRING THE BEST OF THE BEST
by JEFF JABLONOWSKI
As one of the original members of the IBM Watson commercialization team from 2011, Jeff Javlonowski was responsible to build out and lead the software engineering development team that commercialized the IBM Watson Jeopardy winning system developed by IBM Research.
Jeff is currently leading a Watson development organization that is moving Watson onto IBM's Bluemix Cloud Infrastructure.
Scott: This afternoon, we have a featured keynote from IBM Watson with Jeff Javlonowski. He is one of the original IBM Watson commercialization team members from 2011. This is the team that productized the IBM Watson Jeopardy Game Playing System. He is an IBM Watson core technology development manager that has done a considerable amount of hiring and recruiting to grow his team. He will share his experience in their technical interview process and the role of the HireVue tools for building out the IBM Watson team and hiring the best of the best. Thank you for being with us today, Jeff.
Jeff Javlonowski: Thanks for the introductions, Scott. I appreciate the time today. I'd like to spend a little bit of time...again, my name is is Jeff Javlonowski. I'm going to take you through our use of the HireVue CodeVue tools and how we're using it here in the IBM Watson organization to help us hire some of the best of the best in the industry and hope you get something out of the...how we're doing it here in Watson. So let me start with kind of where we're at with Watson and some of the challenges facing us today. So what we're dealing with is, how do you deal with a candidate pool in excess of 1,000 or even 10,000 applicants when you look at our intern pools of people that wanted an internship with Watson this summer? How do you best spend that precious time of yours and your candidates' time with the right folks? And most importantly, how do you drive the offer acceptance rate up? Something north of 80% with some of the best of the best in your candidate pool. So I'm going to spend some time, the next few minutes with you, going through our practical experiences using the HireVue CodeVue tools and its important role and its use in IBM Watson's talent recruiting program.
So let me start with the technical screening process. So the way to do this is it's absolutely essential and critical to have an effective and efficient technical screening process. Here at IBM Watson, we're using two of the tools from the HireVue organization, what's called the CodeVue OnDemand tool and also the CodeVue live tool. So I'll take you through later in the presentation, specifically our own experience but let me show you in the flow here where it fits in the recruiting and talent process for IBM Watson. So the IBM Watson team uses CodeVue OnDemand as a tool to screen our technical positions by giving the candidates an on-demand coding challenge exercise. And followed by that is an electronic whiteboarding set of interviews done if a candidate passes the first set of on-demand autoscored and evaluated exercises. Don't get me wrong, we still value and leverage on-site interviews in our decision process. But these CodeVue tools are very important for us to help us ensure we're spending the right time with the right folks.
If you look at the graphic on the right, you will see what I would call a pass through, which is kind of a representative number and how candidates would pass through a technical screening process. So if you think of the total set of people starting with the on-demand exercise, that would be the full set of a 100%. And then moving next to the on-demand screening process, 75% of the candidates actually took and completed the exercises. Of that...that's when the evaluation of the technical screen process begins and we looked at passing that in the 45 to 50% range is something that you could expect. And after that process...now that is getting you a screen candidate pool that you are comfortable with, has the right skill set, and interests to match your posted positions. Next we move to a live phase, where we're actually doing live electronic whiteboarding interviews. So this could be the first time your technical team engages with the candidates. So that could be...so you have a good solid body of candidates at this point in the process, now we're looking at going through and interacting with the candidate and we'll be seeing experiences maybe in the 70% range of people that will pass that and then proceed into being considered for an offer.
So what I'm showing you is kind of a representative process here and do not think that this on-demand starting point here is your entire candidate pool. There is a talent assessment done at this point. So there is a bit of resume screening and skills matching to ensure those candidates are matching well to your needs in the organization. So let me proceed and give you a little bit of what we've seen and some of the experience we've seen with the Watson teams. So be prepared to be surprised. You will find that candidates will be able to stand out that perhaps didn't stand out on paper and on their resume. Not just resume bloating by people bragging but really looking at the results and evaluating a submission from a candidate. You have more than that one or two pages of paper to really get a feel before you're even interacting and spending your time with the candidates. Look for and find the passion and the valued experiences of your candidates by being able to ask them questions and seeing video responses returned back to you. See the innovation in their coding and problem challenges. So give them challenging exercises and look for that innovation and those creative problem solving skills.
And you will find that candidates are really able to differentiate themselves and find...I was very much surprised by some of our candidates results that we've hired this year in 2015. And we've been able to use this tool to effectively evaluate a larger candidate pool. We were able to send the exercise out to a larger candidate set. Perhaps some of those candidates that you would say they're on the edge of whether or not you want to spend the time with that candidate, you could send them the CodeVue on-demand exercise and let them differentiate themselves and see if they're the right candidate for you to spend time with. So done right, this will allow you to spend the right amount of time with the right candidates to match the needs of your organization.
So how is this done? Done in volume, this is certainly not a one man or one woman exercise. So what we've done here in Watson is enabled a relatively large team of technical screeners to do and review the CodeVue results of the candidates. So we've done enablement across our team, shared what I would call our answer key with our reviewers so that people understand what's to be expected and typical results are and have a larger pool of technical screeners doing these CodeVue on-demand reviews to see if we see the skills and interests match with our candidates for IBM Watson. Next, we move to the...when we move to the live interview phase, we will do a CodeVue live electronic whiteboarding session and we will do that with a hiring manager along with a team of technical leads for that part of the organization where we see the best fit for that candidate. So this will be a combination interview of a fit interview as well as a skill set match and interest interview. So we've kind of combined the conversations between the management team and the technical lead team with the candidate in a very engaging conversation.
Also very importantly, we have the role of what we call a hiring champion by major job function that will assist in matching candidates, perhaps the example would be that the CodeVue whiteboarding exercise saw a very capable candidate but wasn't an exact match for the needs of the team doing the interview. The team doing the live interview can engage the hiring champion and perhaps recommend another part of the organization that would benefit from and just consider that candidate. So we do have the ability to do very focused interviews but we do also utilize the ability to match that candidate with a larger part of your organization. Another large benefit you will see from leveraging these CodeVue tools is flexibility, flexibility built in by design.
You will see that the CodeVue on-demand exercises are asynchronous meaning you send and invite a candidate to take a coding challenge. They take it on their own schedule, when they have time and energy to focus on your interview. You also have the same benefit on your end as the reviewing hiring team to review that candidate submission and see if it's a good match for group. The net result of this, it provides...it's really providing you a much more professional experience where you're not playing phone tag, you're not having people rushed in and out of a phone screen interview to see if this candidate is the right skills match for your group. You will have a much more professional, polished experience and that's the feedback that we receive with Watson and how we're using the tools today. So let's get into some of the practical experience and practical uses of the tools. I mentioned the tool and now you can see one of the screens that you would see as a reviewer. The CodeVue OnDemand tool today that's available, allows you a combination of the ability to record a video introduction to your candidate explaining to them a little bit about your organization. And then, we can get right into a series of questions and challenges presented to your candidate. We've done use of today. Our use has been a combination of thoughtful video response questions, posting a set of text to the candidate and allowing them time to contemplate an answer, provided back as a video response.
So again, you're not seeing a recorded or an audio over the phone response from your candidate. You have a webcam recorded interaction with your candidate to see and get a little bit of a feel for the capabilities and abilities of that candidate to match your needs. You have the ability of submitting coding exercises, coding challenges. So these will be timed exercises. You pick the time to give your candidate, the amount of time deemed necessary. As well as the ability to autoscore and evaluate those exercises. What I'm showing in the image on the right is a picture of what you might see as a candidate looking at a job...a technical exercise that we gave out. So again, you have the ability to give example test set to the candidate, amount of time to code it up, and as well as when you compose this coding challenge question, you also have the ability to give what I would call blind questions where you have test steps to actually run where those are not visible to the candidate. So your score is a combination of the public viewable as well as the private blind question sets. But be careful. Do not get hung up and too focused on absolute, empirical results.
Again, coding and problem solving skills is not always the end result and calculated result. We do spend the time to look at the quality and the approach taken from our candidates. I'm just cautioning. Be careful. Don't look for the absolute scores and try to assess candidates because as we know in the technical industry, it's not all about coding up something that comes out with the expected result. It's also the approach and the creative problem solving and how do you do that effectively is looking and reviewing their code. And we also utilize the combination for the coding challenges and video responses. And what we call video responses to the coding challenge where we actually ask the candidate to describe their approach. So we get a little of the ability to the thought process. We see that as a big benefit here and an innovative way of using the tools together to look at the right candidates and see their problem solving skills. Next for candidates that pass an on-demand autoscored exercise, we will move that candidate forward into the live electronic white boarding. Again, you still have not needed to bring a candidate on-site.
This is where we would have a combination interview with our management and our technical leadership team to have an electronic whiteboarding screen sharing session with the candidate. So this is a very engaging experience and you will see...if you can see the image on the right, you will see that in the background, we will see the video webcams of not only the candidate but the participants as well. So it is a very engaging experience allowing you to flip focus between the whiteboarding session and the video webcam conversation you had with the candidate. So it is very engaging and we find it effective. We're using this across some of our development sites so we're not even needing to be all in the same room to do this interview. We'll have the IBM team at multiple sites having the right people talking to the candidates. So very receptive. We also found a practical approach here was to be prepared and paste the question or exercise into the whiteboard. So when we engage with a candidate, you're still conversation over the phone rather than not losing anything in the verbal quality of the communication. We'll paste the actual exercise.
It's just another way to be more engaging with the candidate. And done right, again, efficient and effective assessment of problem solving skills. So very engaging and at this point, we would be moving...candidates that pass this point, we would consider for coming on-site and evaluating them. So again, a good combination of innovative tools to get to talk to the right people with your team. So achieving a high acceptance rate. There's not a silver bullet here. This is a combination. This is going to be a combination of a set of offerings, your technology itself obviously is an important part but many factors are involved here of...the offer itself, the innovative experience you provided with the candidate. And look at constantly innovating your approach on how you do this. It's innovation in terms of how you interact with your candidate and also how you leverage your team, your extended team. Remember, the candidates here are also interviewing you and looking to see, "Is this an organization I want to join?"
So have the passion and the commitment on both ends of the conversation. Look for that and convey that to your candidates. Remember, they're going to be assessing you and your abilities to meet their needs and their growth needs in their career as part of the interview process. If we look at this, in the process here and obviously a key part of this is a pipeline of talented individuals for your organization. I talked about a process for how to screen those candidates but this process only works if you have the right candidates in evaluating people with the right skill sets for your organization. So what we've done is empowered our team where our current college hires from 2013 are playing a large role in our recruitment process for bringing on our 2015 team. So look to them, empower them, and leverage them in your process. Our on-campus interviews where we're building our pipeline, obviously we're sending our alum to those events but also have your team and reward them to go out and attract new and talented individuals to your team. Build your team with wave...your wave two team should be your wave one team that's hiring and doing the screening as part of the process. Remember, talent will attract talent and you get the right combination of folks, wave two is going to be a very talented and capable team added to your organization.
Remember, be humble. This is my own personal experience going on-site campus recruiting. The candidates did not want to talk as much to me as they did our recent graduates from that organization. So be humble and remember that the candidates will want to and relate to these new hires from you organization as well. You are the decision maker and the key part of the process but again, my message here is leverage your extended team and attract new members through your existing team. And IBM's commitment, so just in summary wrapping up here, IBM is committed to an equal opportunity hiring process. I've talked about just one aspect of that process here with our technical screening process. Again, this is part of a larger talent recruiting process here at IBM but I hope you've enjoyed some time. I know I have, summarizing how I've used the IBM...IBM Watson team has used the the HireVue CodeVue tools today and continue to use them in our screening process. And I hope you see an angle and perspective and ability to use these withing your organization. With that, I'd like to wrap up my session today and I'd invite you to attend two breakout sessions starting at approximately 12:30 today and hope you enjoyed the time. I know I did and thank you.