So once we figured out our criteria, that is our characteristics that differentiate somebody is going to be successful in our organization from less successful, then we need to evaluate candidates. That is we need to figure out for a given candidate do they have these characteristics? Basically choose the right selection techniques. Now it turns out we have over 100 years of research which is really trying to understand which of these selection techniques work and which don't. Of the various different ways that we can evaluate people, ultimately, which are predictive of performance, which do a good job of telling us who ultimately is going to be a good employee and which don't. And we've learned an awful lot from that research. So maybe take a minute ask yourself, when I'm hiring candidates, what do I think is actually going to tell me whether or not somebody can do the job? Of the various ways I can try and figure out information about them, which is going to be most successful? Thought about it? Got your favorite selection techniques there, let's see how they stack up against my list. I'm actually going to start with the ones you don't want on your list ,okay? What are the things that we know from research are really not very predictive of performance? I start with one that's probably not on many of your list, which is graphology, but it's always a good example. So allegedly in a lot of European countries in the late 20th century graphology was a popular way of evaluating candidates. What's graphology? It's studying handwriting. So ask people to write a handwritten statement about why they wanted the job and what would be good for it. They would ask experts to evaluate them based on the content of what they're written, based on their handwriting. Their idea was from the way they formed their letters and all those sorts of things, you could learn a lot about their personality. Obviously the psychologist looked at that one, that's interesting, I wonder if it works? And the answer fairly clearly was no, no it doesn't work. I mean it turned out their predictions weren't terrible, but their predictions weren't terrible because they were reading something somebody written about why they are suitable for the job or something like that. They could learn stuff from that. What they actually learned in addition to that from the actual handwriting appeared to be entirely unpredictive of almost anything. So I'm not sure many people still use graphology. But if you are considering it, don't. Maybe a slightly more surprising thing we have on the list here is reference checks, okay? Should we do reference checks? I mean, they're not a terrible idea. I mean it's probably a good idea just to check somebody isn't a serial killer before you employ them. But if you're going to use reference checks to try and understand are they going to be a good employee or not? They're probably not going to work very well. Why is that? I mean there're various explanations. So a lot of people point out somebody is not really going to put somebody down to give them a reference if that person is unlikely to be positive about them. In the legal environment now makes a lot of people very nervous about giving very much information references, and certainly about giving negative information about them. But I think on top of that, even putting that aside, a fundamental problem with reference checks is they don't tell you that much about the person who the reference is being written by. Remember again, what are we trying to do here? We're trying to make a prediction about somebody's performance based on the information that were given, okay? And from the reference we're doing that based on what somebody else has written about them. But what predicts the quality of that reference? Having read an awful lot of these over the years for various kind of academic hiring stuff. I can tell you that one part of the reference is predicted by the performance of the person the reference being written about. But a huge amount of the references basically predicted by who's writing the reference, right? There are some people who just write these wildly over the top references about how this is the best person they've ever encountered, and they'll surely be running the world. But until that time you should definitely hire them. And there are other people who write these alarmingly honest references, like, yeah, this person seems okay. And he's not obviously a thief, why not hire them, right. The difference between those references is really the person writing reference. That's not to your decision. That's not something that you're terribly interested in. And so it's not something you want to affect your decision. So ultimately, like I say, reference checks, decent due diligence. But comparing the references that people are given to decide whether or not they deserve a job, usually a terrible idea. Just don't predict performance. The third one, and I think the one that surprises most people, is unstructured interviews don't do a very good job of predicting performance.? What's an unstructured interview? It's what we do naturally, we sit down with people, we have a conversation, we ask him a few questions about themselves. We learn about them, see where the conversation takes us. We feel that we're quite good judges of character. Based on that we get a sense of what are they interested in, what are their passions, are they smart? We get a good sense is this person going to be a good colleague or not? We get a good sense, but it's not very reliable. So there's a huge amount of research basically suggesting unstructured interviews don't do a good job of predicting performance. They basically tell you things like with the person seems like me, I'm going to rate them higher. So there's all sorts of similarity biases. There're all sorts of kind of attractiveness biases. People who are just more attractive will rate higher whether or not they'll do the job. It tells you whether you establish an easy report quickly, but in most roles, that's not actually terribly predictive of how they will perform. So unstructured interviews are another piece that we find kind of like the common and exercise before kind of carrying that log over the wall. It sounds good. Does it actually help us hire the right people? No. So it's something else that we really want to stay away from. So I've given you a list of things that don't predict performance. Great, but what does? What should we be using then? Three things that research suggests really is predictive when it comes to understanding the performance of hiring applicants. First of these is cognitive ability tests. And so there's a lot of evidence that just general intelligence levels are surprisingly predictive performance across quite a wide range of jobs. I think it tells us a lot ultimately about people's ability to learn new tasks. Because of that there is evidence that people who score highly on cognitive ability tests tend ultimately to perform better in the job afterwards. There are important reasons to be very cautious when you use these tests as well. Even though in general we find the schools and cognitive ability tests predict performance. There is also evidence that they can be discriminatory, particularly that the questions tend to rely on a certain amount of cultural knowledge and background, and understanding what they're trying to get at that can vary across groups. And because of that I think there are serious concerns that they are discriminatory. And so good at predicting performance, but you want to be very careful in how you use them if you do use them. Second thing that's very good at predicting performance is work samples, right? If you want to understand can somebody do the job well, ask them to do the job and see how they do the job, and if they do it well keep them and if they don't then maybe this isn't the right role for them. And so organizations can do various things to actually try and see somebody in the position. A nice example of this again. The sandwich store with the very poor sandwich selection in the US. So they, in their hiring, one of things that they would do would be to ask an applicant basically just to spend a day working in the shop with the team. And at the end of the day the team that store get together and decide should this person be hired or not. Because what we see from somebody in a 30 minute interview is different from what we see over the course of the day when they have to interact with their team members, when they have to interact with customers, and all of those sort of things. So actually finding opportunities to see somebody grapple with the work is a great way to understand how well they perform. And then the third one is structured interview. Okay, so I just told you don't do unstructured interviews. Now I'm telling you do structured interviews, and you're probably thinking, I don't know whether to interview or not and what's the structure anyway? So what I want to do is spend a few minutes just talking through kind of what's a structured interview and how is it different from an unstructured interview? So what makes an interview structured? It's the idea of a structured interview, is it really fits with what I was talking about before. So I was saying if we want to hire properly, what we want to do is figure out what are the criteria that we want to select people on? What are the characteristics people need to have in order to be successful in this organization? Structured interview is basically an attempt to use the interview not to get a feel for the person, but to use the interview in order to really gather data to take each of these characteristics and say, okay, how well does this person score. So we said what Pramoj, it was passion, clear talking, teamwork. So I'm going to each of those characteristics. I'm going to take a couple of questions that I'm going to use to try and get a sense of how strong are they impassioned? How well are they doing with clear talking? And so I'm going to tie my questions to these things that I'm testing them for. And then I'm really going to try and do a very good job of evaluating people against the standards. I'm going to ask every candidate the same questions so I can put them on the same scale. And I'm going to take each of the answers and I'm going to define beforehand what's a good answer and a poor answer. So we can kind of score them. Actually put numerical scores, this was a three, this was a four in terms of the answers, okay? Then, in order to try to get around some of these biases I was talking about, I like people who seem like me, well, let's have people meet more than one interviewer. Let's make sure all of the interviewers know what they're supposed to be doing, how they're supposed to use these questions, what they're supposed to be looking for. So this is a structured interview process. One that's really treating the interview as an opportunity to gather information on the person in very structured categories. In practice, they tend to be a couple of question types that you might use the same teamwork, we said it was important. How do we learn about this? So, one way is to ask people about past experience. So how they worked in teams before, what do they learn? How did they approach it? So you might for example, ask the classic kind of question about, tell me about a time when you had to work with others to achieve a goal. What approach did you take? What were the results? Really trying to understand how they worked with teams in the past to understand their strengths and weaknesses in doing so. These questions, I think are often seen as quite nice because you really start to see how they approach concrete things. They kind of grounded in somebody's prior experience. They can be a bit limited if somebody's never had those sorts of experience. So I think often for people where you want to get a sense of how they react in situations they haven't experienced before. The other question type is sometimes see is the situational judgment. This kind of question of imagine you're working with the teammates not meeting her commitments. How would you address this? But it's a kind of some of the question types we might use. The important question that we want to ask ourselves is what's a good question, okay? How do we know whether the question I'm working is doing a good job in evaluating people or not? Two things that I ask about any question is, first of all, do you know what the difference is before you even ask the question, do you know what the difference is between a good answer and a bad answer? Okay, if we can define that then we can score people on these criteria and that's good. If we can't define it then you really want to worry. A couple of years ago I was talking to some of my students, they were doing some interviewing. And the question they had chosen to go with is, what's your spirit animal? I was a little nervous about the predictive validity of that question, I have to say. Okay, so I want to know what's a good answer, what's a bad answer, before they open their mouth so I know where to score them. And the second thing that I really want to do is make sure there's variation. There're some questions where I've got the question I know answer I want and everybody gives me that answer, it's so obvious that this is the right answer. I always get that. Not a very interesting question. Not a useful question, right? Because what I'm trying to do is differentiate across candidates. It's not differentiating. So ultimately a good question is one way, I know the right answer and where I differentiate. In an ideal world, if you've got a new question, you'll go to some of your existing employees and you'll ask it of them. So I've got a question on teamwork. I'll go and ask my employees, those employees I see as great team players. I want to be getting different answers from them. Then the answers that I'm getting from those people who I know not to be terribly strong team players. If that's the case, then I know yes, I have a good question. And so this idea of really structured information gathering questions that are keying into particular characteristics, and questions where I know what's a good answering, what's a poor answer. That's what makes a structured interview and is very effective in evaluating candidates.
