The workplace has shifted from being an employer-centric system to employees now understanding their leverage within the workforce. The past few years have shown many companies that creating a better candidate experience is essential for retention and success. But how are they able to quantify the candidate experience?
My guest on this episode created an internal system to analyze his company’s candidate experience, identify the areas in which the company can improve, and quantify the outcomes of candidate experience. Nicholas Bremner is the senior manager of people decision science at Uber and has continuously worked to improve and implement candidate experience surveys to help create a better experience.
The path that led him to his current position was reasonably nonlinear. Nicholas got his first real exposure to rewards and human motivation as a diamond-drilling helper. It piqued his interest in the candidate’s experience at work during his undergrad. By the time he finished, he was obsessed with the concept.
From his first consulting job, where he first experienced pre-hire assessments, to his research for the Canadian military to landing his position at Uber, understanding the statistical data behind candidate experience has been important to Nicholas. Nicholas’s current role with Uber is helping the company by “using data and evidence more broadly to help leaders make better decisions around their strategies.”
In this episode, we discuss the candidate survey that he and his team developed, how that survey has affected Uber and how they are deploying this survey.
What is the Candidate Experience Project?
As a people scientist, Nicholas started this project about three years ago because Uber’s employer brand team wanted to understand what was attracting employees to various roles in the company.
As a research scientist, his first step was looking at the data to understand how it can be leveraged. That is when Uber decided to send out its first survey to candidates. “We had a candidate survey that went out, and I wouldn’t say it was experience-focused, particularly, in a sense that it didn’t tap into some of the constructs that are critical like justice. So I utilized that data. Did some preliminary analysis,” he says.
After that initial survey came back, the Uber team received thousands of open-ended comments to “holistically analyze to figure out what’s going on.” As a result of that initial survey, Nicholas and his team were able to create a global talent survey that was sent out to over 20,000 people and received thousands of responses.
Redesigning Candidate Experience Surveys
The data obtained from that global survey helped Uber redesign its current candidate survey. However, data is constantly changing, so updating the surveys is an ongoing task, even to this day.
Nicholas shared, “When redesigning our current candidate survey, I was able to touch on those subjects like organizational justice, person-organization fit, organizational traction, and I built the tool around that.” Uber hoped to learn what was attracting candidates, put their best foot forward and give them a realistic view of what candidates could expect from them.
These surveys focus on creating an exceptional candidate experience that focuses on that interpersonal interaction and how valued the candidate feels at Uber. “There’s no clear playbook for that. We do have trainings and everything around that, but that’s the art to the candidate experience that really makes it exceptional,” Nicholas says.
How Uber is Deploying the Survey
Initially, the candidate experience survey went out to candidates who made the on-site phase of the selection process. Now, the pool has been expanded to include candidates at almost every stage of the selection process.
Nicholas stated, “We expanded our pool to include anyone who had done a recruiter phone screen because we wanted to know, regardless of how far you make it in the process, ‘How are you being treated?’ ‘How is the experience for you?’”
At Uber, their candidates are their customers. Whether they are brand loyalists or detractors, their voice matters when understanding their experience. While anyone still at the application stage won’t automatically get the survey, once Uber has interacted with a candidate, they’ll be sure to receive communication afterward.
People in this Episode:
- Nicholas Bremner: LinkedIn, Twitter, Mind Your Work Podcast
Read the Transcript
Announcer:
Welcome to Science 4-Hire with your host Dr. Charles Handler.
Hiring is hard. Pre-hire talent assessments can help you ease the pain. Whether you don’t know where to start, or you just want to stay on top of the trends, Science 4-Hire provides 30 minutes of enlightenment on best practices and news from the front lines of the employment testing universe.
So get ready to learn as Dr. Charles Handler and his all-star guests blend old-school knowledge with new-wave technology to educate and inform you about all things talent assessment.
Dr. Charles Handler:
Hello everyone, and welcome to the latest edition of Science 4-Hire. I have a really great guest today, someone that I hadn’t had not actually met before we set the podcast up, but one of the reasons I do this is to be able to meet all kinds of interesting new people. So, we have got Nicholas Bremner, the senior manager of people decision science at Uber. I think everybody’s probably heard of Uber before. So, you’ll have a really interesting frame of reference because I think most of us have used the product before and have enjoyed some of the freedoms that it provides for us. So welcome, Nick.
Nicholas Bremner:
Thanks for having me on. I’m a fan of your podcast, and I have been listening for a little while. I’m a big IO enthusiast in general, so I love to keep up with the latest and consume as much content as I can.
Dr. Charles Handler:
Yeah. Well, now you’re a content maker. How about that? Turned the table for me a little bit. Well, we have some really good stuff to talk about. I was really impressed with some of the things that we talked about in our pre-call and I can’t wait for you to share some of that with our audience.
But just to kick it off, I always let our guests introduce themselves, and I think you can take some time to talk about your background and how you found your way to your present role. I think that’s always really interesting, because we’ve all had varying backgrounds, and a lot of times you find people who end up in a role where they’re touching assessment, where it hadn’t even been something that they had built their career on, say, beforehand. So kick it off for us, Nick.
Nicholas Bremner:
Yeah. Yeah, love to. So as you mentioned, I’m the head of people decision science at Uber. And so, present day, what we’re doing is using data and evidence more broadly to help leaders make better decisions around their talent strategy. And so, love what I’m doing right now. It’s really interesting and rewarding work. But as you alluded to, and what happens with many of us is, my career path to get to there was pretty nonlinear, and I’ve taken on a lot of different roles.
My first real exposure to work was in the diamond drilling industry, where I worked as a diamond-drilling helper. And that really got me exposed to rewards and human motivation, and piqued my interest there as I was learning about how we reward employees and things like that.
My interest in IO psych really peaked towards the end of my undergrad, and I just became pretty obsessed with it. Ended up doing a master’s and PhD in IO in Canada, so I’m fully Canadian-educated. And then, I took my first consulting job, and this is probably the most exposure I’ve had directly to pre-hire assessments. I was doing executive assessment then, executive coaching for a period of time, which I felt totally out of my league, fresh out of school doing this, or I was still in school.
And then I started doing research for the Canadian forces, for the Canadian military. Led reviews on statistical analysis on a whole range of topics, spanning from employee engagement to how to reduce sexual harassment in the military and things like that. So, from there, that led me to Uber, where I started as a people research scientist. Yeah. I’ve done a bunch of cool research projects there. I’ve really loved working for Uber and have eventually made my way into decision science from there.
Dr. Charles Handler:
That is an interesting background. I think a lot of us, I remember early in my career, I was doing some executive assessment stuff, too. I’m like, “Whoa, there’s a pretty big gap here between my experience level with this stuff and what I’m doing, and these are very important decisions that people are making.”
It’s an interesting paradigm though, because you know, when you get into volume hiring with assessments, it’s just so different. You’d like to think that, what would we be able to do if we had the same amount of time that we spend with an executive assessment to get to know any applicant for any job? And certainly it’s not practical to do that, but when we, IOs or psychologists, clinical psychologists, whoever it is, have free reign to really learn about a person, there is so much we can do. We just never really have the time or the practicalities of it, especially at volume level just don’t support it. So, we’ve got to distill it down, but I would suggest that anybody who’s in any kind of assessment get some experience with executive assessment somehow, because it really presents a powerful picture of what’s possible.
Nicholas Bremner:
It does. I mean, we would spend, oh man, four or five hours with candidates doing various assessments. We’d have maybe five or six to run through, and they’d take the better part of a week running the tests and running up the reports. And you’re right, you’d learn an absolute ton about the individual.
Dr. Charles Handler:
Yeah.
Nicholas Bremner:
Yeah. But it’s super time-consuming, you’re right.
Dr. Charles Handler:
And here’s another area that I’d like to call out. This is the man versus machine, right, or human versus machine. One of the things about executive assessment: You can’t really automate that expert person who synthesizes all those various threads of information — four or five different tests, interviews, potentially role plays, all kinds of stuff. The early ones I worked on had word association, which is a lot more out there.
But then you as the trained expert pull that together and write a report, well, AI just can’t do that. It cannot duplicate that super-dense — what I call clinical model of human judgment — that a trained expert can do. I think that’s another reason why it’s important to understand, as psychologists, the value that that kind of paradigm has, that we are not replaceable in that realm.
Nicholas Bremner:
No, and I think you’re right. And I mean, you’re touching on something interesting that when you’re writing about all these different assessments and fusing it together into a report, I was bad at it when I started, obviously. You get better over time, but there’s a lot of tacit knowledge that I think you develop as you get better at combining these and writing a more cohesive narrative, I would say.
But I think part of that is getting to know what the client is looking for specifically, as well, because we’ll use similar assessments, but every client is different, and the demands of the role are different. And so marrying the actual assessment with each of the clients through a narrative form, I think there’s a bit of an art to it, which is understanding why AI can’t copy that right off the bat.
Dr. Charles Handler:
Yeah. And I always think that about the executive stuff is, because typically, there isn’t really a job analysis. I mean, you’re probably talking about a job of one anyway, but it’s typically whoever orders the assessment saying, “OK, this is what we need here.” That’s what everything’s calibrated to. And I’ve always felt like, if the person who gives the standard or the “This is what we’re looking for here” kind of thing isn’t accurate in that regard, then whatever we do to get them information is going to lack accuracy.
So to me, that intake part is just so critical. And if I were doing a business like this, I sure would spend a lot of time making sure my intake was very detailed and almost started to translate it into competencies or key things there. And I’m sure a lot of people do that. I’ve had exposure to that.
But sometimes it seems a little loosey-goosey to me. “Hey, we’re ordering this executive assessment on this candidate. Here’s the job description, here’s what we’re looking for,” and that’s it. And it’s also interesting in the debrief, that’s where it’s a little scary. I even do them once in a while now, and I did one recently. And on the debrief though with the client, who’s hiring this person for hugely important and multimillions of dollars on the line, is just asking me, “So, should I hire this guy or not?” And I’m like, “Here’s the positives, here’s the negatives. You know best what your business is. Here’s where he is going to struggle. Here’s where he is going to probably be pretty good. Now, you make the decision.” I didn’t want to be on the hook for that.
Dr. Charles Handler:
Little sidebar there, but this is what I love talking to people with different experiences. But Uber. So, tell us a little bit about, in general or whatever level of specificity you want really, how does Uber use pre-hire assessments? And I’m going to say, I’m just going to put it out there. I’m sure every listener wants to know, is there any driver-screening measures or anything like that, too? So, just tell us a little bit about what goes on at Uber with hiring assessments.
Nicholas Bremner:
So, I’ll start with the caveat that I’m not directly involved in the pre-hire selection stuff at Uber. But from what I know, and I do work with a lot of the folks who do handle them, I would say that we — so, I mean, Uber a really diverse organization in the first place. We hire for a lot of different roles, we’ve got a pretty complex structure. So, that necessitates a diversity in terms of how we assess.
It’s pretty public knowledge that we use engineering assessments for our tech workers, like through CodeSignal. There’s a lot of coding exercises and things like that. We also use quite a lot of business cases and exercises that are different throughout the business. We try and make sure that they’re as job-focused as possible. For higher-volume roles, we do use some more standard pre-hire assessments, but we’ve typically relied on external vendors for the validation of those.
Dr. Charles Handler:
Yeah. Business cases are the best. You’re starting to bring out all my key — things that I talk about a lot — and I just really feel like if you really want to evaluate someone’s suitability for a job, give them part of the job. From the earliest days in grad school, that’s the concept that resonated with me more than anything. Of course, that’s not always highly practical.
To the extent you can talk about it, do you have any knowledge of the type of — when you talk about business cases, even if you don’t get specific, but just the way those are structured? Are they rated by independent raters? Are they simulated with technology? I’m curious.
Nicholas Bremner:
Yeah. Yeah, for sure. So, I mean, I even use cases for the hiring on my own team within people analytics. There’s a diversity of them across Uber. I would say from the ones that I’m aware of, typically they are designed by the team in partnership with our talent acquisition team, and then they are rated by multiple individuals. When I say rated, you can probably use the term “evaluated,” as well. We have rigorous panel discussions before we make any hires. And so, the exercise or the case is evaluated as a part of that during committees.
The one interesting thing, and we’ll talk about candidate experience in a bit, but one of the things that came out that we have to watch out for in general is that we’re not creating the impression that we’re asking the candidate to solve a problem for us that we currently have in the business.
So, I’ve found it interesting to straddle that line between having it to be realistic, but then not too realistic that the candidate actually thinks that they’re solving a problem for us. This has come out in some of the research that I’ve done, as well, is that, if you have a candidate spend their time on a case, and then you don’t hire them, Uber still technically has that knowledge, the company still has that knowledge and they can go away with, “Oh, that’s a great solution to the problem.” Presumably you hire that person, but it’s all about the tension between selection and attraction, I like to say.
Dr. Charles Handler:
Yeah, that’s very cool, and I bet candidates appreciate it a lot. I mean, I know if I had the chance to do a live-simulated role play or solve a problem in real time while people were interacting with me, I would like that a lot better than a Raven’s Progressive Matrices or something like that, employment test. It’s just so face-valid. And it’s engaging because it’s also showing someone, “Hey, this is the kind of problem you’re going to be solving on the job.”
Nicholas Bremner:
Yes.
Dr. Charles Handler:
If they really like that kind of problem solving, they’re going to be very energized around that. They’re going to see, “This is a little bit of a preview of what I would be doing,” and we all know the power of the realistic job preview. So, I think that’s definitely very cool.
So, one of the things that I got really excited about when we were talking before is just some of the work you’ve done with candidate experience and some quantification of that.
I talk about it a lot. There’s a lot of touch points I talk about that, I wouldn’t say they’re abstract, but they’re not data based or factual. Sometimes they are. But in general, the research, the rigorous research that shows, “Here’s some direct outcomes of candidate experience.” I talk about it a lot from the lens of the things I learned in grad school and that I think are truisms around organizational justice and building commitment. All those nice touchy-feely outcomes are there. I’m sure people or researchers have measured them some, but in the practical way, I haven’t seen much of that.
But I’d love for you to talk a little bit about the candidate experience stuff that you all have been doing, because I think our listeners are going to find that very interesting.
Nicholas Bremner:
I started this candidate experience project when I was a people scientist. It’s going to be maybe three years ago now. This work has been ongoing for quite a period of time. It’s definitely my favorite project that I worked on at Uber, and it’s continuing to pay dividends in the business.
So, the short story is that we partnered with our employer brand team because they were interested in understanding what is attracting employees to various roles in Uber. As a research scientist, I started off with, what data do we already have that exists that I can leverage? And so, we had a candidate survey that went out. I wouldn’t say it was experience-focused particularly, in the sense that it didn’t tap into some of the constructs that you were referring to that are critical ones, like justice.
And so, I utilized that data. Did some preliminary analysis, taught myself NLP, which was super-fun, because we had, I think, 20,000 or something open-ended comments that we could holistically analyze to figure out what was going on. We turned the results of that, in addition to a lit review that I did, into a global candidate survey that we sent out to — I want to say I think 20,000 people or so, and we got thousands of responses back. And really, we used the data from that to redesign our current candidate survey that’s ongoing.
The great part about being an IO psychologist coming to a role like this is you can actually leverage the value of theory. So, when redesigning our current candidate survey, I was able to touch on those subjects like you mentioned, like organizational justice, person-organization fit, organizational traction, and I kind of built the tool around that.
So, I conceived of it in the sense that there’s a tension between attraction and selection. And you want to select the right people for the role, but you also want to make sure that you’re attracting the appropriate people and that you are putting your best foot forward as an organization as well. While at the same time giving them a realistic preview of what to expect, right?
So, fusing all those things together, try to create a reasonable-length survey for candidates. We basically took the results of that, plus the lit review and the theoretical elements, put that into a newer survey that is much more, I would say, psychology-focused that measures aspects of attract organizational justice, person-organization fit, and a lot of the typical talent acquisition or recruiting processes, as well, because it’s important for us to get a feedback loop on how we’re doing, as well, right?
Dr. Charles Handler:
Absolutely.
Nicholas Bremner:
Yeah. So, I consider the process stuff, and we do it at Uber as well, it’s table stakes. We have got to get those right, it’s the bedrock. And then to have an exceptional candidate experience, it’s the interpersonal interaction, how you feel valued as a candidate. There’s no clear, necessary playbook for that. We do have trainings and everything around that, but that’s the art to the candidate experience that really makes it exceptional. So, we focus on all those different layers.
Dr. Charles Handler:
Tell me a little bit about deploying the survey. If I apply for a job, the minute I click “submit my application,” did I get the survey? Or is the survey retroactive later, because you have everybody’s email address, and you’re surveying people who did and didn’t get hired? So, tell us a little bit about how you’re sampling with that survey.
Nicholas Bremner:
Our original survey only went to individuals who made it to the on-site phase. So, the last phase of the selection process. We actually expanded the recipient pool specifically because we wanted to hear the voice of more of our candidates. So, we’ve got a multiple hurdle approach that’s typical for our hiring, starting with a recruiter phone screen to business phone screen, and so on and so forth.
We expanded our pool to include anyone who had done our recruiter phone screen because we wanted to know, regardless of how far you make it in the process, how are you being treated? How is the experience for you? Our philosophy is that our candidates, our customers, they can be brand loyalists or detractors. And so, everyone’s voice matters here. We want to make sure that everyone has a good experience.
So, we expanded as broadly as possible for that reason. It doesn’t go out at the application stage when you click apply on our website, but once we interact with you, there is an automated follow-up afterwards.
Dr. Charles Handler:
Yeah. And look, you all are a globally recognized brand by just about everybody, right? And it’s just so important when — well, anybody it’s important for — the bigger your brand is, the more important it is to be connected with your applicants as consumers, right? Because they are going to consume your product whether they’re hired or not, and they’re going to talk about that, and they’re going to talk about their experience. It’s just so critical.
I mean, we work with a lot of very large brands globally, as well, and a lot of what we try to bring to the table is just an understanding more specifically on the assessment side of what’s going to create that good, solid brand awareness. How can we connect the assessment to your brand? I love that kind of stuff. So, I believe you are able to do some pretty good quantification of brand awareness or just brand presence within the candidate experience. So, maybe speak about that a little bit. That’d be great.
Nicholas Bremner:
I’ll preface this by saying that credit for the idea of the concept goes to our employer brand team, and I just helped to bring it to life. But essentially, what we found in some of the original research — and this is, I think, pretty clearly understood by folks in the talent acquisition industry — is that the candidate experience is not just talent acquisition or recruiting’s responsibility, right? Candidates are also interacting with the business. They’re interacting with hiring managers, they’re interacting with technical interviewers who they will be working with eventually, right, on the team.
And so as they’re doing this, they’re making an assessment of, “Do I want to work here or not based on who I’m speaking with?” Sometimes folks in the business are super-busy and they may be late for interviews, or they may not be totally prepared. This inevitably happens. There’s a lot of folks who are super-busy. And so, that has an impact on the candidate. They may not feel valued, as unintentional as it is, right?
So, part of the challenge here is trying to make sure that everyone in the business takes every interview seriously, that every candidate’s time is valued. One of the ways that we are able to do that is by showing that if an individual has a bad candidate experience, they’re less likely to use our platform as a customer in the future. We were actually able to quantify that by combining the design of the survey that we asked a few questions about future usage intentions with our actual lifetime value of a consumer. So, there was a dashboard internally at the time that we were able to use to quantify the lifetime value of our customers.
Without getting too much of the details, essentially, we were able to get someone’s future behavioral intentions around, “Hey, am I going to continue using this or not based on my experience?” And then quantify that over time to provide a rough estimate of how would it affect our platform revenue. And the purpose behind that was really to go back to the business and say, “Hey, look. How candidates being treated here matters. It translates into a real dollar figure. So, let’s take hiring seriously. Let’s take interviewing seriously.” And it’s currently used as a part of training that goes out to recruiters and also hiring managers to prep them for interviewing the business.
Dr. Charles Handler:
Did you find a relationship?
Nicholas Bremner:
Yes, we did. Yeah. Yeah. So, candidate experience, it does matter. The short version of how we quantified it was, we use Net Promoter Score like many other companies for assessing candidate experience. It’s highly benchmarkable, as well. And so, we were able to quantify, or get an assessment of, what percentage of people have a negative experience? And what is the relationship between those who have a negative experience and their future usage intentions, and then basically calculate a number. We do X number of interviews over the course of a year, X percentage of these individuals are likely going to stop using our platform or switch over. And so, we’re able to put a dollar value to it at the end of the year, which to my understanding has been pretty impactful.
Dr. Charles Handler:
Yeah. I just want anyone who’s listening to really just take a second to reflect on what we just heard is that, we’re always, always those IOs and business partners, looking for that connection with actual revenue or outcomes that are just so valued and measured by companies.
And a lot of times that’s very difficult to do for very many reasons, but you’ve just heard that it is possible to even quantify something — not even the assessment results themselves, but just the opinion of and feelings of a person who’s applying to your company and how that directly impacts bottom-line revenue of your company.
I mean, that is the ultimate IO success story to justify our existence sometimes to companies. And I really want other listeners here who have an opportunity to potentially think about doing something for that for their company to know that it is possible and to see that, or to hear about it here, because I was blown away. If I got to do studies like that all day for a living, I would be pretty pumped.
Nicholas Bremner:
I appreciate it, Charles. One thing I’ll add quickly here is that, it is a model, right? I mean, we know that people are notoriously poor predictors of their future behavior. And so, there’s obviously some heavy assumptions that are rolled into this. But I would say that, don’t let certain assumptions and small limitations prevent you from trying to make that connection where you can. It’s a matter of making that effort. It’s guaranteed not going to be accurate to the dollar by any means, but it’s an approximation. At the end of the day, if you think about what you’re trying to do this for, it’s to change minds and help people take the hiring process seriously, and that’s really the desired effect.
Dr. Charles Handler:
Yeah, yeah, absolutely. Well, we know that, and you obviously are a good researcher and psychologist, because you read any journal article or whatever, there’s always the caveat at the end. So “here’s the things that we couldn’t control. Here’s the reasons that it might not work,” but it’s still valuable for us to talk about it. So, that fits to me. I would appreciate more hearing that than someone trying to claim that this is a highly tuned model that can constantly predict every penny kind of thing.
But really great stuff. Any closing things you want to bring up about what we’ve talked about today, or in general? Thoughts about assessments? And then we can play on out.
Nicholas Bremner:
I just want to say thank you for having me on. It’s been a great conversation. You mentioned at the beginning that I’m a content creator now. I actually do have my own podcast that I —
Dr. Charles Handler:
Oh, nice. Plug it. Plug it. Please plug it.
Nicholas Bremner:
Yeah, I’m going to plug my podcast, Charles.
Dr. Charles Handler:
Do it.
Nicholas Bremner:
We’ve been on hiatus for a little while, but we’re thinking of picking it back up. It’s called Mind Your Work, and you can check us out on Spotify, iTunes, whatever. We’re on all the streaming services, and our website is MindYourWork.io. So, it’s an IO psychology podcast, as well. We talk about a range of stuff, like where the nine-to-five workday came from, is it actually good for us? Talk about personality, leadership, all that good stuff.
Dr. Charles Handler:
Very nice. I did not know about that until we did the prep for this. So, that’s good to file away. I wish there was somewhere a list of all the IO-related podcasts because I know there’s more than I might know about and other people might know about.
I know that SIOP has a few of them listed on their website, but it would be really cool to just have an index of those things somewhere if anybody’s listening and wants to try to take that on. But beyond that, how can people follow you? Any other places they could see you talking about the things that you value and what you’re doing?
Nicholas Bremner:
Yeah, yeah. So, I have a Twitter account, @NLBremner, that you can follow. I’m not super-active on there. I’m trying to be more active these days. And then LinkedIn is usually the best way to reach me and to connect, and probably the area that I’m most active on as well. So yeah, those would be the areas.
Dr. Charles Handler:
Awesome. Well, thanks so much for sharing a half hour with us today. Super enjoyable, and look forward to, maybe I’ll even meet you at SIOP if you end up being out there.
Nicholas Bremner:
Yeah, for sure. Thanks for having me on Charles. Greatly appreciate it. Cheers.
Dr. Charles Handler:
Thanks a lot. Till next time, have a great day and stay healthy.