Data scientists are trained to handle uncertainty. The data we work with, no matter how “big” it may be, remains a finite sample riddled with potential biases. Our models tread the fine line between being too simple to be meaningful and too complex to be trusted. Armed with methodologies to control for noise in our data, we simulate, test and validate everything we can. A great data scientist develops a healthy skepticism of their data, their methods and their conclusions.
Then, one day, a data scientist is promoted and presented with an entirely new challenge: Evaluating a candidate to become a member of their team. The sample size drops fast, experimentation seems impractical, and the biases in interviewing are orders of magnitude more obvious than those we carefully control for in our work.
Many data science leaders resort to following traditional hiring practices — but they shouldn’t.
In setting out to build my latest team, I spoke with many data science leaders to gather their ideas and best practices. I was especially influenced by the ideas of Riley Newman, Head of Data Science at Airbnb, who designed and implemented a radically different way of recruiting data science talent, and who I spoke to several times while devising the system I’ll share with you here. I also learned a great deal from Drew Conway at Project Florida, who has continually evolved his hiring process to select for talent that could squarely land in the middle of his famed data science venn diagram:
In this article, I will outline the goals of a new process that Riley developed and I adapted based on this research, describe its underlying principles, and walk through the implementation we have experimented with at Sailthru. And of course, this guide wouldn’t be complete without looking ahead at opportunities to adapt and improve the process even further.
How to Start a Recruiting Revolution
In developing our recruiting process, we set out to improve the following measurable objectives:
Accuracy: Maximize the chances that new hires will become exceptional employees.
Loss: Minimize the chances that great prospects leave the hiring funnel early.
Success: Maximize the chance that offers will be accepted.
Effort: Minimize the long-term distraction to the hiring team.
At first glance, any experienced manager would think that it’s impossible to improve all four of the above goals simultaneously. The first three tend to work against each other in practice (e.g., the greater the candidate, the harder it is to get them to accept an offer). Beyond that, improving them all would seem to dictate greater ongoing effort by the team.
In a traditional hiring process, most managers feel fortunate if their accuracy is as high as 50%. That is, no more than half of their hires turn out to be exceptional. Loss is hard to measure (after all, candidates who fall out of the process didn’t come to work for you), and most managers worry that they regularly lose amazing talent because their process is so long and cumbersome.
In a competitive field like data science, strong candidates often receive 3 or more offers, so success rates are commonly below 50%.
And the ongoing effort that hiring requires can easily consume 20% or more of a data science team’s time.
After validating this experience with other data science leaders, I sought to implement a process that could achieve the following:
Accuracy: 90% of hires should in fact be exceptional employees.
Loss: We should make offers to 80% of the great candidates who enter our funnel.
Success: 65% of offers extended should be accepted.
Effort: Hiring should consume less than 10% of the team’s time.
By designing a hiring process that is smarter — both in identifying great candidates and simultaneously reducing the risk of losing them — it’s possible to improve on the first three goals simultaneously. And, by investing heavily upfront (an investment that pays off handsomely over time), the ongoing effort and distraction to the team can be managed.
To ensure that we met our objectives, we developed a set of core principles that can be applied to hiring for any function. Principles that keep everyone focused and aligned can significantly help any big process change. They also serve as a healthy foundation when you iterate on that process. Here they are:
Ensure your hiring process is always on and continually improving.
It’s common to think about hiring as either a task that you occasionally participate in, or as a blitzkrieg campaign that is periodically all-consuming. Instead, architect your hiring process to be an engine that is always on, with a predictable funnel of talent moving through clear stages. This ensures that you’re always recruiting, and that whenever great talent comes to the market, you’ll have the opportunity to engage.
Investing in an always-on process will force you to treat hiring as a discipline. This will drive consistency in protocol and results, enable you to collect data about your successes and failures, and force you to manage your talent pipeline with the same care you manage your data pipelines.
Make your process mirror the reality of your hiring needs.
The brutal truth: Standard interviewing questions are fatally flawed.
Ask candidates about their prior experience, and you’ll discover whether they can articulate what happened around them at other jobs. Ask them technical questions, and you’ll uncover their ability to regurgitate knowledge. Make them solve a ‘toy’ problem on a white board, and you’ll discover how quickly they solve toy problems. A candidate that passes all of these hurdles with flying colors may be a completely ineffective data scientist in practice.
To address these flaws, you must first have a very clear understanding of how you want candidates to perform data science. At the highest level, you should be clear on the end product your team will produce. Will it be visualizations and analyses that inform decision makers? Designs and prototypes that are given to developers? Or applications that can be scaled and supported in production environments?
Next, you should have a clear understanding of what you want successful candidates to do. Identify five opportunities you would love to see a data scientist tackle. For each, ensure that you have (or could reasonably collect) the data required, and can envision a solution that would be effective even if you couldn’t design it yourself. These opportunities lie at the intersection of the near-term strategy of your company, the feasibility of how your organization or product functions, and the constraints of the data that you currently have or can reasonably generate.
Knowing answers to how your team performs data science and what challenges you most want candidates to be able to handle, you can design a hiring process that closely reflects your working conditions. This means you should put candidates into an environment that closely resembles what their ‘day-to-day’ would be. If they can succeed in that environment during the interview process, then their chances of succeeding long-term are much greater.
Run objective evaluations first to minimize your biases.
Candidates who would be top performers may fail a traditional interview process.
The culprit is interviewer bias. As soon as you enter the room with a candidate, you begin forming opinions (mostly unconscious) about their abilities. There are a wide array of such biases (check out the list of 100+ cognitive biases here), but the most common bias in interviewing is to prefer people who are similar to ourselves.
Great data scientists must have very strong quantitative and programming skills. That’s non-negotiable. So we designed our process to test these skills first, then move on to more subjective (yet still measurable) skills like problem solving and communication. Only at the end do we get to the most subjective of all — how the candidate works on a team and fits into the culture.
These later stage, more subjective criteria are the most time-consuming to evaluate and are where biases are most likely to creep in. Moving them late in the funnel has the combined benefit of reducing the load on the team (we don’t evaluate culture fit until we’re confident they have the skills we need) and minimizing the risk of losing a great candidate prematurely.
Design your process to sell the candidate.
Most interviewing processes also fail to sell the highest-quality candidates on the role. Interviews are stressful at best and mundane and tedious at worst. Candidates are often forced to repeat their story to 4 or more interviewers and answer questions for hours on end. Afterward, while they may have been able to ask a few questions of their own, they often struggle to imagine what it would be like to work at the company. They then wait for days to receive feedback that is rarely honest or prompt. So how do you fix something so broken?
Create a process where you give candidates the data and problems that reflect the real challenges they’ll face at your organization. On top of that, ensure that your hiring process engages the candidate with your team’s dynamic and culture so that they get a real taste of what it would actually be like to work with you. Each of these candidates should complete the interview process with a trusting sense of what it would be like to join your team.
Make smart decisions with your team, not in your tower.
No matter how you hire, every manager has to make difficult decisions. To decide with confidence, establish clear frameworks for evaluating candidates at every stage of your funnel. This includes defining objectives and metrics that everyone on your team understands.
Also, make decisions openly as a team. This ensures the hiring manager hears direct feedback about candidates from everyone involved in the process. Even more importantly, it makes sure that you’re all looking for the same qualities. An open forum helps to change your recruiting needs and strategy over time.
Finally, engage your cross-functional partners in evaluating your candidates. Data science is never truly done in a vacuum. You will collaborate with decision-makers, engineers, and product managers. Involve key partners in those areas so you select talent that can be successful across departments and divides.
Move faster than the market.
The market for great data science talent is incredibly competitive, so your process should ensure that you move candidates through your funnel as quickly as possible, keeping momentum high and minimizing the chance that they accept a competing offer. Moving fast requires a streamlined process that allows you to build confidence as well as speed. Invest in tools and logistics to track how long candidates stay in each stage of your funnel and aggressively change your system to gain keep your edge.
The Implementation Game
Amazon’s Friction-Killing Tactics To Make Products More Seamless
In the movie The Imitation Game, Alan Turing’s management skills nearly derail the British counter-intelligence effort to crack the German Enigma encryption machine. By the time he realized he needed help, he’d already alienated the team at Bletchley Park. However, in a moment of brilliance characteristic of the famed computer scientist, Turing developed a radically different way to recruit new team members.
To build out his team, Turing begins his search for new talent by publishing a crossword puzzle in The London Daily Telegraph inviting anyone who could complete the puzzle in less than 12 minutes to apply for a mystery position. Successful candidates were assembled in a room and given a timed test that challenged their mathematical and problem solving skills in a controlled environment. At the end of this test, Turing made offers to two out of around 30 candidates who performed best.
At DL Recruiting Partners, our experienced recruiters can help you find the temporary, contract, or direct-hire talent you need, and help you extend an offer once you have found the perfect employee. Contact us today to learn more.