The New York Times is rolling out a new approach to how we hire data analysts.
By Luke Summerlin and MacKenzie Gibson
Over the last few years, the Data & Insights Group at The New York Times has more than doubled as we have integrated data into all our business and newsroom processes. Our data analysts’ responsibilities, and the tools they use, vary greatly depending on the team they work on. Whether a data analyst works with the marketing team to develop media mixed models, partners with editors and product managers to run A/B tests on the homepage or analyzes Crossword engagement, we expect our analysts to foster a culture of curiosity, intellectual honesty and clear communication. The one technical skill that unifies all analyst work is SQL.
In 2018, we implemented a standardized SQL assessment as part of the hiring process for all Times data analyst positions.
Once we launched the assessment, we began holding monthly retrospectives with interview panelists to see what was working and what was not. Through the retrospectives and candidate feedback, we came to the conclusion that our current assessment was not upholding the diversity, equity and inclusion values that The Times seeks to foster through our hiring processes. So, we sought out to change this.
Assessing the Assessment
The core part of our assessment needed to be rethought: the SQL test was not seeing consistent pass rates. Live-coding in a time-boxed environment is neither a realistic working environment nor is it stress reducing, especially given the already high-stress nature of a technical interview.
In our monthly retrospectives, we evaluated the assessment’s strengths and weaknesses. We paid particular attention to where the intent of the assessment diverged with the experience of taking it.
Among its strengths, the format allowed candidates to ask questions on the fly; use documentation; explain their approach while constructing queries; and provide interpretations of results. Additionally, the structure ensured all candidates were able to dedicate equal time to the assessment and it minimized opportunities to cheat.
When considering shortcomings, we found that the live-coding format created a high-pressure environment, which hindered some candidates’ performance. It was clear to us that we needed to create a more equitable format.
We considered other established evaluation practices and we laid out strengths and weaknesses of each approach:
- Normalizes the amount of time spent.
- Standardizes access to resources and subject matter experts.
- Assesses the ability to apply documentation, debug queries and communicate results.
- Creates a high-pressure environment that can impact performance, especially among candidates unfamiliar with the structure.
- Allows candidates to work at their own pace.
- Favors candidates who have fewer time constraints and access to strong knowledge networks.
- Can cause the same pressure as a live-coding format.
- Does not assess fundamental skills we look for in analysts, such as the ability to apply documentation, debug queries or interpret results.
We needed a standardized SQL assessment that had consistent deadlines and access to resources, yet minimized the pressure felt by candidates. The format we chose needed to allow us to evaluate candidates’ abilities to think through analytical problems using SQL; to apply appropriate functions that reference documentation, when necessary; interpret and communicate results; and provide an inclusive environment for neurodiverse candidates.
The New Structure
The format we developed is a hybrid model that combines elements from take-home and live-coding exercises. Our new structure still takes place over a video call. Shortly before the assessment begins, the candidate is given access to the BigQuery browser tool. The assessment is broken into three primary parts: set up, work and interpretation.
Set up (5 minutes): After introductions, the candidate will share their screen so the assessor can help them set up the BigQuery console and walk through the datasets. The assessor will review expectations of the assessment and share a problem set.
Work (30 minutes): The candidate will stop sharing their screen, but stay connected to the video call. With cameras off, they will have 30 minutes to independently work on the problem set. Candidates will write SQL, run queries and return results using documentation when necessary. Assessors will be available to answer questions in the video call, but otherwise candidates will work in an unmonitored environment.
Interpretation (15 minutes): The assessor and candidate will turn their cameras back on and the candidate will share their screen again to walk through each question they answered, run the query and interpret the results.
The assessment screens for core competencies, including the ability to aggregate, categorize and transform data, as well as apply documentation when necessary. Candidates are expected to be familiar with many of the competencies, although they do not need to demonstrate knowledge of everything to pass.
After the assessment is over, the assessor will spend 15 minutes writing in-depth feedback on the candidate’s performance with regard to their understanding of the core competencies and how they interpreted the data.
In the new format, we have extended the time candidates have to work on the problem-set from 20 minutes to 30 minutes without increasing the requirements to pass. By providing 30 minutes of uninterrupted time to work, we hope to reduce the stress of the assessment; if candidates need additional time, they may request it when scheduling the assessment.
We have introduced a survey that we encourage candidates to complete after taking the assessment. Formalizing a feedback loop can help focus where we need to refine the format which we will discuss in monthly retros, and allow us to continue improving upon an inclusive hiring process.
We are currently hiring analysts at various levels. Come work with us.
Luke Summerlin is a Data Manager on the Games team and he barely passed the SQL assessment in the summer of 2018 but has increased his SQL skills a lot since then.
MacKenzie Gibson is a Data Manager on the Messaging and Personalization team. She is a former beekeeper and is a firm believer in the power of public transportation.
Source: New York Times