A day in the life of a Backend Foundation Engineer at Slack
While it’s not always easy to get up before seven, particularly when the sun hasn’t risen yet, being able to get a head start on the day before most of the city begins their commute makes it worthwhile. I sneak out of bed and get ready for work. I’m typically out the door, down the four flights of stairs, and walking to the BART by 6:50. I’ll pop in my earbuds and boot up the latest podcast in my queue: oftentimes an episode of Planet Money, This American Life, or Radiolab. The station is usually pretty quiet by the time I make my way to the platform; there are just a handful of us waiting for the next train.
I walk into the office and (sometimes) turn on the lights. My team recently moved into a new space about two blocks from Slack’s San Francisco HQ. Our desks are on one of the upper floors of a towering skyscraper with beautiful views of the Ferry Building and the SF Bay sprawling behind it on one side and the city stretching out to Twin Peaks on the other. Some mornings, when the fog is dense, it creeps right up to the windows and all I can see is the office building across the street. Other mornings, you can spot just a thin blanket rolling off the hills in the distance. Either way, it’s a truly San Francisco experience.
I absolutely love getting in early. Not only is the commute much more pleasant, but I’m regularly the first one in the office. It’s perfect for quietly catching up on Slack, and figuring out my focus for the day over some cereal and fresh fruit (courtesy of our micro kitchen). If I have a few pull requests ready to merge and deploy, I’ll merge those changes and monitor them as they’re shepherded out to production by our engineers in New York. I normally leave the office around 5pm when some folks are still hard at work, so it’s not uncommon for me to have a few pull requests from the previous evening to review. I pop open GitHub Enterprise and dive into those.
One of the many responsibilities our team is tasked with is shepherding our upgrades to our language stack. At Slack, we’re currently the second only to Facebook as the largest users of Hacklang and HHVM. The language is undergoing rapid development and the team building it out is currently shipping minor version updates nearly every week. If we want to adopt the latest language features, performance enhancements, and security patches, we have to ensure that we’re keeping everything as up to date as possible with each new version. While many of the changes are easy to automate, some are a bit trickier and require us to handle them manually. Either way, there’s always a healthy backlog of language-related code to update or pull requests to review.
At around 8, I’ll head down to the 31st floor to grab an oat milk latte from our barista. Of the many perks, this is one of my absolute favorites! I’ll mingle for a few minutes with our coffee staff and fellow early-morning coffee-drinkers. Eavesdroppers hear us discuss the latest scaling problem (do we have enough capacity to handle adding another million daily active users?) or a recent incident (what additional monitoring do we need to add so we’re alerted earlier when our systems are degrading?). It’s not always business: we’re no strangers to throwing around book recommendations, Netflix suggestions, sharing stories about our roommates, partners, weekends, and upcoming trips. I could easily lounge by the coffee bar for over an hour chatting with these folks but that obviously wouldn’t be great for my productivity so I attempt to strategically spot a lull in the conversation and head back up to my desk.
I jump right back in. I purposefully line up my most difficult work for this part of the day: digging into a pesky bug, ironing out a highly technical design document, or starting on a prototype.
I’m currently the primary developer and point of contact for our backend performance regression monitoring tool, Slerf. (I recently spoke about this tool at Strange Loop in St. Louis, MO; you can see my talk here!) Slerf attempts to identify potential performance regressions while developers are actively writing code, before it gets merged and deployed to production. For every commit pushed up to our GitHub enterprise instance, Slerf runs a set of benchmarks which primarily track our database and memcached usage. It then compares these results with the same benchmarks run against the latest version of master. If it spots a regression, it alerts the code author and provides them with a report as to where the regression might be happening. Every quarter, our team identifies a few high-leverage features we think might take Slerf up a notch: either by surfacing better information to developers or tracking a more diverse set of metrics. Sometimes these features can be tricky to implement, making them great candidates for early-morning prototyping.
At Slack, we’ve been hiring so many new folks to join our ranks that we’re onboarding new engineers nearly weekly! A few of my teammates and I regularly alternate who teach one of the sessions new folks attend in their first week. We give an overview of Slack’s architecture, provide brief insight each of its major components, and trace a message through our whole system from both the HTTPS and web socket perspectives. The course has changed quite a bit over the years, and in just the few months I’ve taught it, we’ve had to update our slides to reflect shifts in our stack.
My husband and I love to cook at home. Slack provides employees with $500 in personal development funds every year, which I use in part to purchase recipe books (I highly recommend Cookie and Kate’s recipe book) and subscribe to the New York Times cooking app (which has an absolutely incredible collection of delicious recipes). We almost always end up with leftovers of our miscellaneous kitchen adventures, so we pack them up into individual containers for lunch throughout the week. We save a couple bucks and don’t have to brave the SOMA lunch lines!
On days where I have few afternoon meetings, I’ll make the time to eat lunch with my team. Our fall intern rejoined the team full-time just a few weeks ago and it’s been fun keeping up to date with all of the shenanigans of life after graduation. He was able to snag a free couch from one of our teammates who just moved into a new place which got us talking about all of the great scores we’ve gotten from other coworkers through our internal channel, #sf-classifieds.
Afternoons are speckled with meetings (except for Mondays which I block off for additional heads-down time). My team syncs up once per week on Wednesdays; we’ll discuss our progress since we last met and what we hope to accomplish in the upcoming week. The meeting frequently devolves into topics that are top of mind: an incident one of us was involved with, our latest mishap with testing infrastructure, impressions of the most recent all-hands.
We recently kicked off an effort to rewrite a significant portion of our backend codebase to follow a more object-oriented approach and promote better abstractions across distinct components in our application. Although most of our files are isolated into directories grouped by functional component, the code contained within those files isn’t particularly well isolated. When fixing a bug, it’s not uncommon for us to have to fix it in multiple places simply because every bit of code is responsible for handling every edge case. By gradually migrating our code to adhere to our new model, we hope to turn spaghetti code into lasagna code.
Every so often I’ll grab coffee with other engineers across the organization; it’s important to build a strong rapport with a wide variety of folks across the company when you’re on a team tasked with keeping folks productive. Plus, the early afternoon is when my brain is the slowest, so it’s a great time to grab a coffee and pick someone else’s brain. While the company doesn’t feel quite so small anymore (engineering has grown nearly four times since I joined the company three years ago), it doesn’t feel quite so big when taking the time to meet with new folks and learn about their challenges as they ramp up on our codebase.
If I haven’t gotten around to answering direct messages from earlier in the day, I take the time to do that. Having been at Slack for a while, I’ve worked on a number of different projects that frequently show up under git blame. I’ll normally get a few pings from folks throughout the day who have a question or two about a variety of things: Slerf, Hacklang syntax, a data warehouse query, finding errors in Logstash, or a few lines of code I authored a while back. If someone pinged me with a question that I think other folks would benefit from knowing the answer to, I kindly ask them to post the question to the most relevant public channel with a promise that I’ll answer it there. This is great for two reasons: first, if folks ever have the same question, they can use Slack search to find my answer again; second, it gives other folks the opportunity to provide an answer too, and give context that can be richer or more recent.
Time to wrap things up. I might leave a few notes for myself to reference tomorrow morning (where to pick back up debugging something, which pull request reviews are still outstanding, which folks to ping with questions). Some days feel more productive than others, and while I’m trying to associate productivity with more than just shipping code, it’s not an easy mindset to develop. If today didn’t feel particularly productive, I spend a few minutes poring over the day’s calendar, the messages I sent, the commits I pushed, and the code I reviewed. I take note of it all in a document I call my “Log of Unproductivity”. Not only does my perspective on the day change drastically, I’m able to spot patterns in how and where I spend my time on the days that I feel the least productive. For example, I’ve recently noticed days I spend primarily doing code review don’t intrinsically feel productive to me, so I’m deliberately trying to find ways to make code review feel just as productive as writing code.
I pack up my things and head home. Once home, I feed my sourdough starter and change into some leggings for either a short run around my neighborhood (it’s very hilly, so my pace varies pretty wildly) or a vinyasa yoga class at my local studio.
Life at Slack
This is the first in a series of posts describing the everyday life of Engineers in different parts of Slack’s ecosystem. Over the coming months, we’ll hear from Engineers on our Product Engineering, Mobile, and Frontend teams. If you’re interested in exploring job openings at Slack, visit our Careers page!
Maude is a Staff Software Engineer, based out of our San Francisco offices. In her time at Slack, she’s worked on a few different teams: first as a product engineer building the Grid product, then as a performance engineer focused on scaling out the backend to handle some of our largest Enterprise customers. She has most recently been working with the Backend Foundation team, focusing on making sure that other backend engineers at Slack are as productive as they can possibly be. The team has its fingers in many pies: shipping the latest updates to our language stack (HHVM/Hacklang), ensuring linters and unit testing framework are in tip-top shape, stewarding core libraries, and building out tools to monitor performance regressions.
An avid conference speaker, Maude’s recently spoken at Strange Loop in St. Louis, MO, the Lead Developer in Austin, TX, and O’Reilly Velocity in NYC. You can find links to all her talks at maudethecodetoad.com.
A Day in the Life of a Backend Foundation Engineer at Slack was originally published in Several People Are Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.