121 articles from Slack Engineering.
Every codebase starts off small and modern. While it’s still small, the team can easily keep it up-to-date with current standards, upgrade libraries, and handle any code hygiene issues that may arise. Updating the API of a framework you built is easy when it’s only called in a handful of places. However, as the codebase…
In this article, “remote development environments” refer to AWS EC2 instances where engineers make code changes and can see a running Slack application with those changes. For years, engineers at Slack isolated and tested their changes by running microcosms of the Slack application on their local computers. This was difficult for many reasons: it involved…
Last September, Slack released Clips, allowing users to capture video, audio, and screen recordings in messages to help distributed teams connect and share their work. We’ve continued iterating on Clips since its release, adding thumbnail selection, background blur, and most recently, background image replacement. This blog post provides a deep dive into our implementation of…
In the first two posts about the Duplo initiative, we described why we decided to revamp our mobile codebases, the initial phase to clean up tech debt, and our efforts to modularize our iOS and Android codebases (post 1, post 2). In this final post, we will discuss the last theme of the Duplo initiative,…
Building load test infrastructure is tricky and poses many questions. How can we identify performance regressions in newly deployed builds, given the overhead of spinning up test clients? To gather the most representative results, should we load test at our peak hours or when there’s a lull? How do we incentivize engineers to invest time…
By Laura Nolan, with contributions from Glen D. Sanford, Jamie Scheinblum, and Chris Sullivan. Assessing conditions Slack experienced a major incident on February 22 this year, during which time many users were unable to connect to Slack, including the author — which certainly made my role as Incident Commander more challenging! This incident was a…
At Slack, the goal of the Mobile Developer Experience Team (DevXp) is to empower developers to ship code with confidence while enjoying a pleasant and productive engineering experience. We use metrics and surveys to measure productivity and developer experience, such as developer sentiment, CI stability, time to merge (TTM), and test failure rate. The DevXp…
In the first post about the Duplo initiative, we discussed the reasons for launching a project to revamp Slack’s mobile codebases, and what we accomplished in Duplo’s initial Stabilization phase. This post will explore modularization, and then there will be a third post to describe how we modernized our codebase and the overall results of…
According to a recent Thoughtworks radar, “the industry is increasingly gaining experience with platform engineering product teams that create and support internal platforms.” They caveated this with a piece of advice: “When creating a platform, it’s critical to have clearly defined customers and products that will benefit from it rather than building in a vacuum.”…
In 2021, we changed developer testing workflows for Webapp, Slack’s main monorepo, from predominantly testing before merging to a multi-tiered testing workflow after merging. This changed our previous definition of safety and developer workflows between testing and deploys. In this project, we aimed to ensure frequent, reliable, and high-quality releases to our customers for a…
In this article, I will talk about how Slack uses Kafka, and how a small-but-mighty team built and operationalized a self-driving Kafka cluster over the last four years to run at scale. Kafka is used at Slack as a pub-sub system, playing an essential role in the all-important Job Queue, our asynchronous job execution framework…
When do you need to overhaul a large code base to address tech debt? What is the best way to address widespread inconsistencies and outdated patterns? How can you make significant architectural improvements to a complex application while still continuing to ship features? These were questions we grappled with at the beginning of 2020, when…
While adding a recent feature to our Kubernetes compute platform, we had the need to mutate newly-created pods based on annotations set by users. The mutation needed to follow simple business rules, and didn’t need to keep track of any state. Surely there must be a canonical solution to this simple problem? Well, sort of.…
Over 70% of the files uploaded on Slack are images, and over 75% of those images are screenshots. What this tells us is that though images are ephemeral, screenshots are often used as a quick way to provide extra detail and context, and typically gain a high level of engagement over a short time period.…
On September 30th 2021, Slack had an outage that impacted less than 1% of our online user base, and lasted for 24 hours. This outage was the result of our attempt to enable DNSSEC — an extension intended to secure the DNS protocol, required for FedRAMP Moderate — but which ultimately led to a series of…
We use plenty of open source tools at Slack and we’ve benefited immensely from the wider Android, Kotlin, and Gradle communities. We also try to be good citizens by giving back. This includes things like sponsoring the Kotlin Lang Slack, contributions to projects we use like Anvil and Insetter, sharing projects of our own like…
At Slack, proactively securing our systems is a top priority. One way we achieve this is by automating the detection of security issues with static code analysis, which are tools that inspect programs without executing them. They’re often used with security-based rules to automate identification of vulnerabilities and insecure programming practices, which frees up more…
About a year ago, I wrote a blog post called Building the Next Evolution of Cloud Networks at Slack. In it, we discussed how Slack’s AWS infrastructure has evolved over the years and the pain points that drove us to spin up a brand-new network architecture redesign project called Whitecastle. If you have not had…
Slack is an integral part of where work happens for teams across the world, and our work in the Core Development Engineering department supports engineers throughout Slack that develop, build, test, and release high-quality services to Slack’s customers. In this article, we share how teams at Slack evolved our internal tooling and made infrastructure bets.…
On July 21st, 2021, Slack officially became a part of Salesforce at the price tag of $27.7 billion. This was undoubtedly Slack’s most significant event in the past year. As a frontend intern on Slack’s Customer Acquisition team this summer, I had the once-in-a-lifetime opportunity to be directly involved with announcing the closing of one…
Agile development methods can bolster company culture and empower teams to move quickly, with a focus on frequently adding value for customers. Whether you are a program manager, game developer, event planner, or architect, within businesses where change is constant, it’s key to have flexibility, and that’s where agile shines. While there are a variety…
We recently rolled out support for Conversation Bubbles for DMs and Group DMs on Android 11. In case you’re not familiar with Conversation Bubbles, take a look at the video below. Basically, they are a way to pop out a conversation from a notification into a Bubble that will draw over other apps, making multitasking…
Reinventing how the world does work inevitably creates a lot of data. Each year, Slack’s scale has increased and the volume of data ingested and stored has kept pace. To make it possible to understand relationships within our data, we’ve invested heavily in an automated data lineage framework. This facilitates producer/consumer coordination, improves risk mitigation,…
More than five years ago, we launched the Slack Platform, giving developers an easy way to build apps in Slack and publish them in our App Directory. Today, millions of users bring their work into Slack, and those apps built by over 885,000 active developers on the platform are key to further improving collaboration in…
With the release of Slack Connect, people can now collaborate both with internal employees and external organizations in the same channel. To make this as smooth as possible, Slack does predictive email analysis to classify and recommend the best way for a user to work with people they want to collaborate with. To accomplish this,…
Note: This article assumes some familiarity with Dagger, Anvil, and Kotlin. We use Dagger heavily in the Slack Android app for compile-time dependency injection. It’s powerful, flexible, supports basic Kotlin idioms, and allows for advanced dependency injection patterns with less boilerplate. It’s not without its sharp edges though. It slows down our builds with kapt,…
At Slack, we believe that designing an optimal keyboard experience is key to delivering a best-in-class product for all our customers. However, despite our design system components being individually accessible, we heard from keyboard users that we were still missing focus transitions in their end-to-end user experience. Non-sighted users who relied on a screenreader constantly…
Artwork courtesy of the Jenkins project. At Slack we manage a sophisticated Jenkins infrastructure to continuously build and test our mobile apps before release. We have hundreds of jobs running in a variety of different environments. One day something very odd happened — our Jenkins UI stopped working although the jobs continued to run. This…
Controlling which users are able to take which actions is no simple task. Building this into Slack has always been an interesting challenge. In large enterprise organizations, the standard types of roles we offered to customers were too broad, and delegating a generic admin role can grant someone with too much power — what if…
Complex systems are difficult to reason about at scale; we often can’t accurately extrapolate system behavior and performance, so we need to derive that data empirically. We use load testing to do just that: find the limits of our systems and weed out bugs at a large scale in a controlled environment. Slack is a…
A customer writes in and says the dreaded words: “My app is slow”. Here we go… Performance problems can be a real struggle to track down, especially if they aren’t easily reproducible. Looking at the customer’s logs, you see that it takes over 1.5 seconds to switch between channels on their Android client! That must…
Slack has a global customer base, with millions of simultaneously connected users at peak times. Most of the communication between users involves sending lots of tiny messages to each other. For much of Slack’s history, we’ve used HAProxy as a load balancer for all incoming traffic. Today, we’ll talk about problems we faced with HAProxy,…
We take developer productivity pretty seriously at Slack. It’s multi-faceted, spreading across build speed, reliability, modernity, and more. One thing we really pride ourselves in is using the latest build tools when possible. Not necessarily bleeding edge (alphas, betas, RC, etc.), but the latest stable versions. Aside from the new features and improvements they bring,…
And now we welcome the new year. Full of things that have never been. — Rainer Maria Rilke January 4th 2021 was the first working day of the year for many around the globe, and for most of us at Slack too (except of course for our on-callers and our customer experience team, who never…
Since its inception, Slack has fostered a culture of inclusion and diversity. The Security organization at Slack is a prime example of how women can thrive in the security space, transitioning to security from different backgrounds and expertises. With Slack’s strong commitment to diversity, it should not be a surprise that nearly a third of…
5:25 am First alarm rings. *Snooze.* 6:30 am The final alarm rings and I know this is the last one, so I get up and immediately start making my bed. I open the curtains to take a look at the weather and the early morning light. I walk to my balcony and stand there to…
In the first installment of the article, we examined why we built a React analytics library. We also looked at how we use the library to share data efficiently, log smarter impressions, and simplify event logging. In this second part of the article, we will focus on how we abstracted the library for use by…
From the very beginning of Slack, MySQL was used as the storage engine for all our data. Slack operated MySQL servers in an active-active configuration. This is the story of how we changed our data storage architecture from the active-active clusters over to Vitess — a horizontal scaling system for MySQL. Vitess is the present…
Last year, we migrated Airflow from 1.8 to 1.10 at Slack (see here) and we did a “Big bang” upgrade because of the constraints we had. This year, due to Python 2 reaching end of life, we again had a major migration of Airflow from Python 2 to 3 and we wanted to put our…
Slack brings the right people, information, and tools together to get work done. Normally, team members can invite one another to Slack via email invitation. Invited users can accept the invitation to activate their account, but if they don’t, then their team members can’t collaborate with them. To solve this problem we created Email Bridge:…
At Slack, we’ve gone through an evolution of our AWS infrastructure from the early days of running a few hand-built EC2 instances, all the way to provisioning thousands of EC2s instances across multiple AWS regions, using the latest AWS services to build reliable and scalable infrastructure. One of the pain points inherited from the early…
A form on a website may seem like a pretty trifling thing. A couple of text fields, a submit button, and you’re good to go, right? <i><span style="font-weight: 400">#lifeisgood</span></i> Wrong! Forms are beautiful and complex creatures. You must dive beneath the surface and explore all the beautiful features that can help create a truly powerful…
Early in my career at Slack I was looking for ways to have an impact. I asked my manager what I could work on to add value to the team. She said that we wanted to catch bugs earlier in the development process and that Cypress testing might be one way to do this. I…
“Why is it slow?” is the hardest problem to debug in a complex distributed system like Slack. To diagnose a slow-loading channel with over a hundred thousand users, we’d need to look at client-side metrics, server-side metrics, and logs. It could be a client-side issue: a slow network connection or hardware. On the other hand,…
The start of any internship brings a wide range of emotions, from excitement to nervousness. After months of anticipating our first day at Slack, reality sunk in that this summer would be extremely different from any other. Due to the pandemic, our entire experience would be virtual. As the two interns for the Product Security…
A fact of life for building an internet service is that, sooner or later, bad actors are going to come along and try to abuse the system. Slack is no exception — spammers try to use our invite function as a way to send out spam emails. Having built up the infrastructure to easily deploy…
This story speaks to the process behind incident response at Slack and uses the May 12th, 2020 outage as an example. For a deeper technical review of the same outage, read Laura Nolan’s post, “A Terrible, Horrible, No-Good, Very Bad Day at Slack” Slack is a critical tool for millions of people, so it’s natural when…
This story describes the technical details of the problems that caused the Slack downtime on May 12th, 2020. To learn more about the process behind incident response for same outage, read Ryan Katkov’s post, “All Hands on Deck”. On May 12, 2020, Slack had our first significant outage in a long time. We published a summary…
Batten down the hatches! The app sandbox is now enabled for all web content. This is a fancy way of saying we’ve dialed up the security of the app. It wasn’t unsafe before, but it’s double safe now. What is the “app sandbox,” what is it protecting against, and why does it matter? This post…
At Slack, Quality is a shared responsibility. The Quality Engineering team is focused on creating a culture of testing, increasing test coverage, and helping the company ship high-quality features faster. We encourage all our developers to write and own end-to-end (E2E) tests. In turn, Quality Engineering (QE) is responsible for the frameworks used and provides…
Over the years at Slack — usually during periods of high growth — teams have defaulted to working in a waterfall where an idea is proposed and researched, then designed or spec’d, then built, in that order. It’s not the most efficient way to work and as a company grows — with more people involved…
6:28 am First alarm rings. Snooze. 6:30 am Second alarm rings. Snooze. 6:34 am Final alarm rings and I know this is the last one, so I hop out of bed and immediately play some music. Music really has a way of waking me up, and I typically play Sofi Tukker or Rufus Du Sol…
Starting From the Top Imagine joining a new team at a new company and you finally get access to a codebase. Slack iOS Xcode File Hierarchy 2017 Where do you add a source file for your team’s project? Great question. Your new team has three source directories for you to add files to, and oh,…
In this article, development environments refer to sandboxes where you can test your code changes before deploying, and should not be confused with integrated development environments (IDEs) like Eclipse or Microsoft Visual Studio. Dev environments have always been a mystery to me. Despite learning about them on my first day at Slack, and using them…
6:45 am My alarm is set for 6:45 but I often wake up before it goes off. I tend to wake up earlier during Vancouver’s long summer days, when the sun is up from 5:10am to 9:10pm and already peeking through my blinds, but it sometimes happens in the winter when it’s still dark out.…
Slack launched in 2014 with a PHP 5 backend. Along with several other companies, we switched to HHVM in 2016 because it ran our PHP code faster. We stayed with HHVM because it offers an entirely new language: Hack (searchable as Hacklang). Hack makes our developers faster by improving productivity through better tooling. Hack began as a superset of PHP, retaining its best…
7am I’m definitely not a morning person, so when my alarm goes off, I can’t help but stay in bed a little while longer. I have two cats, Stella and Orion, who are especially cuddly in the mornings, so it’s hard to leave them and get out of bed. My cats are well known by…
Deploys require a careful balance of speed and reliability. At Slack, we value quick iteration, fast feedback loops, and responsiveness to customer feedback. We also have hundreds of engineers who are trying to be as productive as possible. Keeping to these values while growing as a company means continual refinement of our deployment system. We…
Most new things in technology turn out to be fads: patterns of talking and doing that come and go without leaving a permanent mark. Microkernels; EPIC architectures like IA-64; object request brokers; and 1990s’-style neural nets are gone, and will not return. Sorry for the deep throwbacks; only time proves which things are fads, so for uncontroversial…
6:45am The sound of coffee grinding wakes me up 15 minutes before my alarm. I’m not a morning person, but I always stumble bleary-eyed into the kitchen for some toast and coffee with my partner, which is my favorite daily ritual. I sit down, listen to my morning news podcasts — The Daily and AM…
