
Last week we kicked off GitHub InFocus, a global virtual series just for software teams. We discussed the importance of developer experience and innersource—and how key collaboration really is. Next…

TLDR; If you’re deploying a service with Kubernetes, you shouldn’t have to use all of your cluster management skills just to perform everyday developer tasks (like seeing which pods are experiencing errors or checking autoscaler limits). Backstage Kubernetes simplifies your deployment workflow by connecting to your existing Kubernetes implementation and aggregating the status of all your deploymen
MCMatthew Clarke, Senior Engineer·March 1, 2021SRE & Ops 
What the research is: Silent data corruption, or data errors that go undetected by the larger system, is a widespread problem for large-scale infrastructure systems. This type of corruption can propagate across the stack and manifest as application-level problems. It can also result in data loss and require months to debug and resolve. This work […]
MEMeta Engineering·February 23, 2021SRE & Ops 
We will be hosting a talk about our work on Scaling a Distributed Priority Queue during our virtual Systems @Scale event at 11 am PT on Wednesday, February 24, followed by a live Q&A session. Please submit any questions to [email protected] before the event. The entire Facebook ecosystem is powered by thousands of distributed systems […]

Every workday, Facebook engineers commit thousands of diffs (which is a change consisting of one or more files) into production. This code velocity allows us to rapidly ship new features, deliver bug fixes and optimizations, and run experiments. However, a natural downside to moving quickly in any industry is the risk of inadvertently causing regressions […]
MEMeta Engineering·February 17, 2021SRE & Ops 
For companies like Shopify that practice continuous deployment, our code is changing multiple times every day. We have to de-risk new features to ship safely and confidently without impacting the million+ merchants using our platform. Beta flags are one approach to feature development that gives us a number of notable advantages.
SEShopify Engineering·February 12, 2021SRE & Ops 
Root cause analysis (RCA) is an important part of fixing any bug. After all, you can’t solve a problem without getting to the heart of it. But RCA isn’t always simple, especially at a scale like Facebook’s. When billions of people are using an app on a variety of platforms and devices, a single bug […]

What it is: Thrift is a serialization and remote procedure call (RPC) framework used for cross-service communication. Most services at Facebook communicate via Thrift because it provides a simple, language-agnostic protocol for communicating with structured data. Thrift can already be used in programming languages such as C++, Python, and Java using fbthrift. We are also […]

This is a partner post by Leonid Belkind, the Co-Founder and CTO at StackPulse Over the past decade, engineering-led practices have replaced traditional IT operations across the software development lifecycle.…
GP
GitHub Partnerships·February 4, 2021SRE & Ops 
Introduction In January, we experienced one incident resulting in significant impact and degraded state of availability for the GitHub Actions service. January 28 04:21 UTC (lasting 3 hours 53 minutes)…

In the decade since the word “DevOps” was coined, its goal has stayed the same—but the way organizations implement DevOps is constantly changing. For a closer look , we sat…

Day 51: Fixed my logging and made a couple of puzzles

And now we welcome the new year. Full of things that have never been. — Rainer Maria Rilke January 4th 2021 was the first working day of the year for many around the globe, and for most of us at Slack too (except of course for our on-callers and our customer experience team, who never…

I'm not exactly a large-scale DevOps guy, but I can tell ya we've been moving back toward a monorepo at CodePen and it's rife with advantages over a system

Designing a personalized ranking system for more than 2 billion people (all with different interests) and a plethora of content to select from presents significant, complex challenges. This is something we tackle every day with News Feed ranking. Without machine learning (ML), people’s News Feeds could be flooded with content they don’t find as relevant […]

As GitHub doubled it’s developer head count, tooling that worked for us no longer functioned in the same capacity. We aimed to improve the deployment process for all developers at GitHub and mitigate risk associated with deploying one of the largest developer platforms in the world.

Day 44: Building my VMs with Docker

In December, we experienced no incidents resulting in service downtime. This month’s GitHub Availability Report will provide a summary and follow-up details on how we addressed an incident mentioned in November’s report.

GitHub’s engineering group moved from a monolithic, hero-based on-call rotation to a more balanced on-call culture in order to increase our on-call expertise and improve the experience for our customers.
MM
Mary Moore-Simmons·January 6, 2021SRE & Ops 
Docker Compose: a nice way to set up a dev environment

2020 has been a year of change, with shifts to the way organizations of every size connect, collaborate, and build together. From our 2020 State of the Octoverse report to…

Facebook’s codebase changes each day as engineers develop new features and optimizations for our apps. If not validated, each of these changes could potentially regress the functionality or reliability of our products for billions of people around the world. To mitigate this risk, we maintain an enormous suite of automated regression tests to cover various […]
MEMeta Engineering·December 10, 2020SRE & Ops 
On January 27, 2021 Shipit!, our monthly event series, presented Building a Culture of Resiliency at Shopify. Learn about creating and maintaining resiliency plans for large development teams, testing and tooling, developing incident strategies, and incorporating and improving feedback loops. The video is now available. Each year, Black Friday Cyber Monday weekend represents the peak of activity f
SEShopify Engineering·December 10, 2020SRE & Ops 
During the last year alone, over 56 million developers created more than 60 million new repos and made more than 1.9 billion contributions on GitHub. These developers are building the…

Facebook’s services rely on fleets of servers in data centers all over the globe — all running applications and delivering the performance our services need. This is why we need to make sure our server hardware is reliable and that we can manage server hardware failures at our scale with as little disruption to our […]

Shopify changes a lot. We merge around 400 commits to the main branch daily and deploy a new version of our core monolith 40 times a day. The Monolith is also big: 37,000 Ruby files, 622,000 methods, more than 2,000,000 calls. At this scale with a dynamic language, even with the most rigorous review process and over 150,000 automated tests, it’s a challenge to ensure everything works properly. Dev
SEShopify Engineering·December 7, 2020SRE & Ops 
GitHub Universe is almost here. For more on what to expect from this year’s stream, we sat down with virtual host, Brian Douglas, for a quick Q&A on GitHub Actions,…

GitHub’s team delves into answering the question “what are operations roles in the development and operations (DevOps) environments”. From automating the role of QA in DevOps and more for smaller, faster delivery cycles.

GitHub has the privilege of partnering with thousands of organizations to improve their DevOps practices, from established enterprises transforming the way they work to born-in-the-cloud businesses. We often connect developers,…

If you were using computers in the ‘90s and the early 2000s, you probably had the experience of installing a piece of software you downloaded from the internet, only to discover that someone put some nasty into it, and now you’re dragging your computer to IT to beg them to save your data. To remedy this, software developers started “signing” their software in a way that proved both who they were a
SEShopify Engineering·December 1, 2020SRE & Ops 
At Cloudflare, we are always on the lookout for Single Points of Failure. In this post, we explore the role a failure mode known as a Byzantine fault played in a a real-world incident.
TL
Tom Lianza, Chris Snook·November 27, 2020SRE & Ops 
Cloudflare Workers continues to excel as a JAMstack deployment platform, and be used to power e-commerce experiences, integrating with familiar tools like Stripe, Nuxt.js, and Sanity.io.

Last year, we migrated Airflow from 1.8 to 1.10 at Slack (see here) and we did a “Big bang” upgrade because of the constraints we had. This year, due to Python 2 reaching end of life, we again had a major migration of Airflow from Python 2 to 3 and we wanted to put our…
ASAshwin Sai Shankar·November 19, 2020SRE & Ops 
We've come across many clients who are interested in taking the computational notebooks developed by their data scientists, and putting them directly into the codebase of production applications. Data science ideas do need to move out of notebooks and into production, but trying to deploy that notebooks as a code artifact breaks a multitude of good software practices. Predictably, that results in

What it is: FioSynth is a benchmark tool used to automate the execution of storage workload suites and to parse results. It contains a base set of block level storage workloads, synthesized from production I/O traces, that simulate a diverse range of Facebook production services. It is useful for predicting how a storage device will […]
MEMeta Engineering·November 18, 2020SRE & Ops 
Today we're releasing origin-ca-issuer, an extension to cert-manager integrating with Cloudflare Origin CA to easily create and renew certificates for your account's domains.

What the research is: Twine is our homegrown cluster management system, which has been running in production for the past decade. A cluster management system allocates workloads to machines and manages the life cycle of machines, containers, and workloads. Kubernetes is a prominent example of an open source cluster management system. Twine has helped convert […]
MEMeta Engineering·November 11, 2020SRE & Ops 
Continuous Integration (CI) and Continuous Deployment (CD) are crucial development practices, especially for teams. Every project is prone to error,

In October, we experienced one incident resulting in significant impact and degraded state of availability for multiple services.

Having a semantically versioned software will help you easily maintain and communicate changes in your software. Doing this is not easy. Even after manually

This is the second post in our series on DevOps fundamentals. For a guide to what DevOps is and answers to common DevOps myths check out part one. What role…

How GitHub measures and improves reliability, security, and developer happiness with automated deployments.

In this article, Aleem Isiaka explains how you can deploy a basic NodeJS application to Heroku, automate and schedule a deployment to run at certain intervals of the day, and use other actions from the Github Actions community.

This is our second post on cloud deployment with containers. Looking for more? Join our upcoming GitHub Actions webcast with Sarah, Solutions Engineer Pavan Ravipati, and Senior Product Manager Kayla…

This is a guest post by Jimmy Whitaker, senior data science evangelist at Pachyderm In the last few years, DevOps has begun to shift—from a culture of continuous integration and…

What it is: We all want our workloads to run on our machines as efficiently as possible. With the Facebook resource control demo (resctl-demo), developers can simulate system resource conflicts and test ways to resolve them. It’s like going on a guided tour through a system’s resource control, complete with live demos and detailed explanations. […]

See this post in action during GitHub Demo Days on October 16. What makes a project successful? For developers building cloud-native applications, successful projects thrive on transparent, consistent, and rigorous…

Large-scale companies serve millions or even billions of people who depend on the services these companies provide for their everyday needs. To keep these services running and delivering meaningful experiences, the teams behind them need to find the most relevant and accurate information quickly so that they can make informed decisions and take action. Finding […]

In September, we experienced one incident resulting in significant impact to the GitHub Pages service.

By now, most people in technology are familiar with the term DevOps. What we call “DevOps” will often differ between organizations, yet one thing remains the same: DevOps is defined…

This post is the second in our series on using GitHub for MLOps and data science. Just joining in? Get started with part one. Most continuous integration (CI) tools only…

Earlier this month we were thrilled to welcome the OpenJDK Community to GitHub. The communities migration effort, codenamed Project ‘Skara’, brought JDK 16 main-line development into GitHub. The JDK project is at the…
MW
Martin Woodward·September 30, 2020SRE & Ops 
Sarcasm disclaimer: This article is mostly sarcasm. I do not think that I actually speak for Dylan Thomas and I would never encourage you to foist a light

In this interview, we dig deeper with Maya Kaczorowski on what DevSecOps is, and how to apply it. It’s a mindset shift in how development teams think about security. DevSecOps is about making all parties who are part of the application development lifecycle accountable for security of the application.
GM
Grace Madlinger·September 24, 2020SRE & Ops 
The U.K.’s Royal Academy of Engineering has elected Facebook Research Scientist Mark Harman as a Fellow for his achievements in academia and industry, including his work on search-based software engineering (SBSE), intelligent software testing tools, and web-enabled simulation (WES) approaches. Election to the Academy is by invitation only, and it is one of the highest […]
MEMeta Engineering·September 21, 2020SRE & Ops 
In 2018, we set an ambitious goal of achieving a 75 percent absolute reduction in operational greenhouse gas (GHG) emissions and to support our global operations with 100 percent renewable energy by the end of 2020. On both counts, we are well on track: These commitments have spurred the construction of over 5,400 megawatts (MW) […]
MEMeta Engineering·September 14, 2020SRE & Ops 
Being able to run Google’s Lighthouse analysis suite programmatically provides a lot of advantages, especially for larger or more complex web applications. Using Lighthouse programmatically allows engineers to set up quality monitoring for sites that need more customization than straightforward applications of Lighthouse (such as Lighthouse CI) allow. This article contains a brief introduction to

As our infrastructure has expanded, we have seen an exponential growth in failures that affect a subset of capacity in a data center. These may stem from software or firmware errors, or issues in the mechanical or electrical equipment in the data center. As our data center footprint grows, so does the frequency of such […]

Hardware fails, networks partition, and humans break things. The job of our infrastructure engineers is to abstract these realities away and provide a reliable, stable production environment nonetheless. Two of the technologies we deploy in this pursuit are Twine, our internal container orchestrator, and Apache ZooKeeper. Twine maintains service availability by managing services in containers

Today CenturyLink/Level(3), a major ISP and Internet bandwidth provider, experienced a significant outage that impacted some of Cloudflare’s customers as well as a significant number of other services and providers across the Internet.