Home/SRE & Ops
Topic

SRE & Ops

470 articles on SRE & Ops.

13,309 articles
SRE & Ops — Designing a Better Kubernetes Experience for Developers

Designing a Better Kubernetes Experience for Developers

TLDR; If you’re deploying a service with Kubernetes, you shouldn’t have to use all of your cluster management skills just to perform everyday developer tasks (like seeing which pods are experiencing errors or checking autoscaler limits). Backstage Kubernetes simplifies your deployment workflow by connecting to your existing Kubernetes implementation and aggregating the status of all your deploymen

MCMatthew Clarke, Senior Engineer·March 1, 2021SRE & Ops
SRE & Ops — Mitigating the effects of silent data corruption at scale

Mitigating the effects of silent data corruption at scale

What the research is: Silent data corruption, or data errors that go undetected by the larger system, is a widespread problem for large-scale infrastructure systems. This type of corruption can propagate across the stack and manifest as application-level problems. It can also result in data loss and require months to debug and resolve. This work […]

MEMeta Engineering·February 23, 2021SRE & Ops
SRE & Ops — FOQS: Scaling a distributed priority queue

FOQS: Scaling a distributed priority queue

We will be hosting a talk about our work on Scaling a Distributed Priority Queue during our virtual Systems @Scale event at 11 am PT on Wednesday, February 24, followed by a live Q&A session. Please submit any questions to [email protected] before the event. The entire Facebook ecosystem is powered by thousands of distributed systems […]

CWChris Wiltz·February 22, 2021SRE & Ops
SRE & Ops — Faster, more efficient systems for finding and fixing regressions

Faster, more efficient systems for finding and fixing regressions

Every workday, Facebook engineers commit thousands of diffs (which is a change consisting of one or more files) into production. This code velocity allows us to rapidly ship new features, deliver bug fixes and optimizations, and run experiments. However, a natural downside to moving quickly in any industry is the risk of inadvertently causing regressions […]

MEMeta Engineering·February 17, 2021SRE & Ops
SRE & Ops — Using Betas to Deploy New Features Safely - Shopify

Using Betas to Deploy New Features Safely - Shopify

For companies like Shopify that practice continuous deployment, our code is changing multiple times every day. We have to de-risk new features to ship safely and confidently without impacting the million+ merchants using our platform. Beta flags are one approach to feature development that gives us a number of notable advantages.

SEShopify Engineering·February 12, 2021SRE & Ops
SRE & Ops — Open-sourcing Thrift for Haskell

Open-sourcing Thrift for Haskell

What it is: Thrift is a serialization and remote procedure call (RPC) framework used for cross-service communication. Most services at Facebook communicate via Thrift because it provides a simple, language-agnostic protocol for communicating with structured data. Thrift can already be used in programming languages such as C++, Python, and Java using fbthrift. We are also […]

MEMeta Engineering·February 5, 2021SRE & Ops
SRE & Ops — GitHub Availability Report: January 2021

GitHub Availability Report: January 2021

Introduction In January, we experienced one incident resulting in significant impact and degraded state of availability for the GitHub Actions service. January 28 04:21 UTC (lasting 3 hours 53 minutes)…

KBKeith BallingerKeith Ballinger·February 2, 2021SRE & Ops
SRE & Ops — Slack’s Outage on January 4th 2021

Slack’s Outage on January 4th 2021

And now we welcome the new year. Full of things that have never been. — Rainer Maria Rilke January 4th 2021 was the first working day of the year for many around the globe, and for most of us at Slack too (except of course for our on-callers and our customer experience team, who never…

LNLaura Nolan·February 1, 2021SRE & Ops
SRE & Ops — Monorepo

Monorepo

I'm not exactly a large-scale DevOps guy, but I can tell ya we've been moving back toward a monorepo at CodePen and it's rife with advantages over a system

CCChris Coyier·January 26, 2021SRE & Ops
SRE & Ops — How machine learning powers Facebook’s News Feed ranking algorithm

How machine learning powers Facebook’s News Feed ranking algorithm

Designing a personalized ranking system for more than 2 billion people (all with different interests) and a plethora of content to select from presents significant, complex challenges. This is something we tackle every day with News Feed ranking. Without machine learning (ML), people’s News Feeds could be flooded with content they don’t find as relevant […]

MEMeta Engineering·January 26, 2021SRE & Ops
SRE & Ops — Improving how we deploy GitHub

Improving how we deploy GitHub

As GitHub doubled it’s developer head count, tooling that worked for us no longer functioned in the same capacity. We aimed to improve the deployment process for all developers at GitHub and mitigate risk associated with deploying one of the largest developer platforms in the world.

JNJulian NadeauJulian Nadeau·January 25, 2021SRE & Ops
SRE & Ops — GitHub Availability Report: December 2020

GitHub Availability Report: December 2020

In December, we experienced no incidents resulting in service downtime. This month’s GitHub Availability Report will provide a summary and follow-up details on how we addressed an incident mentioned in November’s report.

KBKeith BallingerKeith Ballinger·January 6, 2021SRE & Ops
SRE & Ops — Building On-Call Culture at GitHub

Building On-Call Culture at GitHub

GitHub’s engineering group moved from a monolithic, hero-based on-call rotation to a more balanced on-call culture in order to increase our on-call expertise and improve the experience for our customers.

MMMary Moore-SimmonsMary Moore-Simmons·January 6, 2021SRE & Ops
SRE & Ops — How do you test your tests?

How do you test your tests?

Facebook’s codebase changes each day as engineers develop new features and optimizations for our apps. If not validated, each of these changes could potentially regress the functionality or reliability of our products for billions of people around the world. To mitigate this risk, we maintain an enormous suite of automated regression tests to cover various […]

MEMeta Engineering·December 10, 2020SRE & Ops
SRE & Ops — Resiliency Planning for High-Traffic Events - Shopify

Resiliency Planning for High-Traffic Events - Shopify

On January 27, 2021 Shipit!, our monthly event series, presented Building a Culture of Resiliency at Shopify. Learn about creating and maintaining resiliency plans for large development teams, testing and tooling, developing incident strategies, and incorporating and improving feedback loops. The video is now available. Each year, Black Friday Cyber Monday weekend represents the peak of activity f

SEShopify Engineering·December 10, 2020SRE & Ops
SRE & Ops — GitHub in the enterprise

GitHub in the enterprise

During the last year alone, over 56 million developers created more than 60 million new repos and made more than 1.9 billion contributions on GitHub. These developers are building the…

EBErica BresciaErica Brescia·December 9, 2020SRE & Ops
SRE & Ops — How Facebook keeps its large-scale infrastructure hardware up and running

How Facebook keeps its large-scale infrastructure hardware up and running

Facebook’s services rely on fleets of servers in data centers all over the globe — all running applications and delivering the performance our services need. This is why we need to make sure our server hardware is reliable and that we can manage server hardware failures at our scale with as little disruption to our […]

CWChris Wiltz·December 9, 2020SRE & Ops
SRE & Ops — The State of Ruby Static Typing at Shopify - Shopify

The State of Ruby Static Typing at Shopify - Shopify

Shopify changes a lot. We merge around 400 commits to the main branch daily and deploy a new version of our core monolith 40 times a day. The Monolith is also big: 37,000 Ruby files, 622,000 methods, more than 2,000,000 calls. At this scale with a dynamic language, even with the most rigorous review process and over 150,000 automated tests, it’s a challenge to ensure everything works properly. Dev

SEShopify Engineering·December 7, 2020SRE & Ops
SRE & Ops — Your DevOps guide to GitHub Universe

Your DevOps guide to GitHub Universe

GitHub Universe is almost here. For more on what to expect from this year’s stream, we sat down with virtual host, Brian Douglas, for a quick Q&A on GitHub Actions,…

GMGrace MadlingerGrace Madlinger·December 4, 2020SRE & Ops
SRE & Ops — The evolving role of operations in DevOps

The evolving role of operations in DevOps

GitHub’s team delves into answering the question “what are operations roles in the development and operations (DevOps) environments”. From automating the role of QA in DevOps and more for smaller, faster delivery cycles.

JMJared MurrellJared Murrell·December 3, 2020SRE & Ops
SRE & Ops — Vouching for Docker Images - Shopify

Vouching for Docker Images - Shopify

If you were using computers in the ‘90s and the early 2000s, you probably had the experience of installing a piece of software you downloaded from the internet, only to discover that someone put some nasty into it, and now you’re dragging your computer to IT to beg them to save your data. To remedy this, software developers started “signing” their software in a way that proved both who they were a

SEShopify Engineering·December 1, 2020SRE & Ops
SRE & Ops — A Byzantine failure in the real world

A Byzantine failure in the real world

At Cloudflare, we are always on the lookout for Single Points of Failure. In this post, we explore the role a failure mode known as a Byzantine fault played in a a real-world incident.

TLTom Lianza, Chris SnookTom Lianza, Chris Snook·November 27, 2020SRE & Ops
SRE & Ops — Migrating Slack Airflow to Python 3 Without Disruption

Migrating Slack Airflow to Python 3 Without Disruption

Last year, we migrated Airflow from 1.8 to 1.10 at Slack (see here) and we did a “Big bang” upgrade because of the constraints we had. This year, due to Python 2 reaching end of life, we again had a major migration of Airflow from Python 2 to 3 and we wanted to put our…

ASAshwin Sai Shankar·November 19, 2020SRE & Ops
SRE & Ops — Don't put data science notebooks into production

Don't put data science notebooks into production

We've come across many clients who are interested in taking the computational notebooks developed by their data scientists, and putting them directly into the codebase of production applications. Data science ideas do need to move out of notebooks and into production, but trying to deploy that notebooks as a code artifact breaks a multitude of good software practices. Predictably, that results in

DJDavid Johnston·November 18, 2020SRE & Ops
SRE & Ops — Automated Origin CA for Kubernetes

Automated Origin CA for Kubernetes

Today we're releasing origin-ca-issuer, an extension to cert-manager integrating with Cloudflare Origin CA to easily create and renew certificates for your account's domains.

TSTerin StockTerin Stock·November 13, 2020SRE & Ops
SRE & Ops — Building a ubiquitous shared infrastructure using Twine

Building a ubiquitous shared infrastructure using Twine

What the research is: Twine is our homegrown cluster management system, which has been running in production for the past decade. A cluster management system allocates workloads to machines and manages the life cycle of machines, containers, and workloads. Kubernetes is a prominent example of an open source cluster management system. Twine has helped convert […]

MEMeta Engineering·November 11, 2020SRE & Ops
SRE & Ops — Getting started with DevOps automation

Getting started with DevOps automation

This is the second post in our series on DevOps fundamentals. For a guide to what DevOps is and answers to common DevOps myths check out part one. What role…

JMJared MurrellJared Murrell·October 29, 2020SRE & Ops
SRE & Ops — Introducing resctl-demo: Better resource control with simulation

Introducing resctl-demo: Better resource control with simulation

What it is: We all want our workloads to run on our machines as efficiently as possible. With the Facebook resource control demo (resctl-demo), developers can simulate system resource conflicts and test ways to resolve them. It’s like going on a guided tour through a system’s resource control, complete with live demos and detailed explanations. […]

CWChris Wiltz·October 14, 2020SRE & Ops
SRE & Ops — Nemo: Data discovery at Facebook

Nemo: Data discovery at Facebook

Large-scale companies serve millions or even billions of people who depend on the services these companies provide for their everyday needs. To keep these services running and delivering meaningful experiences, the teams behind them need to find the most relevant and accurate information quickly so that they can make informed decisions and take action. Finding […]

MEMeta Engineering·October 9, 2020SRE & Ops
SRE & Ops — GitHub Welcomes the OpenJDK Project!

GitHub Welcomes the OpenJDK Project!

Earlier this month we were thrilled to welcome the OpenJDK Community to GitHub. The communities migration effort, codenamed Project ‘Skara’, brought JDK 16 main-line development into GitHub. The JDK project is at the…

MWMartin WoodwardMartin Woodward·September 30, 2020SRE & Ops
SRE & Ops — Lightning Q&A: DevSecOps in five with Maya Kaczorowski

Lightning Q&A: DevSecOps in five with Maya Kaczorowski

In this interview, we dig deeper with Maya Kaczorowski on what DevSecOps is, and how to apply it. It’s a mindset shift in how development teams think about security. DevSecOps is about making all parties who are part of the application development lifecycle accountable for security of the application.

GMGrace MadlingerGrace Madlinger·September 24, 2020SRE & Ops
SRE & Ops — Mark Harman elected Fellow of the Royal Academy of Engineering

Mark Harman elected Fellow of the Royal Academy of Engineering

The U.K.’s Royal Academy of Engineering has elected Facebook Research Scientist Mark Harman as a Fellow for his achievements in academia and industry, including his work on search-based software engineering (SBSE), intelligent software testing tools, and web-enabled simulation (WES) approaches. Election to the Academy is by invitation only, and it is one of the highest […]

MEMeta Engineering·September 21, 2020SRE & Ops
SRE & Ops — The next decade: How Facebook is stepping up the fight against climate change

The next decade: How Facebook is stepping up the fight against climate change

In 2018, we set an ambitious goal of achieving a 75 percent absolute reduction in operational greenhouse gas (GHG) emissions and to support our global operations with 100 percent renewable energy by the end of 2020. On both counts, we are well on track: These commitments have spurred the construction of over 5,400 megawatts (MW) […]

MEMeta Engineering·September 14, 2020SRE & Ops
SRE & Ops — An Introduction To Running Lighthouse Programmatically — Smashing Magazine

An Introduction To Running Lighthouse Programmatically — Smashing Magazine

Being able to run Google’s Lighthouse analysis suite programmatically provides a lot of advantages, especially for larger or more complex web applications. Using Lighthouse programmatically allows engineers to set up quality monitoring for sites that need more customization than straightforward applications of Lighthouse (such as Lighthouse CI) allow. This article contains a brief introduction to

KBKaty BowmanKaty Bowman·September 11, 2020SRE & Ops
SRE & Ops — Fault tolerance through optimal workload placement

Fault tolerance through optimal workload placement

As our infrastructure has expanded, we have seen an exponential growth in failures that affect a subset of capacity in a data center. These may stem from software or firmware errors, or issues in the mechanical or electrical equipment in the data center. As our data center footprint grows, so does the frequency of such […]

CWChris Wiltz·September 8, 2020SRE & Ops
SRE & Ops — Containerizing ZooKeeper with Twine: Powering container orchestration from within

Containerizing ZooKeeper with Twine: Powering container orchestration from within

Hardware fails, networks partition, and humans break things. The job of our infrastructure engineers is to abstract these realities away and provide a reliable, stable production environment nonetheless. Two of the technologies we deploy in this pursuit are Twine, our internal container orchestrator, and Apache ZooKeeper. Twine maintains service availability by managing services in containers

MEMeta Engineering·August 31, 2020SRE & Ops
SRE & Ops — August 30th 2020: Analysis of CenturyLink/Level(3) outage

August 30th 2020: Analysis of CenturyLink/Level(3) outage

Today CenturyLink/Level(3), a major ISP and Internet bandwidth provider, experienced a significant outage that impacted some of Cloudflare’s customers as well as a significant number of other services and providers across the Internet.

MPMatthew PrinceMatthew Prince·August 30, 2020SRE & Ops