Announcing Cloudflare R2 Storage: Rapid and Reliable Object Storage, minus the egress fees
Introducing Cloudflare’s S3-compatible Object Storage service, with zero egress bandwidth charges and automatic migration from S3-compatible services.
How to Implement Logging in a Node.js Application With Pino-logger
Logging, on its own, is a key aspect of any application. Logging helps developers comprehend what it is that their code is doing. It also helps save
Sponsored Post: Wynter, Pinecone, Kinsta, Bridgecrew, IP2Location, StackHawk, InterviewCamp.io, Educative, Stream, Fauna, Triplebyte - High Scalability -
Who's Hiring? * Wynter is looking for system administrators, engineers, and developers to join its research panel. Participate in research surveys, get paid ($90-$180/hr) for your feedback and comments. Super low key commitment, 10-15 mins per survey. Learn more and sign up. * DevOps Engineer: At Kinsta, we set out to create the best managed hosting platform in the world. If you are an experienced
CacheLib, Facebook’s open source caching engine for web-scale services
Caching plays an important role in helping people access their information efficiently. For example, when an email app loads, it temporarily caches some messages, so the user can refresh the page without the app retrieving the same messages. However, large-scale caching has long been a complex engineering challenge. Companies must balance the fast experience people […]
RAMP-TAO: Layering atomic transactions on Facebook’s online graph store
What the research is: RAMP-TAO is a new protocol that improves the developer experience on TAO, Facebook’s online social graph store, by providing stronger transactional guarantees. It is the first protocol to provide transactional semantics over an eventually consistent massive-scale data store while still preserving the system’s overall reliability and performance. RAMP-TAO enables an intuitive
How we built a general purpose key value store for Facebook with ZippyDB
ZippyDB is the largest strongly consistent, geographically distributed key-value store at Facebook. Since we first deployed ZippyDB in 2013, this key-value store has expanded rapidly, and today, ZippyDB serves a number of use cases, ranging from metadata for a distributed filesystem, counting events for both internal and external purposes, to product data that’s used for […]
A linear programming approach for optimizing features in ML models
Whether it’s iterating on Facebook’s News Feed ranking algorithm or delivering the most relevant ads to users, we are constantly exploring new features to help improve our machine learning (ML) models. Every time we add new features, we create a challenging data engineering problem that requires us to think strategically about the choices we make. […]
Migrating Facebook to MySQL 8.0
MySQL, an open source database developed by Oracle, powers some of Facebook’s most important workloads. We actively develop new features in MySQL to support our evolving requirements. These features change many different areas of MySQL, including client connectors, storage engine, optimizer, and replication. Each new major version of MySQL requires significant time and effort to […]
Ribbon filter: Practically smaller than Bloom and Xor
What the research is: The Ribbon filter is a new data structure that is more space-efficient than the popular Bloom filters that are widely used for optimizing data retrieval. One of the ways that Bloom, and now Ribbon, filters solve real engineering problems is by providing smooth configurability unmatched by other filters. Bloom filters work […]
Smart(er) Origin Service Level Monitoring
Today we’re excited to announce Origin Error Rate notifications: a new, sophisticated way to detect and notify you when Cloudflare sees elevated levels of 5xx errors from your origin.
How To Add eBPF Observability To Your Product
Supercharge your Vercel Projects with Integrations
Connect your Vercel project to databases, monitoring tools, commerce providers, developer tools, and more.
Asicmon: A platform agnostic observability system for AI accelerators
Accelerators are special-purpose hardware devices optimized for specific applications, like AI prediction and video encoding. And Application-specific hardware platforms play an important role in meeting the growing latency and compute demands of workloads like deep learning, content understanding, and video encoding. At Facebook, the inevitable rise in use of accelerators in our data centers has
Consolidating Facebook storage infrastructure with Tectonic file system
What the research is: Tectonic, our data center scale distributed file system, enables better resource utilization, promotes simpler services, and requires less operational complexity than our previous approach. Our previous storage infrastructure consisted of a set of use-case specific storage systems. Clusters, or instances of these storage systems, used to scale to tens of petabytes. […] Read M
Network hose: Managing uncertain network demand with model simplicity
Our production backbone network connects our data centers and delivers content to our users. The network supports a vast number of different services, distributed across a multitude of data centers. Traffic patterns shift over time from one data center to another due to the introduction of new services, service architecture changes, changes in user behavior, […]
GitHub Enterprise Server 3.1 is now generally available
GitHub Enterprise Server 3.1 is now generally available for all customers. It helps customers work with large, busy repositories, while enabling developers to develop and deploy with less effort than…
How Facebook deals with PCIe faults to keep our data centers running reliably
Peripheral component interconnect express (PCIe) hardware continues to push the boundaries of computing thanks to advances in transfer speeds, the number of available lanes for simultaneous data delivery, and a comparatively small footprint on motherboards. Today, PCIe connectivity-based hardware delivers faster data transfers and is one of the de facto methods to connect components to […] Read Mo
What is Observability
Improving your monitoring setup by integrating Cloudflare’s analytics data into Prometheus and Grafana
Here at Labyrinth Labs, we put great emphasis on monitoring. Having a working monitoring setup is a critical part of the work we do for our clients. Improving your monitoring setup by integrating Cloudflare’s analytics data into Prometheus and Grafana
Automating Cloudflare Tunnel with Terraform
An overview on how to use Terraform to automatically deploy Named Tunnels into your infrastructure with Cloudflare.
Running Border Gateway Protocol in large-scale data centers
What the research is: A first-of-its-kind study that details the scalable design, software implementation, and operations of Facebook’s data center routing design, based on Border Gateway Protocol (BGP). BGP was originally designed to interconnect autonomous internet service providers (ISPs) on the global internet. Highly scalable and widely acknowledged as an attractive choice for routing, BGP […
Highly available and highly scalable Cloudflare tunnels
Starting today, we’re thrilled to announce you can run the same tunnel from multiple different cloudflareds simultaneously. This enables graceful restarts, elastic auto-scaling, easier Kubernetes integration, and more reliable tunnels.
Achieving observability in async workflows - Netflix TechBlog
Managing and operating asynchronous workflows can be difficult without the proper tools and architecture that puts observability, debugging, and tracing at the forefront. Imagine getting paged…
Most of computing’s carbon emissions are coming from manufacturing and infrastructure
[…]
A brief history of Rust at Facebook
Facebook is embracing Rust, one of the most loved and fastest-growing programming languages available today. In addition to bringing new talent to its Rust team, Facebook has announced that it is officially joining the nonprofit Rust Foundation. Alongside fellow members including Mozilla (the creators of Rust), AWS, Microsoft, and Google, Facebook will be working to […]
Reverse debugging at scale
Say you receive an email notification that a service is crashing just after your last code change deploys. The crash happens in only 0.1 percent of the servers where it runs. But you’re at a large-scale company, so 0.1 percent equals thousands of servers — and this issue is going to be hard to reproduce. Several […]
Cloudflare’s Partnership with HashiCorp and Bootstrapping Terraform with Cf-Terraforming
Learn more about Cloudflare and Hashicorp’s partnership and about our new release for our Terraform bootstrapping tool - cf-terraforming.
DIT — enabling de-identified data collection on WhatsApp
At WhatsApp, privacy is our DNA. That’s why we rolled out end-to-end encryption in 2016 — so that when messages are end-to-end encrypted, only you and your intended recipients can see the messages you send. But securing messages and calls is just one part of how we minimize the information we collect in the process […]
Expanding the Cloudflare Workers Observability Ecosystem
Cloudflare adds Data Dog, Honeycomb, New Relic, Sentry, Splunk, and Sumologic as observability partners to the Cloudflare Workers Ecosystem
Per Origin Host Header Override
Load Balancing as a concept is pretty straightforward. Take an existing infrastructure and route requests to the available origin servers so no single server is overwhelmed. Add in some health monitoring to ensure each server has a heartbeat/pulse so proactive decisions can be made.
How to Build a Web App with and without Rails Libraries - Shopify
Ruby on Rails is a web framework that contains many libraries you’d need to create and deploy a successful web application. We often take for granted the ability to run rails new to create a fully functional web application with tons of built-in features.
Solving the innersource discovery problem
Imagine you’re in an organization with over 2,000 repositories across several different product lines. It can be daunting task to find the right project.
Remove Circular Dependencies by Using Dependency Injection and the Repository Pattern in Ruby - Shopify
Shopify Core is a Ruby on Rails monolith that loads the Shopify admin along with other services such as checkout, APIs, etc. Hundreds of developers work with a continuous deployment cycle that ultimately results in a platform used by over 1.7 million merchants.
Measuring enterprise developer productivity
In a recent paper written by Nicole Forsgren and her colleagues, “The SPACE of developer productivity: There’s more to it than you think,” there is an irony that is hard…
4 things you didn’t know you could do with GitHub Actions
GitHub Actions is a powerful platform that empowers your team to go from code to cloud, all from the comfort of your repositories. In this post, I’ll walk through a…
GitHub Availability Report: February 2021
Introduction In February, we experienced no incidents resulting in service downtime to our core services. This month’s GitHub Availability Report will provide initial details around an incident from March 1…
Boosting the performance of virtual machines with Jump-Start
What the research is: Jump-Start is a new approach for improving the performance of virtual machines at scale. Virtual machines are a modern and popular design to implement programming languages used to build applications in general, including large-scale websites like Facebook and Instagram. However, virtual machines incur well-known performance overhead in terms of the amount […]
Your guide to DevOps automation and CI/CD at GitHub InFocus
Last week we kicked off GitHub InFocus, a global virtual series just for software teams. We discussed the importance of developer experience and innersource—and how key collaboration really is. Next…
Designing a Better Kubernetes Experience for Developers
TLDR; If you’re deploying a service with Kubernetes, you shouldn’t have to use all of your cluster management skills just to perform everyday developer tasks (like seeing which pods are experiencing errors or checking autoscaler limits). Backstage Kubernetes simplifies your deployment workflow by connecting to your existing Kubernetes implementation and aggregating the status of all your deploymen
Mitigating the effects of silent data corruption at scale
What the research is: Silent data corruption, or data errors that go undetected by the larger system, is a widespread problem for large-scale infrastructure systems. This type of corruption can propagate across the stack and manifest as application-level problems. It can also result in data loss and require months to debug and resolve. This work […]
FOQS: Scaling a distributed priority queue
We will be hosting a talk about our work on Scaling a Distributed Priority Queue during our virtual Systems @Scale event at 11 am PT on Wednesday, February 24, followed by a live Q&A session. Please submit any questions to [email protected] before the event. The entire Facebook ecosystem is powered by thousands of distributed systems […]
Faster, more efficient systems for finding and fixing regressions
Every workday, Facebook engineers commit thousands of diffs (which is a change consisting of one or more files) into production. This code velocity allows us to rapidly ship new features, deliver bug fixes and optimizations, and run experiments. However, a natural downside to moving quickly in any industry is the risk of inadvertently causing regressions […]
Using Betas to Deploy New Features Safely - Shopify
For companies like Shopify that practice continuous deployment, our code is changing multiple times every day. We have to de-risk new features to ship safely and confidently without impacting the million+ merchants using our platform. Beta flags are one approach to feature development that gives us a number of notable advantages.
Minesweeper automates root cause analysis as a first-line defense against bugs
Root cause analysis (RCA) is an important part of fixing any bug. After all, you can’t solve a problem without getting to the heart of it. But RCA isn’t always simple, especially at a scale like Facebook’s. When billions of people are using an app on a variety of platforms and devices, a single bug […]
Open-sourcing Thrift for Haskell
What it is: Thrift is a serialization and remote procedure call (RPC) framework used for cross-service communication. Most services at Facebook communicate via Thrift because it provides a simple, language-agnostic protocol for communicating with structured data. Thrift can already be used in programming languages such as C++, Python, and Java using fbthrift. We are also […]
Extending GitOps to reliability-as-code with GitHub and StackPulse
This is a partner post by Leonid Belkind, the Co-Founder and CTO at StackPulse Over the past decade, engineering-led practices have replaced traditional IT operations across the software development lifecycle.…
GitHub Availability Report: January 2021
Introduction In January, we experienced one incident resulting in significant impact and degraded state of availability for the GitHub Actions service. January 28 04:21 UTC (lasting 3 hours 53 minutes)…
What’s next for DevOps: Q&A with Lightstep CEO Ben Sigelman
In the decade since the word “DevOps” was coined, its goal has stayed the same—but the way organizations implement DevOps is constantly changing. For a closer look , we sat…
Day 51: Fixed my logging and made a couple of puzzles
Day 51: Fixed my logging and made a couple of puzzles
Slack’s Outage on January 4th 2021
And now we welcome the new year. Full of things that have never been. — Rainer Maria Rilke January 4th 2021 was the first working day of the year for many around the globe, and for most of us at Slack too (except of course for our on-callers and our customer experience team, who never…
Monorepo
I'm not exactly a large-scale DevOps guy, but I can tell ya we've been moving back toward a monorepo at CodePen and it's rife with advantages over a system
How machine learning powers Facebook’s News Feed ranking algorithm
Designing a personalized ranking system for more than 2 billion people (all with different interests) and a plethora of content to select from presents significant, complex challenges. This is something we tackle every day with News Feed ranking. Without machine learning (ML), people’s News Feeds could be flooded with content they don’t find as relevant […]
Improving how we deploy GitHub
As GitHub doubled it’s developer head count, tooling that worked for us no longer functioned in the same capacity. We aimed to improve the deployment process for all developers at GitHub and mitigate risk associated with deploying one of the largest developer platforms in the world.
Day 44: Building my VMs with Docker
Day 44: Building my VMs with Docker
GitHub Availability Report: December 2020
In December, we experienced no incidents resulting in service downtime. This month’s GitHub Availability Report will provide a summary and follow-up details on how we addressed an incident mentioned in November’s report.
Building On-Call Culture at GitHub
GitHub’s engineering group moved from a monolithic, hero-based on-call rotation to a more balanced on-call culture in order to increase our on-call expertise and improve the experience for our customers.
Docker Compose: a nice way to set up a dev environment
Docker Compose: a nice way to set up a dev environment
How to make DevOps your competitive advantage
2020 has been a year of change, with shifts to the way organizations of every size connect, collaborate, and build together. From our 2020 State of the Octoverse report to…
How do you test your tests?
Facebook’s codebase changes each day as engineers develop new features and optimizations for our apps. If not validated, each of these changes could potentially regress the functionality or reliability of our products for billions of people around the world. To mitigate this risk, we maintain an enormous suite of automated regression tests to cover various […]
Resiliency Planning for High-Traffic Events - Shopify
On January 27, 2021 Shipit!, our monthly event series, presented Building a Culture of Resiliency at Shopify. Learn about creating and maintaining resiliency plans for large development teams, testing and tooling, developing incident strategies, and incorporating and improving feedback loops. The video is now available. Each year, Black Friday Cyber Monday weekend represents the peak of activity f