
Short-lived certificates (SLCs) are part of our latest efforts to further secure our Transport Layer Security (TLS) private keys on our edge networks. SLCs have a very short exposure compared to traditional certificates and lower the chances of a compromised private key being abused. Implementing SLCs has required us to address tradeoffs between operability and […]

Learn how you can structure your enterprise to get the most value out of GitHub and provide the best experience for your developers!
DL
David Losert, Elizabeth Barrord·August 2, 2023SRE & Ops 
Thanks to DevOps, cloud computing and other industry trends, many organizations are shifting from a product mindset to a service mindset. Here’s how you can implement a service-led strategy.

A reflection on Guillermo Roach's Config 2023 talk, "The shape of future builders: from design to deploy."

Developer experience (DevEx) is a key theme when it comes to transforming businesses with GitHub.

Every day, thousands of developers at Meta are working in repositories with millions of files. Those developers need tools that help them at every stage of the workflow while working at extreme scale. In this article we’ll go through a few of the tools in the development process. And, as an added bonus, those we […]

Today, we are thrilled to relaunch Turpentine, and introduce Cloudflare's new Migration Hub. The Migration Hub serves as a one-stop-shop for all migration needs, featuring brand-new migration guides that bring transparency and simplicity to the process

SvelteKit is a framework for building apps using Svelte. In this article, Sriram shows you how to build a server-side rendered (SSR) SvelteKit application and deploy it to Netlify by following this step-by-step guide.

With GitHub Enterprise Importer, you can seamlessly move to GitHub Enterprise Cloud, bringing your code and collaboration history with you so your team doesn’t miss a beat.
TR
Tim Rogers, Melody Mileski·June 12, 2023SRE & Ops 
Could we use our Git repository as the source of truth for operational tasks, and somehow reconcile changes with our real-world view?

TL:DR At Spotify, we run containerized workloads in production across our entire organization in five regions where our main production workloads are in Google Kubernetes Engine (GKE) on Google Cloud Platform (GCP). If we detect suspicious behavior in our workloads, we need to be able to quickly analyze it and determine if something malicious has happened. Today we leverage commercial solutions to
MHMarcus Hallberg, Security Engineer·June 1, 2023SRE & Ops 
Low-code enables developers and non-developers to build custom applications and solutions with less effort. In this blog, we show you how to automate your low-code deployments using GitHub Actions.

We’re rolling out MySQL Raft with the aim to eventually replace our current MySQL semisynchronous databases. The biggest win of MySQL Raft was simplification of the operation and making MySQL servers take care of promotions and membership. This gave the provable safety of Raft and reduced significant operational pain. Making MySQL server a true […]

Create and share your own deployment protection rules, or use the rules from our great partners, like Datadog, Honeycomb, New Relic, NodeSource, Sentry, and ServiceNow, to control your deployments with more confidence. And the API is open for the community to build their own rules to make GitHub Enterprise Cloud even better.

Explore how migrating your source code and collaboration history to GitHub can lead to some surprising benefits.


Buck2, our new open source, large-scale build system, is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient. In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1. Buck2, Meta’s open […]

Many of us are aware of the benefits that a strong focus on automation can bring, particularly in our development workflow and DevOps lifecycle. But silos across businesses can lead to duplication of effort, and potential to lose out on best practices. In this post, we’ll explore how CI/CD can be shared across your entire organization alongside policies, for a well-governed experience with GitHub

Deployment rollbacks provide users the ability to quickly visualize and deploy past versions of their Workers, providing even more confidence in the deployment pipeline

Historical announcement for Remix on Vercel with streaming SSR and multi-runtime support, updated with current Vercel Functions guidance.

This blog post discusses the strategies that Slack uses to manage the lifecycle (development, support, and eventual retirement) of infrastructure projects, through the lens of the migration through three successive internal “platform” offerings. Our challenges Circa 2020, our Cloud Engineering team (now evolved into multiple teams responsible for narrower aspects) was responsible for managing our…

Cloudflare announces one-click ISO certified region, a super easy way for customers to limit where traffic is serviced to ISO 27001 certified data centers inside the European Union

Lock down your services with Cloudflare Aegis: dedicated IPs for Cloudflare egress back to your applications.

We have a huge number of servers of varying kinds, from varying vendors, spread over 285 cities worldwide. We need to be able to rapidly deploy various types of firmware updates to all of them, reliably, and automatically, without any kind of manual intervention.
CH
Chris Howells, Ignat·March 10, 2023SRE & Ops 
Meta is introducing Velox, an open source unified execution engine aimed at accelerating data management systems and streamlining their development. Velox is under active development. Experimental results from our paper published at the International Conference on Very Large Data Bases (VLDB) 2022 show how Velox improves efficiency and consistency in data management systems. Velox helps […] Read M

Scout is an automated system providing constant end to end testing and monitoring of live APIs over different environments and resources. Scout does it by periodically running self explanatory Python tests

Vercel Monitoring

Here at Cloudflare we run over 900 instances of Prometheus with a total of around 4.9 billion time series. Operating such a large Prometheus deployment doesn’t come without challenges . In this blog post we’ll cover some of the issues we hit and how we solved them

We’re excited to announce the general availability of GitHub Actions Importer. GitHub Actions Importer helps you plan, forecast, and automate migrations from Azure DevOps, CircleCI, GitLab, Jenkins, and Travis CI…

Explore how using GitHub and HashiCorp together enables enterprises to develop and ship to their customers faster and more secure with consistent workflows and actions.
MP
Mark Paulsen, Chris Reddington·February 28, 2023SRE & Ops 
Meta’s augmented reality (AR) platform is one of the largest in the world, helping the billions of people on Meta’s apps experience AR every day and giving hundreds of thousands of creators a means to express themselves Meta’s AR tools are unique because they can be used on a wide variety of devices — from […]
MEMeta Engineering·February 13, 2023SRE & Ops 
Today we're announcing Wildebeest, an open-source, easy-to-deploy ActivityPub and Mastodon-compatible server built entirely on top of Cloudflare's Supercloud.

Facebook for iOS (FBiOS) is the oldest mobile codebase at Meta. Since the app was rewritten in 2012, it has been worked on by thousands of engineers and shipped to billions of users, and it can support hundreds of engineers iterating on it at a time. After years of iteration, the Facebook codebase does not […]

What if developers want to leverage branch deployments but don’t have a full ChatOps stack integrated with their repositories? We wanted to set out to find a way for all developers to be able to take advantage of branch deployments with ease, right from their GitHub repository, and so the branch-deploy Action was born!

On January 14, between 00:15 UTC and 03:45 UTC, Spotify suffered an outage. The impact was small at first and increased over the course of an hour until most functionality (including playback) was not working. Spotify engineers were first notified of the problem at 00:40 UTC, and our incident response team was immediately assembled. Due to the nature of the incident, triage took longer than one wo
SESpotify Engineering·February 1, 2023SRE & Ops 
We’ve made architecture changes to Meta’s event driven asynchronous computing platform that have enabled easy integration with multiple event-sources. We’re sharing our learnings from handling various workloads and how to tackle trade offs made with certain design choices in building the platform. Asynchronous computing is a paradigm where the user does not expect a workload […]

The technical journey discusses the motivations, challenges, and technical solutions employed for warehouse schematization, especially a change to the wire serialization format employed in Meta’s data platform for data interchange related to Warehouse Analytics Logging. Here, we discuss the engineering, scaling, and nontechnical challenges of modernizing Meta’s exabyte-scale data platform by migra

Several Cloudflare services became unavailable for 121 minutes on January 24th, 2023 due to an error releasing code that manages service tokens. The incident degraded a wide range of Cloudflare products

Explore how GitHub and cloud native strategies can help you address common DevOps pipeline and team antipatterns.

Safely configure and deploy updates to zone configuration with Zone Versioning now Generally Available for Enterprise customers

Customers can now onboard O365 domains to Area 1, adding more flexibility on deployment methods.

Now, you can standardize and enforce CI/CD best practices across all repositories in your organization to reduce duplication and secure your DevOps processes.

Meta has open-sourced Anonymous Credential Service (ACS), a highly available multitenant service that allows clients to authenticate in a de-identified manner. ACS enhances privacy and security while also being compute-conscious. By open-sourcing and fostering a community for ACS, we believe we can accelerate the pace of innovation in de-identified authentication. Data minimization — collecting th


UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. Infrastructure teams at Meta leverage UPM to build SQL linters, catch user mistakes in SQL code, and perform data lineage analysis […]
MEMeta Engineering·November 30, 2022SRE & Ops 
We’re introducing calendar-based versioning for our REST API, so we can keep evolving our API, whilst still giving integrators a smooth migration path and plenty of time to update their integrations.

We developed a new static analysis tool called Nullsafe that is used at Meta to detect NullPointerException (NPE) errors in Java code. Interoperability with legacy code and gradual deployment model were key to Nullsafe’s wide adoption and allowed us to recover some null-safety properties in the context of an otherwise null-unsafe language in a multimillion-line […]

Page Functions are now out of beta, bringing the ability to add dynamic server-side rendering to your applications. Pages makes it easy to deploy applications built using all the major frameworks.
PB
Peter Bacon Darwin, Igor·November 17, 2022SRE & Ops 
Cloudflare uses the Cloudflare Terraform provider extensively to make changes to our internal accounts as easy as opening a pull request.
MW
Michael Wolf, Dhaynes·November 17, 2022SRE & Ops 
Sapling is a new Git-compatible source control client. Sapling emphasizes usability while also scaling to the largest repositories in the world. ReviewStack is a demonstration code review UI for GitHub pull requests that integrates with Sapling to make reviewing stacks of commits easy. You can get started using Sapling today. Source control is one of […]

GitHub Actions Importer helps you forecast, plan, and facilitate migrations from your current CI/CD tool to GitHub Actions.

We’re sharing Tulip, a binary serialization protocol supporting schema evolution. Tulip assists with data schematization by addressing protocol reliability and other issues simultaneously. It replaces multiple legacy formats used in Meta’s data platform and has achieved significant performance and efficiency gains. There are numerous heterogeneous services, such as warehouse data storage and vario

Developers creating Internet of Things software use a complex stack of software that needs to be custom built into their CI/CD platform. Arm is leveraging the simplicity and scalability of GitHub Actions with a native integration that will revolutionize IoT software development.

We’re sharing how Meta is applying statistics and machine learning (ML) to improve notification personalization and management on Instagram – particularly on daily digest push notifications. By using causal inference and ML to identify highly active users who are likely to see more content organically, we have been able to reduce the number of notifications […]

Today, a change to our Tiered Cache system caused some requests to fail for users with status code 530
JG
John Graham Cumming·October 26, 2022SRE & Ops 
At Slack, we use Terraform for managing our Infrastructure, which runs on AWS, DigitalOcean, NS1, and GCP. Even though most of our infrastructure is running on AWS, we have chosen to use Terraform as opposed to using an AWS-native service such as CloudFormation so that we can use a single tool across all of our…
AGArchie Gunasekara·October 25, 2022SRE & Ops 
At OCP Summit 2022, we’re announcing Grand Teton, our next-generation platform for AI at scale that we’ll contribute to the OCP community. We’re also sharing new innovations designed to support data centers as they advance to support new AI technologies: A new, more efficient version of Open Rack. Our Air-Assisted Liquid Cooling (AALC) – design. […]

The new Cloudflare Radar Outage Center (CROC), launched as part of Radar 2.0, is intended to be an archive of information about Internet outages observed by Cloudflare
DB
David Belson, Stanley·September 30, 2022SRE & Ops 
Commenting in Vercel Preview Deployment

Many of Meta’s products, such as search, ads ranking and Marketplace, utilize AI models to continuously improve user experiences. As the performance of hardware we use to support training infrastructure increases, we need to scale our data ingestion infrastructure accordingly to handle workloads more efficiently. GPUs, which are used for training infrastructure, tend to double […]
MEMeta Engineering·September 19, 2022SRE & Ops