A Safer Landing: Rebuilding ShiftLeft’s Analysis Back-End in Flight

ShiftLeft’s NextGen Static Analysis (NG SAST) is a SaaS platform that scans pull requests for security issues. Recently, the engineering team set out to add two major result types—Secrets and Security Insights—alongside a new v4 API. The challenge was that the original back-end was designed for a single type of output: vulnerabilities. Adding new types would require either a significant retrofitting effort or a full refactor of how results were stored and served.

The team chose the latter path: a near-total rewrite of the data storage layer, executed live, without downtime. For Senior Infrastructure Engineer Preetam Jinka, the analogy is that of changing an airplane’s engine mid-flight without the passengers noticing.

Why Refactor, Not Retrofit?

The quick path would have been to bolt the new result types onto the existing implementation. But doing so would have perpetuated technical debt and made future feature development harder. Instead, ShiftLeft viewed the project as an opportunity to redesign the core architecture, clean up old code, and create a more flexible foundation.

This decision meant a longer project: several weeks spent on design collaboration, iterative coding, and testing against live production data. The UI also required a substantial rewrite to communicate with the new API, which was handled in phases to preserve backward compatibility.

Starting with a Design Doc

At ShiftLeft, no major engineering project starts with code. It begins as a design doc shared across teams. The goal is to iterate on an implementation plan early, allowing the entire team—not just the implementers—to review and refine the approach. This process, commonly used at large tech organizations, helps resolve edge cases and prevents major surprises later in the development cycle.

The design phase proved invaluable. The team made numerous adjustments to the plan upfront, avoiding last-minute roadblocks and ensuring alignment between the Product and Engineering visions.

The New Data Model

The architectural shift centered on introducing two new core types: scans and findings. In the new system, a scan is an explicit representation of each code analysis run, stored in a database table called scans. A finding is a generic type that can now represent any result from code analysis—whether it’s a vulnerability, a secret, or a security insight.

This normalization was a significant departure from the old design, where scan data was scattered across multiple tables with no clear data type. The new model simplifies database mappings, clarifies queries—often making them faster—and makes features like custom tags, severities, and scan comparison much easier to implement.

One of the most user-visible benefits was the ability to add GitHub-style incremental IDs. Instead of long identifiers or hashes, scans and findings for each project now start at 1 and increment. This had been a long-standing customer feature request that was previously difficult to support.

New features made possible by the redesigned data model include:

  • A user-friendly API
  • Human-readable IDs for scans and findings
  • Expanded language support via ShiftLeft Scan
  • Improved build rules leveraging the v4 API
  • Scan comparison
  • Customizable severities and tags

The Unified v4 API

Previously, ShiftLeft operated two distinct APIs: a functional v2 used by the UI and a severely limited v3 offered to customers. The new v4 API replaces both with a single, unified implementation. It is now used for the web interface as well as by customers building their own integrations.

This consolidation provides customers with direct access to all the capabilities once exclusive to the UI. From the engineering side, it eliminates the burden of maintaining two separate API implementations, allowing development effort to be focused on one system.

Running Old and New in Parallel

The production environment was live throughout the project. Without the option to flip a switch, the team implemented a strategy to run both the legacy and new systems concurrently while the new one was built and validated.

As new scans came in, the system wrote data to both the old and new back-ends simultaneously. Since Secrets and Security Insights were brand new features, they were served entirely through the new API. Vulnerabilities, however, remained on the old API because the historical data migration was still pending. The UI made the switch to the new API in phases, initially displaying only the new result types while vulnerabilities were still fetched from the old back-end.

The Final Migration

When the new API was ready for vulnerabilities, the team initiated the large data migration. They created a vuln_migration_status tracking table referencing each old scan by its internal ID (sp_id) and mapping it to the appropriate new project and mutable scan ID.

A small Go script (under 250 lines) performed the heavy lifting. It iterated through the tracking table and replayed the production code for each scan, creating records with expected timestamps in the past. The process was run in careful batches over several hours, successfully migrating thousands of scans with zero failures.

The design provided a built-in safety net: since the migration wrote exclusively to new tables, a failure would only require a wipe and read of the old data, with no consequences for the new system state.

Validating with Feature Flags

After the data migration, the next step was to compare the output from the old and new systems against live customer data. ShiftLeft relied on feature flags to toggle the functionality. By appending ?findings=enable to a URL, developers could activate the new back-end views in the UI, allowing for direct side-by-side comparisons with results from the legacy system.

The project stands as one of the largest in the company’s history. Beyond the successful launch of Secrets, Security Insights, and the v4 API, it provides the architectural groundwork needed to rapidly add new result types in the future. The practical processes employed during the project—collaborative design, incremental production rollouts, and the ability to test with real customer traffic—have since become the template for other large-scale initiatives.