Compliance Controls and Their Unintended Consequences

Regulatory standards such as Sarbanes-Oxley, ISO-27001, and governmental Authority to Operate (ATO) mandates all require teams to meet compliance obligations before deploying software to production. A core tenet of DevOps is shifting quality and security concerns left into the developer feedback cycle, but Compliance Controls and Audit activities remain a separate, essential function for meeting those standards.

Organizations have developed several methodologies for implementing compliance evidence collection and validation within CI/CD pipelines. Each approach carries distinct tradeoffs in terms of team autonomy, speed of delivery, and the integrity of the evidence gathered.

Core Concepts and Terminology

Establishing a common vocabulary helps align engineering teams with an organization's Office of Compliance (OoC). The OoC defines desired outcomes (the “what”), while teams self-solution on the “how” to meet those outcomes, pending OoC agreement.

  • Property: An objectively measurable attribute of the system.
  • Measurement: A test that measures a property of the system.
  • Evidence: The measured value of that property, typically boolean or numerical.
  • Constraints: Defined limits within which the property value must fall to satisfy requirements.
  • Validation: Comparison of evidence against a constraint to prove compliance for a requirement.

A fitness function combines the measurement and validation steps. Example: extract open ports from Kubernetes configuration (measure), gather the resulting port list (evidence), and check that only ports 80 and 443 are present (validation against the port constraint). More complex examples involve variable constraints like the Common Vulnerabilities and Exposures (CVE) database, where a container scan produces evidence that must be validated against parameters that change over time.

Auditing the Compliance Process

Compliance is not a one-time activity, and the integrity of the process itself warrants inspection. The Audit process is a form of meta-compliance: it validates that the compliance workflow is operating consistently, accurately, and per the requirements. Though it follows the same measurement-and-validation pattern, auditing operates at a higher level. Its rule set tends to be smaller and simpler, and auditors typically evaluate a random sample of compliance activities rather than exhaustively verifying every release candidate.

Building a threat model is crucial to understanding the risks to a given compliance approach. A common assumption is that developers may be compromised or might unintentionally bypass checks; therefore, important questions to ask include whether changes can be introduced into the final system without visibility or auditability. If developers can access build infrastructure directly, for example, one can modify code mid-build in ways that are neither tracked in SCM nor reflected in audit logs. Awareness of the threat model guides decisions about the level of security and segregation that must be implemented in CI/CD infrastructure.

Evidence Anti-Patterns

The choice of how to collect and present compliance evidence shapes both developer experience and the value of the audit itself. Certain approaches are measurably weak in terms of automation and integrity.

Artifacts and Tests in a Separate Repository

Teams sometimes store compliance-related scripts and reports in a repository separate from the source code being validated, sharing a link to an externally hosted PDF or test report. While this often supports mandated “sign-off” workflows, automation is limited by the manual maintenance of those artifacts. Expectation management is the only mechanism ensuring owners keep reports up to date; gaps commonly surface only during audit sampling, followed by documentation archaeology to trace when changes were applied.

No inherent guarantee exists that the software in production matches the artifact being reviewed. The audit results, or a description of audit events, are typically stored with the evidence rather than attached to the version under review.

Static Test Results in the Source Repository

One common approach puts generated results in the source repository with the artifact being validated. That could mean zipped reports or a static file for the latest master build. This approach offers slightly better lineage, since the results exist in the repository despite having no link to the reviewed code. However, developers must be extremely disciplined to update static results for each change; otherwise, the file becomes stale and shows misleading indication that revised code was proven correct. After a branch cut, the results stop reflecting the code truly released.

(Continued…)

Evaluating Compliance Implementation Patterns

With the core Compliance process components in mind, we can assess how different implementation strategies map back to the foundational concepts of measurement, validation, and audit. Each pattern makes distinct tradeoffs, and the root causes of those tradeoffs become visible when viewed through this lens.

The patterns we explore here increase in complexity, primarily as a response to organizational and delivery scale. The simplest form is manual, checklist-driven compliance. As the organization grows, compliance checks get embedded into the delivery pipeline itself. When the scope of solutions expands, organizations often attempt to reuse pre-approved components as compliant building blocks. Finally, a single-responsibility split of the measurement and validation functions makes it possible to shift compliance verification to the moment a change is submitted.

Manual Compliance

In this approach, people execute the compliance checks. The Office of Compliance defines the properties to be measured and encodes them as forms or questionnaires. These often include simple yes/no checkboxes, lists of open ports or URLs, or narrative descriptions of security practices like access control and backup procedures.

The development team manually supplies evidence for each release by filling out the form for every item. A compliance officer then reviews the submission, applying objective comparison heuristics. An example rubric might ensure that only ports 80 and 443 are listed, or that no disallowed CVEs appear. Auditing this process is a matter of reviewing a random sample of forms to confirm the answers are accurate and pass the validation rules.

While the process is simple to launch—a basic form and a review meeting may be sufficient—its inadequacy at scale shows up quickly in several dimensions.

  • Capacity: A central Compliance team handling a growing volume of deployment requests becomes a bottleneck, and the variability in request arrival makes it costly to size the team for rapid turnaround.
  • Complexity: New failure modes arise as the organization and its systems grow, leading to lengthier and more complicated forms. This demands more effort from both the development teams and the Compliance office.

The leading indicator of trouble is the process itself turning into a limiting factor for value delivery. Teams respond by batching more changes into single releases to amortize the wait time. With a hard ceiling on deployment frequency, the only way to increase throughput is to increase batch size. If the office becomes overloaded, queues lengthen, and the pressure often results in developers finding ways to circumvent or falsify the process — a dynamic we have observed across commercial and government clients. In extreme cases, manual compliance was a six-to-nine month ordeal blocking production deployment.

Experience shows that this pattern tends toward “security theater”: it delays delivery without materially improving system safety. Crucially, it can also distort organizational structure, architecture decisions, and staffing far beyond the delivery process itself.

When to use it: Manual compliance can be sensible for a small property set, a low release cadence, and simple answers. It fails when required to operate at scale due to the capacity and complexity issues noted above.

Pipeline Compliance

This pattern embeds Compliance enforcement into the CI/CD pipeline via fitness functions and manual gates. Fitness functions that are straightforward to automate—open port checks, software supply chain validation, code coverage—run during the build. Any failure halts the pipeline, preventing deployment. Manual gates remain necessary for validations that are hard to automate, such as approving specific CVEs.

A successful pipeline run implies that all automated checks have passed. Passing the manual gates indicates that any human-verified requirements have also been satisfied. For the Office of Compliance, pipeline logs serve as an audit trail; for developers, feedback on compliance is immediate.

Efficiency gains exist in homogeneous environments where standard stacks make it easy to copy a common pipeline with the same compliance fitness functions. But the strategy sours when organizations respond to production incidents by locking the pipeline down. Centralizing pipeline ownership creates a single friction point for every team’s evolution.

There are two counterproductive responses we have seen. First, organizations add manual verification steps, such as requiring a manager and security officer sign-off that “all work is complete.” Those roles are rarely familiar with the changes they approve, so this adds delay but no security value. Second, a centralized “golden pipeline” becomes a bottleneck; teams with legitimate needs for variation find ways around it. Over time, the pipeline team—often a DevOps group optimized for stability rather than delivery—drifts in priorities and creates persistent friction for developers.

When to use it: Automating compliance tests is inherently beneficial for both auditors and developers. Pipeline compliance works well in a medium-scaled, homogeneous delivery environment. It fails when central control is used as a mechanism to slow releases down, which is sometimes an unstated attempt to shift responsibility for delivery quality away from engineering teams.

Composition Compliance

Composition Compliance relies on an assumption: compliance is distributive over composition. If a function C is distributive over addition (that is, C(A) + C(B) = C(A+B)), then a system built only from compliant components should itself be compliant. In practice, capabilities that have passed a traditional compliance process are distributed as locked-down “golden images” to development teams. As long as the team’s needs fit within the image’s constraints, their system inherits the image’s compliance status.

For instance, a team needing persistence can select from a compliant PostgreSQL or Cassandra image, as long as their configuration choices (PII restrictions, encryption-at-rest requirements) align with the image defaults. The critical maintenance load is on the image providers: CVEs can invalidate a golden image, and then every downstream team must update and re-test.

Auditing becomes a two-stage exercise: the team verifies that its infrastructure only references golden images without violating customizations, while the image provider maintains a separate audit process for the images themselves.

When needs fit the available building blocks, this pattern is efficient. The team inherits delivery, tuning, and operations capabilities already passing compliance, no duplicate work required. This is most effective in a homogeneous environment with a mature, comprehensive set of building blocks and low business variance.

The weaknesses appear when assumptions break. Any team requiring capabilities outside the golden image catalog must run a full compliance process for those custom components — a risk that often steers architecture and design decisions toward what’s pre-approved. As every team manages special cases, the organization begins to lose the economy of scale that justified the pattern in the first place. Marginal compliance costs may be higher than expected because the unit of compliance is the whole image, and coupling between teams grows with shared baselines.

We have observed this pattern to work well at small scale, where teams share similar needs. But once the solution landscape diversifies, the friction points resemble those of manual compliance, pushing teams toward shadow IT and deploying unapproved components.

Better facilities do not make the composability assumption true: compliance may be distributive only over a very narrow set of configurations, and the more customization the images allow, the more likely the claim unravels. Evaluate carefully what configurations remain compliant for your use cases.

When to use it: Composition is appropriate for teams whose requirements are all and only those covered by a set of mature, well—maintained compliant building blocks. It is a fragile fit for a heterogeneous or rapidly evolving product portfolio.

Point-of-Change Compliance

This pattern enforces compliance at the point of change—before a change is accepted—by validating that all constraints are met. The trick is to split fitness functions into their single-responsibility halves: measurement and validation. The measurement is the act of gathering evidence; the validation is the comparison of that evidence to a compliance policy.

The development team owns the measurement activity. They gather evidence in their pipeline, typically through automation, such as reading the list of open ports from Kubernetes configuration files, running CVE scans, or recording code coverage. This evidence is stored in a System of Record (SoR). The SoR may be the tool that performs the measurement natively, such as a SaaS code coverage or CVE scanning platform. Some measurements, like unpacking configuration files manually, require a custom evidence record to be stored in an appropriate SoR.

Because the measurement is separated from the validation when the SoR is not the source of the measurement itself, we face an authenticity question: how do we know the evidence in the SoR is reliable? Cryptographic signing of evidence closes this trust gap. The validation step can then simply verify a valid signature and provenance chain.

The validation step is performed at the point of change, commonly by a Kubernetes admission controller or by external operators managing non-Kubernetes infrastructure. Each application carries a set of exceptions—some permanent, like controls not applicable to the system type, and some temporary, like a newly-discovered CVE that the team has an agreed timeline to patch. Therefore the deployment is allowed even if the raw evidence shows a violation, as long as the appropriate exception is logged.

The single most important benefit of this decomposition is that compliance becomes externalized as Policy-as-Code. Measurement is a stable activity that does not change with policy adjustments. Policy changes can be evaluated against the accumulated evidence in the SoR, even for already-deployed systems. This allows an organization to see the impact of a policy change without first mandating a new round of deployments.

A centrally provided baseline pipeline, or starter kit, can offer a pre-approved set of measurement tests. This gives the economies of scale seen in Pipeline Compliance without forcing teams into a rigid workflow. Any custom measurement a team needs can be self-managed through an approval process with the Office of Compliance. The scope of that approval is much smaller and more focused than re-certifying a whole container image.

Auditing simplifies to reviewing the deployment transaction logs. A check of the cryptographically signed evidence and the outcome of the validation at that moment is sufficient.

When to use it: Point-of-Change Compliance removes most of the bottlenecks inherent in earlier-stage coordination. The complexity is real, but it is centralized: only the Office of Compliance needs deep expertise in the SoR and its key management infrastructure. The development teams interact with a well-documented pipeline and a set of small approval items.

Teams can start with centrally provided controls and adopt custom measurements as needed. Frequent policy changes become tractable, and audits become queries rather than manual document reviews.

For teams using this pattern, we were able to deploy custom admission controllers for large enterprise clients that would validate deployment requirements, security settings, and container registry provenance in real time, and integrate with existing Policy-as-Code solutions to run internal CIS Benchmark tests on Kubernetes configs prior to deployment.

Risks: The separation of measurement from validation introduces a layer of trust management that is the single biggest operational hurdle. The central team must protect signing keys, and a failure in the signing infrastructure can block changes. Shadow IT still remains a theoretical risk, but the structural ability to side-step compliance measurement is significantly reduced compared to earlier-stage manual or composition patterns.

Compliance Quanta: Where the Savings Accumulate

Stepping back from the four compliance patterns, a useful way to understand their relative strengths is to borrow the concept of the Architectural Quantum from Building Evolutionary Architectures, as popularized by Zhamak Dehghani in her writing on Data Mesh. Applying a similar lens to compliance—thinking in terms of distinct Compliance Quanta—clarifies how each approach changes the locus of work and the potential for reuse.

In the manual compliance model, the system is both the Architectural Quantum and the Compliance Quantum. Each system, however similar to another, demands its own bespoke compliance effort. Any shared characteristics between systems represent duplicated, wasted work that scales poorly as the estate grows.

Pipeline compliance begins to separate these two concepts. The system remains a unique Architectural Quantum, but the Compliance Quantum is now decomposed into smaller units, encapsulated as fitness functions. This fragmentation allows the organization to share compliance work across systems by reusing the same fitness functions in shared or similar pipelines. The individual team retains responsibility only for the fitness functions unique to their service.

Composition compliance goes a step further by decomposing the system itself into smaller, composable Architectural Quanta, each carrying its own compliance. The composed system is no longer responsible for running the entire validation process end-to-end. Instead, it consumes the results of previously executed compliance steps—shared as compliant container images. The team now only oversees the compliance of the specific images it produces.

Point-of-change compliance completes this trajectory. It decomposes the Compliance Quantum into the individual properties that predict outcomes, and further splits the fitness function into separate measurement and validation steps. This yields the maximal reusability of the compliance machinery (the shared, general-purpose measurements and validations) while reducing the team’s unique burden to custom measurements only.

The pattern is consistent: each successive solution shrinks the volume of unique, team-specific validation work by narrowing the scope of both the Architectural and Compliance Quanta. As that unique work diminishes, so do the second-order effects of maintaining bespoke compliance infrastructure. New capabilities also appear as the domain boundaries are reorganized; most notably, ownership of the pipeline returns to the development team while economies of scale are retained where they make sense.

In Thoughtworks’ Digital Platform Strategy engagements, developer friction is a primary focus. Across the broad set of changes introduced for clients, returning pipeline ownership to development teams consistently ranks in the top three value-returning investments for the engineering organization.

Making the Tradeoff

As with most architectural decisions, the aim is not to find a perfect solution but to evaluate the tradeoffs. The optimal balance of cost and benefit will differ per organization, shaped by the nature of the work, organizational scale, and the complexity of the delivered solutions.

From a purely architectural standpoint, however, adopting a fundamental separation of concerns in the compliance architecture positions the solution to evolve smoothly. If tight coupling between measurement and validation—bundled into fitness functions within a pipeline—suits the current organizational context, that freedom remains. And once the organization reaches sufficient scale, these same principles enable a clean refactor toward the loose coupling that drives team efficiency.