Uncovering the Seams in Mainframes for Incremental Modernisation

Finding the Breaking Points in a 40-Year-Old Mainframe

For a UK-based client, the mandate was clear: move a core Consumer Subsystem off a Mainframe that had been in production for four decades and onto cloud-native technology. The subsystem alone consisted of roughly 7 million lines of code, consumed ~80% of the estate's MIPS, and was a tangle of domain concerns scattered across multiple layers. A "Big Bang" replacement was never a realistic option. The risks were too high, the testing burden too great, and the business could not afford downtime or data inconsistency.

The strategy that emerged was one of incremental displacement. Rather than planning every detail upfront, the team opted for a "just enough, just in time" design phase followed by immediate engineering. The goal was to find seams in the legacy system: precise points where logic flow could be diverted toward new, cloud-based services with minimal changes to the Mainframe code itself. Rebuilt capabilities would then dual-run in parallel with their Mainframe counterparts, be verified for outcome parity, and eventually allow the legacy code to be retired.

Why the Mainframe Had to Go

The client's motivations for modernisation were familiar but pressing:

• Speed of change: Every modification to the Mainframe was slow and expensive. The business could not keep pace with market demands or innovate quickly enough.
• Cost pressure: Operational costs were high, and a core software vendor had announced an imminent price increase, adding commercial risk.
• A shrinking skills pool: While existing staff could run the system, finding new engineers with Mainframe expertise had become nearly impossible. The limited job market in this niche gives newcomers little incentive to learn the stack.

The Shape of the Consumer Subsystem

The subsystem handled two distinct kinds of workloads. The first was batch processing, which operated like a data pipeline: ingesting semi-structured data from external sources and other internal Mainframe systems, then cleansing and modelling it to fit the Consumer Subsystem's requirements. A significant piece of this logic was identity searching. In the UK, there is no universal identifier equivalent to a US Social Security number, so the system relied on custom algorithms to accurately match records to the right individuals.

The second workload was online transactions for the API layers. Request orchestration was handled by internally developed frameworks that looked up execution flows in datastores and used the output of code to decide conditional branches. Customisation was extensive: some flows were wired by hand, with ad-hoc configuration per customer or per interacting system. What started as exceptional cases became the norm as the client expanded its online products. An Entitlements engine ran across layers to ensure that only authenticated, authorised customers could retrieve the raw or aggregated data exposed through APIs.

Principles for a Safe Exit

Given the system's complexity, the team anchored its approach on two principles. Early risk reduction came first. Engineering started on day one to flush out unknowns before they became schedule-killers. Two types of risk were singled out: outcome parity — the need to generate identical attributes for each consumer as the legacy system did, which was non-negotiable under strict industry regulations — and cross-functional requirements, since the Mainframe is extraordinarily performant and there was genuine doubt a cloud solution could match its service levels.

The second principle was delivering value early. The team identified the most critical business capabilities that could be delivered early and used them as thin slices of the overall system. This approach reduced the cognitive load on developers and prevented analysis paralysis. Supporting this were patterns like dark launching and canary release, which would let the team transition customers to the new system seamlessly, without perceptible impact. Comparison testing and continuous output monitoring from both systems were essential to make this "silent migration" real.

Running the Old and New Side by Side

The pragmatic conclusion was an incremental dual run strategy. In this context, that meant using a transitional architecture to support moving capability away from the legacy environment slice by slice, with the target system and the as-is system temporary running in parallel. Each rebuilt slice would receive the same inputs as its Mainframe counterpart. The outputs from both would be checked for an exact match or, at minimum, an acceptable tolerance.

This approach curbed the risk of a risky full cutover, made parity issues visible and addressable early, and gave the client confidence in the new technology. It did not promise a short project — this was always going to be a multi-year programme — but it did promise a safer one, where business continuity was never put at the mercy of a single unproven release. Over the first twelve months, the team tested multiple techniques for introducing the seams this strategy requires, building a roadmap and a business case even as the engineering work revealed what would and would not hold up in practice. The work did not reach production in that first year, but the groundwork for what is achievable in breaking modernisation down into verifiable, dual-run steps is now on the table.

Mapping the Boundaries for Incremental Migration

To break the Consumer subsystem down into migratable pieces, the delivery team worked with Mainframe SMEs and the client’s engineers. The goal was not exhaustive documentation, but a working understanding of the current state — both technical and business — sufficient to design a Transitional Architecture between the Mainframe and the new Cloud platform, which other workstreams were building in parallel.

The decomposition started by grouping the subsystem into business and technical domains: data load, data retrieval and aggregation, and the external product layer exposed via APIs. Because the client’s workload was primarily analytical — ingesting mostly external data and selling the derived insight — the team found a high-level split early on. The programme could be divided into two independent tracks: one for data curation, another for data serving and product use cases, with data interactions as the seam between them.

Each track then needed to be split into smaller delivery increments.

Data Curation: Data Sets as Natural Units

On the curation side, investigation showed that data sets were largely managed independently. Although upstream and downstream dependencies existed, there was no entanglement during curation itself — each ingested data set mapped one-to-one to its input files. This meant any given data set could be treated as an independent migration unit.

The team then worked with SMEs to identify seams inside the technical implementation, at several levels of granularity (the database writers processing pipeline, the batch pipeline step handoff, and the most granular data characteristic seam). As long as data exchange with up- and downstream systems could be routed through the new cloud environment, each workload could be modernised in any order.

Data Serving: A Customer-Segment Approach

The serving and product side did not decompose as neatly. Any given product used roughly 80% of the client’s data sets and capabilities, so a data-centric split was not viable. Instead, the team examined how access was sold and found that a “customer segment” approach would work. They identified an initial subset of customers who had purchased a smaller subset of the product capabilities and data, which kept the first increment small. Later increments then built on that foundation, progressively cutting over more customer segments from the as-is to the target architecture.

This track required different seams and a different use of the transitional architecture, detailed in the database readers section and downstream processing as a seam.

In effect, the team analysed which components formed a cohesive whole from a business viewpoint but were nonetheless built as separate elements. Those independent elements became the units of a sequenced modernisation programme, each deliverable migrating to the Cloud on its own timeline.

Finding the seams in a mainframe estate

Our transitional architecture was shaped largely by the legacy seams we could uncover inside the mainframe. Seams are the junction points where code, programs, or modules meet. In an older system, these may have been deliberately placed for modularity, extensibility, or maintainability — but decades of continued development tend to hide them inside accumulated complexity. When found, seams are strategically useful: they give you places to intercept data flows and offload capability to a new platform without breaking what still runs on the old one.

Identifying seams and defining valuable delivery increments was a two-way exercise. Technical possibilities informed which increments we could plan, and those increments in turn shaped the transitional architecture the programme needed. The mechanisms below were refined continuously as our knowledge grew; some were deployed to test environments while others remained spikes. We expect them to evolve further as we apply the same approach on other large-scale mainframe modernisation programmes.

External interfaces as seams

We looked first at the external interfaces the mainframe exposed to data providers and to the client's customers. Applying event interception at these integration points meant external-facing workload could move to the cloud silently, from the outside world's perspective. Two types of interface existed: a file-based transfer for providers supplying data, and a web-based API set that customers used to interact with the product layer.

The batch input seam

The file-transfer service was the first external seam we found. Providers could upload semi-structured files through a web GUI or directly via FTP for programmatic access. The service decided, per provider and per file, which datasets on the mainframe should be updated, and dataset triggers on the batch job scheduler then ran the relevant pipelines.

Our plan assumed each pipeline could be rebuilt in full on the cloud (we address breaking down larger pipelines later). For each pipeline we would build a cloud version and dual-run it alongside the mainframe to confirm identical outputs. This was achievable by applying additional configuration to the file-transfer service to fork uploads to both platforms. We validated the approach against a production-like file-transfer service running with dummy data in test environments.

Dual-running could continue as long as needed to build confidence. Once satisfied, we would apply a further configuration to the file-transfer service to prevent mainframe dataset updates, deprecating the old pipeline while leaving it in place. We never executed that final step because we didn't complete an end-to-end pipeline rebuild, but our technical SMEs confirmed the required configurations were available.

The API seam

The externally facing APIs offered a parallel opportunity. The existing API Gateway — the customer entrypoint to the consumer subsystem — was an obvious seam. Our design drew on dual-run concepts and proposed a proxy high up in the HTTPS call chain, as close to the user as possible. The proxy would run the as-is mainframe flow and the new cloud APIs in parallel and report on both outcomes.

Effectively this was dark launching the new product layer to gain early confidence through continuous comparison of outputs. Building the proxy was not a first-year priority: it only becomes valuable once a meaningful portion of the product-level functionality has been rebuilt. We intended to build it as soon as meaningful comparison tests at the API layer became possible. Analysis also flagged side-effects originating from the products layer: the mainframe produced things like billing events. Preventing customer double-billing would require intrusive mainframe changes once requests were forked.

Like the batch seam, parallel running could continue indefinitely. The eventual cutover would use a canary release at the proxy layer, moving customers to the cloud one at a time and incrementally reducing mainframe workload.

Internal interface seams

We then examined internal components to find seams that would allow more granular migration. The pervasive database accesses were an important focus. Identifying the programs that read or wrote the database — and treating the database itself as a seam — let us break apart flows whose only connection was shared data.

Database readers and writers

For reading programs, both platforms needed access to the same data to support new cloud-side data APIs. We analysed the tables accessed by the product chosen as the first migration candidate and worked with client teams to deliver a data replication solution. Change Data Capture (CDC) synchronised the required tables from a test database to the cloud. The replication gave us a chance to redesign data models too: the client gained access to non-relational stores (document, event, key-value and graph were considered). Access patterns, query complexity, and schema flexibility determined the target technology per data subset. In the first year we built replication streams from DB2 to both Kafka and Postgres. With data flowing to the cloud, capabilities built around reading from the database could be rebuilt and migrated incrementally.

The writers were mostly batch workloads. By analysing data flowing through and out of them we could apply an extract-product-lines pattern to identify domains that could run independently — the fact they ran as a single flow was an implementation detail we could change. These atomic units, with their seams, let other workstreams rebuild pipelines on the cloud and compare outputs with the mainframe.

Our team also built supporting tooling: mainframe batch jobs, triggered by dropping a file in the file-transfer service, which would extract and format pipeline journals for automated comparison testing. Once outcomes matched, other teams could cut over sub-pipelines one by one.

Artefacts produced by a cloud sub-pipeline might still be needed on the mainframe — for online transactions, for example. Our design used a legacy mimic approach to replicate processed data back to the mainframe while dependent capabilities remained there. Events on a stream were the source for this return path, but direct stream consumption by the mainframe was too complex and invasive. Instead we designed an adaptation layer to transform data back into the format the mainframe expected, as if it had been produced there. Simple transformations might be handled by the chosen replication tool, but we anticipated custom software alongside it to meet additional cloud-side requirements. This is a common scenario when businesses use a rebuild as an opportunity to improve processing efficiency.

Close work with client SMEs was essential. They helped challenge the existing batch implementations and identify discrete pipelines with clearer data boundaries. Critically, the pipelines we dealt with did not overlap on the same records — later work examined cases where they did.

Pipeline step handoff as a coarse seam

The database is rarely the only seam available. Some data pipelines served curated data to downstream pipelines for further processing, in addition to persisting outputs.

We identified handshakes between pipelines — typically state in flat or VSAM files, or possibly temporary storage queues. One example involved a downstream pipeline reading a curated flat file stored upstream. That downstream pipeline produced a VSAM file queried by online transactions. Our plan for the cloud-based, event-driven rebuild was to use the CDC tool to get data off the mainframe as a stream of events. As before, the transitional architecture needed an adaptation layer (for schema translation) plus CDC to copy cloud-produced artefacts back to the mainframe.

We built and tested this interception for one exemplar pipeline. The same handshakes then informed designs for migrating upstream and downstream pipelines, with legacy mimic feeding the mainframe the data it needed for downstream processing. The approach still required non-trivial mainframe changes for data extraction and feeding back, but reusing the same batch workloads at the core — with different job triggers at the edges — kept the risk contained.

Data characteristics as a granular seam

Occasionally these strategies are insufficient, as we found with one discrete module fed by the data-load pipelines: identity curation. This space was a client differentiator. They could not accept an outcome from a new system that was less accurate than the mainframe for the UK&I population. Migrating the whole module at once would have demanded building tens of identity search rules and their database operations — too large a change for safe delivery. We needed finer-grained seams.

Working with SMEs and engineering teams, we looked for characteristics in the data and rules that could act as seams for incremental cutover. This analysis produced two groups:

• Simple rules could run on either system, provided they operated on different data segments from separate upstream pipelines. They formed roughly 70% of the rules and were triggered during file ingestion, establishing associations between an existing identity and a new data record. They were candidates for incremental migration.
• Complex rules were triggered when a data record indicated an identity change — creation, deletion, or update. They could not be migrated incrementally: multiple data segments could trigger updates to a single identity, and running such rules on both systems in parallel would cause identity drift and data quality loss. A single system had to own identity minting at any point in time, dictating a big-bang migration for this group.

Our original understanding was that ingesting pipelines triggered DB2 changes that kept identities, data records, and their associations up to date. Further analysis with SMEs refined this into a discrete identity module fed by multiple data pipelines, applying both simple and complex rules.

The constraints drove a design that split the work differently: simple rules were handled on cloud while complex rules stayed on the mainframe, with the constraint that no identity data was maintained on both systems at once. We used the pipeline-step-handoff and event-interception techniques we had already established, capturing and forking data feeding the identity pipeline on the mainframe. This allowed a divide-and-conquer approach over ingested files: a parallel cloud workload executed the simple rules and applied identity changes back on the mainframe. Because many rules fell in the simple bucket, the target cloud module also needed a fallback path to the mainframe for any rule not yet implemented.

As cloud builds improved, fewer simple rules needed fallback. Eventually only complex-rule processing would be visible on that leg. Those rules — built incrementally against a cloud database replica and validated by comparison testing — would then be released as a single unit. At that point the fallback leg would be disabled entirely, turning the mainframe's identity and association data into an effective replica of the new cloud primary store. Replication would then keep the mainframe functional.

As in other parts of the programme, the design relied on a legacy-mimic layer and an anti-corruption layer of adapters between systems. Data flowed out of the mainframe as streams for cloud event-driven pipelines and returned as flat files for existing batch jobs. Diagrams throughout this work simplified these adapters away, but they were integral to every data crossing regardless of seam grain. This particular initiative did not progress beyond analysis and spikes. We validated that a CDC tool and the file-transfer service could move data in and out of the mainframe in the required format, but building the scaffolding around the mainframe and reverse-engineering the as-is pipelines exceeded the first phase's timeframe.

Downstream processing handoff as a granular seam

The online flow required the same legacy-mimic adapter pattern for its migration. In the existing system, a customer API call triggered programs that produced side-effects such as billing and audit trails, persisted in mainframe journals.

To move the online flow incrementally, each side-effect either needed handling in the new cloud system — increasing scope there — or an adapter back to the mainframe to execute the underlying programs. We chose the latter, using CICS web services. The adapter was tested for functional requirements only: production-like mainframe test environments were hard to obtain in the first phase, so latency and performance could not be validated.

These adapters were always meant as temporary scaffolding. Once the cloud could handle the side-effects itself, their purpose would be gone; the plan was to revert to plain data replication back to the mainframe for as long as continuity required.

Replicating Data to Accelerate New Products

Incremental modernisation also opens the door to product ideas that rely mainly on analytical or aggregated data drawn from the core Mainframe stores. These use cases—reporting, or summaries over trailing periods—typically don’t depend on up-to-the-second information. In such scenarios, data replication can unlock business value with a relatively modest upfront investment, and that early momentum can build support for the wider modernisation effort.

In our recent engagement, the client had already started down this path, using a CDC tool to replicate core DB2 tables to the Cloud. While this enabled new products to launch, it brought its own problems. Without explicitly abstracting the schema during replication, new cloud products become coupled to the legacy data model from day one. That coupling creates a drag on any subsequent innovation in the target environment: you’ll resist changing the core schema because you don’t want to reinvest in reworking a product you’ve only just funded.

To avoid this, our proposed design added further projections from the replica database into purpose-built stores and schemas, on top of which new products would be developed. That gave us room to refactor the schema and sometimes shift parts of the data model into non-relational stores better suited to the query patterns we observed with the subject-matter experts.

Once batch workloads migrate, you need to decide how to keep the stores in sync. Two options stand out: write back directly to the new Primary (the former Replica), which in turn feeds DB2 on the Mainframe—though this raises coupling between the batches and the old schema—or reverse the CDC and adaptation layer so the Optimised store is the source and the new Primary is the target. In practice, you’ll likely need to manage replication per data segment; one segment may replicate from Replica to Optimised store, while another flows in the opposite direction.

Key Considerations for the Offloading Journey

Offloading from the Mainframe involves many moving parts. For large systems, this is a multi-year effort, and the cost of running the old and new environments in parallel is far from negligible. You should not expect dual running to save money: it's an investment that enables a safe transition. The business case, therefore, depends on generating value early to secure stakeholder buy-in that will fund the full programme. Incremental Dual Run supports that by letting teams respond quickly to business demand—complementing Agile and Continuous Delivery practices.

Start by understanding the overall system landscape and its entry points. Those interfaces are essential: they let external users and applications migrate across to the new system you're building. You have freedom to redesign external contracts during the migration, but that requires an adaptation layer sitting between the Mainframe and Cloud.

Next, identify the business capabilities the Mainframe system provides and find the seams between the underlying programs that implement them. A capability-driven approach stops you rebuilding another tangled system and keeps responsibilities separated at the right layers. You will end up building a set of Adapters that expose APIs, consume events, or replicate data back to the Mainframe, letting other Mainframe systems continue functioning unchanged. Treat these adapters as reusable components—you can apply them in multiple places across the system, depending on specific requirements.

Third, if the capability under migration is stateful, you'll almost certainly need a replica of the data the Mainframe holds. A CDC tool can do this work. Understand the cross-functional requirements (CFRs) for data replication up front: some data will need a fast lane to the Cloud, and the tool you choose should provide that where necessary. The tooling landscape is crowded, so evaluation matters. In our case, for instance, we looked at Qlik Replicate for DB2 tables and Precisely Connect specifically for VSAM stores.

Cloud providers are also entering this space. Google Cloud's Dual Run, for example, recently launched its own proprietary data replication approach.

Testing, Tooling, and Next Steps

Testing deserves its own emphasis. Automated testing shortens the feedback loop for the teams building the target system, while comparison testing verifies that both systems behave identically from a technical standpoint. Combine those with synthetic data generation and production data obfuscation, and you get precise control over the scenarios you want to trigger and the outcomes you validate. Finally, production comparison testing confirms over time that the system in Dual Run produces the same results as the legacy one running alone. At minimum, compare outcomes from an external observer's point of view—such as a customer interacting with the system—where useful, go further and compare intermediate system results as well.

Our involvement came in the first few months of a multi-year programme, so some of the approaches we've described were still at an early stage of inception. Nonetheless, the lessons we drew from that work are worth sharing. Breaking a Mainframe exit into viable, valuable steps will always depend on context, but we hope these ideas help you get started and eventually carry your own modernisation roadmap into production.

For a wider view on mobilising a team of teams to deliver a programme at this scale, see our colleague Sophie Holden's article “Eating the Elephant.”