You Can't Buy Integration

Integration Is Not a Product

The market for commercial software emerged after the 1974 CONTU ruling made computer programs copyrightable. Today, "build versus buy" is a standard architectural conversation, and rightly so: building software is risky and expensive, and product vendors can spread that risk across many customers. But the decision framework that works for fraud detection or CRM systems does not carry over cleanly to every category of software.

Integration tooling is one of those categories. Despite decades of ESBs, ETL tools, API platforms, and cloud integration services, you can't buy integration. What you can buy is a programming language—bundled with a toolchain and runtime, and marketed under the guise of a product.

The Low-Code Reality

Most integration tools are low-code platforms. They offer a graphical design palette where you drag and drop workflow steps, and the platform translates your interactions into a markup representation like JSON or XML. The runtime interprets that markup, not unlike how Micro-Soft's Altair BASIC interpreter converted source into machine code in the 1970s. AWS Step Functions is a typical example: a workflow can be authored either through the visual palette or directly in JSON. Changing one representation is immediately reflected in the other, a property worth calling source-diagram isomorphism.

In normal software development, by contrast, the developer edits source code directly; there's no diagram that can be translated back and forth. Call that source endomorphism if you like, or just "normal." The graphical palette exists to provide a domain-specific language for integration, a way to organize thought around wiring systems together without being distracted by the incidental complexity of a general-purpose language. Excel's functions are a similar idea: they make budgeting problems easier to express than writing Java from scratch.

But this is precisely where the buy decision gets distorted. When you acquire an integration tool, you are not buying an integration. You are agreeing to build it yourself, in a commercial DSL. The purchase merely promises the implementation will be simpler than writing the same logic in a general-purpose language. The architect's job is to judge where that promise holds.

Some integration DSLs are genuinely simpler projections of their problem space—like a calculator. Others are Turing complete, meaning they technically match the algorithmic power of any general-purpose language. But computability theory fails to capture software engineering, which Google engineers define as "programming over time." Programming requires abstractions; programming over time requires evolving abstractions within a complex, changing ecosystem, under team agreements and quality practices. That is where the calculus changes.

A Broader View of Integration

The conventional framing of integration is about connecting systems. Digital organizations, however, treat it differently: integration is primarily about putting clean interfaces in front of digital capabilities, where capabilities take precedence over systems. This reframing changes which parts of the problem lend themselves to simplifying DSLs and which require the full expressive power of a general-purpose language to manage evolution and governance over the long term.

To understand that distinction, it helps to recognize what exactly an integration tool is. It is a programming language with a runtime—the source markup alone has no meaning until interpreted. The question, then, is not whether such tools can work, but where they fit within a broader architecture that treats interfaces as first-class elements and must survive the passage of time.

Interfaces Over Implementations

There is a common assumption that integration is about connecting systems and keeping them in sync by sharing data. For a modern digital business, that definition falls short. The real goal of integration is to create clean interfaces over digital capabilities.

When integration is framed as connecting systems, success is measured by how fast a new system can be wired into the existing estate. The systems themselves become the primary value driver, and integration is merely a necessary evil to make them behave. Shifting the focus to clean interfaces over capabilities changes the metric of success: it becomes organizational digital agility over time, and the capabilities—rather than the systems—are the true value drivers. That distinction is more than semantic.

Digital organizations emphasize interfaces over implementations. A simple interface belies a complex implementation; that is the trade-off that enables scaling inside a complex ecosystem and is one of the critical elements of a successful product.

Consider the keyboard I'm typing on. I have little understanding of the mechanical-electrical implementation, the input system drivers, or the operating system interrupts that turn a keystroke into text on a screen. Many people had to figure those pieces out — the keyboard, the driver, the OS, the monitor, and the application are each separate products. All I have to do is press the right key at the right time to integrate my thoughts into on-screen words.

The corollary is that simplifying an interface often requires accepting a more complex implementation. We routinely accept this for digital products. Google Search is unimaginably complex under the hood, yet uncannily easy for a digitally unsavvy user. Business users who push for Salesforce understand that while its interface may be more intuitive, it requires significant effort to maintain and evolve, which justifies the subscription fees. Yet we treat integration differently. We intuitively accept that the two-dimensional boxes in an architecture diagram hide considerable complexity, but we expect the one-dimensional lines between those boxes to be simple. They are not. You can buy the boxes—the digital products—but you cannot buy the lines. Integration is not something you can buy.

Historically, project plans and budgets are built around the boxes. The lines connecting them are the hidden, often primary driver of organizational tech debt. They are why things just take longer now than they used to. Simplifying the glue code is a noble effort, and integration tools can help, but not if doing so displaces the effort of building clean interfaces over capabilities. The only effective judges of how easy an interface is to use are the people actually using it. Google could have asked users for more information to simplify their implementation—geographical, recency, and popularity data, for example. Instead, they offered a single text box and handled those factors in their algorithm. The same holds for API design, broadly defined to include synchronous calls and asynchronous events.

Avoid Leaking Implementation Details

Clean interfaces hide implementation details—including the programming languages of the systems involved. No architecture diagram that I've seen puts primary focus on the implementation languages, yet plenty of diagrams do exactly that for integration. Presenting integration views that way reinforces a tactical understanding of integration as wiring systems together, emphasizing the wiring toolchain instead of the underlying digital capabilities.

Another implementation detail that API consumers would be happy to ignore is which upstream system the data comes from. Outside of the business users who work in SAP and the IT staff around them, nobody should have to care about the quirks of the SAP system. They only care about how to access customer data or create an order. This is one of the most commonly violated principles in integration strategies, and one of the strongest indicators that a company implicitly views integration as wiring systems together rather than creating clean interfaces. You don't need an SAP API, because your API users don't care about SAP. You might need an order management API. Abstract the capability, not the system.

Interfaces Evolve Over Time

Users don't stand still, and a good API adapts through reuse. Over-indexing on reuse as the primary goal is a trap—taming complexity is more important—but reuse is still a useful aspiration. Keeping up with evolving user needs means breaking previous assumptions.

In one of my own integrations, the keyboard, I routinely type in the Colemak layout. Because my physical keyboard couldn't adapt to the software layout, there was an impedance mismatch between the printed letters and what appeared on screen. As a touch typist that wasn't normally a problem—until I had to look at an on-screen QWERTY mapping during the learning process, making it painfully confusing. There are backlit keyboards that can project a letter on the physical key to match the layout. The price of that improved interface is more implementation complexity, an evolution that is a programming-over-time concern.

Integration interfaces that fail to adapt to users over time, or that change too easily with the underlying systems for implementation convenience, become point-in-time integrations. Those are really point-to-point integrations with multiple layers. They may wear API clothing, but they show their true stripes whenever a new system is wired into the estate and the API is duplicated or abused to solve an implementation problem.

Treating integration primarily as a systems problem leads to a landscape littered with point-in-time integrations, which decreases organizational agility. Systems of record, particularly ERPs, resist being put in a box. An ERP is designed to do everything, so externalizing a newly desired capability will conflict with it. Containing the resulting complexity and hiding it from the user requires significant architectural skill. The alternative is increasing organizational tech debt—adding another noodle to the spaghetti of point-to-point integrations. The only way to pay that debt down is to hold the line on a clean interface for your users and externalize the necessary transformations, caching, and orchestration internally. If you don't do this, you are forcing every API consumer to tackle that complexity themselves.

The mindset must be inverted. Instead of asking how to solve integration problems with tools, ask how to build the right interfaces to maximize organizational agility.

Managing Integration Complexity in a General-Purpose Language

Most commercial integration platforms pitch themselves as the center of your integration landscape, calling out to general-purpose languages only when absolutely necessary. That messaging serves the vendor’s interests, not your architecture. The better approach is almost always the reverse: manage interface evolution in a general-purpose language, and keep the integration tool at the edges. Two benefits follow. First, you can actually maintain a clean interface over time. Second, you avoid making strategic decisions through the lens of the vendor’s tooling.

Why general-purpose languages win at long-term maintenance

Integration DSLs offer source-diagram isomorphism — the markup you edit mirrors the visual workflow. That sounds convenient but it makes “programming over time” significantly harder than working in Java or similar languages. The practical difference shows up in the most basic developer activity: diffing changes between commits.

1. Modularity is out of your hands. In Java, you decide where boundaries live. Packages map to directories, classes to files, and a commit typically touches lines within natural, understood boundaries. With an integration DSL, the tool’s design palette has a say in how the underlying textual source is organized. It is not unusual for an entire workflow to live in a single file.
2. Syntax adds noise. The tools do a decent job of pretty-printing their markup with line breaks for readability, but structural changes still reorder elements and produce diffs far larger than the logical change warrants. XML, in particular, adds a lot of visual noise that hides what actually happened.
3. Diffing requires a second mental step. With Java, you read changed lines and understand them directly. With an integration DSL you first read the changed lines, then translate them into what they mean for the workflow you see on the palette. Text is the only medium for representing change over time — graphical designs are not built to show it — so this extra cognitive load is unavoidable.

Gregor Hohpe’s The Software Architect Elevator makes the debuggability point memorable. After vendors demonstrate how easily they can drag and drop a solution, he asks the salesperson to step out for two minutes while he randomly changes something in the underlying markup. Then she has to debug it when she returns. As of the book’s publication, no vendor had taken him up on it.

Scaling a team within one codebase is also harder on these platforms. With general languages, developers control modularity, which is why a dozen Java developers can work on parallel changes and merge without constant conflict. With integration DSLs, the modularity constraints and the extra abstraction step make merging painful. The common workaround is to split the work into separate components developed in parallel — effectively dodging the problem instead of solving it.

Testing and promotion suffer too. Every test run requires starting the runtime that interprets your XML markup, which makes short red-green-refactor loops impractical. You are also locked into the vendor’s testing framework. And while ecosystems for general-purpose languages evolve quickly with new testing tools, IDEs, observability libraries, and supply chain scanning, low-code ecosystems move far slower because their community is so much smaller.

Operational support in integration runtimes is comparatively weak as well. Observability and resiliency patterns get real attention in general-purpose platforms; they are not the main focus of integration tools. I have watched multiple large-scale low-code adoption projects develop serious performance problems over time. The typical first answer is buying more licenses, until that stops working. By then, the platform lock-in is effectively permanent.

A colleague once ran a bake-off in a contentious environment where TIBCO BusinessWorks was mandated. He proposed building an integration to a COTS product’s SOAP web services — coded in Apache Axis — and challenged the TIBCO team to match him. His Java/Spring developer had a working implementation by lunch. The TIBCO team could not get there at all because the tool did not support that older version of Axis. The mandate would have meant going back to the vendor to change their roadmap, or writing an extension in a general-purpose language anyway. Fred Brooks called that “accidental complexity” in No Silver Bullet: complexity added by the choice of solution, unrelated to the problem at hand. A mandate to route all integration through low-code tools guarantees a steady accumulation of exactly this kind of complexity.

Implementation-focused thinking is built into the tools

Integration tools exist and thrive because connecting SAP, Siebel CRM, mainframe billing systems, and Oracle ERP is genuinely hard. When a business wants to swap out Siebel for Salesforce, they understand there will be configuration work. What they do not accept is a long IT timeline just to rewire the glue. Historically those long timelines came from point-to-point integration, where every new connection meant relearning how to link systems, interpret data, and route messages. Integration platforms broke the work into reusable pieces, particularly the connectivity into each system.

The abstraction they use is revealing. AWS Step Functions, like most of its peers, describes everything in terms of actions on some service — a DynamoDB table name in a step, a Lambda invocation in the next. The palette is entirely implementation-oriented. This is a modern example, but the pattern goes back to the enterprise service bus. ESBs separated out system connectivity from orchestration and routing, aiming to remove the “donkey work” of integration. Though the industry has largely moved past ESBs, their thinking persists in modern API platforms. MuleSoft’s three-layer API architecture — experience, process, and system APIs — preserves that same separation of concerns, now dressed up in API terminology.

The system layer focuses on connectivity; the process layer on orchestration. The difference between that picture and the old ESB is hard to spot. MuleSoft’s real value sits in the library of connectors for systems like SAP and Salesforce — shortcuts at the edges of the system. The layered architecture exists to encourage reuse of those connectors. Conceptually it aims for more reuse across layers, but in practice you run into the programming-over-time problem of evolving process APIs for multiple consumers. I have seen many APIs that are really ETL in API clothing: the system layer extracts, the process layer transforms, the experience layer loads.

The channel API idea has genuine merit — it narrows access for attackers and lets a channel evolve faster than the underlying capabilities. I prefer “channel API” to “experience API” or “backends for frontends” because the name makes the broader scope clear. What I would push back on is the prescriptive separation between process and system APIs. That emphasis on orchestration and connectivity pulls thinking toward integration as a tactical concern — about systems, not capabilities. The interface matters more than the implementation behind it, yet the tools keep directing attention to the implementation layer.

Integration tools are attractive because they make the tactical wiring of systems cheaper and lower the risk of custom software. The trap is that the framing of the problem comes along with the tool. Once you let the tool define the terms, you have let the tool do the thinking for you.

Commercial Tools Have a Place—If You Draw the Boundary Correctly

Being skeptical of enterprise-wide integration tool mandates doesn’t mean rejecting the tools themselves. Vendors naturally push for maximum footprint and lock-in, and their product strategy should not be mistaken for your architectural strategy. The architect’s job is to define a bounded context for the tool—where it helps—and keep it from owning the interface to the rest of the system.

Within that boundary, commercial integration DSLs offer real value in at least two areas.

Accelerating Workflow and Connectivity

Implementation speed is the primary selling point of these tools, and that value is legitimate as long as the tool sits behind an abstraction written in a general-purpose language. The market pushes integration DSLs to “own” the landscape and call out to code only when necessary. To handle evolution complexity well, you want to invert that: keep the parts of the system likely to change over time in a mainstream language, and let the DSL handle the plumbing that is unlikely to shift.

One team I worked with uses Camunda for microservices orchestration, taking advantage of its Java library mode with Spring and Spring Boot integrations. That setup allows traditional software engineering discipline to manage the evolving interface in Java, while the workflow tool simplifies specific implementation steps. Open source in that case, but a commercial equivalent would serve just as well.

System connectors and adapters provide similar lift. They can be hidden behind a capability abstraction written in a general-purpose language—advice consistent with Mulesoft’s system API guidance, even if your overall API strategy does not emphasize the systems underneath. Graphical workflow visualizations can also speed up wiring multi-step processes for simple cases, much like the AWS Step Functions example described earlier.

Be cautious about putting transformations inside the integration DSL. Transformations sit between source system data and the interface consumers expect, so they face evolutionary pressure from both sides: changes in the system of record and changes in the consumer-facing contract. That makes them a hotspot for long-term complexity, and you should expect to reimplement them in a language like Java over time. Keep performance optimizations and resilience code like caching in the general-purpose layer as well—those tend to grow intricate quickly.

Handling the Long Tail of B2B Integration

B2B scenarios frequently demand integration with parties outside your organization, and you cannot always count on clean APIs. Partners range from sophisticated enterprises to small businesses with minimal IT capacity. The combination of system diversity and low technical capability creates a recurring challenge I have seen in three different industries:

• An energy company transacting through distributors, sharing sales information to automate stock replenishment.
• A heavy machinery retailer coordinating parts delivery with third-party dealers.
• A health care services firm working with payers to detect fraud, waste, and abuse.

Even when partners do maintain proper IT systems, the variety can be overwhelming, and you often lack the leverage to require them to integrate to your API contract. Many B2B partners operate in legacy industries, where FTP file transfers, EBCDIC conversions from mainframes, and EDI remain practical necessities.

This domain changes the tradeoffs. Slow-moving IT means programming-over-time concerns are reduced, so putting transformations directly in the integration DSL becomes more acceptable. Commercial tools likely already support the required integration patterns and transformations, making them an inexpensive adapter between the partner and your dedicated channel API. Tackling the long tail of B2B integration with a general-purpose language can be prohibitively expensive; using tools built for quick, non-rapidly-evolving solutions is often the better economic choice.

Integration Is Strategic, Not Tactical

The most common justification for buying integration tools is some variation of “we’re not a software company.” It’s an understandable principle for prioritizing investment, and there is truth to it: developers comfortable with integration DSLs are cheaper in the labor market than those fluent in general-purpose languages. But the reasoning is penny wise and pound foolish. You’re probably not a math company either, yet at scale you rely on advanced mathematical skills. You don’t solve that by handing your statisticians a weaker calculator and asking them to reshape every problem to fit the tool’s complexity ceiling.

Software feels different because custom development is notoriously risky and expensive. Graphical integration tools offer a more approachable form of custom software, and each line between boxes on an architecture diagram becomes simpler to draw. But the complexity ceiling of such tools means the number of lines explodes. The result is like pouring slow-hardening concrete over your architecture, accumulating technical debt with time.

When the API Doesn’t Fit the Use Case

A few years back, I worked with a telecom building self-service eCommerce for new phone purchases. Telco services are fundamentally more complicated to buy than retail products because they have a lifecycle, hidden behind the clean interface of a phone number. An API already existed, but it was built for call center agents, not websites. It expected a transaction log of the agent’s actions before returning available plans — an obviously wrong abstraction for a website.

We worked around the limitation by creating a fake transaction to receive an XML payload full of system details:

      <x:offerDetails>
        <id>2207891</id>
        <program>2205442</program>
        <filter>
          <typeCode>C</typeCode>
          <subTypeCode>E</subTypeCode>
          <contractTerm>24</contractTerm>
        </filter>
      </x:offerDetails>

Coordinating with experts to decode those leaky abstractions from the underlying billing system was only half the battle, as a subsequent call for pricing returned over 1,000 lines of XML, of which roughly 100 mattered for eCommerce.

The path forward was to work with the IT organization to build new APIs that represented eCommerce concerns directly, translating the legacy complexity into clean interfaces that required no knowledge of billing mechanics. The architecture diagrams shifted from underlying systems to digital capabilities, deliberately excluding downstream complexity and implementation languages. That telecom became the first in its country to offer fully automated self-service when a new iPhone launched, beating direct competitors and Apple itself.

Capabilities, Not Wiring

The famous Jeff Bezos mandate — communicate only through externalizable APIs — may be apocryphal, but its consequences are visible in Amazon’s dominance. The mandate flipped the integration conversation from wiring systems together to exposing business capabilities, and it created secondary revenue streams from internal operations. Infrastructure provisioning, call centers, and fulfillment became reified as programmable interfaces: new boxes on the architecture diagram where lines used to be.

An integration strategy built this way is the key architectural component of organizational agility. Treating integration as a tactical nuisance to be outsourced to a product will always keep it tactical. Reframing it as self-service interfaces between business capabilities can produce significant business value, through principles like:

• Design your interface from your users’ perspective. Your APIs are digital products designed to make your developers’ and integrators’ lives easier, not yours.
• Abstract the capability, not the system. The underlying system is an implementation concern. Avoid leaky abstractions and simplify the view of the capability.
• Hide implementation complexity, even through evolution. Build abstractions that can evolve, even if the implementation becomes more complicated underneath.
• Create the future; adapt the past. Don’t push legacy complexity onto consumers, who have less context than you do to wrestle with it.
• Integration is strategic to your business. At scale, rationalizing business complexity requires simplifying abstractions behind clean interfaces.

In The Software Architect Elevator, Gregor Hohpe argues that digital organizations operate in the first derivative, focusing on their rate of change. A strong integration strategy lives in the second derivative: investing time and money to simplify interfaces to organizational capabilities accelerates digital transformation. It may slow you down at first, but over time those interfaces become the gas pedal.

So buy your CRM, your revenue management system, and your ML-driven sentiment analysis add-on. Buy your API gateway and your container orchestration system. Adopt product operating models and autonomous team structures. None of it makes you competitive in a digital world if integration remains a tactical nuisance. You can’t buy integration — and that’s fine. Building it yourself may be the most strategic software in your portfolio.