The case for boring infrastructure at scale

Wave is a $1.7B company with roughly 70 engineers, and its product is a CRUD app that adds and subtracts numbers. Unsurprisingly, the architecture is a standard CRUD app setup: a Python monolith backed by Postgres. Choosing simple architectures and solving problems in straightforward ways has allowed the company to scale to this size while keeping engineers focused on user-facing work.

Wave isn't an outlier here. Stack Overflow ran a monolith through its growth to a $1.8B acquisition, all while ranking among the top 100 highest-traffic sites on the internet. Many valuable companies have been built on monoliths. While certain application categories genuinely require more exotic designs, for most use cases—even at top-100 site traffic levels—modern hardware is fast enough that simple architectures can handle the load, and they are far cheaper and easier to build than complex alternatives.

Despite this, the engineering press and conference circuit overwhelmingly favor complexity. At a recent generalist tech conference, there were six talks on building or managing microservice-based architectures and zero on building a simple monolith. A large enterprise conference in SF had a double-digit number of talks on managing sophisticated architectures and none on simple ones. Many attendees at such events work on low-scale applications that would run perfectly well on a monolith, yet they have adopted the latest complex techniques popular on Hacker News and conference stages.

Boring choices that pay off

Wave's current setup uses boring, synchronous Python, meaning server processes block during I/O waits. The team previously tried Eventlet, an async framework, but encountered enough bugs that the theoretical efficiency gains were not worth the operational headaches. Other well-known Python async frameworks have similarly caused significant issues for users at scale. The cost of synchronous Python is wasted CPU during network waits, but when handling only billions of requests per month (for now), that expense is negligible, even with a slow language and retail public cloud pricing. Engineering salaries dwarf infrastructure costs by a wide margin. For long-running tasks that shouldn't block responses, the monolith simply offloads them to a queue.

One area where Wave cannot stay fully boring is datacenter operations. Initially operating entirely in the cloud in Senegal and Côte d'Ivoire, expansion into Uganda and other countries requires on-prem deployments to comply with data residency laws. This is not simple, but it is significantly less painful than it would be with a complex service-oriented architecture that must be replicated across every new jurisdiction.

Building vs. buying

Wave started with a strong preference for buying software over building it, since a small team cannot build everything. While the buy option often produces tools that don't work well, it was the right call initially. However, when vendors cannot fix critical blockers, it does make sense to build internal tools and expertise—even outside the company's core competency. The vendor problem is inherently harder: they must solve a generalized problem for every customer, while Wave only needs a solution for one customer, itself. Yet in many product categories, even after extensive research, no vendor can deliver a working solution.

Where mistakes were made

A notable early misstep involved database transaction boundaries. The SQLAlchemy session is a request-global variable; any access to a DB attribute implicitly begins a transaction, and any function can commit the session, flushing all pending updates. This makes transaction timing difficult to control, increasing subtle data-integrity bugs. It also complicates using database features like idempotency keys or transactionally-staged job drains, and raises the risk of holding long-running transactions open, which makes schema migrations operationally difficult.

Certain tech choices, while not disasters, are questionable in hindsight. RabbitMQ could probably be replaced by Redis for task queuing, reducing operational burden. Celery is overcomplicated for Wave's needs and has caused outages, particularly around backwards compatibility during upgrades. SQLAlchemy obscures what database queries the code will emit, creating debugging challenges. Python was the right initial call given the founding CTO's background, but its concurrency model, performance, and dynamism raise questions for a large-scale backend codebase. None of these were major mistakes, and for some—especially Python—the cost of migrating outweighs the ongoing maintenance burden. Still, a similar project starting from scratch today should carefully evaluate these choices.

Complexity that is worth it

Some seemingly complex choices have been justified for Wave's specific context:

  • Kubernetes: The team knew that business success would bring expansion into countries requiring in-country service operation. Kubernetes was chosen because regulatory requirements vary by country, and some markets mandate a primary datacenter or failover capability within the country. This would be far harder to retrofit later.
  • GraphQL: Despite imperfect library support (the base Python library is a port of the JavaScript one, Graphene requires boilerplate, and Apollo-Android generates poorly optimized code), the benefits have proven worth it: self-documenting return types, safer clients through code generation, interactive exploration with GraphiQL, a shared API across multiple apps, single-roundtrip exact data fetching without special-purpose endpoints, and eliminating REST API bikeshedding. One downside is redundant default encoding, which matters for customers with low bandwidth.
  • Custom protocols: Wave previously used a custom UDP-based protocol with SMS and USSD fallback for performance reasons. The rollout of HTTP/3 allowed replacing the custom protocol with the standard, though USSD remains necessary for events like internet shutdowns in Mali.

The unavoidable complexity of telecom

Telecom integrations are inherently complex. A SaaS SMS provider would be ideal, but the major providers do not operate everywhere in Africa, and their costs would be prohibitive for a business that competes on price. While the engineering team generally costs more than infrastructure, this would not hold if outsourcing all SMS needs—the internal telecom integration team pays for itself many times over, especially since competitors have had to slash prices to match Wave's.

By keeping the application architecture as simple as possible, the complexity budget can be spent where it creates business value. This philosophy has enabled building a large business with a relatively small engineering team, in a domain—African finance—that many consider extremely difficult. One of Wave's earliest advisers, who was critical to its success, initially advised the founders to abandon the idea entirely, foreseeing too many potential difficulties.