Technology
The platform
The most immediate pleasure of working at Heroku was the product itself. Apps could be created and deployed in seconds, which made prototyping easy and allowed fast iteration on production software. That frictionless path from idea to running code is something every company should provide to its developers.
That doesn't necessarily mean every company should run on Heroku — it's one way to get that capability without a major infrastructure investment. As a company grows, it may make sense to build an internal version of the same idea. Remind did this with Empire, SoundCloud with Bazooka. GitHub's model of deploying experiments and small apps to Heroku and promoting them to dedicated infrastructure when needed is a workable compromise between agility and performance.
Dogfooding
We used our own products wherever possible. Every production app at the company ran on the Heroku platform except for a small set of core services that couldn't. Internal Salesforce apps were migrating over each year as well, showing the approach was valuable enough to spread organically to the parent company.
Internal apps that required login — the Dashboard, the Help system, the add-ons SSO portal — all used the same Heroku OAuth provider available to third parties, which kept services loosely coupled and easy to build.
One of the most valuable consequences: the Dashboard, which customers use to manage their apps through a web interface, runs against the same public V3 API available to everyone. That practice surfaced an untold number of bugs that would otherwise have been found by frustrated customers or third-party developers.
Twelve-factor and conventions
Twelve-factor methodology gave internal apps a shared set of principles. Every app got configuration from the environment, had a Procfile, and emitted logs to standard out. That common ground made it easier for engineers to reason about unfamiliar services.
The criticism that twelve-factor is just a set of workarounds for platform limitations doesn't hold up. It's a solid foundation, but some of us wanted even stronger conventions. The service toolkit Pliny was designed to offer a powerful out-of-the-box stack that would be a sensible default for most internal apps. The only misstep: we should have pushed it earlier and harder. Even the basic form it took eliminated the problem of every new service becoming a special snowflake of its author's favorite ideas. With automatic distribution of updates, built-in rate limiting, and service discovery, it could have gone much further. Internal service frameworks are important enough that most mid-sized I/P/SaaS companies should have dedicated people building them.
The HTTP API design guide
Left to their own devices, engineers will design HTTP APIs slightly differently every time. One engineer will name a new resource /resource_with_underscores even though 78 out of 78 existing resources look like /resource-with-hyphens. Keeping a cohesive API when dozens of people contribute requires codified, opinionated conventions.
The HTTP API design guide was written from the decisions made building the V3 API. The result is that Heroku's public API is one of the most self-consistent HTTP APIs in the real world.
Postgres
Given a blank slate, I might avoid some of Heroku's staples like Ruby. Postgres is the exception — I would use it without hesitation. It's powerful, flexible, and incredibly stable. Having seen how other popular database software operates in production, I have a sober view of just how good Postgres is relative to the alternatives.
We may have missed some cutting-edge technologies by not jumping on every new data store, but the resources saved by that restraint are incalculable. There was probably room for an HA store eventually, but it was right to delay until mature options existed. In the meantime, we got very good at operating Postgres and it handled almost everything.
The only thing better than Postgres was the Heroku Data team. Those engineers saved me an untold number of times on messy operational problems; I was told more than once that as the operator of the largest internal database, I was their highest-maintenance customer.
Ephemeralization
Ephemeralization means doing more with less. Beyond the efficiency gain, reducing the number of moving parts lowers a system's cognitive burden and makes it easier to learn. Picking one true way forward from similar options keeps engineers productive as they move between components. Some examples:
- Choose a single "right" technology from each set of like options — Ruby over Python, and focus on just Ruby and Go (and JavaScript) over the long run.
- Zero in on one library for each function, such as preferring Puma over Unicorn or Thin for Ruby HTTP stacks, and Sequel over ActiveRecord.
- Standardize deployment images so components share one Chef recipe and differ only at the application level.
- Use a single type of data store consistently — Postgres.
- Never create internal forks of libraries, obvious as that should be.
Use services
Use hosted services whenever possible instead of operating them yourself. The cost of infrastructure and bringing a service online is usually well understood. The full personnel cost of maintaining that service — who will upgrade it and migrate data a year from now, who will retire it when the time comes — rarely is.
Culture
Leadership & inspiration
Working at Heroku meant being surrounded by people who inspired me on a fundamental level, especially in my early days. The company had everything at one point: great leaders, inspiring thinkers, and incredibly ambitious engineers. As someone relatively inexperienced and new to the Bay Area, my first few months felt like a constant stream of new ideas about technology and organizational structure. This motivated me to build great things and made the learning exciting.
Self-service
Rather than doing work for someone, give them the tools to do it themselves. Heroku’s core API service, for instance, had a private administrative branch that employees could access through a CLI plugin to perform special actions like re-sending a sign-up e-mail. This sets a strong precedent for people to try solving problems on their own before leaning on others. With sufficient coverage, this approach reduces constant disruption on open communication channels so people actually have time to work.
Cross-team contribution
Want a new feature or improvement? Send a pull request. There’s no better way to show commitment to an idea. It also gives engineers a wider view of how the whole system works, forcing them to look beyond the narrow confines of the projects they maintain day to day. This obviously doesn’t scale infinitely, but it scales far further than many would have you believe.
Shipping cadence
We shipped services fast and frequently, with tooling that made it safe to do so. Changes often went out the same day, keeping endless possibilities open for new products or improvements. This had to be discovered at the organizational level too. There was a period when projects were hard to ship due to a weak process for getting them across the finish line. The problem was examined and corrected, and products have been making it out the door regularly ever since.
Strong engineers
At its core, this means hiring good engineering talent. But it gets murkier up close. You want people who are good at what they do, but more importantly, flexible enough to jump into almost any project and fix bugs or make changes. That requires independent learning and problem-solving skills not everyone has. When achieved, it results in fewer disruptions for the rest of the team and higher output overall. These ideal candidates might not invert a binary tree on a whiteboard or have a Stanford education, and the interview process may need adjusting to accommodate them.
For quite some time we had a team that synced weekly and plowed through huge workloads the rest of the time. Communication was largely asynchronous, except for instances where a higher-bandwidth channel was more suitable. It was the most productive environment I’ve ever seen.
Technical culture
Technical culture was actively fostered, which I believe led to a high degree of technical excellence in our products. It showed up as papers being passed around, discussions on the engineering mailing list, and plenty of forward-thinking water cooler speculation on improving products and internal architecture. For a long time we also held a weekly technical event called “Workshop” every Friday, where engineers could show off interesting projects. It was designed to educate and inspire, and it worked.
Flexible environment
Traditional organizations often insist everyone punch in at 9 AM, leave at 5 PM, five days a week year round. At Heroku people regularly worked from home or elsewhere. It made little difference to productivity but had a profoundly positive effect on happiness. I visited family in Calgary for weeks at a time two or three times a year, and worked from Berlin for roughly three weeks almost every year I was there. This works if you hire well — the right people do the right things without constant oversight.
Coffee
Admittedly self-indulgent, but I came to appreciate coffee for the first time at Heroku. For the longest time there wasn’t even a coffee machine — just Chemex pots, a grinder, and paper filters. Making coffee became a five-to-ten-minute process during which you’d interact with colleagues who dropped by. The system worked. I learned both Chemex and AeroPress, and still use both regularly.
Process & organization
GitHub
GitHub has been one of the best pieces of software on the Internet for years, and is the right way to organize code and projects. Companies should use tools developers can extend to optimize their workflows and maximize efficiency. With a well-maintained API and a healthy ecosystem of supporting tooling like hub and ghi, as well as complementary turnkey services like Travis, GitHub is one of those tools. Time developers don’t spend supporting custom infrastructure or fighting bad tooling is time they can spend building your product.
Access to resources
If an engineer needed a new resource for a service being deployed, a prototype, or even a one-off experiment, they were free to provision it and keep working — even if it wasn’t free. Resources might include anything from dynos to run an app, to a Postgres backend, to a few extra EC2 boxes for deployment to bare metal. Having Heroku’s catalog of add-on providers and being fully deployed on AWS helped enormously — no internal personnel were ever needed to help with provisioning.
This works because despite a nominal cost to the organization, it keeps engineer momentum up and the cost of prototypes down. It’s hopefully becoming standard practice in many newer companies, but it’s easy to get wrong. I’ve seen the other side where provisioning a job queue is a multi-month process with endless meetings, territorial ops people, and mountains of paperwork. Some care is needed to avoid shooting from the hip with new technology, but that approach doesn’t help anyone.
Total ownership
Our version of “devops,” total ownership meant the team responsible for developing a component was also responsible for its maintenance and production deployment. This mechanical sympathy has huge benefits: faster feature and bug-fix delivery, less esoteric production manipulation, easier coordination for tasks like data migrations, and more personal responsibility for the product — which leads to more uptime. Total ownership was instrumental in improving my engineering skills, but I’m still a bit on the fence. I don’t miss multi-week deployment schedules, but I do miss regular blocks of daily focus without interruptions from production.
Technical management
When I started at Heroku, my manager knew the codebase better than I did, knew Ruby better, and pushed more commits in a day than I would in a week. In planning sessions we’d sketch features in broad strokes and leave the details to each engineer’s self-initiative. There wasn’t time or interest for micromanagement. Eventually we moved toward a model where a virtuous manager didn’t commit code, wasn’t on the pager rotation, and never looked at a support ticket — probably the situation most big organizations have. Though technical management didn’t last, it was a very good place to be an engineer while it did.
