Modern software engineering supplies a familiar toolkit — AI-assisted development, automated testing, CI/CD, observability, automation, simulation, safe deployment — good enough that the assumptions underneath it rarely get questioned. Our team had to question them while modernizing a retail IoT fleet of roughly 800,000 battery-powered sensors across 3000 stores. The existing connectivity technology was reaching end of life, and the replacement had to happen without swapping out installed devices, while newer stores came online with next-generation hardware.
We knew web applications, cloud platforms, mobile applications, IoT and other distributed systems. What we lacked was deep experience in embedded firmware and large-scale device networks. The software engineering principles carried over unchanged; the default practices did not.
Rebuilding system knowledge
After initial spikes on firmware development technology, writing the firmware stopped being the hard part. The hard part was understanding an IoT system that had run for years — organizational changes had left gaps in knowledge of the existing firmware, hardware and network architecture.
Documentation helped, but not enough. Some things were undocumented, and some documents described hardware or behaviour we could not reproduce. Certain answers existed only on the physical devices, discovered while working under limited production access.
So we built and measured. We set up an IoT lab with physical sensors, wire harnesses and learning kits, created environments for radio experimentation, reused sensors with dead batteries for testing, and measured power consumption and network behaviour. What began as preparation and trials became the engineering work itself. The lab turned into the way we learned and reverse engineered — in embedded systems, the fastest answer to a question is often connecting the device rather than reading another document.
The physical world in the development loop
A conventional development environment is made of software: clone the repository, install dependencies, run the application from anywhere. The equivalent here needed physical devices. We built a sensor farm — a controlled collection of real devices on the new network with monitoring around them — so engineers could work with multiple devices and observe communication, sleep, wake-up and connectivity loss. Physical placement, radio conditions and power all became part of development and testing.
Not every developer needed a laboratory on their desk, but the development process had to expose the system's physical behaviour and return fast feedback. Practices such as TDD, automation and XP still applied, only with hardware in the loop.
Devices that explain themselves
With a working device inside the development environment, a new question appeared: what happens when the device is somewhere we cannot reach? It sounds operational; it is architectural. An engineer may get a report that a sensor is misbehaving while the sensor sits in a store or a test setup — no debugger attached, nobody available to open it, the device possibly asleep and the network possibly absent.
Ordinary applications explain themselves through logs and traces; a physical device has no such capability by default. We built diagnostics into the firmware: which firmware is running, device health, last communication time, network state, what preceded a failure, sleep state, repeated reconnection, firmware upgrade status. Health checks and diagnostic information, local access paths, and APIs to retrieve the same data remotely all followed, feeding dashboards and operational tooling.
This reframed observability. We were not adding logging to an application; we were designing a system that operates devices we cannot physically see, where each device must explain itself. Useful for a small number of devices — essential for hundreds of thousands.
Engineer-facing interfaces are system interfaces
Diagnostics covered only part of the need. Engineers also required ways to interact with devices without opening them physically: local interfaces such as NFC and BLE, plus remote interfaces through the gateway and platform, letting them inspect configuration, retrieve diagnostics, understand device state and perform operations remotely. Teams routinely invest in interfaces for users while treating those for engineers and operators as an afterthought. At fleet scale, they are part of the system.
Emulating what cannot be reproduced at scale
Scaling up brought the next surprise. Small stores and a handful of devices were fine; larger stores exposed network infrastructure limits never seen in lab experiments. At one point we found the new network hardware could support only a limited number of devices — far below what some larger stores required. Testing a few devices is a different problem from testing thousands: network behaviour, infrastructure limits, message rates, physical placement constraints and disconnections all surfaced at scales hard to reproduce physically.
Our response was a virtual device emulator: special firmware for the same device that simulates hundreds of unique devices, each configurable to behave differently under different network and environmental conditions. Running many together simulated a large store and tested the parts of the system where scale itself was the problem. It did not replace physical testing — real devices remained necessary to validate firmware, hardware and radio behaviour. We also built a virtual store behaving like a real one, with a gateway publishing simulated sensor readings to the platform, exercising downstream systems at store scale without hundreds of physical devices per test.
Designing around disconnection and energy
Enterprise applications assume the network is available; disconnection is an infrastructure concern to be handled with a friendly message and an immediate reconnect. A battery-powered device has a different relationship with the network. It may sleep when idle to save battery, lose connectivity temporarily, reconnect repeatedly, or be unreachable exactly when the platform wants to talk to it — and a connection attempt is the heaviest operation it performs. Intermittent connectivity therefore had to be designed for rather than treated as exceptional failure, shaping firmware, gateway communication, commands, telemetry and device management alike. Accepting that the device would sometimes be unavailable made many decisions clearer.
Energy was the other easy-to-underestimate constraint: limited energy, years of required operation. A server inefficiency costs CPU, memory or cloud spend; unnecessary computation or communication on a sensor costs useful life. We folded power measurement and profiling into development — when the device is awake, how often it communicates, what each operation costs the battery. Optimization was secondary to the realization that every design decision can have a physical consequence, down to a log statement or a few lines of code that look unused.
Memory management assumptions shifted too. Rather than aggressively cleaning up allocated memory after every operation, we sometimes chose a simpler approach: let the device run for 24 hours and then reboot it as part of its normal operating cycle. On a device, the simplest software solution is often not the one a server would choose.
Deployment and recovery are one design problem
Cloud applications can usually be rolled back in minutes. A fleet spread across thousands of stores has no such option, and firmware raised the stakes further.
Battery spent downloading and installing an image cannot be recovered. A device may be offline mid-update, a network may drop partway through a download, an image may fail validation, or a fault may only surface after the rollout. Each of these changes what "deploy" and "recover" actually mean.
We therefore treated recovery as part of the deployment mechanism rather than something that follows it. Over-the-air updates were built on secure downloads, validation and partition swapping. Updates could target individual devices, groups or whole stores, with progress visible through the gateway and, where useful, locally over NFC. Each device persisted its upgrade state so an update could resume after an interruption or failure.
Rollout was staged. The operating question was no longer "how do we ship this firmware?" but "how do we ship it safely, and what do we do when it breaks?"
From throughput to confidence
Early on the team asked software engineering questions: how fast can we build, how fast can we deploy, how much of the pipeline can we automate. Those questions never stopped mattering, but the working vocabulary shifted toward trust, recoverability and explainability. Can we trust this update? Can we recover if it fails? Can we explain why a device failed or what is happening in a store? Can we update thousands of devices safely? What does it cost in battery, and how long does a store take to upgrade?
That reframing is best described as optimizing for confidence. Speed was not abandoned; the point is that speed without confidence is not progress. On a physical fleet, a fast deployment that leaves thousands of devices unhealthy is a very efficient way to create a very large problem.
Observability at fleet scope
Making a single device explain itself was necessary but insufficient. Nobody inspects hundreds of thousands of devices individually, so the question moves from "what is this device doing?" to "what is happening across the fleet?"
Dashboards and analytics were built to surface communication failures, anomalous behavior, outages, firmware distribution and other fleet-level patterns. Operations teams needed alerts and tooling that could distinguish a one-off fault from a problem spreading through a much larger slice of the fleet.
The surrounding platform becomes the system
Tools that began as engineering conveniences ended up as load-bearing parts of the product: the physical sensor farm for learning and testing, the virtual device emulator for testing at scale, diagnostic interfaces for remote understanding, the gateway for communication and management, and dashboards and analytics for fleet operations.
The resulting shape:
Device → Firmware, Diagnostics and Configuration → IoT Gateway → Telemetry, Commands and OTA → Fleet Platform → Analytics, Alerts and Operations
Figure 1: An engineering environment: Hardware, Software, and Feedback
These were not developer niceties. Together they are what allowed the organization to build, test, operate, migrate and manage a physical fleet at this size — and the value of that platform grew with every device added.
AI against the cost of trying
LLMs entered the workflow in a different place than expected. They were used to explore unfamiliar territory, prototype solutions and shorten the gap between a question and a usable experiment, and AI-assisted development sped up dashboards, migration trackers, outage detection and other internal tooling.
The gain was less about producing code and more about lowering the cost of an attempt.
Document the discovery, not only the decision
Knowledge accumulated through experiments, failures, measurements and incidents. Left unrecorded, the next engineer relearns it the hard way.
Documentation therefore outgrew the architecture decision record. The team wrote down what it had learned about devices, networks, testing, diagnostics, deployment and operations, and spread it through technical articles, training and its engineering community. Conventional documentation states what the system is; in an unfamiliar domain, it is just as valuable to record how the team found out why it is that way.
Same principles, different defaults
No new methodology came out of this. Fast feedback, automation, good testing, observability, maintainability, safe deployment and continuous learning were all still the goals. What changed was the practice of them, until the changes effectively formed a new set of defaults.
| We started with this assumption | We ended up with this sensible default |
|---|---|
| Documentation should tell us enough | Learn by building an experimentation environment |
| Hardware can be tested later | Consider the physical world part of the development lifecycle |
| A device performs its function, and platform can always explain its health | Every device must be able to explain itself |
| Interfaces are primarily for users | Design interfaces for engineers as deliberately as interfaces for users |
| Network behavior, physical environment can be tested and reproduced with a few devices in lab | Simulate what is expensive to reproduce physically |
| Devices should normally be reachable | Design for disconnection, not just connection |
| Battery is mainly a hardware concern | Treat energy consumption as a software design concern |
| Deployment means installing firmware | Design deployment and recovery together |
| Optimize primarily for delivery speed | Optimize the engineering process for confidence at scale |
| Observability means logs and metrics | Design observability for the fleet, not just the device |
| Build the product, then supporting tools | The engineering environment became part of the system |
| AI helps write code | Use AI to shorten the learning loop, not just the coding loop |
| Discovery happens before development and capture decisions in the final design | Capture discoveries, not just decisions |
That table is a record of problems met, not a design handed down: every default was earned.
The reliance on "sensible defaults" as a frame is deliberate. Almost none of these practices came from first principles. The team began with an assumption that seemed reasonable, hit something that contradicted it, experimented, and gradually altered how it worked. That is how practice usually evolves on real projects — the environment shows you where your assumptions are wrong, and if you pay attention, a failure becomes a better default.
That was our experience. When software meets hardware, the engineering principles hold; the sensible defaults may not.



