Overview

Designing a large-scale instant messenger like WhatsApp involves building a system that can handle enormous traffic while keeping messages temporary. The core requirements include one-on-one conversations, delivery receipts, last-seen timestamps, image sharing, and push notifications.

Capacity estimates based on WhatsApp-scale traffic provide a useful baseline: 1 billion monthly users, 650,000 active users per second at peak, and 40 million messages per second at peak. Server requirements can be estimated using:

servers in chat microservice = (#messages per second × latency) / #concurrent connections per server

With 100K concurrent connections per server and a 20 ms latency per message, the chat server fleet would need roughly 8 servers. In practice, additional servers should be added to handle failures. WhatsApp's engineering team has demonstrated how tuning an Erlang-based server on a FreeBSD kernel can support millions of concurrent connections per server, significantly reducing server footprint.

System Architecture

Two primary microservices model the required functionality. The Chat Service handles traffic for online chat messages. When a user is online, the message is routed instantly. For offline recipients, the Transient Service steps in: it stores messages temporarily until the offline user reconnects. This approach mirrors the design pattern WhatsApp uses in production.

API Design

A REST endpoint exposes the chat functionality through a single message-sending method:

sendMessage(String fromUser, String toUser, ClientMetaData clientMetaData, String message)

  • fromUser: The userId of the sender
  • toUser: The userId of the recipient
  • clientMetaData: Device details and location information
  • message: The content being communicated

Data Model

The data model for user information includes fields for user identity and, critically, tracking the activity status of each user. The last_seen timestamp and an active/inactive flag enable the system to determine whether to route messages directly or queue them transiently.

Message Delivery Flows

Text Message to an Online User

When Alice sends a message to an online Bob, the flow involves several coordinated steps:

  1. Alice's message reaches the chat server she is connected to (Chat_Server_A).
  2. An acknowledgement is returned to Alice; the message is marked SENT.
  3. Chat_Server_A queries the data store to identify which server hosts Bob.
  4. The store responds with Chat_Server_B.
  5. The message is forwarded from Chat_Server_A to Chat_Server_B.
  6. Bob receives the message via a push mechanism.
  7. Bob's device sends an ACK to Chat_Server_B.
  8. The ACK is propagated back to Chat_Server_A.
  9. Alice receives the ACK; the message is now marked DELIVERED.
  10. A second ACK fires when Bob reads the message.
  11. Chat_Server_B looks up Alice's current server.
  12. The store identifies Chat_Server_A.
  13. The read ACK is forwarded to Chat_Server_A.
  14. Alice sees the message status as READ.

Image Message to an Offline User

For sending an image to an offline Bob, the sequence deviates from the text message path:

  1. Alice sends the image to Chat_Server_A.
  2. The file is uploaded to a File Server and stored within a directory structure.
  3. The File Server returns a URL for the uploaded image.
  4. Alice's client receives the URL for rendering, and her message is marked SENT.
  5. Chat_Server_A requests the routing info for Bob.
  6. The data store returns Bob's offline status.
  7. The message body containing the image URL is routed to a transient server.
  8. The transient server persists the message in transient storage.
  9. When Bob comes online, he performs a heartbeat with Chat_Server_B.
  10. Chat_Server_B retrieves queued messages from the transient server.
  11. The messages are delivered to Bob.
  12. Bob's device fetches the actual image from the file server, and the transient message references are cleaned up.
  13. Bob's device sends an ACK to Chat_Server_B, noting the image's delivery.
  14. The server information for Alice is fetched.
  15. The ACK is forwarded to Chat_Server_A.
  16. Alice's message is marked DELIVERED.

Transient Storage

A queue-based mechanism using FIFO ordering is recommended for storing offline messages. Existing cloud services such as Amazon SQS or Windows Azure Queue Service meet this requirement. References in these queues are purged as soon as the offline user picks up the messages, ensuring no permanent storage footprint.

Delivery Methods and Status Features

Two fundamental approaches exist for delivering messages to clients: polling and server push.

Client pull via polling offers two options:

  • Short polling — Simple to implement and light on server resources if the polling interval is long. The trade-off: undesirable delay for real-time event notifications.
  • Long polling — Offers immediate server-event notification. Success comes at the cost of the additional complexity to implement it along with higher server resource usage.

Server push approaches are better suited for real-time chat:

  • WebSocket is the de-facto protocol for chat applications. It provides full-duplex communication over a single TCP connection, making two-way conversation flows natural.
  • Server-Sent Events (SSE) enables one-way asynchronous data flow from server to client after an initial connection. Applications involving pub-sub patterns, such as streaming stock prices or live feeds, are a better match — though the lack of a bidirectional channel makes them less viable for chat.

The standard "last seen" indicator relies on data stored in the user information model, tracking the last activity timestamp. Similar status data supports rich presence features for delivery confirmation within the push framework.

Optimizing the Design

Several parameters directly influence the system's behavior and cost profile:

  • Latency: In-memory caching is the primary lever for cutting response times. A distributed cache such as Redis can store user activity statuses and recent chat histories, reducing the load on persistent storage. Alternatively, database-native caching layers like Amazon DynamoDB Accelerator can serve the same purpose without adding a separate cache tier to the architecture.
  • Infrastructure Cost: Chat servers are the dominant infrastructure expense. Keeping this cost in check requires maximizing the connection density per host. The number of servers needed is inversely proportional to how many connections each can sustain. Tuning both the server application and the operating system kernel is necessary; as a reference point, WhatsApp's engineering team optimized its Erlang-based servers and tuned the FreeBSD kernel to handle millions of concurrent connections per host.
  • Availability: Temporary messages held for offline users are a single point of failure. To guard against data loss, the design should maintain multiple copies of this transient data across distinct storage instances. When the recipient comes online, the client or server retrieves messages from these multiple queues and merges them into a single ordered timeline.

Handling Failures

Two components are particularly susceptible to outages: chat servers (which hold live connections) and transient storage (which holds undelivered messages).

Chat Server Failure. A chat server dropping offline disconnects every user attached to it. Two recovery approaches are possible. The first is transparent TCP connection fail-over to a healthy server; this is technically complex and rarely worth the implementation overhead. The second, simpler approach is to have the client automatically re-establish a connection when it detects a drop. In both cases, the authoritative database must be updated with the new server assignment.

Transient Storage Failure. If a transient storage instance becomes unavailable, in-transit messages can be lost permanently. Maintaining a replica of each user's message queue solves this. Upon reconnect, the system queries both the original and the replica, fetches messages from each, and merges them before delivery.

Monitoring SLAs

Two service level agreements should be defined for the sendMessage API to guarantee performance and reliability:

  1. Availability: p99.999 — an alarm fires when more than 1 in 1,000 requests fail.
  2. Latency: p99.99 of 5 milliseconds — an alert triggers when response time exceeds 5ms for more than 1 in every 100 requests.

Beyond threshold-based alerts, failure alarms should be wired into specific error paths. One critical scenario occurs when a chat server attempts to fetch offline messages from all transient storage replicas (Step #10 in the original flow, where Chat_Server_B retrieves messages for the offline user). If the system maintains two copies of that user's messages and neither can be retrieved due to a storage issue, the operation fails silently and requires immediate debugging. This path merits its own dedicated monitor.

Extending to Groups and Security

The single-user messaging model can be generalized to group chats. A GroupChatID data entity maintains the roster of participants. The delivery logic scales across the existing architecture: the component responsible for routing messages checks the activity status of each group member and either pushes the message (if online) or routes it to that user's transient storage for later retrieval.

Encryption is end-to-end by design. Each user holds a public/private key pair. When Alice sends a message to Bob, she encrypts it with Bob's public key; the server only sees and routes ciphertext. Bob decrypts with his private key. This scheme ensures the server can never read the content of messages exchanged between parties.