Why btoa() and atob() Fail on Unicode

JavaScript's built-in btoa() and atob() functions provide base64 encoding and decoding. They are commonly used for transforming binary content into web-safe text, such as data URLs for inline images. The basic usage is straightforward:

const encoded = btoa("hello world");
const decoded = atob(encoded);

However, these functions only work correctly with ASCII strings—characters that can be represented by a single byte. Attempting to encode a string containing Unicode characters, such as emoji, throws an error:

btoa("🧀");  // Throws an error

To understand why, you have to look at how JavaScript handles strings internally.

JavaScript Strings and UTF-16

Unicode assigns each character a numerical code point. For instance, h is code point 104, while the cheese emoji 🧀 is code point 129472. The two dominant encodings for converting these code points into bytes are UTF-8 (1–4 bytes per code point) and UTF-16 (2 bytes per code point).

JavaScript processes strings as UTF-16. The problem is that btoa() expects a "binary string"—one where each character is treated as a single byte. This assumption breaks for any character whose UTF-16 representation exceeds one byte.

Encoding Unicode with TextEncoder

The recommended workaround is to first convert the UTF-16 string into UTF-8 bytes using the TextEncoder interface, then base64-encode those bytes. Here's the pattern:

function bytesToBase64(bytes) {
  let binary = '';
  for (const byte of bytes) {
    binary += String.fromCodePoint(byte);
  }
  return btoa(binary);
}

const encoder = new TextEncoder();
const uint8Array = encoder.encode("I love 🧀");
const base64 = bytesToBase64(uint8Array);

The steps are:

  1. TextEncoder.encode() converts the UTF-16 string into a Uint8Array of UTF-8 bytes.
  2. String.fromCodePoint() turns each byte into a code point that fits in one byte.
  3. btoa() then encodes that binary string.

Decoding reverses the process using TextDecoder.

The Lone Surrogate Trap

This solution works for well-formed Unicode but silently corrupts data containing lone surrogates. Consider this example:

// Encoding and decoding a string with a lone surrogate
const input = "I love \uDE75";  // Lone high surrogate
const encoded = bytesToBase64(new TextEncoder().encode(input));
const decoded = new TextDecoder().decode(base64ToBytes(encoded));

The decoded string now contains the replacement character \uFFFD (�) instead of the original \uDE75. No error is thrown—the data has simply been changed.

The issue lies in how UTF-16 handles supplementary characters. Code points above 65535 are represented by a surrogate pair: a high surrogate indicating the group, and a low surrogate specifying the entry. A lone surrogate is a pair with one missing half—technically malformed UTF-16.

Some APIs, like TextDecoder, reject or replace malformed input. By default, TextDecoder substitutes malformed data with the replacement character \uFFFD.

Validating Input Before Processing

To avoid silent data corruption, check for well-formed strings before encoding. The modern approach uses String.prototype.isWellFormed(), supported in recent browsers (Chrome 111+, Safari 16.4+). For older environments, encodeURIComponent() throws a URIError on lone surrogates and can serve as a fallback:

function isWellFormed(str) {
  if (typeof str.isWellFormed === 'function') {
    return str.isWellFormed();
  }
  try {
    encodeURIComponent(str);
  } catch (error) {
    return false;
  }
  return true;
}

A Complete Solution

Combining these pieces gives a robust function that handles both Unicode and lone surrogates without silent replacement:

function base64Encode(str) {
  if (!isWellFormed(str)) {
    throw new Error("String is not well-formed");
  }
  const bytes = new TextEncoder().encode(str);
  let binary = '';
  for (const byte of bytes) {
    binary += String.fromCodePoint(byte);
  }
  return btoa(binary);
}

function base64Decode(base64) {
  const binary = atob(base64);
  const bytes = new Uint8Array(binary.length);
  for (let i = 0; i < binary.length; i++) {
    bytes[i] = binary.charCodeAt(i);
  }
  return new TextDecoder().decode(bytes);
}

You can further refine this code—parameterizing the TextDecoder to throw on malformed data, for example, or deciding instead to allow explicit replacement. The key takeaway is that careful text validation matters, especially when handling data from external or user-generated sources. A simple encoding routine can silently change content if you're not aware of how JavaScript's string internals interact with the API you're using.