Why Salt's crypto needed a closer look
Cloudflare has been preparing its infrastructure for quantum computers, which will eventually break widely used cryptography like RSA. For most systems, that means swapping out TLS algorithms for post-quantum ones—a routine upgrade. But Salt, the open-source infrastructure management tool Cloudflare relies on for provisioning and automation, uses a bespoke cryptographic protocol, so making it quantum-secure required a deeper investigation. That investigation surfaced several security vulnerabilities, which have since been patched.
Salt's architecture centers on masters and minions (servers and clients). By default, communication runs over ZeroMQ on top of TCP, with an experimental option for a custom TCP transport. The cryptographic protocol is largely the same across both. A setup phase gives each master and minion its own long-term RSA-2048 keypair. Salt then defines a handshake that generates a shared secret, plus a record protocol that uses that secret for symmetric encryption.
Under default settings (as of Salt 3004), the key exchange is an RSA exchange: the server chooses a secret and encrypts it to the client's public key. Clients trust the server's key on first use, and servers only trust a client's key after out-of-band acceptance. The shared secret is fleet-wide, not per client-server pair, so the server can encrypt a broadcast once.
What the symmetric channel looked like
The shared secret feeds an Encrypt-then-MAC scheme using AES-192 in CBC mode with SHA-256 for HMAC. Some more sensitive messages use variations: commands get signed with the server's long-term key, and "pillar data" is encrypted per-client using a freshly generated key.
Vulnerabilities found and fixed
The pillar-data variation contained a design flaw. A monster-in-the-middle attacker sitting between a master and minion could substitute arbitrary pillar data to the client, because neither the freshly generated key nor the payload was authenticated as coming from the server. The only prerequisite was knowing the client's public key, which clients broadcast during key exchange. Since pillar data can specify packages to install or carry credentials and cryptographic keys, this could let an attacker compromise a vulnerable machine.
That issue was reported to Salt on November 5, 2021, and assigned CVE-2022-22934. Salt released a patch on March 28, 2022, adding server signatures to pillar messages. Two other issues were also found and fixed:
- CVE-2022-22935: certain messages could be manipulated to crash a client. Patched with the addressee's name, a nonce, and a signature.
- CVE-2022-22936: messages could be replayed to the same or different clients, potentially serving a file meant for one client to another. Also patched with an addressee, nonce, and signature.
Sites running Salt should update to 3002.8, 3003.4, or 3004.1.
Rethinking the design for quantum-readiness
The patches added signatures to nearly every message, and that's costly: signatures are computationally far more expensive than symmetric encryption, and post-quantum signatures are much larger (Dilithium2, for instance, is 2.4 kB versus 256 bytes for RSA-2048). The original design traded a single shared secret for broadcast efficiency, but that trade-off doesn't hold up once every message needs a signature.
A better approach is to switch to a separate shared key per client and create a single long-lived mutually authenticated channel per client. In practice, that means moving toward mutually authenticated TLS (mTLS). Salt already has an experimental TCP transport option for TLS, but it doesn't authenticate clients and creates a new TCP connection per request, producing unnecessary TLS handshakes.
We tested an mTLS-based design by implementing a third transport using WebSocket over mTLS (WSS). Clients would pin the server certificate, and the server would pin client certificates—no traditional public key infrastructure, which aligns with how Salt handles keys today. Long-lived connections offer confidentiality, mutual authentication, and forward secrecy, and mitigate replays, message swapping, reflection attacks, and denial-of-service pathways by default.
Preliminary experiments with a single server handling a thousand clients showed no meaningful difference versus the default ZeroMQ transport across several metrics. Resource-intensive operations like fetching pillar and state data used less CPU with the mTLS transport, and long-lived connections reduced data volume, sometimes significantly.
These results have been shared with Salt maintainers, and work is underway to add an mTLS-based transport upstream. The path to a post-quantum Salt runs through mTLS, and the same system-by-system review continues across the rest of Cloudflare's infrastructure.
