OpenSSL bug hunt: when allocator calls go out of sync

Most software projects rely on third-party components—GitHub’s State of the Octoverse puts open source dependency usage as high as 94% of projects. That means security work on widely used libraries has an outsized impact. One of those libraries is OpenSSL, which handles SSL/TLS and cryptographic operations for a huge portion of the software ecosystem.

While researching OpenSSL as part of GitHub security work, I found two bugs. The first is documented in GHSL-2020-056. The second, which I focus on here, was already being fixed independently by the OpenSSL team when I was writing my report—a common occurrence with heavily scrutinized projects, and a sign that active maintenance is paying off.

What BN_CTX does and where it breaks

At the heart of this bug is BN_CTX, a memory allocator for BIGNUM objects. Its API is minimal: call BN_CTX_start to create a new slab of contiguous memory, use BN_CTX_get to carve chunks out of that slab, and call BN_CTX_end when done to reclaim the whole slab at once. Individual allocations aren't freed; the entire context goes away in one call.

This design has a catch: calling BN_CTX_end without a matching BN_CTX_start will free memory belonging to an enclosing context that's still in use. Any BIGNUM references held by the caller become dangling pointers into reclaimed memory that may be reallocated elsewhere—possibly in another thread.

The consequences can be serious. If an attacker can influence the memory layout such that process-critical objects land in those freed slabs, they may be able to use the erroneous references to overwrite sensitive data, such as a function pointer, before the application uses it. That opens the door to arbitrary code execution, though exploitation details tend to be application-specific.

A concrete case: commit a9612d6c

Commit a9612d6c introduced exactly this mismatch. The problematic code path occurs when an elliptic curve group (EC_GROUP) lacks Montgomery data (mont_data). In that scenario, the logic skips the BN_CTX_start call and jumps straight to BN_CTX_end:

static int ec_field_inverse_mod_ord(const EC_GROUP *group, BIGNUM *r, const BIGNUM *x, BN_CTX *ctx)
{
   BIGNUM *e = NULL;
   int ret = 0;
#ifndef FIPS_MODE
   BN_CTX *new_ctx = NULL;
 
   if (ctx == NULL)
       ctx = new_ctx = BN_CTX_secure_new();
#endif
   if (ctx == NULL)
       return 0;
 
   if (group->mont_data == NULL)
       goto err;
 
   BN_CTX_start(ctx);
   if ((e = BN_CTX_get(ctx)) == NULL)
       goto err;
 
   // ...
 
err:
   BN_CTX_end(ctx);
#ifndef FIPS_MODE
   BN_CTX_free(new_ctx);
#endif
   return ret;
}

Hunting variants with CodeQL

Control flow reasoning is hard, but CodeQL can automate much of it. The goal was simple to state: find a path in the control flow graph that reaches a BN_CTX_end call without ever passing through BN_CTX_start. The implementation requires a bit more nuance.

I started by writing a predicate that captures the basic relationship between control flow nodes:

predicate isInteresting(ControlFlowNode a, ControlFlowNode b) {
  not a instanceof BN_CTX_start and
  b = a.getASuccessor()
}

This defines pairs (a, b) where a is not a call to BN_CTX_start and b is a direct successor of a. If each pair is a link in a chain, a transitive closure can extend that chain from a function's entry point all the way to a BN_CTX_end call:

import cpp
 
class BN_CTX_start extends FunctionCall {
 BN_CTX_start() { getTarget().getName() = "BN_CTX_start" }
}
 
class BN_CTX_end extends FunctionCall {
 BN_CTX_end() { getTarget().getName() = "BN_CTX_end" }
}
 
predicate isInteresting(ControlFlowNode a, ControlFlowNode b) {
 b = a.getASuccessor() and
 not a instanceof BN_CTX_start
}
 
from BN_CTX_end end, ControlFlowNode entryPoint
where
 entryPoint = end.getEnclosingFunction().getEntryPoint() and
 isInteresting+(entryPoint, end)
select entryPoint, end

The + operator turns the predicate into a transitive closure, which finds all paths from any start node a to any end node b that satisfy the predicate at every step. The results on OpenSSL: 36 candidate paths, including the original bug.

Raw control flow nodes aren't very readable, though. Converting the query to a @kind path-problem—by renaming the predicate to edges and adding the proper metadata—lets the CodeQL UI render full path explanations:

/**
* @kind path-problem
*/
 
import cpp
 
class BN_CTX_start extends FunctionCall {
 BN_CTX_start() { getTarget().getName() = "BN_CTX_start" }
}
 
class BN_CTX_end extends FunctionCall {
 BN_CTX_end() { getTarget().getName() = "BN_CTX_end" }
}
 
query predicate edges(ControlFlowNode a, ControlFlowNode b) {
 b = a.getASuccessor() and
 not a instanceof BN_CTX_start
}
 
from BN_CTX_end end, ControlFlowNode entryPoint
where
 entryPoint = end.getEnclosingFunction().getEntryPoint() and
 edges+(entryPoint, end)
select end, entryPoint, end, "Finalised context may not have been started."

Results for this query were available on LGTM.com. The rendered paths showed each step from entry point to the offending BN_CTX_end call:

Screenshot of CodeQL UI

That's an improvement, but the paths still operate at the level of individual control flow nodes. Switching the edges predicate to use BasicBlock types instead gives a coarser, more human-readable view of the paths:

query predicate edges(BasicBlock a, BasicBlock b) {
 b = a.getASuccessor() and
 not a.getANode() instanceof BN_CTX_start
}
 
from BN_CTX_end end, ControlFlowNode entryPoint
where
 entryPoint = end.getEnclosingFunction().getEntryPoint() and
 edges+(entryPoint.getBasicBlock(), end.getBasicBlock())
select end, entryPoint.getBasicBlock(), end.getBasicBlock(),
 "Finalised context may not have been started."

This version, viewable on LGTM.com, made manual triage far easier:

Screenshot of CodeQL UI

Results and fixes

All 36 reported paths were manually reviewed. Four were genuine mismatches. The remaining findings involved similar patterns that weren't exploitable security issues in context, but were still worth correcting. All were addressed in a single pull request.

The exercise shows the strength of hybrid approaches to auditing: an automated query to narrow the search space, followed by manual review to separate true positives from benign patterns. CodeQL's ability to express reachability requirements—matching a call while avoiding another—makes it well suited for finding this class of allocator misuse.

What the bug can actually reach

The crash sits in ec_field_inverse_mod_ord, a helper that computes the multiplicative inverse of a BIGNUM using Montgomery modular multiplication for efficiency. But a bug in that routine only matters if an attacker can control the inputs that reach it. That turns out to be easy: the same function underpins certificate signature verification through the X509_verify API, which is part of how a client validates a server's certificate.

The trigger condition is specific. The vulnerable code path requires a multiplicative inverse operation inside an EC_GROUP that has no mont_data assigned. That field is populated only by ec_precompute_mont_data, and that function is called only when the group order is odd:

if (BN_is_odd(group->order)) {
       return ec_precompute_mont_data(group);
   }

So the practical question becomes: does an elliptic curve with an even group order exist? If one does, the bug can be reached. The way to find out is to build a proof of concept.

Building the trigger

Since OpenSSL has already patched the issue, the library has to be compiled from the vulnerable commit:

git clone https://github.com/openssl/openssl.git
cd openssl
git checkout -b vulnerable a9612d6c034f47c4788c67d85651d0cd58c3faf7

To avoid clobbering the system OpenSSL, install the debug build into a dedicated environment:

export INSTALL_DIR="$HOME/Installations/openssl-debug"
./config --debug --openssldir=$INSTALL_DIR --prefix=$INSTALL_DIR no-shared no-threads no-tests enable-asan enable-ubsan
make
make install

With a debug library in place, the next step is a small program that generates a certificate on the right curve and then runs X509_verify against it. The full test case is available in a public gist.

The tricky part is locating a curve with an even group order. Checking every curve OpenSSL ships reveals at least one candidate:

std::optional<EC_builtin_curve> EC_get_even_curve()
{
   // Get the builtin curves.
   size_t count = EC_get_builtin_curves(nullptr, 0);
   std::vector<EC_builtin_curve> curves(count);
   EC_get_builtin_curves(curves.data(), count);
 
   // Find a curve that has an even order.
   for (auto curve : curves)
   {
       auto eckey = EC_KEY_new_by_curve_name(curve.nid);
       auto group = EC_KEY_get0_group(eckey);
       auto ctx = BN_CTX_new();
       auto order = BN_new();
       EC_GROUP_get_order(group, order, ctx);
       if (!BN_is_odd(order))
           return curve;
   }
 
   return {};
}

The rest of the proof of concept is straightforward. A key is generated on the even-order curve and assigned as the certificate's public key. The certificate is then signed using a key on a different curve — the even-order curve would trip the bug during signing, so a second curve avoids that interference. Finally, the certificate is verified with X509_verify to see whether the vulnerable path fires.

Compiling the test case is just two commands:

export PKG_CONFIG_PATH=$HOME/Installations/openssl-debug/lib/pkgconfig
clang++ test.cc -o test -fsanitize=address $(pkg-config --libs --cflags openssl) -g -std=c++17

Then run it under a debugger:

$ lldb test
(lldb) target create "test"
Current executable set to '/Users/goose/Work/Research/projects/openssl/bugs/bug2/test' (x86_64).
(lldb) r
Process 66228 launched: '/Users/goose/Work/Research/projects/openssl/bugs/bug2/test' (x86_64)
Process 66228 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x60c40000fffc)
   frame #0: 0x0000000100167ad1 test`BN_STACK_pop(st=0x0000607000009f10) at bn_ctx.c:300:12
  297
  298  static unsigned int BN_STACK_pop(BN_STACK *st)
  299  {
-> 300      return st->indexes[--(st->depth)];
  301  }
  302
  303  /***********/
Target 0: (test) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x60c40000fffc)
 * frame #0: 0x0000000100167ad1 test`BN_STACK_pop(st=0x0000607000009f10) at bn_ctx.c:300:12
   frame #1: 0x0000000100167468 test`BN_CTX_end(ctx=0x0000607000009ef0) at bn_ctx.c:216:27
   frame #2: 0x0000000100439b99 test`ossl_ecdsa_verify_sig(dgst="\x060��!����:�#\x13�\�E\x94, dgst_len=32, sig=0x0000602000016ed0, eckey=0x0000608000003220) at ecdsa_ossl.c:421:5
   frame #3: 0x000000010043a945 test`ECDSA_do_verify(dgst="\x060��!����:�#\x13�\�E\x94, dgst_len=32, sig=0x0000602000016ed0, eckey=0x0000608000003220) at ecdsa_vrf.c:24:16
   frame #4: 0x0000000100438b3a test`ossl_ecdsa_verify(type=672, dgst="\x060��!����:�#\x13�\�E\x94, dgst_len=32, sigbuf="0\"\x02\x0f", sig_len=36, eckey=0x0000608000003220) at ecdsa_ossl.c:310:11
   frame #5: 0x000000010043ac79 test`ECDSA_verify(type=672, dgst="\x060��!����:�#\x13�\�E\x94, dgst_len=32, sigbuf="0\"\x02\x0f", sig_len=36, eckey=0x0000608000003220) at ecdsa_vrf.c:39:16
   frame #6: 0x000000010043084f test`pkey_ec_verify(ctx=0x00006080000034a0, sig="0\"\x02\x0f", siglen=36, tbs="\x060��!����:�#\x13�\�E\x94, tbslen=32) at ec_pmeth.c:146:11
   frame #7: 0x00000001005f2f35 test`EVP_PKEY_verify(ctx=0x00006080000034a0, sig="0\"\x02\x0f", siglen=36, tbs="\x060��!����:�#\x13�\�E\x94, tbslen=32) at pmeth_fn.c:82:12
   frame #8: 0x00000001005d8713 test`EVP_DigestVerifyFinal(ctx=0x0000607000009e10, sig="0\"\x02\x0f", siglen=36) at m_sigver.c:207:12
   frame #9: 0x00000001005d8caa test`EVP_DigestVerify(ctx=0x0000607000009e10, sigret="0\"\x02\x0f", siglen=36, tbs="0R\x02\x01", tbslen=84) at m_sigver.c:217:12
   frame #10: 0x000000010007ed40 test`ASN1_item_verify(it=0x0000000100cc89e0, a=0x0000613000002588, signature=0x0000613000002598, asn=0x0000613000002500, pkey=0x0000611000002c00) at a_verify.c:166:11
   frame #11: 0x0000000100a00bbf test`X509_verify(a=0x0000613000002500, r=0x0000611000002c00) at x_all.c:170:13
   frame #12: 0x00000001000053b0 test`main(argc=1, argv=0x00007ffeefbff898) at test.cc:80:5
   frame #13: 0x00007fff20338621 libdyld.dylib`start + 1
   frame #14: 0x00007fff20338621 libdyld.dylib`start + 1
(lldb)

The proof of concept ends in a crash on an unmapped memory address. That alone is not code execution, but the amount of program state an attacker can influence before the crash suggests exploitation is plausible with more work.

Triggering the bug through X509_verify models a client checking a server's certificate. Since the multiplicative inverse operation shows up in several places, other entry points may exist — possibly ones that turn this into a server-side remote bug. That hunt is left as an exercise for the reader.

This bug came out of a broader look at cryptography-adjacent code. Most of the earlier results in that effort were not flaws in OpenSSL itself but API misuse by the library's consumers:

Thanks to Pavel Avgustinov for help with the CodeQL query and to Matt Caswell from OpenSSL for shepherding the fix into the codebase.