When a Jenkins Job Takes Down the Jenkins UI

Slack runs a large Jenkins fleet to build and test mobile apps before release, with hundreds of jobs across many environments. One day, the UI stopped responding even though jobs kept executing. The cause turned out to be a subtle interaction between a custom integrity check, a Jenkins security fix, and the Groovy sandbox.

The Symptoms and the Suspicious Upgrade

Slack continuously runs integrity jobs that inspect Jenkins nodes for configuration failures. When a check fails, the job marks the node offline and alerts the Mobile Build & Release team. After a routine upgrade of Jenkins and all its plugins, an integrity job failed on one node — and the UI broke shortly after. A reboot of the main instance restored the UI, but the investigation had just begun.

Reproducing in Staging

The UI error message gave little detail, so the team turned to a staging Jenkins environment that mirrored production's Jenkins version and plugins. By deliberately manipulating a node and rerunning the integrity job, they reproduced the UI failure and captured a telling error in the job logs:

expected to call
hudson.slaves.SlaveComputer.setTemporarilyOffline
but wound up catching OfflineMessage.toString; see:
https://jenkins.io/redirect/pipeline-cps-method-mismatches/

The relevant code used a common pattern: calling the Jenkins Java API with an instance of an OfflineMessage class to supply the reason a node went offline:

def markNodeOffline() {
  def node = getCurrentNode(env.NODE_NAME)
  node.toComputer().setTemporarilyOffline(
    true, OfflineCause.create(new OfflineMessage())
  )
}

class OfflineMessage extends org.jvnet.localizer.Localizable {
  def message

  OfflineMessage() {
    super(null, null, [])
    def timestr = new Date().format(
      "HH:mm dd/MM/yy z", TimeZone.getDefault()
    )
    this.message = "The node was taken offline at ${timestr} due to corrupted host file"
  }

  String toString() {
    this.message
  }

  String toString(java.util.Locale l) {
    toString()
  }
}

Adding the typical @NonCPS annotation to the affected toString() method cleared the error from the job logs, but the UI still broke. Tailing Jenkins logs while reloading the UI surfaced the real problem:

2020-12-05 02:44:44.201+0000 [id=24] WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID a4bf874a-5029-4f37-b6e6-217ca8bb76de org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-2.235.5.jar!/hudson/model/Computer/index.jelly:63:66: Rejecting unsandboxed property get: OfflineMessage.message

Root Cause: A Security Fix Meets a Custom Class

Searching GitHub for the error message led to the source: the Groovy sandbox's RejectEverythingInterceptor was rejecting the interaction. Tracing the change back revealed that Jenkins maintainers had tightened sandbox restrictions in response to CVE-2020-2279. The fix shipped in the Script Security Plugin, which had been updated during the recent Jenkins upgrade. Downgrading that plugin in staging and rerunning the pipeline confirmed the theory — the UI survived.

The Fix: Swap the Custom Class for a Built-In One

The custom OfflineMessage class was the problem, so the goal became replacing it with a simpler, sandbox-safe approach. The Jenkins API offers hudson.slaves.OfflineCause.ByCLI, which takes a plain String for the offline cause instead of requiring a class that implements Localizable. The fix was a direct swap of the custom class for the built-in constructor:

def createMessage() {
  def timestr = new Date().format(
    "HH:mm dd/MM/yy z", TimeZone.getDefault()
  )
  "The node was taken offline at ${timestr} due to corrupted host file".toString()
}

def markNodeOffline() {
  def node = getCurrentNode(env.NODE_NAME)
  node.toComputer().setTemporarilyOffline(
    true, new OfflineCause.ByCLI(createMessage())
  )
}

With the change deployed, the integrity check job ran again without breaking the UI.

Operational Takeaways

The incident reinforced several practices for running Jenkins at scale:

  • Mirror production in a staging environment. A separate Jenkins instance with matching core and plugin versions lets you safely reproduce and validate hypotheses. It requires ongoing effort to keep versions aligned and to adapt jobs for non-production dry runs, but it paid off here by enabling a quick root-cause confirmation.
  • Keep Jenkins API integrations minimal. Jenkins exposes convenient APIs for pipeline customization, but binding custom Groovy code tightly to internal classes creates fragility. Leaner integrations reduce the risk of breakage from Jenkins or plugin releases.
  • Watch the Groovy sandbox and update notes. The sandbox that executes custom Groovy code is actively hardened by maintainers, often with tightened defaults that accompany security fixes. Reading changelogs for core plugins is essential, especially during routine maintenance upgrades.
  • Maintain a thorough runbook. Documented upgrade processes for the Jenkins core, plugins, and all pipelines that call the Jenkins API saved time during this investigation and shortens the path to recovery in future incidents.