Testing Tomorrow's Build Tools Today

Developer productivity at Slack depends on more than just build speed or reliability. It also hinges on keeping our toolchain current. We make a point of adopting the latest stable releases of core build tools as soon as we responsibly can. Newer versions bring improvements, but they also carry risk — especially for tools like the Android Gradle Plugin (AGP) and Kotlin, which have long release cycles. No maintainer's test suite can match the scrutiny of a real production codebase.

Our solution is what we call Shadow Jobs. These are continuous integration (CI) workflows — running on GitHub Actions in our case — that apply pre-release versions of our core build tools on top of our main branch and run the full build against them. By doing this nightly, we catch regressions early, give upstream maintainers useful feedback, and gain the confidence to upgrade quickly when stable versions ship.

Designing a Shadow Job

The pattern is straightforward: take your existing build, layer on a set of version overrides, and run it against a matrix of upcoming tool releases. Here's a simplified workflow that tests newer Java versions, AGP versions, and the Kotlin IR compiler backend:

name: (🔮) Build Tools

on:
  schedule:
    - cron: '0 8 * * *'

jobs:
  build:
    # Names are shortened for readability in CI
    name: 'J=${{ matrix.java }} | AGP=${{ matrix.agp }} | IR=${{ matrix.kotlinIR }}'
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        java: [1.8, 11, 15]
        agp: [4.1.2, 4.2.0-beta04, 7.0.0-alpha05]
        kotlinIR: [true, false]
        exclude:
          - agp: 7.0.0-alpha05
            java: 1.8
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Gradle Wrapper Validation
        uses: gradle/wrapper-validation-action@v1

      - name: Install JDK ${{ matrix.java }}
        uses: actions/setup-java@v1
        with:
          java-version: ${{ matrix.java }}

      - name: Build
        id: gradle
        uses: eskatos/gradle-command-action@v1
        with:
          arguments: :app:assembleExternalRelease :app:lintExternalRelease test --stacktrace -PagpVersion=${{ matrix.agp }} -PkotlinIREnabled=${{ matrix.kotlinIR }}

The workflow triggers on a nightly cron schedule. Running once per day at 8am UTC works well for a mostly North American team — frequent enough to stay current without wasting CI minutes.

on:
  schedule:
    - cron: '0 8 * * *'

We explicitly set fail-fast: false so every combination in the matrix runs to completion. The default behavior of canceling pending jobs on the first failure would hide useful results — we want all the data, not just the first error.

fail-fast: false

The matrix itself defines the versions to test:

matrix:
  java: [1.8, 11, 15]
  agp: [4.1.2, 4.2.0-beta04, 7.0.0-alpha05]
  kotlinIR: [true, false]
  exclude:
    - agp: 7.0.0-alpha05
      java: 1.8

Two details deserve explanation:

  • Android binaries currently require JDK 8, but we expect to move to a higher version soon. AGP 7.0 will mandate JDK 11, so we test against both JDK 8 and JDK 11 to keep our options open.
  • Running AGP 7 with Java 8 is not a valid combination, so we use the exclude: directive to remove it from the matrix. This keeps the job list free of doomed configurations.

The matrix values flow into the setup-java action, which configures the JDK for each job:

- name: Install JDK ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}

Passing the other matrix values into the Gradle build requires a slightly different approach. We expose them as Gradle properties using the -Pproperty=value command-line syntax:

- name: Build
  id: gradle
  uses: eskatos/gradle-command-action@v1
  with:
  arguments: :app:assembleExternalRelease :app:lintExternalRelease test --stacktrace -PagpVersion=${{ matrix.agp }} -PkotlinIREnabled=${{ matrix.kotlinIR }}

The build script then reads those properties and applies the corresponding AGP version and enables the Kotlin IR backend:

buildscript {
  dependencies {
    // AGP dependency. Must go before Kotlin's
    val agpVersion = findProperty("agpVersion")?.toString() ?: "4.1.2"
    classpath("com.android.tools.build:gradle:$agpVersion}")
    classpath(kotlin("gradle-plugin", version = "1.4.30"))
  }
}

apply(plugin = "com.android.application")
plugins {
  kotlin("android")
}

val irEnabled = properties.findProperty("kotlinIREnabled")?.toBoolean() == true

tasks.withType<KotlinCompile>().configureEach {
  kotlinOptions {
    useIR = irEnabled
  }
}

// Rest of the build file...

With this setup, every night produces a matrix of builds covering all valid version combinations. This gives us early visibility into how our codebase interacts with these new tools — and how the tools interact with each other.

Beyond the Basics

The same pattern extends well beyond Java and AGP. We've used or currently use shadow jobs for:

  • Kotlin milestone and RC releases, selected via a Gradle property
  • New Android SDK versions, also via Gradle property
  • Upcoming Gradle RCs and nightly builds, using the eskatos/gradle-command-action action which accepts different Gradle versions as arguments (see its documented configuration options)
  • Snapshots and RCs of regular library dependencies, including Sonatype snapshots, Kotlin EAP builds, and AndroidX snapshots
  • Internal experiments as well — an invasive change can get weeks of extra runtime before you commit to it. We used this approach to incubate an RxJava 3 migration script, which let us switch over with virtually no friction

Results and Recommendations

In roughly a year of running shadow jobs, they've surfaced dozens of issues before they could affect our stable builds. If your project depends heavily on Kotlin or Gradle, this is an especially valuable practice — JetBrains has explicitly asked developers to exercise the new JVM IR backend, and nightly shadow jobs are an ideal way to do exactly that.

The benefits work in both directions. Your team gets confidence in upcoming upgrades, and tool maintainers get early reports of regressions in real-world codebases. These pre-release builds also frequently surface latent problems in your own code — particularly after Kotlin updates — that you can fix on your own schedule rather than during an emergency upgrade.

Staying current on core tools also unblocks your engineers to experiment with dependent technologies like Android Studio Arctic Fox or Jetpack Compose, even if they only use them locally. The list below shows the range of issues we've filed with upstream projects as a direct result of these jobs:

  • Kotlin: KT-44420, KT-43538, KT-44045, KT-43242, KT-43241, KT-42915, KT-41493, KT-42257, KT-42752, KT-41484, KT-41494, KT-41702, KT-41732, KT-41486, KT-40351, KT-40485, KT-40091, KT-37944, KT-36478, KT-33052
  • Android: b/177562904, b/170026127, b/162155191, b/161920328, b/161727305, b/159516509, b/157723604, b/157681341, b/156657359, b/148284064, b/148284065, b/162528480, b/162446295, b/162259266, b/161381697, b/160815393, b/160714171, b/158597249, b/154315491, b/148929520
  • Gradle: #15674, #13333, #13302, #13811, #12997, #12452, #12198

The infrastructure cost is modest — a handful of nightly CI jobs. The payoff is early warning, smoother upgrades, and a team that gets to work with modern tools instead of waiting months for a sanctioned migration window.