Genie 2.0: Rethinking Netflix's Job Execution Engine

A year after releasing Genie, Netflix's distributed job and resource management tool, the team has shipped a major overhaul. Genie 2.0 is a rewrite aimed at moving beyond the narrow Hadoop 1-focused design of its predecessor to support the broader execution ecosystem the company now relies on.

The Limits of Genie 1.0

Genie 1.0 achieved its initial goals: abstracting execution environments from Hadoop, Hive, and Pig submissions, and enabling horizontal scaling of client resources. But its rigid architecture made it hard to adapt as Netflix adopted Hadoop 2, Parquet, and Presto, and built tooling like Inviso.

Key limitations included:

  • Hadoop 1-only support
  • A fixed data model with limited entity fields — cluster configs, for example, had specific fields for s3CoreSiteXml and s3HdfsSiteXml rather than a generic collection
  • Rigid cluster selection criteria limited to SLA, ad hoc, or bonus schedule types

The decision to redesign rather than patch was driven by the need to support a growing number of use cases and an accelerating pace of technology adoption.

Design Goals

The team set four objectives for Genie 2.0:

  • Build a generic data model broad enough for any multi-tenant distributed processing cluster
  • Create a flexible cluster and command selection algorithm
  • Expose richer REST APIs
  • Refactor the codebase for flexibility and extensibility

The New Data Model

The model now uses four core entities, each supporting metadata tags for selection purposes:

Cluster — Stores connection info and properties for execution clusters, such as Hadoop 2, Spark, or Presto. A cluster can be linked to multiple commands.

Command — Encapsulates the configuration for an executable invoked by Genie to submit jobs, including the path, environment variables, and config files. Examples include Hive, Pig, Presto, and Sqoop. If an executable isn't already on the Genie node, a command can link to an application for runtime installation.

Application — Supplies the components required to install a command's executable at runtime: jars and binaries, extra configuration files, and an environment setup file. Netflix's Presto client is configured this way.

Job — Captures all details of a job request and execution, including command-line arguments derived from the selected cluster and command. Jobs can pull files from attachments or accessible file systems via file dependencies.

Execution Environment Selection

Job launches now rely on tag-based resolution rather than hardcoded schedules. A job request provides:

  • Command Tags: a set of tags mapping to one or more commands
  • Cluster Tags: a priority-ordered list of tag sets, each mapping to one or more clusters

Genie walks the cluster tag list in order, testing each set with the command tags to find a viable environment. This ordered fallback is a core feature. Netflix's nightly ETL jobs, for instance, first target the bonus clusters spun up during low-traffic hours; if those are down, the next tag set routes the job to the production cluster instead.

Richer APIs and Codebase Improvements

Genie 1.0's REST API required full-object updates for any resource change. Genie 2.0 supports fine-grained APIs that can manipulate entity collections directly.

The codebase rewrite leveraged Spring for dependency injection, JPA support, and transactions; Jersey and Swagger for standardized REST APIs with generated documentation and multi-language clients; and JUnit and Mockito for a significantly expanded test suite of nearly 600 tests. Swagger codegen also produced the core of a Python client, published to Pypi.

The three-tier decoupling and removal of boilerplate make the application easier to deploy both inside and outside Netflix.

Netflix Deployment Experience

Genie 2.0 has been running in production for several months, with all Genie 1.0 jobs migrated. It fronts all Hadoop and Presto clusters across production, test, and ad hoc environments, autoscaling between 12 and 20 i2.2xlarge AWS instances to handle hundreds of concurrent jobs.

Presto and Sqoop commands are tied to applications pointing to S3 locations containing the necessary binaries. Files are downloaded and installed at runtime for every job invocation, letting the team update clients without redeploying Genie. Plans call for moving Pig and Hive commands to the same pattern.

Cluster lifecycle is managed through Genie's status fields. New clusters are registered as "available" once configured, marked "out of service" before shutdown to stop new job acceptance, and "terminated" once drained and shut down in AWS. Notably, adding Presto support took days and Sqoop under an hour in Genie 2.0 — both would have required code changes and a redeploy in the prior version.

What's Next

The team has identified several areas for future work:

  • Improving job execution and monitoring for better fault tolerance, host efficiency, and finer-grained status feedback
  • Abstracting Netflix OSS component usage to ease adoption by external users
  • Enhancing the admin UI, such as showing all clusters a command is registered with

With Genie 2.0 in place, Netflix has a configuration-driven execution layer positioned to absorb the next wave of big data tooling without a rewrite.