How the Cinder JIT’s function inliner helps us optimize Instagram

Why Python needs a JIT

Instagram runs one of the largest Django deployments anywhere, so squeezing performance out of Python directly benefits our production fleet. We recently open-sourced Cinder, a fork of CPython that layers on optimizations including immortal objects, shadowcode (our name for inline caching and quickening), Static Python, and Strict Modules. This post drills into the just-in-time (JIT) compiler and its newest capability: the function inliner.

Even with Static Python and shadowcode active, the bytecode interpreter still carries fixed costs: the dispatch switch, the stack-based push/pop of operands, and the heavy generality of CPython 3.8's interpreter. The JIT removes most of that by emitting native code. It compiles one function at a time through a pipeline: bytecode → control-flow graph (CFG) → high-level IR (HIR) → SSA-form HIR → low-level IR (LIR) → register-allocated LIR → assembly. The switch to a register-based IR kills the stack overhead, native code kills dispatch overhead, and type inference plus other passes specialize the generic bytecode.

Consider a trivial pair of functions:

def callee(x):
	return x + 1	
def caller():
	return callee(3)

callee is fully generic—it knows nothing about the type of x, so the addition cannot be specialized. In caller, the lookup of the global name callee must happen on every invocation because Python permits rebinding of globals. The dis module shows what the interpreter actually walks through:

callee:
          	0 LOAD_FAST            	0 (x)
          	2 LOAD_CONST           	1 (1)
          	4 BINARY_ADD
          	6 RETURN_VALUE

caller:
          	0 LOAD_GLOBAL          	0 (callee)
          	2 LOAD_CONST           	1 (3)
          	4 CALL_FUNCTION        	1
          	6 RETURN_VALUE 

Each instruction line has four columns: bytecode offset, human-readable opcode, byte-sized argument, and a context-aware interpretation of that argument (normally pulled from an auxiliary structure in PyCodeObject, not the bytecode stream itself).

Executing that bytecode through the interpreter triggers a costly call path: argument count and default checking, __call__ resolution, heap allocation of a call frame, and more. The JIT, by contrast, can eliminate many of those checks at compile time. It can bake in constant types, avoid dictionary lookups for stable globals, and use shadow frames—two stack-allocated words of metadata per function that let us reify a PyFrameObject later if we need to deoptimize.

From bytecode to typed IR

Our JIT first dissects bytecode into basic blocks; since jumps, returns, and raises terminate blocks, both functions above consist of a single block. Then an abstract interpreter walks the stack-based bytecode and emits our infinite-register HIR:

# Initial HIR
fun __main__:callee {
  bb 0 {
	v0 = LoadArg<0; "x">
	v0 = CheckVar<"x"> v0
	v1 = LoadConst<MortalLongExact[1]>
	v2 = BinaryOp<Add> v0 v1
	Return v2
  }
}

fun __main__:caller {
  bb 0 {
	v0 = LoadGlobalCached<0; "callee">
	v0 = GuardIs<0xdeadbeef> v0
	v1 = LoadConst<MortalLongExact[3]>
	v2 = VectorCall<1> v0 v1
	Return v2
  }
}

This IR already carries information the bytecode lacked, with object pointers shown as 0xdeadbeef for illustration. LoadConst is parameterized by the exact class of the constant, and we automatically insert a GuardIs after LoadGlobalCached. Because we assume globals rarely change, the guard does a fast pointer comparison and deoptimizes on failure rather than re-walking the module dictionary every time.

Before optimization passes can run, the IR must become SSA. The conversion pass also performs basic flow typing:

# SSA HIR
fun __main__:callee {
  bb 0 {
	v3:Object = LoadArg<0; "x">
	v4:Object = CheckVar<"x"> v3
	v5:MortalLongExact[1] = LoadConst<MortalLongExact[1]>
	v6:Object = BinaryOp<Add> v4 v5
	Return v6
  }
}

fun __main__:caller {
  bb 0 {
	v3:OptObject = LoadGlobalCached<0; "callee">
	v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
	v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
	v6:Object = VectorCall<1> v4 v5
	Return v6
  }
}

Type annotations now appear on definitions: constants carry their own type, while globals carry the type they had at compile time. With the global-stability assumption, we can even infer function targets after the guard, baking the resolved address into the generated code—see MortalFunc[function:0xdeadbeef] above.

Our current optimizers eliminate CheckVar (CPython guarantees non-null arguments) but otherwise leave these functions alone. The generic BinaryOp<Add> and the generic VectorCall<1> survive because we lack type information: this is a method-at-a-time JIT, so type specialization only happens within one function's body, and functions are compiled before they ever run.

What inlining buys us

If we splice callee into caller, the argument type flows directly into the addition. That alone justifies the pass: it removes call overhead and enables specialization. It also eases inline-cache pressure (monomorphizing caches within the callee) and reduces register spills caused by native calling conventions.

Manually inlined, the code would look like this:

# Hypothetical inlined HIR
fun __main__:caller {
  bb 0 {
	v3:OptObject = LoadGlobalCached<0; "callee">
	v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
	v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
	# Inlined "callee"
	v13:MortalLongExact[1] = LoadConst<MortalLongExact[1]>
	v16:Object = BinaryOp<Add> v5 v13
	# End inlined "callee"
	Return v16
  }
}

Now the optimizer has enough type information to replace BinaryOp with LongBinaryOp, which calls int.__add__ directly:

# Hypothetical inlined+optimized HIR
fun __main__:caller {
  bb 0 {
	v3:OptObject = LoadGlobalCached<0; "callee">
	v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
	v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
	# Inlined "callee"
	v13:MortalLongExact[1] = LoadConst<MortalLongExact[1]>
	v16:LongExact = LongBinaryOp<Add> v5 v13
	# End inlined "callee"
	Return v16
  }
}

The memory effects of the operation become precisely known—we can see exactly which built-in runs. Sometimes constant folding goes further and collapses everything:

# Hypothetical inlined+optimized HIR II
fun __main__:caller {
  bb 0 {
	v3:OptObject = LoadGlobalCached<0; "callee">
	v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
	# Inlined "callee"
	v17:MortalLongExact[4] = LoadConst<MortalLongExact[4]>
	# End inlined "callee"
	Return v17
  }
}

One inlining pass cascades into a constant. The LoadGlobalCached and GuardIs remain only to handle a hypothetical reassignment of callee; their runtime cost is minimal.

Implementing the inliner pass

The pass receives the optimized HIR of caller, which looks roughly like this:

# Original HIR, pre-inlining
fun __main__:caller {
  bb 0 {
    v3:OptObject = LoadGlobalCached<0; "callee">
    v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
    v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
    v6:Object = VectorCall<1> v4 v5
    Return v6
  }
}

It scans all VectorCall instructions and collects those whose target function is statically known—in this example, v4 resolves to a specific function. Call sites are collected before any CFG mutation.

For each candidate call, we follow a sequence of steps, bailing out where the callee cannot be inlined (for instance, when argument counts don't line up with parameters):

1. Build the callee's HIR inside the caller's CFG without merging graphs yet. The caller is already in SSA; we separately SSA-ify the callee's graph, and rewrite all its Return instructions into a single unified return point so entry and exit are unique.

fun __main__:caller {
  bb 0 {
    v3:OptObject = LoadGlobalCached<0; "callee">
    v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
    v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
    v6:Object = VectorCall<1> v4 v5
    Return v6
  }

  # Non-linked callee
  bb 1 {
    v7 = LoadArg<0; "x">
    v8 = CheckVar<"x"> v7
    v9 = LoadConst<MortalLongExact[1]>
    v10 = BinaryOp<Add> v8 v9
    Return v10
  }
}

2. Split the caller's basic block after the call instruction.

fun __main__:caller {
  bb 0 {
	v3:OptObject = LoadGlobalCached<0; "callee">
	v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
	v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
	v6:Object = VectorCall<1> v4 v5
  }

  # Non-linked callee
  bb 1 {
	v7 = LoadArg<0; "x">
	v8 = CheckVar<"x"> v7
	v9 = LoadConst<MortalLongExact[1]>
	v10 = BinaryOp<Add> v8 v9
	Return v10
  }

  bb 2 {
	Return v6
  }
}

3. Insert bookkeeping and branch instructions. Shadow frames are maintained via BeginInlinedFunction and EndInlinedFunction; the original call instruction is removed.

fun __main__:caller {
  bb 0 {
	v3:OptObject = LoadGlobalCached<0; "callee">
	v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
	v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
	BeginInlinedFunction
	Branch<1>
  }

  # Linked callee
  bb 1 (preds 0) {
	v7 = LoadArg<0; "x">
	v8 = CheckVar<"x"> v7
	v9 = LoadConst<MortalLongExact[1]>
	v10 = BinaryOp<Add> v8 v9
	Return v10
  }

  bb 2 {
	EndInlinedFunction
	Return v6
  }
}

4. Rewrite LoadArg to Assign, since the call boundary no longer exists; argument registers map directly to parameters.

fun __main__:caller {
  bb 0 {
	v3:OptObject = LoadGlobalCached<0; "callee">
	v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
	v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
	BeginInlinedFunction
	Branch<1>
  }

  # Linked callee with rewritten LoadArg
  bb 1 (preds 0) {
	v7 = Assign v5
	v8 = CheckVar<"x"> v7
	v9 = LoadConst<MortalLongExact[1]>
	v10 = BinaryOp<Add> v8 v9
	Return v10
  }

  bb 2 {
	EndInlinedFunction
	Return v6
  }
}

5. Rewrite the inlined Return into an Assign to the original call's output register, reusing that register because there's only one exit.

fun __main__:caller {
  bb 0 {
	v3:OptObject = LoadGlobalCached<0; "callee">
	v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
	v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
	BeginInlinedFunction
	Branch<1>
  }

  # Linked callee with rewritten Return
  bb 1 (preds 0) {
	v7 = Assign v5
	v8 = CheckVar<"x"> v7
	v9 = LoadConst<MortalLongExact[1]>
	v10 = BinaryOp<Add> v8 v9
	v6 = Assign v10
	Branch<2>
  }

  bb 2 (preds 1) {
	EndInlinedFunction
	Return v6
  }
}

6. Run CleanCFG to prune the now-redundant branches in straight-line code.

fun __main__:caller {
  bb 0 {
	v3:OptObject = LoadGlobalCached<0; "callee">
	v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
	v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
	BeginInlinedFunction
	v7 = Assign v5
	v8 = CheckVar<"x"> v7
	v9 = LoadConst<MortalLongExact[1]>
	v10 = BinaryOp<Add> v8 v9
	v6 = Assign v10
	EndInlinedFunction
	Return v6
  }
} 

7. Re-run type inference to reflow types over the newly added untyped code.

fun __main__:caller {
  bb 0 {
	v3:OptObject = LoadGlobalCached<0; "callee">
	v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
	v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
	BeginInlinedFunction
	v7:MortalLongExact[3] = Assign v5
	v8:MortalLongExact[3] = CheckVar<"x"> v7
	v9:MortalLongExact[1] = LoadConst<MortalLongExact[1]>
	v10:Object = BinaryOp<Add> v8 v9
	v6:Object = Assign v10
	EndInlinedFunction
	Return v6
  }
}

8. Re-run optimization passes—CopyPropagation removes useless Assigns, Simplify removes the now-unnecessary CheckVar.

fun __main__:caller {
  bb 0 {
	v3:OptObject = LoadGlobalCached<0; "callee">
	v4:MortalFunc[function:0xdeadbeef] = GuardIs<0xdeadbeef> v3
	v5:MortalLongExact[3] = LoadConst<MortalLongExact[3]>
	BeginInlinedFunction
	v9:MortalLongExact[1] = LoadConst<MortalLongExact[1]>
	v10:LongExact = LongBinaryOp<Add> v5 v9
	EndInlinedFunction
	Return v10
  }
}

The result is callee compiled in the context of caller. But the inliner doesn't force that coupling: callee can still be compiled standalone for other call sites that fail the inline checks.

Engineering around the shadow stack

Several parts of Python’s runtime and tooling assume they can always view an accurate, non-inlined view of the call stack. The Cinder JIT must preserve that illusion even when it has replaced calls with direct jumps into the callee’s machine code.

Profiling and deoptimization

The sampling stack profiler cannot execute any code; it walks pointer chains to discover active functions. Shadow frames are what let it see through inlined calls. Likewise, when a function raises an exception or transfers control back to the interpreter, the runtime has to reconstruct a PyFrameObject containing every variable, line number, and other state that existed at that point — some of which the JIT may have optimized away entirely. That reconstruction relies on mapping machine-code positions back to the original Python semantics.

Coroutines and external frame requests

Inlining ordinary functions into coroutines is conceptually fine, because both run from top to completion. The complication is that coroutines must yield control and materialize a coroutine or generator object when called, and their frame layout differs slightly. The current frame implementation doesn’t yet support multiple shadow frames, so this work is deferred.

The deeper issue is that frame materialization isn’t limited to deoptimization events. Python programmers can call sys._getframe (an implementation detail, not a guarantee), and C extension or standard library developers can use PyEval_GetFrame. Even with no deoptimization in the middle of an inlined function, managed or native code may demand a real frame at any time. JIT-ed code must therefore handle the case where its shadow frames have been replaced by genuine Python frames.

Deciding when (not) to inline

Inlining every callee would bloat code for functions that are rarely or never called. Even for popular callees, the cost of enlarging the caller may outweigh the savings from removing a call. Runtimes typically rely on heuristics tuned over long periods for specific workloads. Cinder is still at the stage of determining what rules work best for its use cases.

Handling dynamic changes

Python’s dynamism creates edge cases, but some are less troublesome than they appear. If a function’s __code__ object is reassigned, Cinder detects it from Python code and invalidates the affected JIT-ed code. For inlined functions, the options are to check for changes before each execution (slow) or patch the generated code in response — neither is implemented yet, but both are straightforward. Native extensions that modify PyFunctionObject fields are expected to notify the runtime; Cinder trusts them to behave well.

What if the callee itself changes, e.g., a global variable points to a different function? The inliner doesn’t need to handle that directly. It begins with the assumption that the callee is known, and if the value later changes, a guard instruction maintains correctness for the native code and deoptimizes to the interpreter when the invariant breaks.

Next steps in Cinder’s development

With the inliner in place, the focus shifts to collecting performance data on real workloads and deriving heuristics for which functions benefit most from inlining. The team also plans to evaluate existing research on the topic. Those interested are invited to experiment with the GitHub repository, which includes a Dockerfile and a prebuilt Docker image.