The mismatch between cloud and internal identity
Long-lived organizations tend to run two identity systems in parallel: the cloud provider's IAM roles, instance profiles and execution roles, and an internal system that services actually consult before answering a request. On self-managed infrastructure you control how your own identity is bootstrapped. On managed compute you do not — the provider hands the process a cloud identity and nothing more.
Closing that gap is the subject here, using Apache Spark on Amazon EMR as the worked example: a workload that begins with only an AWS identity must finish holding a first-class internal identity. Performing the exchange is easy; making it trustworthy is where the design effort went. Almost none of it is specific to Spark or EMR.
The internal side: Metatron and Data Projects
Internal service-to-service authentication runs on a private PKI called Metatron. Each workload receives a short-lived X.509 certificate, and mutual TLS is how services authenticate one another. Certificates are not handed out on request: the identity service issues one only after satisfying itself that the requester genuinely is the workload it claims to be — a verification step called attestation. Attestation in each supported environment (VMs, containers, functions) rests on an environment-specific proof the platform can check independently.
Sitting alongside that is the Data Project, the unit of data ownership. It owns tables, has authorized users, and carries its own identity. Jobs execute as the Data Project rather than as the human who launched them, which keeps table-level access control and audit consistent whether the workload is scheduled or interactive.
The consequence: a Spark job on managed compute starts with an AWS execution role and no internal identity, yet internal operations — reading an encrypted column, evaluating table ACLs, holding an interactive session past the life of a short-lived token — all require one.
The mapping that makes translation possible
The design depends on a decision taken entirely outside the Spark stack. Every Data Project identity maps 1:1 onto a dedicated IAM role, and the service that owns Data Project metadata keeps that mapping authoritative.
That mapping is what allows a provider-vocabulary statement — this process runs as role R — to become one in our vocabulary: this process is workload W. Cryptography alone gets you nowhere without something to translate into. The difficult half of bridging two identity systems is generally not the protocol but committing to a mapping and keeping it authoritative.
One immediate objection: a single AWS account cannot hold tens of thousands of IAM roles, and data projects are expected on the order of ten thousand. The answer was to shard data project roles deterministically across a small pool of dedicated accounts. That scales comfortably past the projected count and has a welcome side effect — workload roles end up on the far side of an account boundary from the control planes that launch them.
Who participates
Five components are involved.
- Control plane — the only service permitted to launch Spark jobs. It resolves the Data Project's IAM role, builds and signs a workload metadata payload, and submits the job with that role as the execution role. Its signing key is issued for this purpose alone.
- Data Project service — authority for the identity-to-role mapping.
- Identity service — receives attestation requests, verifies them, issues certificates.
- Spark plugin — supplies the hook. Spark 3.0+ ships a plugin interface with driver- and executor-side components that initialize during process bootstrap.
- AWS STS — acts as a notary, which is not its normal role.
Claim, proof of possession, corroboration
The control plane's signed claim
On submission, the control plane validates the caller, looks up the Data Project's IAM role, and assembles a small payload describing the workload about to launch: the application identity, the mapped role, and the environment, stack and detail the credentials should be scoped to. It signs the payload and passes payload and signature through as ordinary job configuration.
Two things matter here. Only one service can generate a valid signature, so the set of parties able to assert that a workload is legitimate stays small and auditable. And the payload is a claim, not an authority. By itself it proves nothing: job configuration travels through infrastructure we do not fully control, so anything able to read it can replay it.
Proving the cloud identity
The managed service launches the driver under an OS user with the execution role's credentials available. The driver-side plugin initialises, then uses those credentials to sign — without sending — a request to the provider's identity endpoint (sts:GetCallerIdentity). What comes back is a short-lived pre-signed URL.
That URL is transferable proof of possession. Anyone can fetch it, but only a holder of the role's credentials could have produced it, and the response originates with AWS rather than with the workload, stating which role signed. The workload does not supply the answer, so it cannot lie about it.
The pattern is established rather than novel — the same idea underlies AWS IAM authentication in HashiCorp Vault and AWS's own function attestation flows. It generalises: any environment that gives a process cloud credentials and nothing else can still yield a verifiable statement about its identity.
Both artefacts travel to the identity service in a single attestation request.
Corroborating two statements
The identity service then does four things.
- Fetch the pre-signed URL against AWS STS and read back the signing role — the provider's statement.
- Verify the metadata signature against the control plane's key — the platform's statement.
- Corroborate: the role AWS reports must be the role the control plane said it dispatched, for the identity the metadata names.
- Issue certificates for that identity, scoped to the environment, stack and detail in the signed metadata.
Step 3 justifies the whole design, because the two statements are individually insufficient and fail in complementary directions. The provider's statement cannot be forged but is under-specified — it yields a role, and a role is not a workload; it says nothing about why the process exists or what it should be allowed to be. The platform's statement is well-specified but unverifiable alone, for the reasons above. Together, the signature establishes legitimate dispatch and the pre-signed URL establishes that this really is that workload. Neither party has to accept the workload's own account of itself.
Expect the tension we resolved deliberately: the provider's answer names a role, not an application. Deriving an application name from a role name is fragile string matching, and platform-assigned session identifiers are outside our control. We take application identity from the signed metadata and use the provider's answer for corroboration only — at the cost of a network round trip and a stricter signing requirement, in exchange for a much clearer trust story.
Driver, executors, and identity amplification
A Spark application is one driver plus up to thousands of executors, created and destroyed across the job's life. Attestation built around one process per host does not survive that fan-out. Two answers are defensible.
Independent attestation by every executor: uniform, no new trust boundary, and every process's identity grounded in the same provider-verified proof. The cost is amplification — one large job can emit thousands of STS calls and thousands of attestation requests in a burst, which resembles an attack and creates a hard dependency on provider-side rate limits.
Inheritance from the driver: the driver attests once and distributes credentials to executors over Spark's internal RPC, configured with authentication and AES-GCM encryption so only processes in the same application can participate. Identity service load stays constant. The cost is a second trust boundary, plus a driver that is now a credential distribution point.
At our scale we took the second. The broader lesson: identity amplification is a capacity question, and it is better answered deliberately than during an incident.
Renewal, not just bootstrap
Since certificates are short-lived by design, bootstrapping is half the job. Self-managed hosts run a system service that handles renewal; managed compute offers no equivalent hook, so the driver JVM runs a timer that re-attests on a fixed interval, well ahead of expiry. Credential material goes to a temporary directory readable only by the process owner and is deleted when the process exits.
Executors do not refresh. They are normally short-lived, and an executor that outlives its credentials can exit and be replaced more cheaply than every executor can serve as a renewal client.
The transferable rule: attestation must be repeatable, not a bootstrap step. Anything that can happen only once at process start will eventually be why a long-running job dies nine hours in.
What carries over to other systems
When your own identity system is meeting managed compute, the specific services matter less than the shape of the solution.
- Anchor on one exchangeable primitive. A single authoritative 1:1 mapping between your identity and a provider identity is what makes translation possible; the rest is plumbing.
- Require two independent claims and corroborate them. One unforgeable but under-specified from the provider; one well-specified but replayable from your control plane. Trust only the intersection.
- Keep the signer scarce. With exactly one service able to make the platform's claim, the trust story fits in a sentence.
- Hook the layer you still own. Host and init system are usually out of reach on managed compute, but the runtime is not: a plugin, an agent, an entrypoint.
- Decide amplification policy deliberately. Distributed engines multiply per-process operations by their parallelism.
- Make attestation repeatable and credentials short-lived. Renewal is a requirement, not a follow-up.
The result is trustworthy because no participant issues an identity alone — not the workload, not the control plane, not the provider. And the workload, least able to be trusted, is never asked to vouch for itself.
