Plugin workload identity

Give plugin workloads short-lived Core credentials by exchanging their Kubernetes ServiceAccount token at IAM.

Plugin workloads do not hold long-lived service user tokens. A token sidecar in the plugin Pod presents the Pod’s projected Kubernetes ServiceAccount token to IAM and receives a short-lived token for the installation’s platform principal. The platform principal is the authorized subject. It holds no role. What the plugin may do follows from the Core service grants in the Plugin Manager installation, and Core confines each effect to one organization.

This exchange has the contract level v1alpha.

plugin Pod                         IAM                               Core
  token sidecar
    projected token  ───────────>  verifies issuer, keys, audience,
    (audience of the binding,      expiry and subject, finds an
     600 seconds)                  active binding
                     <───────────  issues a PWT- token, at most
                                   15 minutes, stored only as a hash
    writes the token to a memory volume
  plugin container
    reads the token for each request ────────────────────────────>  authenticates as
                                                                     the platform principal

Platform principals #

A platform principal is an IAM service user with the scope platform. It differs from the service users of an organization in these ways:

• It belongs to no organization, so one principal serves every organization that the installation is enabled for.
• It holds no role, and IAM cannot give it one. IAM denies every organization, project, and Environment authorization check for it, organization administrators cannot list, change, or delete it, and removing an organization’s access leaves it active.
• IAM creates the plugin’s principal together with its workload binding. It has no static token and authenticates only with tokens that it receives through the binding.
• A second kind of platform principal has a static token. The Data Plane admission checker and the Control Plane’s Deployment admission identity use it. See The admission checker.

Its reach is the plugin boundary that Core gates per organization: the service grants of the manifest, reads within active operations, dispatch within the frozen scope of an operation, and event publication. A principal of one installation cannot act for another installation.

Role holders get no workload credential. IAM refuses to bind a service user that holds a role. If a bound service user gains a role, IAM stops issuing it workload tokens and its issued tokens stop authenticating at the next request.

Requirements #

• Kubernetes 1.30 or newer. The sidecar runs as a native sidecar container.
• An IAM image that contains the workload token exchange, as listed in the release notes.
• The plugin runtime chart with serviceAccount.create=true. A binding names exactly one ServiceAccount.

Configure IAM #

1. Look up the cluster’s ServiceAccount token issuer:

   kubectl get --raw /.well-known/openid-configuration | jq -r .issuer
   

2. Choose the IAM principals that may manage bindings, typically the operator or automation that installs plugins. Their UUIDs go into PLUGIN_WORKLOAD_BINDING_WRITERS. This list is separate from the operators of the Plugin Manager, which IAM configures with PLUGIN_MANAGER_WRITERS and PLUGIN_MANAGER_READERS; see Install and operate native plugins. Apart from IAM’s internal system principal, only these principals hold the platform-level permission plugin_workload_binding_write. Organization roles cannot grant it, and changing the list requires an IAM restart. The same permission guards the /platform-principals routes. IAM issues and regenerates static tokens only for a user token or IAM’s internal system identity, so automation that authenticates as a service user cannot create the admission checker.

3. Enable the profile in the Platform Orchestrator chart:

   iam:
     configurationProfiles:
       pluginWorkloadIdentity:
         enabled: true
         config:
           PLUGIN_WORKLOAD_TOKEN_ISSUER: https://kubernetes.default.svc.cluster.local
           PLUGIN_WORKLOAD_BINDING_WRITERS: 11111111-1111-1111-1111-111111111111
   

IAM then reads the cluster’s key set from https://kubernetes.default.svc/openid/v1/jwks with its own ServiceAccount token and the cluster CA. The chart binds IAM’s ServiceAccount to the built-in, read-only ClusterRole system:service-account-issuer-discovery and grants nothing else.

PLUGIN_WORKLOAD_TOKEN_TTL sets the lifetime of issued tokens, from 1m to 15m. The default is 15m. Do not set it to an empty value; an empty value stops IAM from starting.

Static key set #

If IAM cannot reach the Kubernetes API server, supply the key set as a file:

1. Export it with kubectl get --raw /openid/v1/jwks > jwks.json and store it as jwks.json in a Secret.
2. Enable iam.configurationProfiles.pluginWorkloadStaticJWKS with that Secret name, and set serviceAccountIssuerDiscovery: false in pluginWorkloadIdentity.
3. Roll IAM whenever the cluster rotates its signing keys. The file is a subPath mount and does not refresh in place.

Create the plugin’s platform principal #

The installation UUID that Core assigns when you install the package is part of the binding, so you create the plugin’s principal and its binding in one request after the installation exists. With octl, run octl plugin-manager create-identities, which also creates the admission checker and records both with the installation; see Create the identities. With the API, call this as a principal listed in PLUGIN_WORKLOAD_BINDING_WRITERS:

curl -X POST "${PO_API_URL}/plugin-workload-bindings" \
  -H "Authorization: Bearer ${PLATFORM_ORCHESTRATOR_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "issuer": "https://kubernetes.default.svc.cluster.local",
    "namespace": "stellwerk-plugins",
    "service_account": "example-plugin-native-plugin-runtime",
    "audience": "platform-orchestrator-plugin-workload",
    "platform_principal": {"display_name": "example-plugin"},
    "plugin_id": "dev.stellwerk.example",
    "installation_id": "44444444-4444-4444-4444-444444444444"
  }'

The answer names the new principal as service_user_id and reports "service_user_scope": "platform". Record that UUID as service_user_id in the Core Plugin Manager installation.

• issuer must equal PLUGIN_WORKLOAD_TOKEN_ISSUER.
• audience is the exact and only audience that the projected token must carry. Choose a value that no other system uses.
• Send exactly one of platform_principal and service_user_id. The second names an existing platform principal that has no binding, for example after a revocation.
• IAM refuses to bind an organization service user, a principal with a static token, or a service user that holds a role (HTTP 400).
• A ServiceAccount and a principal each have at most one binding. A second binding for either returns HTTP 409.
• installation_id is the UUID that Core assigned when you installed the package with POST /plugin-installations. Create the binding after that and before you record the identity.

Core records the principal of an installation with PUT /plugin-installations/{id}/identity only when IAM binds exactly that plugin ID, installation UUID, and principal, and the principal is a platform principal. Otherwise it answers HTTP 409 with the code plugin_workload_binding_missing or plugin_service_user_not_platform_principal and changes nothing.

List bindings with GET /plugin-workload-bindings and inspect one with GET /plugin-workload-bindings/{bindingId}.

Enable the token sidecar #

Set the same audience in the runtime chart, and point it at the API gateway:

serviceAccount:
  create: true
installationId: 44444444-4444-4444-4444-444444444444
workloadIdentity:
  enabled: true
  audience: platform-orchestrator-plugin-workload
  expirationSeconds: 600
  gateway:
    url: https://api.orchestrator.example.com

installationId is the installation UUID that Core assigned, and the chart requires it with workloadIdentity.enabled. The sidecar behaves as follows:

• Only the sidecar mounts the projected token. The plugin container gets the token directory read-only and never sees the Kubernetes token. expirationSeconds is fixed at 600, the Kubernetes minimum and the contract maximum.
• The sidecar exchanges the token before the plugin starts. Its startup probe holds back the plugin until the first exchange has succeeded. Its readiness probe keeps the Pod unready while no unexpired token exists.
• It writes the token with mode 0400 to a memory-backed volume and replaces it atomically. The plugin reads it from the file named by STELLWERK_CORE_TOKEN_FILE for every request. The file exists only while the token is valid.
• It refreshes at two thirds of the issued lifetime and reads the projected token again for every exchange.
• Temporary failures, such as HTTP 503, are retried with jittered exponential backoff from one second up to 30 seconds. A token that expires meanwhile is removed.
• A refusal, such as HTTP 400 invalid_grant, removes the token and stops the sidecar with an error, so the container restarts visibly. No token is ever logged.
• workloadIdentity.gateway.caSecret optionally replaces the system roots with a public CA bundle. Redirects are refused.

The admission checker #

The Data Plane holds one admission checker token for the plugin dispatches of every installation and organization. Make it a static-token platform principal. octl plugin-manager create-identities creates it. With the API, call this as a user with plugin_workload_binding_write:

curl -X POST "${PO_API_URL}/platform-principals" \
  -H "Authorization: Bearer ${PLATFORM_OPERATOR_USER_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"display_name": "Data Plane admission checker", "expiry_in_days": 365}'

The answer contains the token once, in token, and the principal’s id (1 to 3660 days for expiry_in_days). Store the token in the Secret behind EXTERNAL_DEPLOYMENT_CORE_BEARER_TOKEN_FILE of the Data Plane and record the id as admission_checker_id of every installation in the Core Plugin Manager. Core refuses any other identity as the checker with HTTP 409 and plugin_admission_checker_not_platform_principal.

The routes are GET and POST /platform-principals, GET and DELETE /platform-principals/{principalId}, and POST /platform-principals/{principalId}/actions/regenerate.

Ordinary Deployment admission #

The Data Plane can also ask Core to guard each ordinary Deployment against concurrent Pin changes before it commits it. It does so as one static-token platform principal for every organization, and the admission checker can be that principal. Enable the deploymentAdmission profiles of the chart on both planes:

control-plane:
  configurationProfiles:
    deploymentAdmission:
      enabled: true
      config:
        DEPLOYMENT_ADMISSION_IDENTITY: "<principal-id>"
data-plane:
  configurationProfiles:
    deploymentAdmission:
      enabled: true
      config:
        DEPLOYMENT_ADMISSION_CORE_GATEWAY_URL: https://api.orchestrator.example.com
      secretFiles:
        - name: deployment-admission-core-token
          secretName: plugin-admission-checker-token-v1
          key: core-bearer-token
          mountPath: /var/run/secrets/stellwerk/deployment-admission/core-bearer-token

DEPLOYMENT_ADMISSION_IDENTITY is the principal’s id, and the Data Plane presents its token. Core admits the Deployments of every organization as this one identity, including organizations that you create later, so a new organization deploys without a configuration change or a restart. Core refuses to start if the value is not the canonical UUID of a service identity. Core looks the caller up in IAM on every callback and refuses any other identity with HTTP 403 and deployment_admission_service_forbidden. If IAM cannot answer, the callback fails with HTTP 503 and authorization_unavailable. This is a change for installations that used a service user of an organization for it.

Exchange contract #

POST /auth/plugin-workload-token needs no prior authentication. It uses the field names of RFC 8693 token exchange:

A successful response carries access_token (an opaque PWT- token), issued_token_type urn:ietf:params:oauth:token-type:access_token, token_type Bearer, and expires_in in seconds, at most 900. It is sent with Cache-Control: no-store.

IAM checks the signature, the exact issuer, the exact and only audience, the expiry, the not-before time, and the subject system:serviceaccount:<namespace>:<name>. It then requires an active binding and an active platform principal that holds no role. An issued token never outlives the projected token.

Rotation, revocation, and rebinding #

The plugin’s workload tokens rotate on their own: the sidecar exchanges again at two thirds of the token’s lifetime, and the plugin principal has no long-lived credential to rotate. To rotate the admission checker, regenerate its token:

curl -X POST "${PO_API_URL}/platform-principals/${CHECKER_ID}/actions/regenerate" \
  -H "Authorization: Bearer ${PLATFORM_OPERATOR_USER_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"expiry_in_days": 365}'

The previous token stops authenticating, which can take up to a minute because of the authentication cache. Update the Secret and restart the Data Plane, which reads the file at start-up. The token expires like every service user token, so regenerate it in time. Regenerating a workload principal answers HTTP 409.

Revoking the binding is the kill switch for the whole plugin, across all of its organizations. It needs no change in Core, because the principal stays, and a rebinding restores the plugin. Workload tokens bypass the positive authentication cache, so revocation takes effect on the next request. After a refusal, the sidecar removes the token and the Pod becomes unready.

Limits #

• IAM validates tokens against the issuer’s key set, not against the Kubernetes API. It cannot detect a deleted Pod before the Pod’s exchanged token expires.
• IAM reloads the key set when it sees an unknown key ID or after 10 minutes, at most once every 30 seconds. A failed reload keeps the previous keys, so a key that the cluster removed stays trusted while IAM cannot reach the issuer.
• IAM tolerates up to five minutes of clock skew for the not-before time only.
• A workload principal registers no IAM capabilities, because it holds no role. Core registers the package’s capabilities for every enabled organization. Operators with plugin_registration_write keep their authority.
• The binding does not list operations. What the plugin may do follows from the service grants in the Plugin Manager installation, which Core checks per organization. A compromised plugin reaches every enabled organization within that boundary, and a leaked workload token does so until it expires, after at most 15 minutes.
• The Data Plane admission checker authenticates with the static token of a platform principal. A static token can be used from anywhere until it expires or you regenerate or delete it.
• IAM refuses new bindings of organization service users. A binding of that kind that was created earlier shows "service_user_scope": "organization" and stops serving while the service user holds a role. Replace it: revoke it, create a new binding with platform_principal, and record the new principal in the installation.

Transitional static tokens #

PLUGIN_WORKLOAD_STATIC_TOKEN_ADAPTER: v1alpha keeps static tokens of bound service users usable. A platform principal for a plugin workload never has a static token, so the adapter matters only for bindings of organization service users that were created earlier. It exists only to migrate an existing installation and is off by default. Leave it empty in production. Any other value stops IAM from starting.