Install and operate native plugins

Install, upgrade, disable, and uninstall a native plugin with the generic plugin runtime.

A native plugin consists of a signed OCI package and a Core installation. The generic plugin runtime chart verifies and runs the package in its own workload. The Plugin Manager in the Control Plane stores the installation in Core’s database and decides which organizations can use the plugin. You change it through an operator API, without a restart. Read Plugin host architecture first.

Before you begin #

You need:

• A Platform Orchestrator release that contains the plugin host, with every Core component at the versions listed in its release notes.
• Kubernetes 1.30 or newer for the plugin workload. The token sidecar of the workload identity runs as a native sidecar container, and the plugin and UI containers use the sleep preStop action (preStopDelaySeconds, default 5) so rolling upgrades do not drop requests. The runtime chart refuses to render the sidecar on older clusters.
• A network implementation that enforces Kubernetes NetworkPolicies. Verify allowed and denied traffic, including DNS and communication between nodes; the presence of a policy object alone does not establish isolation. See Network access for the enforcement limits.
• An OCI registry that you operate, reachable from the cluster. The runtime pulls only from this mirror, never from Stellwerk’s registry.
• From Stellwerk: the plugin package or offline bundle, its detached signature bundle, its release.json, the plugin’s release notes, and a trust bootstrap publishers.json. Obtain the trust bootstrap through a channel that is separate from the package download and verify it before use.
• A platform operator user that is listed in the IAM setting PLUGIN_MANAGER_WRITERS, as described in step 5. To create the plugin’s identities, the same user must also be listed in PLUGIN_WORKLOAD_BINDING_WRITERS; see Plugin workload identity.
• For a licensed plugin: an issuer trust document and a signed license bundle. See License and entitlement trust.
• A PostgreSQL database for the plugin, with separate owner, migration, and runtime principals. The plugin never uses a Core database.
• For a plugin with UI pages: a DNS name and TLS certificate on a hostname that differs from the Console. Prefer a separate registrable domain, because sibling subdomains of one domain are same-site.

mirror ──> trust ──> database ──> migration Job
  ──> Plugin Manager: package, installation ──> platform principals
  ──> runtime ──> activate ──> enable organizations ──> license

If a step fails, fix the cause and repeat that step. Record each completed step, including the package digest and the Helm revision, so that an interrupted installation can resume where it stopped.

Step 1: Mirror the package and images #

Copy the package by digest into your registry. For an offline bundle, unpack it and copy the OCI layout:

mkdir bundle
tar -xf example-plugin-1.0.0.tar -C bundle
jq . bundle/release.json
oras cp -r --from-oci-layout bundle/oci-layout:1.0.0 \
  registry.example.com/stellwerk/example-plugin:1.0.0
oras resolve registry.example.com/stellwerk/example-plugin:1.0.0

The digest that oras resolve prints must equal packageDigest in release.json. Note manifestSHA256 as well; the runtime chart and the Core package registration use it.

The signature, the SBOM, and the provenance attestation are OCI referrers of the package manifest. oras cp -r copies them with the package and keeps their digests. release.json lists the digests of the SBOM and provenance referrers under attestations; you set them in the runtime chart in step 4.

Mirror the runtime image and the ORAS and Cosign images that the runtime chart uses for fetching and verification. Reference all three by digest.

Optionally, verify the signature offline before the cluster does. Use the publisher key from the trust bootstrap:

cosign verify-blob --insecure-ignore-tlog \
  --key stellwerk-publisher.pub \
  --bundle bundle/oci-manifest.sigstore.json \
  bundle/oci-manifest.json

--insecure-ignore-tlog is correct here: Stellwerk signs private packages with a managed key and does not upload them to a public transparency log. The runtime repeats this check, and its own policy gate first confirms that the publisher is pinned with the managed-key-no-tlog policy. The gate also verifies the SBOM (SPDX 2 JSON) and the source provenance attestation, signed with the same publisher key, as one release set with the signature. Each attestation must name exactly the package digest as its subject.

Offline bundle for air-gapped installations #

If the cluster and your workstation have no route to Stellwerk, you receive one directory per plugin release instead of separate files:

bundle/
├── bundle.json           plugin, minimum Core and chart versions, every component's digest
├── bundle.sigstore.json  publisher signature over bundle.json
├── publishers.json       the public trust bootstrap
└── oci/                  one OCI image layout: package, runtime image, ORAS and
                          Cosign images, and the runtime chart

The bundle holds no licence and no sources. The runtime, ORAS, and Cosign images contain linux/amd64 and linux/arm64. Import it into your registry with the script from the runtime release. You need only ORAS and a registry that the cluster can reach:

./scripts/import-offline-bundle.sh bundle registry.example.com/stellwerk \
  --trust-bootstrap-digest sha256:<digest from the delivery note> --out import

The trust bootstrap digest reaches you out of band, together with the licence. It makes the publishers.json in the bundle authentic. Before it pushes anything, the import verifies without network access that:

• the publisher pinned for the plugin signed bundle.json with a pinned, valid, and unrevoked key;
• the bundle holds exactly the expected files, no symlinks, no unreferenced blob, and every blob matches its digest and size;
• the package, its signature, SBOM, and provenance verify as one release set;
• each image index holds exactly linux/amd64 and linux/arm64;
• the chart is native-plugin-runtime at the recorded version and not older than the minimum version.

Then it copies every component with its referrers and keeps all digests, into <registry>/<plugin-id>, /native-plugin-runtime, /oras, /cosign, and /charts/native-plugin-runtime. Each gets the tag <plugin-id>-<version>, which protects it from garbage collection of untagged manifests. The import refuses to move such a tag to another digest. It then reads the registry back and prints the installation steps. --out receives:

• values.json with the package reference, attestation digests, images by digest, the trust mode, and the Secret names;
• publishers.json and oci-manifest.sigstore.json for the trust Secrets in step 2;
• instructions.txt.

You still add networkPolicy.registryMirrors and, when needed, the migration Secrets. Continue with step 2; the manual copy above is not needed. Use --plain-http, --registry-config, and --ca-file to configure the target registry. Pass --verifier with a trusted runtime binary, because the bundle’s own runtime image cannot vouch for itself.

Step 2: Install trust and registry credentials #

The trust bootstrap is your authentic statement of which publisher may sign which plugin IDs. It never comes from a package:

{
  "schemaVersion": 1,
  "publishers": [
    {
      "id": "stellwerk",
      "policy": "managed-key-no-tlog",
      "pluginIds": ["dev.stellwerk.example"],
      "keys": [
        {
          "keyId": "<base64 SHA-256 of the DER public key>",
          "algorithm": "ecdsa-p256-sha256",
          "publicKey": "-----BEGIN PUBLIC KEY-----\n...",
          "validFrom": "2026-10-01T00:00:00Z",
          "validUntil": "2027-10-01T00:00:00Z"
        }
      ]
    }
  ],
  "revokedKeyIds": []
}

• Each plugin ID belongs to exactly one publisher.
• A publisher without a policy, or with sigstore, is refused. Transparency-log verification is not part of v1alpha.
• A key’s validity window gates new installations. Several keys with overlapping windows allow rotation.
• revokedKeyIds lists compromised keys. A revoked key blocks new installations and re-fetches of already installed digests.
• For managed-key-no-tlog, the runtime requires the SBOM and the provenance attestation by default. A missing attestation fails closed, and a supplied one is always verified. A publisher can opt out with requireSBOM: false or requireProvenance: false.

The runtime verifies the trust bootstrap when a Pod starts. A changed bootstrap, for example after a key rotation or revocation, applies to Pods that start after the change. The chart mounts the bootstrap, the signature bundle, and the registry configuration as plain files (subPath), and Kubernetes does not refresh those in place. Restart the runtime to apply a trust change, for example with kubectl -n stellwerk-plugins rollout restart deployment/<name>.

Create the Secrets in the plugin namespace. Keep registry credentials out of shell arguments:

kubectl -n stellwerk-plugins create secret generic example-plugin-trust \
  --from-file=publishers.json=./publishers.json
kubectl -n stellwerk-plugins create secret generic example-plugin-signature \
  --from-file=oci-manifest.sigstore.json=./bundle/oci-manifest.sigstore.json
kubectl -n stellwerk-plugins create secret generic example-plugin-mirror \
  --type=kubernetes.io/dockerconfigjson \
  --from-file=.dockerconfigjson=./mirror-config.json

Step 3: Provision the database #

1. Create the plugin database with three principals: an owner, a migration principal, and a runtime principal. The runtime principal must not be able to change the schema. The plugin’s release notes list the grants it needs.
2. Store the migration profile (mode: native-migrate) and the migration principal’s database URL in two separate Secrets. The chart mounts the URL at /run/secrets/stellwerk-plugin-migration/database-url, and the profile must refer to that path. Store the runtime profile and the runtime database URL in Secrets for pluginSecretFiles. The runtime chart refuses to mount migration Secrets into the runtime.

The plugin’s identities do not exist yet. IAM binds them to the installation UUID, which Core assigns when you install the plugin in step 5, and licenses bind to the same UUID. You create the identities in step 5. They are platform principals: service users that belong to no organization and hold no role, so you assign no IAM role to them.

Step 4: Run the migration phase #

Database migration is an explicit phase, not a startup side effect. Install the runtime chart with the runtime off and the migration on. The chart rejects both phases at once:

enabled: true
plugin:
  id: dev.stellwerk.example
  version: 1.0.0
package:
  reference: registry.example.com/stellwerk/example-plugin@sha256:<package-digest>
  manifestSHA256: <manifestSHA256 from release.json>
  attestations:
    sbom: sha256:<attestations.sbom.digest from release.json>
    provenance: sha256:<attestations.provenance.digest from release.json>
  registryConfigSecret:
    name: example-plugin-mirror
trust:
  mode: managed-key-no-tlog
  bootstrapSecret:
    name: example-plugin-trust
  signatureBundleSecret:
    name: example-plugin-signature
images:
  runtime: registry.example.com/stellwerk/stellwerk-plugin-runtime@sha256:<runtime-digest>
  oras: registry.example.com/mirror/oras@sha256:<oras-digest>
  cosign: registry.example.com/mirror/cosign@sha256:<cosign-digest>
networkPolicy:
  registryMirrors:
    - cidr: 192.0.2.10/32
      port: 443
runtime:
  enabled: false
migration:
  enabled: true
  profileSecret:
    name: example-plugin-migration-profile
  databaseSecret:
    name: example-plugin-migration-database

helm upgrade --install example-plugin \
  oci://registry.example.com/stellwerk/charts/native-plugin-runtime \
  --version <runtime-chart-version> \
  --namespace stellwerk-plugins -f migrate.yaml
kubectl -n stellwerk-plugins wait --for=condition=complete \
  job/example-plugin-native-plugin-runtime-migration --timeout=10m

The migration Job fetches and verifies its own copy of the package and its attestations. If it fails, the plugin stays off. Fix the cause and run the phase again; do not start the runtime until the Job has succeeded.

On a fresh installation this step creates the runtime Deployment paused, without Pods. Wait for the Job itself, not for every resource. With persistence.enabled and a storage class that binds on first use, the state volume binds when the first Pod starts in step 6.

Step 5: Register the installation in Core #

The Plugin Manager in the Control Plane is the only source of plugin installation state. It lives in Core’s database, so registration, enablement, and licence changes take effect without a Core restart. Helm and the runtime chart reconcile workloads only and never write it.

Configure the operators #

The Plugin Manager API is platform-scoped, outside any organization. Two platform permissions guard it, and IAM grants them to named users. The Control Plane asks IAM on every request and does not read the holder lists itself:

Set them in the Platform Orchestrator chart as iam.config.PLUGIN_MANAGER_READERS and iam.config.PLUGIN_MANAGER_WRITERS. Both are empty by default, so nobody can manage plugins until you name operators. IAM reads them at start, so roll IAM after you change them.

• Only users qualify. IAM refuses to start if a list names a service user, the internal system identity, or the nil UUID, because plugin workloads authenticate as service users.
• Organization roles are never consulted, and a plugin’s platform principal never holds either permission.
• A caller without the permission receives HTTP 403 with the code plugin_manager_forbidden.
• With the plugin host off, every route answers HTTP 404 with the code plugin_host_disabled.

The Control Plane keeps one tuning setting for the Plugin Manager: control-plane.config.PLUGIN_MANAGER_REFRESH_INTERVAL sets how often each replica checks for changes. The default is 2 seconds, and the range is 100 milliseconds to one minute.

Create an Ed25519 key for the request contexts that the Control Plane signs. The Data Plane and the plugin receive only the public key:

openssl genpkey -algorithm ED25519 -out plugin-gateway-signing-key.pem
openssl pkey -in plugin-gateway-signing-key.pem -pubout \
  -out plugin-gateway-public-key.pem

Store the signing key and the CA certificate that issued the plugin backend’s serving certificate in Secrets in the Orchestrator namespace. Use versioned Secret names, because subPath mounts do not refresh in place. Then name the operators and enable the Core profiles. Helm replaces lists, so repeat every secretFiles entry:

iam:
  config:
    PLUGIN_MANAGER_WRITERS: "<operator-user-uuid>"
control-plane:
  configurationProfiles:
    nativePluginGateway:
      enabled: true
      secretFiles:
        - name: native-plugin-signing-key
          secretName: plugin-gateway-signing-key-v1
          key: signing-key.pem
          mountPath: /var/run/secrets/stellwerk/plugins/signing-key.pem
        - name: native-plugin-ca
          secretName: plugin-backend-ca-v1
          key: plugin-ca.pem
          mountPath: /var/run/secrets/stellwerk/plugins/plugin-ca.pem
data-plane:
  configurationProfiles:
    nativeExternalDeployment:
      enabled: true
      config:
        EXTERNAL_DEPLOYMENT_CORE_GATEWAY_URL: https://api.orchestrator.example.com
      secretFiles:
        - name: native-plugin-verification-key
          secretName: plugin-gateway-public-key-v1
          key: gateway-public-key.pem
          mountPath: /var/run/secrets/stellwerk/plugins/gateway-public-key.pem
        - name: native-plugin-core-token
          secretName: plugin-admission-checker-token-v1
          key: core-bearer-token
          mountPath: /var/run/secrets/stellwerk/plugins/core-bearer-token
console:
  gatewayApi:
    route:
      pluginUiOrigins:
        - https://plugins.example.net
global:
  gatewayApi:
    backend:
      cors:
        allowOrigins:
          - https://console.orchestrator.example.com

nativeExternalDeployment is needed only by plugins that request Deployments. The admission checker token is the static token of the admission checker, a platform principal that you create in step 5. The Secret must exist before the Data Plane Pods start, so apply the nativeExternalDeployment part after you created the checker. To guard ordinary Deployments with the same principal, enable the deploymentAdmission profiles as described in Ordinary Deployment admission. pluginUiOrigins lets the Console frame the exact origin; it adds no script or network permission.

A plugin UI must stay outside the reach of the user’s session. The chart refuses to render when a plugin UI origin:

• uses the Console hostname;
• uses the API hostname or one of its subdomains, which receive the session cookie;
• is matched by an entry of global.gatewayApi.backend.cors.allowOrigins, which would let the plugin UI call the API with the user’s session.

The default allowOrigins is a wildcard. Once you configure plugin UIs, list the exact Console origins instead, as in the example, and serve plugin UIs from a hostname outside the API hostname, preferably under a separate registrable domain.

Apply the values with helm upgrade. This rollout is the last Control Plane restart that plugin registration needs. IAM fails to start when a permission holder is invalid. The Control Plane fails fast when the refresh interval or the first load of its state is invalid, or when the signing key or CA bundle is. An installation whose stored state no longer validates is not served and is logged; the others are.

The Plugin Manager API #

Every write has the same shape and rules:

• It takes an Idempotency-Key header, a reason of 1 to 1000 characters, and the expected_revision of the Plugin Manager that you decided on. Every read returns the current revision.
• It runs in one transaction that serialises with every other write. A replayed key returns the stored response. A different request under a known key fails with HTTP 409 and plugin_manager_idempotency_conflict. A stale revision fails with plugin_manager_revision_conflict; read the state and retry.
• A change advances the revision and writes one audit event with the actor, reason, request ID, and the state before and after. A step whose target state already holds changes nothing and writes no audit event, so an interrupted installation resumes by running its steps again.

An installation is in one of the states installed, active, disabled, uninstalled, or purged. A plugin has at most one installation that is not purged. One installation serves every organization that is enabled for it, with one runtime, one database, and one platform principal; see step 8. A UI route prefix belongs to one installation.

GET /plugin-installations and GET /plugin-installations/{id} also return the installation’s derived availability, the package it serves, the target of a pending upgrade, and the last status that Core read from the running plugin (observation). They also return the organization_mode, the organization_defaults, the pending_organizations, and for every enabled organization the capability_registration described in step 8. If that status does not prove the package, diagnostic carries the reason; see Activate.

Each Control Plane replica serves read decisions, such as discovery, routing, and UI, from a snapshot that it reloads when the revision changes. The replica that ran a command reloads at once, and the others follow within the refresh interval. A change can therefore reach another replica a moment later. Effects never lag: every effect that Core performs for a plugin checks again, in its own transaction, that the installation, the package digest, and the organization enablement are still active. Safety work stays permitted while either is disabled.

Core does not verify package signatures. The plugin runtime verifies the package against your publisher trust before it starts the process. Core binds the installation, its licence grants, and every effect check to the package digest that you register, so anyone who holds plugin_manager_write is trusted to register the manifest that belongs to a digest.

Use octl #

octl plugin-manager (alias pm) wraps the write routes for platform operators, and octl get reads the state:

Every write requires --reason. It sends the current Plugin Manager revision unless you pass --expected-revision, and it derives the idempotency key from the request, so repeating an identical command replays the recorded answer. Each command prints the resulting state and revision. The output is YAML unless you pass -o json.

install takes an installation as far as its flags say: it creates the installation, records --service-user and --admission-checker when you pass them, activates, and enables --enable-org. It stops at the first step that cannot succeed yet and names the installation UUID. On a new installation that is the activation, because the identities do not exist yet: create them with create-identities, start the runtime, and run the same install command again with --resume. It skips every step whose target state already holds and keeps no local state.

Register and install #

Read the current revision, then register the package. The manifest comes from the plugin release. It carries the plugin’s identity, operations, UI contribution, capabilities, service grants, authorization policies, and licence declaration. manifest_sha256 is the SHA-256 of the package manifest that the runtime verifies; Core rejects unknown fields:

REVISION=$(curl -s "${PO_API_URL}/plugin-installations" \
  -H "Authorization: Bearer ${PLATFORM_ORCHESTRATOR_TOKEN}" | jq .revision)

curl -X POST "${PO_API_URL}/plugin-packages" \
  -H "Authorization: Bearer ${PLATFORM_ORCHESTRATOR_TOKEN}" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d "$(jq -n --argjson rev "${REVISION}" --slurpfile manifest registration-manifest.json \
    '{expected_revision: $rev, reason: "Register example-plugin 1.0.0",
      manifest_sha256: "<manifestSHA256 from release.json>", manifest: $manifest[0]}')"

Every successful write advances the revision, so read it again before the next write.

Manifest rules for capabilities, operations, and routes #

Core checks these rules when you register a package. A manifest that breaks one is refused with HTTP 400 and the code plugin_manifest_invalid, and Core registers nothing.

• Every entry in capabilities is an object with the fields id, display_name, description, category, level, and scopes. The id has the form plugin.<plugin-id>.<name>. The <plugin-id> is the manifest’s own plugin ID and <name> is one segment of lowercase letters, digits, and hyphens that starts with a letter. The id has at most 255 characters. The texts must not be empty, level is read, write, or manage, and scopes lists each of organization, project, and environment at most once. A capability of Core, of another plugin, or without a name is refused, and so is a capability that is a plain string. Core registers these declarations in IAM, so the manifest must carry everything IAM needs.
• Each capability is declared once, and each operation ID is declared once. Operation IDs never start with core. or iam.. Core routes an operation below its plugin ID, so two plugins can use the same operation ID without meeting.
• The UI route_prefix is a slash followed by lowercase segments, for example /rollouts. A trailing slash, uppercase letters, and empty segments are refused, because the Console serves only this spelling and so every route has exactly one prefix string.
• An installation that is active serves its UI route prefix alone. If another installation already serves the exact same prefix, the activation answers HTTP 409 with the code plugin_registration_conflict and changes nothing. This also applies to the activation that completes an upgrade, which leaves the installation on its current package. Change the prefix in the plugin’s next package version, or uninstall the installation that holds the prefix, which releases it. A nested prefix such as /rollouts/reports is no conflict, because the Console routes to the longest matching prefix.

Then install the package:

curl -X POST "${PO_API_URL}/plugin-installations" \
  -H "Authorization: Bearer ${PLATFORM_ORCHESTRATOR_TOKEN}" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d "{\"expected_revision\": ${REVISION}, \"reason\": \"Install example-plugin\",
       \"package_sha256\": \"<manifestSHA256>\",
       \"backend_url\": \"https://example-plugin-native-plugin-runtime.stellwerk-plugins.svc:8080\",
       \"ui_origin\": \"https://plugins.example.net\"}"

The response status is 201, the installation is in state installed, and installation_id is the UUID that Core assigned. It never changes, and licences bind to it. Replaying the command returns the same UUID.

The installation stays in installed until you create its identities and activate it. With octl, install stops at the activation, because a package with service grants cannot be served without identities. Core then answers HTTP 409 with the code plugin_installation_incomplete.

Create the identities #

The installation acts through two IAM platform principals. A platform principal is a service user that belongs to no organization and holds no role, so it has no IAM authority in any organization. See Plugin workload identity.

• The plugin’s principal serves every organization that is enabled for the installation. IAM creates it together with the workload binding of the plugin’s ServiceAccount, which names the installation UUID. With serviceAccount.create=true, the ServiceAccount is named <release>-native-plugin-runtime.
• The admission checker is a platform principal with a static token. The Data Plane presents the token when it checks Deployments that a plugin requests.

IAM can create the binding only after Core assigned the installation UUID, so create the identities after install. Both calls need the IAM permission plugin_workload_binding_write, and IAM issues the static token only to a user token, never to a service user:

octl plugin-manager create-identities <installation-id> \
  --workload-issuer https://kubernetes.default.svc.cluster.local \
  --workload-namespace stellwerk-plugins \
  --workload-service-account example-plugin-native-plugin-runtime \
  --workload-audience platform-orchestrator-plugin-workload \
  --reason "Create the identities of example-plugin" -o json > identities.json

The issuer and the audience must equal the IAM setting and the runtime chart value. The command reuses a binding that IAM already holds for the installation and creates one otherwise. It creates the admission checker with a token that expires after 365 days (--checker-expiry-days, 1 to 3660), unless you pass an existing one with --admission-checker. Then it records both identities with the installation. The output holds the checker’s token once, as admission_checker.token. Store it in the Secret that you named in step 5 for the Data Plane, and delete identities.json:

umask 077
jq -r .admission_checker.token identities.json > core-bearer-token
kubectl -n platform-orchestrator create secret generic plugin-admission-checker-token-v1 \
  --from-file=core-bearer-token=./core-bearer-token
rm core-bearer-token identities.json

If recording fails after IAM created the checker, the output still carries the token. Run the command again with --admission-checker <id> instead of creating a second checker.

Without octl, create the binding with platform_principal and the checker with POST /platform-principals as described in the workload identity page, then record both with PUT /plugin-installations/{id}/identity. The body holds service_user_id (the plugin’s principal), admission_checker_id, expected_revision, and reason. The two UUIDs must differ. A package without service grants takes no identities. Core records the identities only while the installation is installed.

Core records the identities only when all of the following hold. Otherwise it answers HTTP 409 and changes nothing:

Core asks IAM on every identity request, including a replayed one. It never asks IAM about identities when you enable, change, or remove an enablement, so you can still disable an organization while IAM is down.

Step 6: Start the runtime #

Switch the runtime chart to the runtime phase. Set installationId to the UUID that Core assigned, and configure workload identity, the plugin’s own Secrets, and the UI:

runtime:
  enabled: true
migration:
  enabled: false
installationId: <installation UUID from step 5>
workloadIdentity:
  enabled: true
  audience: <audience of the workload binding>
  gateway:
    url: https://api.orchestrator.example.com
pluginSecretFiles:
  - name: plugin-profile
    secretName: example-plugin-profile
    mountPath: /var/run/secrets/stellwerk/plugin-profile
ui:
  enabled: true
  consoleOrigin: https://console.orchestrator.example.com
  gateway:
    enabled: true
    className: <gateway-class>
    hostname: plugins.example.net
    certificateSecretName: example-plugin-ui-tls
networkPolicy:
  additionalIngress:
    - <exact peer of the UI Gateway on ui.port>

Set probes.scheme to match the plugin’s backend listener, HTTPS when the plugin terminates TLS itself.

Set replicaCount above 1 for high availability; see Availability and the plugin writer. Merge these values with the values from step 4 and run helm upgrade. Every replica fetches and verifies its own package copy in init containers before the token sidecar and the plugin start. The Pod becomes ready only after the first token exchange succeeded and the plugin’s readiness probe passes.

installationId is required with workloadIdentity.enabled. The runtime passes it to the plugin as STELLWERK_INSTALLATION_ID, together with STELLWERK_PACKAGE_SHA256, the digest of the package manifest that it verified before it started the process. The plugin reports both to Core in step 7. The runtime refuses to start a plugin when STELLWERK_INSTALLATION_ID is not a canonical lowercase UUID.

After the first successful start, set package.installedDigest to the package digest (sha256:<package-digest>) and apply it. Pod restarts and new replicas then re-fetch exactly that digest even after the signing key’s validity window ended. A revoked key still blocks them.

Step 7: Activate the installation #

Activate with POST /plugin-installations/{id}/actions/activate. The body holds expected_revision and reason. With octl, run octl plugin-manager install <package-sha256> --backend-url <url> --resume, which activates and continues with the enablement, or octl plugin-manager activate <id>.

Before it serves anything, Core reads the status of the running plugin from /.well-known/stellwerk/plugin-status on the installation’s backend_url. It uses your plugin CA, does not follow redirects, and gives up after 10 seconds. The plugin reports the installation UUID that the runtime configured, the package digest that the runtime verified, and the migration generation applied in the plugin’s database. The path /.well-known/ is reserved for Core, and no plugin operation routes there.

Core activates only when the plugin reports exactly the installation UUID, the package digest, and the migration generation that the package declares. Then it materialises the package’s registrations, serves them, and the state becomes active. Otherwise it changes nothing and answers HTTP 409 with one of these codes:

The installation also shows the last reading as observation and the refusal as diagnostic, and its availability stays unavailable until the first activation. A refusal does not advance the revision, so repeat the same command once the runtime has caught up. Activating an active installation changes nothing.

Core reads the status only when you activate. It does not poll a running plugin, so it learns of a later outage from failing calls, which report plugin_unavailable.

Step 8: Enable organizations and import the license #

One installation serves every organization that is enabled for it. It has one runtime, one platform principal, and one plugin database. Core keeps every effect of the plugin inside one organization: each call names its organization, and Core checks the installation and that organization’s enablement inside the transaction of the effect. Disabling one organization stops its next effect on every Control Plane replica while the other organizations continue.

1. Choose the organization mode of the installation. The installation must be active (all also accepts a disabled one).

   Mode	Behavior
   selected	The default. You enable each organization yourself
   all	Core enables every active organization at once, and every organization that you create later, inside that organization’s creation transaction

   To enable one organization, call PUT /plugin-installations/{id}/organizations/{orgId} (octl plugin-manager enable-org <id> <org-id>). The body holds state: enabled, expected_revision, reason, and optionally timezone (an IANA name, default UTC) and approval_policies, a map from deployment approval scope (environment or project) to an authorization policy that the package owns. Each organization has its own timezone and approval policies. Any organization can be enabled, and the identities of the installation belong to none of them.

   To serve every organization, call PUT /plugin-installations/{id}/organization-mode (octl plugin-manager organization-mode <id> all --reason "<reason>"). The body holds mode, expected_revision, reason, and optionally default_timezone and default_approval_policies. Core gives every enablement that it creates the defaults. With the API, an omitted default means UTC and no policies. octl keeps the current defaults unless you pass --default-timezone or --default-approval-policy. The defaults change no existing enablement, and Core checks the default policies against the package at activation and upgrade (plugin_installation_incomplete).

   The switch is one audited command. It enables every active organization that has no enablement of the plugin, and it never changes an existing enablement. Setting the mode and defaults that already hold changes nothing.

2. To opt an organization out, disable its enablement with octl plugin-manager disable-org <id> <org-id>. A disabled enablement is the opt-out: Core never enables that organization again, across mode switches and restarts. Removing an enablement is refused with HTTP 409 and plugin_serves_all_organizations while the mode is all, because it would leave the organization without an enablement. To enable an opted-out organization again, call enable-org yourself.

   With selected, every enablement stays as it is when you switch back, organizations that you create afterwards are not enabled, and removing an enablement works again. An uninstall removes every enablement, including the opt-outs, and returns the installation to selected.

3. Check pending_organizations in the installation. In mode all, it lists organizations that Core could not enable: an organization created while the plugin host was off, or while the installation-wide license grant’s organization limit was reached. Such an organization is still created. Run organization-mode <id> all again to enable every pending organization, for example after you raised the limit.

4. For a licensed plugin, install the issuer trust and import the license bundle as described in License and entitlement trust. An organization grant matches only an installation that is enabled for its organization, so import it after the enablement. An installation-wide grant matches the installation, so you can import it before any organization is enabled.

5. Assign the plugin’s IAM capabilities to users through roles. Plugin capabilities have the form plugin.<plugin-id>.<name>. Core registers the capability declarations of the package’s manifest in IAM for every enabled organization, as its own internal identity, and the plugin registers nothing itself. The registration runs right after Core enables an organization, after every activation, and every minute for every enabled organization, and it replaces only what drifted. Disabling or removing an enablement keeps the registration. The installation shows the result per organization as capability_registration:

   state	Meaning
   registered	IAM holds the declarations of the served package for the organization
   pending	Core has not yet reconciled the served package for the organization, for example right after an upgrade or in the response to the enabling command. Read the installation again
   failed	The last attempt failed. detail says why, and Core retries every minute

   The role holders of the organization can use a capability once its state is registered.

6. Check the result with GET /orgs/{orgId}/plugins/{pluginId}. The state must be available. Core derives it: available while the installation is active and the organization is enabled, not_enabled (safety-only) while either is disabled, then refined by the signed license.

When an organization is created while the plugin host is on and the installation serves all organizations, Core registers the capabilities right after the creation, so the first administrator can grant them at once. If the creation fails, Core removes the enablements together with the organization. With ordinary Deployment admission on, the new organization also deploys at once: Core admits every organization as the Data Plane’s one admission identity, so the organization needs no change of the chart values and no restart.

Codes for organizations #

The codes plugin_service_user_organization_mismatch and plugin_enablement_limit no longer exist. An installation can serve any organization, and the number of organizations is bounded by the license only.

See the installations in the Console #

Platform operators with plugin_manager_read find a read-only Plugin Manager view in the Console, at /platform/plugins. The navigation entry appears only for them, and the page asks IAM for the permission before it sends any Plugin Manager request. The list shows for each installation the plugin, the package version and digest, the state, the organization mode, the number of enabled organizations, the license state, and organizations whose capability registration is pending or failed. The detail page adds the enablements with their timezone, capability registration and covering grant, the pending organizations, the installation-wide grant against its limit, the packages of the plugin, and the newest audit events. The view has no controls: you change the state with octl or the API.

Network access #

The runtime chart enables a NetworkPolicy by default:

• Ingress only from the Control Plane Pods, plus the peers that you add to networkPolicy.additionalIngress.
• Egress only to the API gateway proxy (port 10443), the plugin database (port 5432), DNS, and the registry mirrors.

The defaults target the platform-orchestrator release in the platform-orchestrator namespace, with Envoy Gateway and CloudNativePG. Check that every selector matches the actual Pods, including the gateway proxy. Override selectors where needed; do not broaden them to every Pod in a namespace. The API hostname that the plugin uses must resolve to the selected gateway Pods.

The cluster must enforce these policies. Kubernetes does not expose a signal that policy processing is complete, and a new Pod can start before its rules take effect. A Ready Pod, a Ready controller, or successful allowed traffic is not proof that denied traffic is blocked. Test allowed and denied paths with reachable control endpoints, new Pods, and controller restarts. This contract does not promise zero-first-packet isolation or a sandbox for hostile native code. See NetworkPolicy Pod lifecycle .

networkPolicy.registryMirrors accepts single addresses (/32 or /128) or an exact in-cluster peer with namespace and Pod selector, each with a port. Ranges, namespace-only peers, and free-form rules are rejected. The chart grants no mirror access by default; with policy enforcement in effect, fetching fails until you list the mirror. NetworkPolicy applies per Pod: the plugin and UI containers can reach the mirror endpoints too, but they hold no registry credentials. Registry credentials and the trust bootstrap reach only the fetch and verification init containers.

Availability and the plugin writer #

A v1alpha plugin keeps exactly one active writer. A replica writes only while it holds a lease, and the lease carries a monotonic fencing token, so a replica that lost its lease cannot overwrite a newer writer. A standby takes over after the lease expires. Read APIs and UI can serve from every replica.

replicaCount (default 1) sets the number of runtime replicas:

• One replica is replaced with the Recreate strategy. Recreate avoids overlapping Pods within one Deployment, but it is not a fencing protocol. The plugin’s lease stays authoritative, for example when an interrupted upgrade leaves a second Pod running.
• Several replicas roll one at a time (RollingUpdate with maxUnavailable: 0 and maxSurge: 1), so at least the configured number stays available during a package upgrade.
• A PodDisruptionBudget (minAvailable: 1, unhealthy Pods stay evictable) keeps one replica through node drains. A preferred anti-affinity on kubernetes.io/hostname spreads the replicas. An explicit affinity value replaces that default.
• The chart elects no writer. The plugin’s fenced database lease admits exactly one, and every supported plugin binary takes it.
• Replicas keep no local state, so persistence.enabled requires replicaCount: 1.
• Every replica fetches and verifies its own package copy, so replicas need no shared storage and can run on any node.

Every new Pod fetches the package from your mirror. Keep the mirror available, and keep the last installed package, its signature bundle, and the runtime images in it for as long as you run that version.

Upgrade a plugin #

Read the plugin’s release notes first. They state whether the new version changes the plugin’s database schema and what it declares.

An upgrade moves an installation to a later package of the same plugin. The installation UUID, and with it the licence binding, never changes. Core keeps serving the current package, its registrations, and every effect bound to its digest until the new package is running. Then one activation switches package and registrations in one transaction, so a Control Plane replica serves either package, never a mix.

1. Mirror the new package and images as in step 1, and register the package with POST /plugin-packages (octl plugin-manager register-package).
2. Request the upgrade with POST /plugin-installations/{id}/upgrade (octl plugin-manager upgrade <id> <package-sha256>). The body holds package_sha256, expected_revision, and reason. The installation shows the package as target_package_sha256 and keeps package_sha256. Core refuses the request with HTTP 400 and plugin_upgrade_invalid if the package belongs to another plugin, does not have a later version, declares a lower migration generation, or cannot be served with the installation’s origins, identities, and approval policies. Requesting the served package withdraws a pending upgrade.
3. Update plugin.version, package.reference, package.manifestSHA256, and package.attestations in the runtime chart. A new digest needs a publisher key that is valid now.
4. Apply the new package with migration.enabled=true. The migration Job runs, and the runtime Deployment is paused, so it creates no Pod from the new package. Wait for the Job to succeed and record the result.
5. Apply the same package with migration.enabled=false. The Deployment resumes and starts or rolls to the new package. Then set package.installedDigest to the new digest.
6. Activate with POST /plugin-installations/{id}/actions/activate (octl plugin-manager activate <id>). Core reads the plugin’s status as in step 7 and switches only when the plugin reports the installation UUID, the target digest, and the target’s migration generation. Otherwise it answers HTTP 409 with the same diagnostic codes, changes nothing, and keeps serving the current package. Repeat the command once the rollout has finished. A disabled installation stays disabled.

During the rolling update, Core still routes the current package’s registrations to the new Pods. A new package must therefore keep serving the registrations of its predecessor. Effects that a replica authorised from its snapshot of the predecessor fail with HTTP 409 and plugin_not_installed for at most one refresh interval after the switch, and succeed on retry.

The migration.compatibility value of the runtime chart says how the runtime behaves while the migration runs:

The Core and runtime images do not change for a plugin upgrade. The Plugin Manager refuses an upgrade to an earlier version, so there is no downgrade through it. Recover by rolling forward, or by switching the plugin off as described in Plugin recovery. Never downgrade the plugin database.

Disable a plugin #

You can disable a plugin for one organization or for the whole installation. Both keep the plugin in safety_only mode, which Core reports as not_enabled:

• For one organization, call PUT /plugin-installations/{id}/organizations/{orgId} with state: disabled (octl plugin-manager disable-org <id> <org-id>). The other organizations continue. When the installation serves all organizations, this is how an organization opts out.
• For the installation, call POST /plugin-installations/{id}/actions/disable (octl plugin-manager disable <id>). Call .../actions/enable to return it to active.

While disabled:

• Reads and safety operations, such as cancel and rollback, stay available where the plugin’s license still allows them.
• New forward work fails with HTTP 409 and the code plugin_not_enabled.
• The workload and its data stay in place.
• The organization’s resource event journal pauses when no subscriber remains. After a later resume, cursors issued before the pause fail with plugin_resource_cursor_expired, and the plugin reads again from current Core resources.

Keep the runtime running while organizations depend on safety operations.

Uninstall a plugin #

Uninstalling removes the installation’s registrations and every organization enablement, including the opt-outs of the all mode, and returns the installation to the mode selected. It keeps the installation UUID, its package, its identities, its licence binding, and the plugin’s own data, so you can reinstall it later.

1. Disable the installation with POST /plugin-installations/{id}/actions/disable to stop new forward work. Let the plugin finish, cancel, or roll back its running work. Keep the plugin workload running meanwhile, because removing the safety operations strands in-flight work.
2. Uninstall with POST /plugin-installations/{id}/actions/uninstall (octl plugin-manager uninstall <id>). Core refuses it with HTTP 409 and plugin_work_unsettled while it holds safety work of the plugin in any organization: operation grants that are not released, dispatches that are not settled, or active Module reservations. The message gives the counts. There is no force option. Until the command succeeds, the plugin’s cancel, rollback, and read paths stay available. The organization’s resource event journal pauses when no subscriber remains.
3. Delete the workload binding with DELETE /plugin-workload-bindings/{bindingId} and uninstall the runtime release with helm uninstall. The binding’s platform principal stays and keeps no credential. Keep the plugin’s platform principals if you plan to reinstall. Delete them with DELETE /platform-principals/{principalId} when you purge.
4. If no plugin remains, disable the nativePluginGateway profile and, if unused, nativeExternalDeployment and nativePluginEntitlements. The plugin then reports plugin_not_installed, and Core stops journaling resource events for it.
5. Decide what to do with the plugin database. Core never deletes it. Back it up before you drop it.

An uninstalled installation reports not_installed. Uninstalling again changes nothing.

To reinstall, run POST /plugin-installations/{id}/actions/reinstall (octl plugin-manager reinstall <id>). The installation returns to installed with its UUID, and its licence grants match again once you enable the organizations. Treat it like a new installation from step 5 on: create a new IAM binding for the same installation UUID, naming the existing platform principal (service_user_id in the binding request; create-identities would create a new principal), start the runtime with the same installationId, activate, and enable the organizations or set the mode to all again. A new installation of the plugin is refused with plugin_installation_exists while a retained one exists.

To retire the installation for good, purge it with DELETE /plugin-installations/{id} (octl plugin-manager purge <id>). The body holds expected_revision and reason. Core refuses it with HTTP 409 and plugin_installation_state unless the installation is uninstalled. Purging retires the UUID: licence grants bound to it never match again, and a later installation of the plugin gets a new UUID and needs new grants. Core does not own the plugin’s data, so delete it through the plugin runtime.

Core keeps its own neutral records about the plugin, such as operation grants, dispatch receipts, license evidence, outcome acknowledgements, and the Plugin Manager audit events. Their down migrations refuse to discard them. A Deployment that the plugin dispatched and whose outcome is unknown still blocks its Environment after the uninstall; see Plugin recovery.

Limits #

• The runtime chart does not register the installation with the Control Plane. You register it through the Plugin Manager API or octl plugin-manager, as described in step 5.
• Core reads the plugin’s status only when you activate. It does not probe the health of an active plugin, and the Plugin Manager checks collisions only for the UI route prefix, which it compares as an exact string.
• Uninstall does not ask the plugin whether it is ready to be decommissioned, and removing a single organization’s enablement does not wait for that organization’s safety work. Disable the plugin and let its work settle first.
• Purge retires the installation but deletes no plugin data. You delete the plugin database through the plugin runtime and your database tooling.
• Downgrading Core below the Plugin Manager schema is refused once an audit event exists, and below the purged state once an installation is purged.
• The runtime executes a sealed in-memory copy of the executable. It needs Linux with memfd_create, vm.memfd_noexec must not be 2, and the copy counts toward the container memory limit. Size resources.limits.memory for the executable plus its working set.
• Signatures authenticate code; they do not sandbox it. Native plugins share the node kernel with other workloads.
• All organizations of an installation share one plugin database. A bug in the plugin can mix the data of organizations inside that database, while Core refuses every effect of the plugin across organizations. Keep this in mind when you decide which organizations share a plugin installation.
• A compromised plugin reaches every organization that is enabled for its installation, within the plugin boundary: the service grants of the manifest, reads within active operations, dispatch within the frozen scope of an operation, and event publication. A new operation still needs a fresh proof of the delegating user. A leaked workload token reaches the same organizations until it expires, after at most 15 minutes. The plugin’s platform principal holds no IAM role in any organization.
• The licence limit limits.organizations caps enablements at the time of the enablement. It never disables an organization that is already enabled; see License and entitlement trust.