Plugin recovery

Switch plugins off, release an Environment that a plugin-dispatched Deployment blocks, and handle expired event cursors.

The supported recovery paths for v1alpha are rolling forward and switching the plugin off while Core keeps running on schema-compatible images. Neither path downgrades a database or deletes data.

Switch a plugin off #

Switch a plugin off when its workload misbehaves, or when you need Core without the plugin while you investigate a plugin release.

1. Stop the plugin workload. Set runtime.enabled=false in the runtime chart, or uninstall the runtime release.
2. Disable the plugin profiles in the Platform Orchestrator chart: control-plane.configurationProfiles.nativePluginGateway, control-plane.configurationProfiles.nativePluginEntitlements, and data-plane.configurationProfiles.nativeExternalDeployment.
3. Keep the Core images of the installed release, or schema-compatible images of it. Do not downgrade the Control Plane, Data Plane, or IAM databases.
4. Keep the plugin database and its backups. Do not delete Core’s plugin tables.

With the plugin host off:

• Plugin discovery returns an empty list, and plugin operations fail with plugin_not_installed.
• Ordinary Deployments, Module Version Management, and every other Core function work as usual.
• The Control Plane writes no extension journal entries and does not prune the journal.
• A failed Deployment that the plugin dispatched and whose outcome is unknown still blocks its Environment. Release it as described in the next section.

To switch the plugin back on, restore the previous values. Changes that happen while the plugin host is off never appear in the event journal. The plugin’s release notes describe how it reconciles them from current Core state.

Do not edit database statuses, and do not replay an uncertain request by hand. An older plugin package is not a fallback after a newer package’s migration ran, unless the package declares compatible migrations.

Acknowledge an unknown Deployment outcome #

A plugin can request ordinary Deployments. If such a Deployment fails and the Data Plane has no proof that its Runner stopped, the Runner might still change the Environment. The Data Plane then rejects every new Deployment, rollback, and destroy of that Environment with HTTP 409:

failed plugin-dispatched Deployment '<deployment-id>' may still have an active Runner; outcome is unreconciled

The block ends when a genuine late Runner result arrives or when an authorized user acknowledges that the attempt can no longer change the Environment. Only plugin-dispatched Deployments block their Environment. Failed ordinary Deployments never do.

Before you acknowledge #

• Confirm that no Runner process for the Deployment can still run, for example because its Kubernetes Job and Pods are gone.
• Wait until the Runner dispatch window has closed. It ends one hour after the first dispatch attempt. An earlier request returns HTTP 409.
• The Deployment must be the current head of the Environment and a failed plugin-dispatched Deployment without stop evidence.

Permission #

The acting user needs deployment.outcome.acknowledge on the Environment. A grant at organization scope also applies. The built-in Admin role includes the permission. For other users, add it to a configurable role; see RBAC. The system user and the deprecated organization token are rejected with HTTP 403, because the acting user is the audit record.

Acknowledge #

Send the request with a reason of 1 to 1000 characters:

curl -X POST "${PO_API_URL}/orgs/${ORG_ID}/projects/${PROJECT_ID}/envs/${ENV_ID}/deployments/${DEPLOYMENT_ID}/actions/acknowledge-outcome" \
  -H "Authorization: Bearer ${PLATFORM_ORCHESTRATOR_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"reason": "Runner job deleted after node loss; no pod remains"}'

With octl, run octl deployments acknowledge-outcome <deployment-id> --reason "<reason>". The command looks up the project and Environment, then calls the route above.

The Data Plane stores the Deployment, Environment, plugin, operation, acting user, reason, and time. The Control Plane keeps no second copy and logs only the identifiers of a new acknowledgement, not the reason.

An acknowledgement applies to exactly one Deployment:

• It never changes the Deployment’s status, history, or outputs.
• The next Deployment of the Environment, and an explicitly authorized rollback to the captured base, may proceed.
• The plugin sees the failure instead of an unreconciled outcome.
• It does not carry over to a later failure. There is no bulk or Environment-wide reset.

Acknowledging never starts queued work. Queued items of the Environment, requests that wait in the deployment queue, are rejected with the reason code outcome_unreconciled, either by the next promotion or at the latest by the acknowledgement itself. An Environment with an unreconciled outcome accepts no new queue item and answers HTTP 409 outcome_unreconciled. After the acknowledgement, new requests work normally, so an incident is not followed by an automatic burst of Deployments. Submit the work again once you have checked the Environment.

The acknowledgement is a Core route and does not need the plugin host, so it also works while the plugin is switched off or uninstalled. No infrastructure retry or rollback starts automatically.

Expired event cursors #

Plugins read Core resource events from the extension journal with an opaque cursor. Presenting a cursor acknowledges every entry up to it. The Control Plane removes an entry only when it is older than the retention window and every active consumer in the organization has acknowledged it. Retention never prunes past the cursor of an active consumer.

A plugin that presents a cursor older than the retained history receives HTTP 409 with the code plugin_resource_cursor_expired, never a silent gap. This happens when a consumer was inactive for longer than the retention window, or when a plugin database was restored from an old backup. The consumer then rebuilds its state from current Core resources and reads again without a cursor, which starts at the oldest retained entry.

Set it with control-plane.config.PLUGIN_EVENT_JOURNAL_RETENTION. Choose a window longer than your longest expected plugin outage and older than the plugin database backups that you might restore.

• An active consumer that never presented a cursor holds its organization’s entire history. Release operations that a plugin abandoned, so the journal can shrink.
• Every Control Plane replica runs retention at startup and then every ten minutes, removing at most 10,000 entries per organization and run. Replicas coordinate through a PostgreSQL advisory lock, and request handling never waits for retention.
• Once any entry was pruned, the journal retention migration refuses to downgrade, because older images could not report expired cursors.