Skip to content

Approval-gated reliability remediation

Reliability remediation is a closed, provider-neutral workflow for exactly three actions:

  • silence one alert rule for 60 seconds to 24 hours;
  • move one incident from an exact current state to an exact new state;
  • execute one host-registered runbook with bounded scalar parameters.

There is no shell, URL, remote method, MCP tool name, or free-form executable payload. Provider-specific names remain inside the pack-owned adapter. A model or provider result may suggest a target, but it never grants authority to use it.

Planning resolves the suggested resource through trusted inventory under the requesting principal’s effective authority. The resulting canonical dry-run plan is committed to the current RunAggregate before it can be shown for approval.

The operator review display must show all of these committed values:

  • proposal, plan, task, run, generation, attempt, and expected aggregate version;
  • requesting principal and approval expiry;
  • trusted tenant, resource, target type, connector identity, and connector schema digest;
  • exact current-state preconditions and mutations;
  • effective resource and verb scopes plus their validity window;
  • blast radius, maximum duration, and expected evidence;
  • policy and configuration digests;
  • declared rollback mutation or typed compensating action;
  • stable plan digest.

Approval uses the kernel’s durable one-shot interrupt. The grant records the resolver, reason, resolution time, and execution identity and is bound to the exact review values. Drift in the plan, target, authority, connector schema, policy, configuration, owner fence, or aggregate version requires a new plan and a new approval. Expired, copied, already-resolved, or already-consumed approval identities fail closed.

The host owns secret resolution and supplies distinct read and write credential profiles. Every write profile is pinned to one connector, tenant, closed action, resource set, verb set, and validity window. The request may only narrow that profile; extra tenants, resources, verbs, time, or a reused read profile are rejected before dispatch.

Every transition checks the current RunAggregate owner, generation, attempt, and expected aggregate version. Approval consumption is committed before the provider call, and the fence is checked again immediately before dispatch. Superseded workers cannot reuse a grant or commit an outcome.

The provider mutation receives a stable idempotency key derived from the exact approval request and plan digest. A definitive success, conflict, or failure is committed as a durable outcome. Cancellation before dispatch commits a cancelled terminal event without touching the provider.

A disconnect or post-send timeout is different: it records ambiguous with the provider operation identity and fences all redispatch. Recovery calls only the adapter’s fixed reconciliation operation for that identity. It never repeats the original mutation blindly. Reconciliation evidence is committed alongside the original ambiguous receipt.

A partial alert silence or incident-state update invokes only the rollback declared in the approved plan, with a stable :rollback idempotency key. A successful rollback terminates as rolled_back. A rejected, ambiguous, partial, or failed rollback records a typed compensation_required event and the operator action needed to recover.

Registered runbooks deliberately have no invented inverse. Partial runbook execution always records its plan-defined compensating action. Durable events link the proposal and plan, approval request, approval actor and reason, provider outcome, reconciliation, rollback, compensation, and terminal evidence.

Fail closed and create a new plan instead of continuing whenever:

  • trusted inventory cannot re-resolve the exact target and current state;
  • requested authority is broader than the host-owned write profile;
  • the connector, schema digest, policy digest, or configuration digest changed;
  • the owner fence, attempt, aggregate version, principal, or approval binding is stale;
  • provider operation identity is absent or changes during reconciliation;
  • a provider result is malformed, proposes another tool, or requests more scope;
  • the declared recovery cannot produce a durable terminal result.

An emergency never authorizes bypassing approval, changing the requested target, using ambient credentials, or invoking an arbitrary provider tool. Escalate to a separate operator-controlled incident procedure outside this remediation workflow.