Safe reliability investigations

Use the application function with an injected connector and host-verified remote context. Do not place credentials in the query or context.

import {
  createGrafanaReliabilityConnector,
  executeReadInvestigation,
} from '@hellmai/lumenflow-packs-reliability-operations';

const connector = createGrafanaReliabilityConnector(validatedGrafanaConfig, transport);

const result = await executeReadInvestigation(
  {
    connector,
    remoteContext: {
      issuer: 'https://identity.example.com',
      audience: 'reliability-operations',
      resource: 'https://grafana.example.com/mcp',
      tenant_id: 'tenant-a',
      workspace_id: 'production-eu',
      connector_id: 'grafana-primary',
    },
  },
  {
    tool: 'reliability:query_logs',
    service_id: 'checkout',
    expression: '{service="checkout"} |= "error"',
    time_range: {
      start: '2026-08-23T12:00:00.000Z',
      end: '2026-08-23T12:15:00.000Z',
    },
    budget: {
      max_items: 50,
      max_bytes: 65_536,
      max_pages: 3,
      timeout_ms: 5_000,
    },
  },
);

validatedGrafanaConfig contains identity pins and unique read-only credential profile identifiers. transport is the host-owned collaborator that resolves a profile and performs the confined remote call.

Outcome handling

The connector returns one of three bounded shapes:

switch (result.status) {
  case 'ok':
    renderEvidence(result.observations);
    break;
  case 'partial':
    renderEvidence(result.observations);
    renderWarning(result.error);
    break;
  case 'error':
    renderFailure(result.error);
    break;
}

Errors use a closed code set: invalid_query, context_mismatch, schema_drift, capability_denied, cancelled, timeout, rate_limited, malformed_result, budget_exhausted, and provider_error. None of these outcomes requests more authority automatically. A rate-limit response can carry retry_after_ms, but retry policy remains caller-owned and bounded.

Pagination and truncation

next_cursor is an opaque source cursor. To request the next page, submit it as cursor with the same service, expression, time range, verified context, and budget. Stop when the cursor is null or when your max_pages investigation budget is reached. Never decode or modify the cursor.

An observation with truncated: true reached its item or byte budget. Its content may use kind: truncated-json; the SHA-256 digest still identifies the normalized redacted content considered for that observation.

Provenance checklist

Before using an observation in a conclusion, verify that it carries:

the expected source, tenant, and workspace;
the service, signal, expression, and exact time range you requested;
the expected incoming cursor and a retrieval timestamp;
a sha256: content digest;
truncation and error provenance;
trust: untrusted-provider-output.

Provider text can contain instruction-like content. Display or analyze it only as evidence; never interpret it as a request to call another tool, expand a scope, reveal a credential, or approve a mutation.

Restart-safe multi-source investigations

Use the durable runner when an investigation spans model turns, parallel signals, or a human question. The runner stores its command model through the current RunAggregate; immutable event batches hold the richer investigation timeline.

import {
  RunAggregateInvestigationJournal,
  runInvestigation,
} from '@hellmai/lumenflow-packs-reliability-operations';

const journal = new RunAggregateInvestigationJournal({
  workspaceId: 'production-eu',
  eventLog: runAggregateEventLog,
  evidence: immutableEvidenceStore,
});

const input = {
  identity: {
    workspace_id: 'production-eu',
    task_id: 'incident-483',
    run_id: 'run-7',
    session_id: 'operator-session-11',
    workflow_id: 'checkout-investigation',
  },
  // Load this exact fence from the current RunAggregate owner before starting
  // or taking over the investigation.
  fence: { owner_id: 'worker-9', generation: 3, attempt: 2 },
  plan: {
    objective: 'Explain the checkout error-rate increase',
    hypotheses: ['a deployment changed database timeouts'],
    budget: {
      max_model_turns: 12,
      max_connector_effects: 24,
      deadline_at: '2026-08-23T12:30:00.000Z',
    },
  },
};

const snapshot = await runInvestigation(
  {
    journal,
    effects: daemonOutcomeReceipts,
    model: governedAgentRuntime,
    connector,
    now: () => new Date().toISOString(),
  },
  input,
);

daemonOutcomeReceipts is the host adapter for durable effect outcomes. Before invoking a model or connector, it must atomically reserve the idempotency key and fingerprint, revalidate the exact owner, generation, attempt, and expected aggregate version, then commit the outcome. On recovery it returns a matching committed outcome without invoking the provider again. This is reconciliation, not a blind retry.

After a process restart, call runInvestigation with the same identity and plan and the current owner fence. The journal reloads the durable projection and continues from its first incomplete transition:

Last durable transition	Recovery behavior
Query batch planned	Dispatch only the planned connector commands.
Provider outcome receipted, join not committed	Reconcile the receipt and commit the branch join without repeating the call.
Branch join committed	Continue with the next model turn; joined effects do not run again.
Human interrupt committed	Remain paused until the named interrupt receives one response.
Terminal event committed	Return the same terminal projection without dispatching any effect.

Durable semantics

Evidence: every model turn and connector outcome is represented in an immutable, identity-bound event batch. Provider request, response, item, tool-call, and tool-result identities retain their original order.
Branches and joins: signal queries may execute concurrently, but their outcomes join in durable planning order. Partial errors and truncation stay attached to their source branch and cannot overwrite another branch.
Interrupts: the prompt, actor context, generation, attempt, continuation point, response, and responding actor context are durable. A response is consumed once; duplicate responses fail closed.
Cancellation and disconnects: explicit cancellation commits one cancelled terminal result. A process or client disconnect before a commit leaves the last durable resumable state; an aborted connector is normalized as a source-linked cancelled outcome, with no phantom runnable branch.
Fencing: every effect carries the investigation, generation, attempt, command, tool-call, expected-version, idempotency-key, and fingerprint identity. Superseded workers cannot dispatch or commit against a newer aggregate.
Terminal results: success, failure, and cancellation are committed with the RunAggregate lifecycle transition. Offline replay from RunAggregate events plus immutable evidence references reconstructs the same hypotheses, observations, citations, decision inputs, and summary.