Evidence Store

The Evidence Store records every tool call that passes through the kernel. It produces immutable, append-only records that prove what was requested, what was allowed, and what happened — regardless of whether the tool succeeded, failed, or was denied.

What Gets Recorded

Every tool call generates:

1. TOOL_CALL_STARTED

Written before the tool executes. Contains:

receipt_id — unique identifier for this tool call
task_id — which task this call belongs to
run_id / session_id — the exact persisted execution selected by the call
tool_name — which tool was called
capability_fingerprint — hash of the full minted authority context, including caller, selector, workspace/config hash, lane, runtime version, capability, pack integrity, policy profile, scopes, and deny overlays
input_hash — SHA-256 hash of the serialized input
input_ref — protected-state URI for the content-addressed input blob
scope_enforced — the computed scope intersection result
timestamp — when the call was initiated
owner — optional { instance_id, pid, host } minted by the runtime that wrote the trace (see Crash Recovery); earlier traces have none
trace — optional W3C span identity { trace_id, span_id, parent_span_id, trace_flags }, present only when the host supplied a trace context for the call (see Trace spans)

2. TOOL_CALL_PROGRESS (optional, repeatable)

Written between the started and finished entries when a tool reports progress snapshots. It is a durable checkpoint, not a live feed: a streamed model turn writes one final entry for the assembled reply, and its per-delta stream progress is delivered in memory to the calling host instead of being persisted one blob and one entry per delta. Contains:

receipt_id — matches the started entry
sequence — monotonic snapshot order for this receipt
payload — partial progress data such as streaming turn snapshots
timestamp — when the snapshot was recorded

3. TOOL_CALL_FINISHED

Written after the tool completes (or fails). Contains:

receipt_id — matches the started entry
result — one of success, failure, denied, or crashed
duration_ms — execution time
output_hash — SHA-256 hash of the output
policy_decisions — every policy rule that was evaluated, with its decision
artifacts_written — files the tool reports having created or modified

Non-streaming tools usually produce only started and finished entries. Streaming tools such as agent-runtime:execute-turn can emit multiple progress entries before the final result.

Trace spans

A host that already has a W3C trace context (for example the traceparent of its own request) passes it per call as ToolExecutionOptions.traceContext ({ traceparent, tracestate? }). Each tool dispatch, including every model turn (a dispatch of agent-runtime:execute-turn), then records a trace on its TOOL_CALL_STARTED entry: the trace id, a fresh random span id, the parent span id taken from the supplied traceparent, and its flags. The dispatch result (executeWithApprovalCandidate) returns the same trace, so a host builds the tree by passing a turn’s span as the parent of the tool calls that turn requested:

consumer span ── turn 1 ── tool A
              │         └─ tool B
              └─ turn 2 ── tool C

The tool receives a frozen context whose parent is its own dispatch span: in-process handlers read options.traceContext, subprocess tools read context.trace_context (a field of the worker envelope that only the host sets), and a lumenflow_cloud model turn forwards it to the inference route. A tool can neither change the context the kernel holds nor the recorded span. Without a context nothing is invented: no trace, no context for the tool. A traceparent that is not valid W3C is refused with INVALID_INPUT before any evidence exists. Span ids appear only in trace; input_hash and output_hash do not depend on them, so identical inputs keep identical digests. There is no exporter: the identities are evidence only.

Content-Addressed Inputs

Before a tool executes, its input is:

Serialized to a canonical JSON string (deterministic key ordering)
Hashed with SHA-256
Written to evidence/inputs/<hash> using exclusive-create mode (O_EXCL)

This means:

Identical inputs produce the same hash — deduplication is automatic
Inputs are immutable — the exclusive-create flag means an existing blob is never overwritten
Inputs are verifiable — given a trace entry’s input_hash, anyone can recompute the hash from the blob and confirm it matches

Model-turn inputs are stored as a manifest

A model-turn input carries the whole conversation, and the conversation only grows between turns. Storing each turn as one blob would write every earlier message again on every turn. An input that is an object with a non-empty messages array is therefore stored as one blob per message, one blob for tool_catalog, and a small manifest naming them by hash. A message, system prompt or tool catalogue that did not change is already present and is not written again. In a recorded 28-turn run this cut the stored turn-input bytes from 2.65 MB to 0.21 MB (12.7 times).

input_hash does not change: it is still the SHA-256 of the complete canonical input, so receipts, approvals and replay keep comparing it with the hash of the raw input. To verify it, rebuild the input from the manifest and its blobs, hash it, and compare. A changed message blob, a missing blob or a manifest that names another input fails verification. Evidence written before this change is a single blob named by input_hash and verifies exactly as it did. An input that carries an inline image keeps the redacted single-envelope form described in the image guidance.

Storage Model

The evidence store uses append-only JSONL (JSON Lines) files:

.lumenflow/kernel/evidence/
  traces/
    tool-traces.jsonl           ← active append file
    tool-traces.lock            ← file-based mutex
    tool-traces.00000001.jsonl  ← compacted segment
    tool-traces.00000002.jsonl  ← ...
  inputs/
    a1b2c3d4e5f6...             ← content-addressed blobs (SHA-256 filename)
    <input_hash>.manifest       ← points a model-turn input at its manifest blob
  retention/
    chain-checkpoints.jsonl     ← signed chain checkpoints and retention anchors

Append-only guarantee

The evidence store never modifies existing records. New entries are appended to tool-traces.jsonl. Reads never acquire the file lock — only writes hold the mutex.

Write lock and dead writers

Each append holds tool-traces.lock for the length of one write. The lock file names its owner: the same runtime identity (instance, process id, host and process start time) that every started trace carries. A writer that is killed between taking and releasing the lock therefore leaves a lock that can be attributed instead of an anonymous file.

Dead owner. When the owner is on this host and the host’s process-identity rule proves it dead (its process id is gone, or the id now belongs to a process with a different start time), the next append removes that exact lock, writes one structured level=warn line to stderr naming the dead owner, and carries on with no wait. A runtime can also pass an onLockTakeover callback to receive that event instead.
Live or unprovable owner. A live owner, an owner on another host, an owner when the host supplies no identity rule, and a lock that names no readable owner (for example one written by an older release) are waited on and never taken over. Elapsed time is not evidence of death.
Wait exhausted. An append that waits its whole budget raises EvidenceLockTimeoutError (code EVIDENCE_LOCK_TIMEOUT, with the lock path and the holder). The tool host refuses to run a call whose started receipt cannot be recorded, so a run is never continued unaudited. If only the finished receipt is blocked after the tool ran, the tool’s output is still returned, the typed error is reported through onTraceError, and the unfinished start is closed as crashed by the next orphan reconciliation. A physical call is stricter: any started-trace failure refuses it, and a finish that cannot be recorded is reported crashed and holds the resource (see Physical-Effect Class).

Directories under the evidence root are verified once per store instance. A steady-state append or blob write walks no directories, and the cache is dropped after any failed write.

Compaction

When the active trace file exceeds the compaction threshold (default: 10 MiB), it is atomically renamed to a numbered segment file (e.g., tool-traces.00000001.jsonl). The next append creates a fresh active file. This keeps individual files manageable while preserving the full history.

Compaction is transparent — reads automatically discover and include segment files.

Predecessor-digest chain

Every trace line carries prev_hash: the SHA-256 of the trace line written before it, or null for the first line the workspace ever wrote. The store seals each line while it holds the write lock, after reading the newest line in history (the active file’s last line or, right after a rotation, the last line of the newest sealed or archived segment). The chain therefore continues across compaction: the first entry of a new active file carries the digest of the last entry of the sealed one. Indexed traces returned by readTraces() are unchanged; the field lives in the stored line.

EvidenceStore.verifyChain() walks the whole history in order and returns { ok: true, entries, unchained_prefix, head_hash, checkpoints_verified, checkpoints_unverified }, or { ok: false, reason, location, message } naming the segment and line:

Reason	Meaning
`evidence-chain.broken-link`	An entry before this line was removed, edited or reordered.
`evidence-chain.unchained-entry`	A line without `prev_hash` appears after chained lines.
`evidence-chain.prefix-missing`	The first line in history names a predecessor that is not there.
`evidence-chain.malformed`	A trace line is not a JSON object.

Lines written before chaining shipped have no prev_hash; they are reported as unchained_prefix and the first chained line links to the last of them. The chain detects changes made by anyone who does not also recompute every later link. A writer with access to the store can still do that, and can remove entries from the tail; signed checkpoints narrow both.

Signed checkpoints

EvidenceStore.checkpointChain(signer) proves the chain verifies, then signs and timestamps its head through the host’s EvidenceSignerPort (timestamp(digest), sign(message), verify(...)), checks the result with the port’s own verify, and only then appends the signed checkpoint to retention/chain-checkpoints.jsonl and returns it for export. A port whose checkpoint does not verify (for example one that timestamps the wrong digest) records nothing. The kernel builds the signed bytes (a domain tag and the canonical JSON payload: head_hash, entry_count, the timestamp and the key id) and names no provider: a host can implement the port with a local key, a timestamping authority or a remote signed-receipt service. createEd25519EvidenceSigner is the local-key adapter; its timestamp is the local clock signed by the same key (authority: local-clock), not a trusted third-party time. A chain that does not verify is never signed. Hold the private key only in the physical-host process (its dedicated identity) or behind an out-of-process signer, never in a process that runs as an agent identity.

Verification needs only public material. createEd25519EvidenceVerifier({ publicKeyPem }) builds an EvidenceCheckpointVerifier (it has verify and nothing else), and every verification path takes that type, so an auditor, a remote log or a CLI run as an agent identity never holds the signing key.

verifyChain({ verifier, checkpoints }) verifies every local record and every checkpoint you pass in checkpoints (exported earlier, or fetched from a remote log), exactly alike. Deleting the local record therefore hides nothing from a verifier that holds an exported checkpoint. Refusals: evidence-chain.checkpoint-signature (the signature or timestamp does not verify), evidence-chain.checkpoint-mismatch (entries the checkpoint covers were rewritten, even if every later link was recomputed), evidence-chain.checkpoint-beyond-head (entries it covered were removed), evidence-chain.checkpoint-malformed (a record or supplied checkpoint is not the exact signed shape; the location is chain-checkpoints.jsonl or supplied-checkpoints with its 1-based position) and evidence-chain.checkpoint-unverifiable (the verifier threw, for example an unreachable authority, or checkpoints were supplied without a verifier; this says nothing about tampering). Without a verifier the local record is counted as checkpoints_unverified and never parsed, so a damaged record cannot break a links-only verification.

Export each checkpoint off the device: tampering is detectable once a checkpoint has left it, and removing entries written after the latest exported checkpoint stays undetectable locally. Known limits: lines written before chaining shipped are bound only through the last of them, so an edit to an earlier legacy line is not detected; signature_key_id is recorded but one verifier checks every checkpoint, so after a key rotation verify older checkpoints with the old key’s verifier; and the payload binds no workspace identity, so with one key shared across workspaces a whole evidence directory, checkpoints included, could be substituted for another’s.

Recovery after a detected break. Once verifyChain reports a break, checkpointChain and a signed prune refuse until it is resolved, by design. Preserve the evidence directory as found, verify it against your exported checkpoints to establish which entries are still vouched for, and record the finding before any repair. A damaged line in chain-checkpoints.jsonl only fails verification with a verifier (checkpoint-malformed, naming the line); move that file aside and verify against the exported checkpoints instead. Do the same for a record the kernel wrote through a port that accepted its own bad checkpoint: it later reports checkpoint-signature, which there means a lax port, not tampering.

Torn trace line. A write cut short by a crash or a full disk leaves a fragment that is not JSON. The next append starts on a new line and is sealed over the newest complete line, never over the fragment. Readers skip the fragment and report it through EvidenceStore.readTornTraceLines() (reason: 'evidence-trace.torn-line', its SHA-256, and whether a later append has ended it), so they keep reading every entry. verifyChain reports it as evidence-chain.malformed with its segment and line. Repair: copy the fragment aside for the record (its digest matches the note), then remove that line from the file. The chain, every exported checkpoint, checkpointChain and a signed prune all verify again with nothing rewritten. A verifier that answers anything other than the boolean true never verifies a checkpoint: false is checkpoint-signature, any other answer checkpoint-unverifiable.

Retention

Nothing deletes evidence on its own, so a workspace grows with every run. pnpm evidence:prune reclaims what no retained receipt needs. It is an operator-only maintenance command (it is not an MCP tool, and agents must not run it with --apply). It does three things and never touches anything an open run, a pending receipt or a retained receipt still needs.

Unreachable blobs. A blob in inputs/ is live when a trace (archived, sealed or active) names it (a started trace’s input_ref, a finished trace’s output_ref, a progress trace’s snapshot_ref, or an artifact), when a live turn-input manifest lists it (messages, tool_catalog), when a pointer (<hash>.manifest, <hash>.redacted) that a trace names points at it, or when the event log mentions its digest (approval inputs, completion and delivery evidence). Any of those references can be a manifest, not only an input. Everything else is unreachable. Names the store does not manage are never touched.
Sealed trace segments are archived, not deleted. A sealed segment past the configured age or size, and at least one day old, whose runs are all positively terminal in the event log, is gzipped into retention/archive/, read back and proven identical, and only then replaced by an empty placeholder (the segment is never removed). Every reader (indexing, replay, verification) reads archives merged by segment number, and archived traces keep their input, output and artifact blobs. With no rule configured, or no readable event log, nothing is archived. When the caller passes a signer (EvidenceStore.pruneEvidence({ ..., signer })), an apply that archives any segment first records a signed retention-anchor checkpoint of the chain head (and a chain-anchored prune-log line), and refuses to archive a chain that does not verify. The pnpm evidence:prune command does not configure a signer yet.
Progress snapshots of completed archived runs. For a run that is positively terminal and archived, progress snapshot blobs are no longer kept; receipts depend on inputs and outputs, not on progress snapshots. Each dropped digest is recorded in the prune log with its run and receipt, and verification reports it as pruned (expected). A snapshot of a run that is open or unknown is never dropped.

pnpm evidence:prune --workspace-root <path>             # dry run: counts and bytes, read-only
pnpm evidence:prune --workspace-root <path> --apply     # prune, then verify every retained receipt

The command prints each setting with where it came from: workspace:, env:, flag:, default or unset. The workspace block software_delivery.evidence_retention (or the default) is the base, and a flag or environment variable can only make retention more conservative than it, never less; only the workspace key switches segment archiving on (see the configuration reference). A blob younger than min_blob_age_ms (default one hour, at least ten minutes on an apply) is kept even when unreachable, because a writer stores a blob before it appends the trace that names it.

Missing traces/. Removing the traces/ directory does not hide archived history. Readers still read the archives the prune log recorded, check each against its recorded digest, and raise an integrity error if a recorded archive is missing. An archive the log never recorded is refused. A workspace with no traces/, no prune log and no archive reads as empty.

Compatibility. After a segment is archived its file stays, empty, as a placeholder, so a kernel that predates archives (a long-lived server that has not restarted since the upgrade) still counts and numbers segments correctly; it just cannot read the archived history until it restarts. Run the command as the workspace’s own OS user, because it reads the event log through a read-only SQLite connection that may create empty -wal and -shm files beside the database.

Safeguards. Segments are never archived sooner than one day. A retained blob that does not hash to its name, a malformed manifest, or a pointer that names no digest stops the prune before anything changes. An unreadable event log stops an apply. Every --apply appends to retention/prune-log.jsonl: who asked (operator, session, work unit when known) and with which settings, the SHA-256 and the receipt and run ids of each archived segment, and the final counts. The full threat model is in the engineering documentation (kernel-evidence-reconciliation).

Concurrency and crash safety. The deciding scans run under the same lock appends take, held only for the active file and the final decision. Blobs are not deleted directly: each is first renamed to <name>.pruning, the traces are scanned again after settle_ms (at least five seconds on an apply), any quarantined name a trace now reaches is renamed back, and only then are the rest removed. A writer that reuses a blob refreshes its age, and a prune bumps an epoch so a long-lived writer re-checks what it remembers. A kill at any point leaves a workspace whose retained receipts verify; a store restores a killed prune’s leftovers before it reads (and again if a verification fails), and a rerun completes.

Protected host binding

Evidence code in @hellmai/lumenflow-kernel addresses files with immutable ProtectedStatePath intent values and the backend-neutral ProtectedStatePort. The normal local runtime factory binds that port through createProtectedStateAdapter() in @hellmai/lumenflow-host. The kernel does not derive a host path from a receipt, task, run, or workspace identifier, and never selects an adapter itself.

The binding is required. A custom event-log factory must expose its protected-state binding or runtime initialization fails closed; the kernel does not fall back to direct filesystem access. Callers also cannot replace the bound task-spec or evidence roots with taskSpecRoot or evidenceRoot options. Tests and explicitly ephemeral hosts can bind the in-memory adapter instead. The public kernel evidence barrel exports only the port-backed evidence store. Node filesystem compatibility helpers belong to @hellmai/lumenflow-host and are never evaluated by importing @hellmai/lumenflow-kernel/evidence.

createProtectedStateAdapter() selects one of two mechanisms by host, both satisfying the same confinement, owner-isolation, atomic-replacement, and fail-closed contract:

Linux and macOS — the descriptor-pinned PosixProtectedStateAdapter. It opens the configured root and every ancestor directory with O_DIRECTORY | O_NOFOLLOW, keeps the selected parent descriptor open, and performs the leaf operation through that held descriptor. Exclusive creation, append, replacement, rename, range reads, and removal therefore stay attached to the held directory even if an attacker renames the visible parent and substitutes a symlink. Leaf opens use O_NOFOLLOW; files with more than one hard link are rejected before reads or mutation.
Windows — the canonical-path WindowsProtectedStateAdapter. Node’s fs module exposes no O_NOFOLLOW/O_DIRECTORY/descriptor-relative open on win32, so confinement is instead an lstat check immediately before every ancestor and leaf use, plus OS-native realpath canonicalization against the confined root (catching a junction or reparse point an lstat symlink check alone might miss). Owner isolation uses a Windows DACL (icacls) and atomic replacement uses the shared bounded-retry MoveFileEx wrapper; both are host chokepoints this adapter reuses rather than reimplementing.

Durability costs follow what a call creates. On Linux and macOS a write fsyncs a parent directory only when that call created a directory entry in it; walking directories that already exist adds no fsync. A directory whose parent sync was interrupted is synced again by the next write through it, so a newly created file and its new parent directories are durable before the call returns. One governed tool call (two content-addressed blobs and two trace lines) costs 6 fsyncs on a warm store, down from 54. The trace lock is a file created with an exclusive create. If the holder releases the lock between a competing writer’s failed create and its validation, the competing writer sees an ordinary lost create and retries; it is never reported as a hard-linked state file. A file with more than one hard link is still refused.

ProtectedStateConfinementError is the adapter-facing failure on every platform. It includes a stable code and the logical path intent, but never the configured host root or an escaped target path. A plain lexical-plus-realpath check followed by an ordinary path open is not a conforming port.

Incremental hydration

The evidence store maintains an in-memory index for fast lookups. On first access, it reads all segment files and the active file to build the index. On subsequent accesses, it reads only new bytes appended since the last read — making the hot path O(new data), not O(total data).

Crash Recovery

If the kernel crashes mid-execution (between TOOL_CALL_STARTED and TOOL_CALL_FINISHED), the evidence store detects the orphaned started trace on next startup and synthesizes a finished trace with result: crashed. This works even when progress traces already exist for the same receipt, so streaming calls still reconcile to a complete audit trail.

Several runtimes on one evidence root

More than one runtime can share an evidence root (concurrent agents or CLI processes in one workspace, or two hosts in one process). Reconciliation therefore closes a started receipt only when its owner is provably dead. Each evidence store mints its own owner (a random instance_id, the process pid and the host name) and stamps it on every started trace it appends. A caller can never supply it: any owner on the entry you pass is discarded.

Owner of the unfinished receipt	Reconciled to `crashed`?
The runtime that is shutting down	Yes, its in-flight calls are abandoned
Same host, host identity port proves it dead (pid gone or reused)	Yes
Same host, still running	No, the call is genuinely in flight
Same host, no host identity port (kernel-only use)	No, the kernel cannot judge it
A different host	No, local liveness says nothing about a remote process
No owner (written by an earlier release)	Only once the trace is older than 24 hours

The kernel never probes a pid itself, because a pid alone is not an identity: the operating system reissues freed pids. The host supplies the rule through the optional ProtectedStateBinding.processIdentity port, and createLocalEventLogFactory binds it to the host’s pid-plus-start-time check, so Windows, macOS and Linux behave as that check does. The owner records the start time next to the pid; a pid now held by a process with a different start time counts as dead. On a platform where the host has no start-time rule (macOS), only a vanished pid proves death.

Limitation of kernel-only use. A runtime built without a host binding (no processIdentity port, for example a custom event-store factory or an in-memory test binding) reconciles only its own receipts when it closes. A crashed same-host runtime’s receipts stay unfinished until a runtime with the host port starts. This is deliberate: without the host rule the kernel cannot tell a live owner from a dead one, and guessing would mark live calls crashed.

A synthetic crashed finish never shadows a real one. If both lines exist for a receipt, the indexed view (inspectTask and the trace indexes) shows the real finish whichever was appended first, and both lines stay in the append-only file for audit. The in-memory index also notices traces another runtime appended between its own reads and writes.

Querying Evidence

Evidence can be queried by task:

readTracesByTaskId(taskId) — returns all traces for a specific task
readTraces() — returns all traces across all tasks
inspectTask(taskId) — returns the full task inspection including receipts and policy decisions

When a task completes, its in-memory index entries are pruned to free memory. The on-disk records are permanent.

Why This Matters

The evidence store answers three questions that traditional logging cannot:

What was the agent asked to do? — Content-addressed input blobs provide the exact request, verifiable by hash.
Was the action authorized? — Scope intersection results and policy decisions are recorded on every call.
What actually happened? — Success, failure, denial, and crash outcomes are all captured with timing.

The fingerprint is produced from the kernel-minted authority, not from a caller-supplied evidence payload. Mutating a selector object, capability descriptor, scope collection, pack config, or policy profile after issuance cannot rewrite the recorded authority. Canonical trace correlation holds across a composed Connected Compute host boundary, and a durable, idempotent outbox with restart reconciliation delivers receipts in that boundary.

This is the foundation for compliance, audit, and post-incident analysis in agent-driven workflows.

Pack-owned workflow continuations are a separate concern. For example, the agent-runtime pack stores session resume state and wakeup continuations under .agent-runtime/workflow/, while the evidence store remains the immutable record of actual tool executions.

Next Steps

Kernel Runtime — How evidence recording fits into the tool execution pipeline
Policy Engine — How policy decisions are recorded in evidence
Scope Intersection — How scope enforcement results are captured