Evidence Store

The Evidence Store records every tool call that passes through the kernel. It produces immutable, append-only records that prove what was requested, what was allowed, and what happened — regardless of whether the tool succeeded, failed, or was denied.

What Gets Recorded

Every tool call generates:

1. TOOL_CALL_STARTED

Written before the tool executes. Contains:

• receipt_id — unique identifier for this tool call
• task_id — which task this call belongs to
• run_id / session_id — the exact persisted execution selected by the call
• tool_name — which tool was called
• capability_fingerprint — hash of the full minted authority context, including caller, selector, workspace/config hash, lane, runtime version, capability, pack integrity, policy profile, scopes, and deny overlays
• input_hash — SHA-256 hash of the serialized input
• input_ref — protected-state URI for the content-addressed input blob
• scope_enforced — the computed scope intersection result
• timestamp — when the call was initiated

2. TOOL_CALL_PROGRESS (optional, repeatable)

Written between the started and finished entries when a tool reports progress snapshots. Contains:

• receipt_id — matches the started entry
• sequence — monotonic snapshot order for this receipt
• payload — partial progress data such as streaming turn snapshots
• timestamp — when the snapshot was recorded

3. TOOL_CALL_FINISHED

Written after the tool completes (or fails). Contains:

• receipt_id — matches the started entry
• result — one of success, failure, denied, or crashed
• duration_ms — execution time
• output_hash — SHA-256 hash of the output
• policy_decisions — every policy rule that was evaluated, with its decision
• artifacts_written — files the tool reports having created or modified

Non-streaming tools usually produce only started and finished entries. Streaming tools such as agent-runtime:execute-turn can emit multiple progress entries before the final result.

Note

Even denied tool calls produce evidence. If scope intersection fails or a policy denies the call, the started trace is still written. This means the evidence store records attempted actions, not just successful ones — critical for security auditing.

Content-Addressed Inputs

Before a tool executes, its input is:

1. Serialized to a canonical JSON string (deterministic key ordering)
2. Hashed with SHA-256
3. Written to evidence/inputs/<hash> using exclusive-create mode (O_EXCL)

This means:

• Identical inputs produce the same hash — deduplication is automatic
• Inputs are immutable — the exclusive-create flag means an existing blob is never overwritten
• Inputs are verifiable — given a trace entry’s input_hash, anyone can recompute the hash from the blob and confirm it matches

Storage Model

The evidence store uses append-only JSONL (JSON Lines) files:

.lumenflow/kernel/evidence/
  traces/
    tool-traces.jsonl           ← active append file
    tool-traces.lock            ← file-based mutex
    tool-traces.00000001.jsonl  ← compacted segment
    tool-traces.00000002.jsonl  ← ...
  inputs/
    a1b2c3d4e5f6...             ← content-addressed blobs (SHA-256 filename)

Append-only guarantee

The evidence store never modifies existing records. New entries are appended to tool-traces.jsonl. Reads never acquire the file lock — only writes hold the mutex.

Compaction

When the active trace file exceeds the compaction threshold (default: 10 MiB), it is atomically renamed to a numbered segment file (e.g., tool-traces.00000001.jsonl). The next append creates a fresh active file. This keeps individual files manageable while preserving the full history.

Compaction is transparent — reads automatically discover and include segment files.

Protected host binding

Evidence code in @lumenflow/kernel addresses files with immutable ProtectedStatePath intent values and the backend-neutral ProtectedStatePort. The normal local runtime factory binds that port to LinuxProtectedStateAdapter in @lumenflow/host; the kernel does not derive a host path from a receipt, task, run, or workspace identifier.

The binding is required. A custom event-log factory must expose its protected-state binding or runtime initialization fails closed; the kernel does not fall back to direct filesystem access. Callers also cannot replace the bound task-spec or evidence roots with taskSpecRoot or evidenceRoot options. Tests and explicitly ephemeral hosts can bind the in-memory adapter instead. The public kernel evidence barrel exports only the port-backed evidence store. Node filesystem compatibility helpers belong to @lumenflow/host and are never evaluated by importing @lumenflow/kernel/evidence.

On Linux the adapter opens the configured root and every ancestor directory with O_DIRECTORY | O_NOFOLLOW, keeps the selected parent descriptor open, and performs the leaf operation through /proc/self/fd/<dirfd>/<leaf>. Exclusive creation, append, replacement, rename, range reads, and removal therefore stay attached to the held directory even if an attacker renames the visible parent and substitutes a symlink. Leaf opens use O_NOFOLLOW; files with more than one hard link are rejected before reads or mutation.

ProtectedStateConfinementError is the adapter-facing failure. It includes a stable code and the logical path intent, but never the configured host root or an escaped target path. A plain lexical-plus-realpath check followed by an ordinary path open is not a conforming port.

Caution

The descriptor-held adapter currently supports Linux filesystems with /proc/self/fd, O_DIRECTORY, and O_NOFOLLOW. It fails closed with UNSUPPORTED_PROTECTED_STATE_PLATFORM elsewhere. A future platform adapter must prove equivalent parent and leaf swap resistance before advertising the same guarantee.

Incremental hydration

The evidence store maintains an in-memory index for fast lookups. On first access, it reads all segment files and the active file to build the index. On subsequent accesses, it reads only new bytes appended since the last read — making the hot path O(new data), not O(total data).

Crash Recovery

If the kernel crashes mid-execution (between TOOL_CALL_STARTED and TOOL_CALL_FINISHED), the evidence store detects the orphaned started trace on next startup and synthesizes a finished trace with result: crashed. This works even when progress traces already exist for the same receipt, so streaming calls still reconcile to a complete audit trail.

Querying Evidence

Evidence can be queried by task:

• readTracesByTaskId(taskId) — returns all traces for a specific task
• readTraces() — returns all traces across all tasks
• inspectTask(taskId) — returns the full task inspection including receipts and policy decisions

When a task completes, its in-memory index entries are pruned to free memory. The on-disk records are permanent.

Why This Matters

The evidence store answers three questions that traditional logging cannot:

1. What was the agent asked to do? — Content-addressed input blobs provide the exact request, verifiable by hash.
2. Was the action authorized? — Scope intersection results and policy decisions are recorded on every call.
3. What actually happened? — Success, failure, denial, and crash outcomes are all captured with timing.

The fingerprint is produced from the kernel-minted authority, not from a caller-supplied evidence payload. Mutating a selector object, capability descriptor, scope collection, pack config, or policy profile after issuance cannot rewrite the recorded authority. WU-3361 verified canonical trace correlation in a composed Connected Compute host boundary. WU-3409 added the durable, idempotent outbox and restart reconciliation needed for production receipt delivery in that boundary.

This is the foundation for compliance, audit, and post-incident analysis in agent-driven workflows.

Pack-owned workflow continuations are a separate concern. For example, the agent-runtime pack stores session resume state and wakeup continuations under .agent-runtime/workflow/, while the evidence store remains the immutable record of actual tool executions.

Next Steps

• Kernel Runtime — How evidence recording fits into the tool execution pipeline
• Policy Engine — How policy decisions are recorded in evidence
• Scope Intersection — How scope enforcement results are captured