App Backends Conformance Kit

A written adapter interface cannot stop a provider from reporting that nothing changed after a lost connection, or from sharing an idempotency record between two tenants. The conformance kit attacks the adapter instead. It is exported from @hellmai/lumenflow-packs-app-backends/conformance.

Running the suite

runConformance(subject, options?) runs every case in CASE_CATALOG against a ConformanceSubject, which is { adapter, harness }, and returns a ConformanceReport.

import {
  assertConformant,
  formatReport,
  runConformance,
} from '@hellmai/lumenflow-packs-app-backends/conformance';

const report = await runConformance({ adapter: myAdapter, harness: myHarness });
console.log(formatReport(report));
assertConformant(report); // throws with the full report when the adapter is not conformant

The adapter is your implementation of AppBackendAdapter. The harness is the part only your adapter’s environment can supply. The kit drives the SPI itself; the harness lets it look inside a backend, break the provider on purpose and, for Postgres-family adapters, try to escape.

Options:

• only: an array of case ids. Use it while fixing one case. Cases you do not list are absent from the report, so a filtered run is not evidence of conformance.
• onCase: called with each CaseResult as it finishes.

Each case provisions its own backends under a fresh isolation key from harness.newIsolation(label) and removes them when it ends. A case that leaves a backend behind that deprovision cannot remove fails.

For a Postgres server you operate, use the Postgres adapter and its ready-made harness. For an in-memory run, see the overview.

The harness

ConformanceHarness has these members.

Member	Purpose
clock	The clock your adapter was built with. The kit builds deadlines from it.
advanceClock?(ms)	Moves that clock forward. Needed to prove seven-day operation-record retention; without it c1d.record-retention is harness_incomplete.
newIsolation(label)	A fresh isolation key never used before, so reruns on a shared server cannot collide.
data	A DataProbe: privileged access to the data inside a backend.
faults	A FaultInjector: break the provider on purpose.
objectStore	An ObjectStoreProbe: where exports land and imports read from.
sql?	A SqlProbe: engine-level attack probes. Required when the adapter declares engineFamily: 'postgres'.
canary	A secret you planted as the adapter’s provider credential. It must never surface.
capturedLogs()	Every log line the adapter emitted so far.
dispose?()	Cleanup.

DataProbe supplies migrationFixtures() (engine-native SQL for an owner-scoped table with a policy, a bare table with none, a second table, a failing statement and, optionally, a statement the admission lint must refuse), tableExists, insertRows, readRows, plantMutation and a policy probe (readAs, writeAs) that reads and writes as an end user with a given verified subject, or as the anonymous role. Rows have the shape { id, owner, body }.

FaultInjector can arm(operation, fault) the next matching provider call with one of unavailable_before_send, rate_limited_before_send, capacity_exhausted, quota_exceeded, connection_lost_after_send, slow, partial_after_first_step or internal_fault_with_secret; disarm(), restart() (drop in-process caches, keep durable state), holdNextProvision() and armCrashAfterProviderCallOnProvision().

SqlProbe can attempt a statement as a backend’s own login role, list visibleCatalogNames, list a backend’s loginRoleNames, try connectAcrossBackends and read the role’s privilegeFlags.

ObjectStoreProbe creates sealed sinks (newSink), presents the same bytes as a source (sourceFromSink) and reads every object written (bytesOf) so the kit can scan it for the canary. In production the data plane resolves sealed sink and source references; the kit has no custody, so the harness plays that role.

Reading the report

ConformanceReport has adapterId, startedAt, cases, the counts passed, failed, notApplicable and harnessIncomplete, and conformant.

Each CaseResult has an id, title, kind, status, a reason for anything that did not pass, and durationMs. status is one of:

• passed
• failed: the adapter did the wrong thing. reason says what was expected and what happened.
• not_applicable: the case does not apply to this adapter and reason says why. See below.
• harness_incomplete: the harness lacks a probe the adapter’s declared capabilities require.

conformant is true only when no case failed, no case was harness_incomplete and at least one case passed.

No case is skipped quietly

There is deliberately no skip option. A case either runs, is not_applicable with a stated reason, or is harness_incomplete, which fails the run. A kit that could be told to skip what it cannot pass would pass a bad adapter. If you see HARN, add the missing probe to your harness; do not filter the case out.

formatReport(report) prints one line per case (PASS, FAIL, N/A or HARN, then id, kind, title and reason) and a summary line. To read the pass list, check three things: that CONFORMANT is on the summary line, that every N/A is one you expect, and that the count of passed cases matches the catalogue (CASE_IDS).

The cases a run reports as not_applicable:

• The policy cases (c3.policy-default-deny, c3b.policy-owner-scoped) when capabilities.rowLevelPolicies is false. An adapter that cannot host the policy contract cannot pass them, and cannot be selected for hosted use.
• c5.restore-roundtrip and c5d.swap-discard when capabilities.restorePoints is false. c7.capability-honesty then asserts that the unsupported operations answer unsupported_capability.
• c4e.migration-forbidden when the harness declares no forbiddenSql, because the engine has no admission lint to attack.
• The engine-level attack cases (c2c.cross-backend-credentials, c2d.opaque-names, s3.engine-escapes, s3.shared-roles and s3.least-privilege) when the engine family is not postgres and the harness has no sql probe. For a postgres adapter a missing sql probe is harness_incomplete, not not_applicable.

Case kinds

• positive: the adapter must do the right thing.
• negative: a deliberately bad input or defect must be caught. For example, a planted one-row mutation must fail verify and name the table.
• adversarial: the adapter must refuse an attack, such as cross-tenant access or an engine escape.

The catalogue has 32 cases. The ids are the public vocabulary of the kit and never change meaning; a new check gets a new id.

Negative cases

Case id	What it proves
c1.mismatch	The same key with another hash is idempotency_mismatch and changes nothing
c1e.request-hash-binding	A context whose requestHash does not match the call is refused
c4b.migration-atomic	A bundle with a failing migration applies none of it
c4c.migration-edit-refused	An edited, already applied migration is refused
c5b.verify-detects-mutation	A planted one-row mutation fails verify and names the table
c5c.verify-detects-extra-row	An extra row fails verify on count and hash
c7.capability-honesty	Each false capability yields unsupported_capability, never a silent no-op
c9.error-taxonomy	Injected provider faults map to the right code, retryable and outcome

Adversarial cases

Case id	What it proves
c1b.tenant-scoped-keys	Two tenants reusing one idempotency key get independent results, no mismatch and no shared handle
c1c.concurrent-replays	Concurrent replays of one key run exactly one operation
c2.isolation	Every handle operation with the wrong isolation key is not_found and changes nothing
c2b.list-scoping	List shows only the caller’s backends; adapter scope needs operator privilege
c2c.cross-backend-credentials	One backend’s credentials cannot connect to another backend’s database
c2d.opaque-names	Catalog enumeration from an app role reveals only opaque names
c3.policy-default-deny	A table without policies is unreadable and unwritable by the end-user roles
c4e.migration-forbidden	A statement outside the allowlist is refused and nothing is applied
c8.secret-non-disclosure	A canary credential never appears in any result, error, log line or manifest
s3.deadlines	An expired deadline does no work; a deadline passing mid-call is timeout with unknown outcome
s3.crash-mid-provision	A crash mid-provision is reconciled through list, status and reconcile
s3.engine-escapes	Extension, program, file and large-object escapes are refused
s3.shared-roles	SET ROLE to another backend and ALTER ROLE on the shared end-user roles are refused
s3.least-privilege	Backend roles hold no superuser, create, bypass or file-access privilege

Positive cases

Case id	What it proves
cap.document	The capability document is valid, consistent and honest about optional methods
c1.lifecycle	Provision, status and deprovision, with opaque handle references
c1.replay	The same key and hash returns the first result, a failure included
c1d.record-retention	An operation record still answers a replay just under seven days later
c3b.policy-owner-scoped	A per-row predicate on the verified claim admits only the owner’s rows
c4.plan-readonly	Planning a migration does not change the database
c4d.migration-ledger	The ledger is complete, ordered and carries the digests
c5.restore-roundtrip	Restore point and point-in-time round trip: a new handle, equal to the marked state, original untouched
c5d.swap-discard	Swap promotes the restored candidate and keeps the previous; discard removes it
c6.export-import-roundtrip	Export then import round trips in every declared format, verified by counts and checksums

Proving the kit catches a bad adapter

The reference adapter in /testing can be built with one of the defects in MEMORY_DEFECTS, each of which makes it behave like a plausible buggy implementation (for example idempotency-ignores-tenant, restore-in-place, verify-always-passes, leaks-secret-in-error, ignores-deadline). MEMORY_DEFECT_EXPECTED_FAILURES names the case ids that must then fail. The kit’s own tests run every defect and require that the suite catches it. You can do the same for your adapter: introduce a known defect and confirm the matching case fails.