App Backends Export Format (neutral-jsonl-v1)

neutral-jsonl-v1 is the engine-neutral export of an app backend: its tables, rows, primary keys, sequences, row-security flags, grants and policies, as plain text files. It is written by the postgres-url adapter and by any adapter that declares exportFormats: ['neutral-jsonl-v1'].

An export is yours to keep and to check. This page specifies the format completely, so you can verify an export and load it into a plain Postgres database with the tools the pack ships, or with your own code, without a control plane, a LumenFlow workspace, a licence key, the pack’s meta schema or any network access beyond the target database.

The key words MUST, MUST NOT, SHOULD and MAY are used as in RFC 2119.

Contents of an export

An export is a set of named text objects. In a directory they are files; in an object store they are keys. Names use / as the separator.

Object	Count	Holds
manifest.json	exactly one	The description of the export: schema, digests, what is missing
sequences.jsonl	exactly one	One line per sequence with its position
tables/<name>.jsonl	one per table	One line per row

All objects are UTF-8 text, without a byte order mark. Line separators are a single line feed (U+000A). A reader MUST NOT normalise anything (line endings, Unicode, whitespace, number formatting) before it hashes or parses an object.

Table object names

The object for a table is tables/ followed by the percent-encoded display name, followed by .jsonl.

• The display name of a table is its bare name when its schema is public, and <schema>.<table> otherwise.
• Percent-encoding is the one of ECMAScript encodeURIComponent: every UTF-8 byte is written as %XX (uppercase hexadecimal) except the unreserved characters A-Z a-z 0-9 - _ . ! ~ * ' ( ).

Examples: table kv in public is tables/kv.jsonl. Table Tab/le in schema my schema is tables/my%20schema.Tab%2Fle.jsonl.

The manifest names each table object in its file field. A reader MUST reject a manifest whose file is not exactly the name this rule derives from the table’s schema and table, and MUST reject a manifest that lists the same table twice.

manifest.json

manifest.json is one JSON object. The producer writes it with two-space indentation and a trailing line feed, but a reader MUST accept any valid JSON text. Every object below is strict: a reader MUST reject a key that is not listed. There is no extension space inside a neutral-jsonl-v1 manifest; a change to the format gets a new format value.

Field	Type	Meaning
format	string	Exactly neutral-jsonl-v1. Any other value is an unknown format and the export MUST be refused.
exportId	string, 8 to 128 characters	Identifies the export. An importer into a managed backend records it to refuse a second import of the same export.
createdAt	string	When the export was made, ISO 8601 UTC. Informational.
expiresAt	string	When the producer’s copy of the objects stops being kept. Informational; a verifier MUST NOT refuse an export for it.
schema	object	{ tables, sequences }, below.
unsupported	array	What the export does not carry, below.

schema.tables

An array with one entry per table. The producer orders it by schema name then table name, and a reader MUST NOT depend on the order for correctness. An entry has these fields:

Field	Type	Meaning
schema	string, 1 to 63	The schema of the table.
table	string, 1 to 63	The table name.
file	string	The object name, derived as above.
rows	integer, 0 or more	The number of rows in the object.
objectSha256	64 lowercase hex digits	The SHA-256 of the entire table object. It protects the transfer.
chainedSha256	64 lowercase hex digits	The chained-sha256-v1 hash of the rows. It proves the data after an import.
columns	array, at least 1	The columns in table order, below.
primaryKey	array	Primary-key columns in key order, each { name, collatable }. Empty when the table has no primary key.
rowSecurity	boolean	True when row-level security is enabled on the table.
grants	array	{ grantee, privilege }. The producer emits grants to PUBLIC, anon and authenticated only; a reader accepts any role name. Privilege is one of SELECT, INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER, MAINTAIN.
policies	array	Row-security policies, below.

Names (schema, table, column, role and policy names) are 1 to 63 characters. They are never interpreted: an importer quotes every one as an SQL identifier (double quotes, with embedded double quotes doubled) and MUST refuse a name that is empty, contains a NUL character or is longer than 63 bytes in UTF-8, because the server would silently truncate it.

A column is { name, type, notNull, expression, identity, generated }:

Field	Type	Meaning
name	string	The column name.
type	string, at most 128 characters	The type as Postgres format_type prints it, for example integer, numeric(12,2), timestamp with time zone, text[].
notNull	boolean	The column is NOT NULL.
expression	string, at most 4096 characters, or null	The default expression, or the generation expression when generated is true. SQL text as pg_get_expr prints it.
identity	always, by_default or null	The column is an identity column of that kind.
generated	boolean	The column is a stored generated column; its value is computed, not loaded.

type MUST match this grammar, which admits built-in type names and refuses everything else: one or more words separated by single spaces, each word [a-z_][a-z0-9_]* optionally followed by a modifier list (n) or (n,m) of digits, then zero or more array suffixes [] or [n]. A type outside the grammar (a user-defined type, a quoted or schema-qualified name) is a malformed manifest.

A policy is { name, permissive, command, roles, using, withCheck }: permissive is a boolean; command is one of ALL, SELECT, INSERT, UPDATE, DELETE; roles is an array of role names (empty, or containing public, means every role); using and withCheck are SQL expression text of at most 4096 characters, or null.

schema.sequences

An array of { schema, name, lastValue, identity }, one per sequence, ordered by schema then name. lastValue is the sequence’s last value as a decimal integer string (-?[0-9]+), or null when the sequence has never been used. identity is true for the sequence behind an identity column, which is created together with its table; the other sequences are created by name.

unsupported

An array of { kind, name }: things the source has that the format does not carry. kind is one of index (any index that is not a primary key and does not back a constraint), constraint (foreign key, unique, check and exclusion constraints), view, materialized view, function and trigger; name is the qualified name. A reader MUST accept any kind string of up to 64 characters. User-defined types and extension-owned objects are not listed. They come back through migrations, so an importer reports the list and does not guess at them.

Row encoding

Each line of a table object is one row, encoded as the text Postgres produces for to_jsonb(row)::text, with the session settings pinned like this before the rows are read:

set local timezone = 'UTC';
set local datestyle = 'ISO, YMD';
set local intervalstyle = 'postgres';
set local extra_float_digits = 1;
set local bytea_output = 'hex';
set local row_security = off;

So a line is a JSON object whose keys are the column names, in table order, including generated columns. Postgres writes ": " after each key and ", " between members; a reader MUST treat the line as opaque text for hashing and MUST NOT re-serialise it. The usual mappings hold: integers and numeric are JSON numbers, boolean is true or false, bytea is a string "\\x..." of hexadecimal digits, timestamps and dates are ISO 8601 strings in UTC, json and jsonb values are embedded as JSON, arrays are JSON arrays, and SQL NULL is JSON null.

A table object is the lines of its rows in the order below, each followed by one line feed. A table with no rows is a zero-length object. There are no blank lines. A non-empty object that does not end with a line feed is malformed.

Row order, which the chained hash depends on:

• With a primary key, by the primary-key columns in key order. A column whose manifest entry has collatable: true (a column with a collation, such as text) is compared with COLLATE "C", byte by byte, so B sorts before _ and _ before a.
• Without a primary key, by the line itself (to_jsonb(row)::text) compared COLLATE "C".

A verifier that only hashes the lines does not need to sort: it hashes them in file order. An importer proves the data after loading by reading the table back in this order.

Table digests

objectSha256

The SHA-256, as 64 lowercase hexadecimal digits, of the whole table object: the UTF-8 bytes of every line including each line feed. The hash of an empty object is e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.

chained-sha256-v1

chainedSha256 is computed from the rows alone, so it can be recomputed from the JSONL and, after an import, from the database. Let row be the line without its line feed.

1. Cut the rows, in order, into chunks of 10,000. The last chunk may be shorter.
2. The digest of a chunk is the SHA-256 of the concatenation of row followed by \n for every row in it, as lowercase hexadecimal.
3. The table hash is the SHA-256 of the concatenation of each chunk digest followed by \n, in order, as lowercase hexadecimal.
4. A table with no rows has no chunks, so its hash is the SHA-256 of the empty string.

Worked example: a table with two rows {"id": 1, "body": "one"} and {"id": 2, "body": "two"} has one chunk. Its digest is the SHA-256 of

{"id": 1, "body": "one"}\n{"id": 2, "body": "two"}\n

which is 259469db026b6de348bfa0647d2fdcc4d56b728695905588193047b2b6bfc3b1. The table hash is the SHA-256 of that digest followed by \n, which is 41fe0ec9ecfe2abca6ea394fe02c68d7d02d11388a137d656a9f5ee8202c4623. In this example the object hash and the chunk digest are equal because the object holds exactly the bytes the chunk hashes; they differ for any table that has more than one chunk.

The algorithm name is chained-sha256-v1. Verification reports carry it so a future algorithm can be told apart.

sequences.jsonl

One line per sequence: the compact JSON (no spaces) of {"schema":…,"name":…,"lastValue":…,"identity":…} with the keys in that order, followed by a line feed, in the order of schema.sequences. With no sequences the object is empty. It MUST list exactly the sequences of the manifest, with the same positions; it carries no information the manifest does not, so a mismatch is a corrupt export. A reader MAY compare the parsed values instead of the text.

Verifying an export

A verifier needs only the objects. It MUST fail closed: any check below that does not pass, and any object it cannot read, makes the export unverified. Do not import an export that does not verify.

1. Read manifest.json. If it cannot be read, refuse.
2. If it is JSON with a format other than neutral-jsonl-v1, refuse as an unknown format.
3. Validate the manifest against the field rules above, including the strict key sets, the derived file names, and the objectSha256 and chainedSha256 shapes. If it does not validate, refuse.
4. For each table: read its object (refuse if missing); compare its SHA-256 with objectSha256; check it is well-formed (empty, or lines that each end with a line feed, none blank, each a JSON object); count the lines and compare with rows; recompute the chained hash and compare with chainedSha256.
5. Read sequences.jsonl (refuse if missing) and compare it with schema.sequences.
6. If the source lists its objects, refuse when it holds a tables/ object the manifest does not list.

The objectSha256 check and the chained hash catch different things. An attacker or a fault that changes a row and also rewrites objectSha256 in the manifest is still caught by the chained hash, and by every later check the importer does against the database.

The manifest is not signed

Digests inside an export prove that its objects agree with its manifest. They do not prove who wrote the manifest. If the export crosses a boundary you do not trust, record the SHA-256 of manifest.json when you take the export, over a channel you do trust, and compare it before you import.

The shipped verifier reports these codes: manifest_unreadable, unknown_format, manifest_invalid, object_missing, object_sha256_mismatch, object_malformed, row_count_mismatch, chained_hash_mismatch, sequences_mismatch and unexpected_object.

Importing into plain Postgres

The target is any Postgres server reachable with a role that may create the objects, and that is all it needs to be: the import creates no meta schema, registry, ledger or extension. The importer works in one transaction and, after a successful verification, performs these steps in order. The first failure rolls everything back, so a refused or failed import leaves the target as it was.

1. Pin the rendering settings listed under Row encoding for the transaction.

2. Check the target. If a relation named like any table or sequence in the manifest already exists in its schema (a relation of any kind), refuse with “target not empty”, unless replace was requested. With replace, drop those tables, sequences, views, materialized views and foreign tables with CASCADE (tables first) and carry on; refuse even then if the clashing relation is of another kind.

3. Resolve roles. Collect every role named by a grant (other than PUBLIC) or a policy (other than public). A role that does not exist on the target is not an error: its grants and policies are not applied and are reported as skipped, unless the caller asked to create missing roles, in which case each is created NOLOGIN.

4. Create each non-public schema (CREATE SCHEMA IF NOT EXISTS), then each sequence that has identity false (CREATE SEQUENCE), then each table:

   CREATE TABLE "schema"."table" (<column definitions>, PRIMARY KEY ("a", "b"))

   A column definition is the quoted name, then the type text, then the first that applies: for a generated column with an expression GENERATED ALWAYS AS (<expression>) STORED; for identity always GENERATED ALWAYS AS IDENTITY; for identity by_default GENERATED BY DEFAULT AS IDENTITY; for a column with an expression DEFAULT <expression>. Then NOT NULL when notNull is true and the column is not an identity column. The primary-key clause is omitted when primaryKey is empty.

5. Load the rows. For each table with rows, read its object again, check its SHA-256 and line count, and insert the lines in batches (the shipped importer uses 1,000 rows per statement). Each batch binds one JSON array of the batch’s lines as a parameter:

   INSERT INTO "schema"."table" (<non-generated columns>) [OVERRIDING SYSTEM VALUE]
   SELECT <non-generated columns> FROM jsonb_populate_recordset(null::"schema"."table", $1::jsonb)

   OVERRIDING SYSTEM VALUE is present when any column is GENERATED ALWAYS AS IDENTITY. Generated columns are skipped: the server computes them.

6. Restore access. For each table: ALTER TABLE … ENABLE ROW LEVEL SECURITY when rowSecurity; one GRANT <privilege> ON … TO <grantee> per grant (PUBLIC unquoted, other roles quoted); one CREATE POLICY "name" ON … AS PERMISSIVE|RESTRICTIVE FOR <command> TO <roles> [USING (…)] [WITH CHECK (…)] per policy, with public when roles is empty.

7. Restore sequences. For each sequence with a lastValue: SELECT setval('"schema"."name"'::regclass, <lastValue>, true). A sequence with lastValue null is left as created.

8. Prove the import. Still inside the transaction, read every table back in the row order above and recompute its row count and chained-sha256-v1 hash; read every sequence’s position with pg_sequence_last_value. Any difference from the manifest aborts and rolls the import back.

9. Commit. Then report the manifest’s unsupported list, the replaced objects, any roles created and any grants or policies skipped.

Every statement the importer generates from the manifest contains text that came from the export (type names, default and policy expressions). The importer MUST check that each generated statement is exactly one SQL statement (it splits on semicolons outside quotes, comments and dollar-quoted strings) and refuse otherwise. That check does not make an untrusted manifest harmless: the expressions run with the importing role’s rights, so import only exports you trust, and verify the manifest digest as described above.

Not restored: whatever is in unsupported, table and column comments, ownership, tablespaces, extensions and the data of other schemas. An extension type or function that a column type or default needs MUST be installed on the target before the import.

The shipped tools

The pack ships the verifier and importer as a library subpath and as a command. Both need Node.js 26 or later; only the importer needs the pg package (version 8.11 or later), which is an optional peer dependency of the pack.

Command line

npm install @hellmai/lumenflow-packs-app-backends pg
npx app-backends-export verify ./export
PGURL=postgres://user@host/db npx app-backends-export import ./export

Command	Does
verify <directory> [--json]	Runs the checks above on an unpacked export. Needs no database.
import <directory> [--url <string>] [--replace] [--create-roles] [--batch-rows <n>] [--json]	Verifies, then imports.

The connection for import is --url, else the PGURL environment variable, else DATABASE_URL, else the standard PGHOST, PGPORT, PGDATABASE, PGUSER and PGPASSWORD variables. Prefer the environment to --url, so the password stays out of the process list. --replace drops and recreates clashing tables and sequences. --create-roles creates missing roles NOLOGIN. The command reads no LumenFlow configuration, workspace or licence key and opens no connection but the one to the target.

The exit code is 0 on success, 1 when the export does not verify or the import fails or is refused, and 2 for a usage error. Error text never contains the password or the connection string.

Library

import {
  verifyNeutralExport,
  importNeutralExportIntoPostgres,
  directoryExportSource,
} from '@hellmai/lumenflow-packs-app-backends/export-format';

const source = directoryExportSource('./export'); // or any { get(name), list() }
const report = await verifyNeutralExport(source);
if (!report.ok) throw new Error(report.issues.map((i) => i.message).join('\n'));
const result = await importNeutralExportIntoPostgres(source, process.env.PGURL!, {
  replace: false,
});

verifyNeutralExport(source) resolves to a VerificationReport: ok, format, algorithm, exportId, a tables entry per table (declared and found rows, whether the object hash, the count and the chained hash agree, and the recomputed hash), sequencesOk, the manifest’s unsupported list and issues. It does not throw for a problem with the export.

importNeutralExportIntoPostgres(source, connection, options) verifies first. connection is a connection string, { connectionString } (or {} for the PG* variables), or an open session with a query(text, params) method that the caller owns. Options are replace, createMissingRoles and batchRows. It resolves to a report with the imported tables and their recomputed hashes, the sequences, what was replaced, roles created, grants and policies skipped, and the unsupported list. It throws ExportFormatError with a code: verification_failed (nothing was touched), target_not_empty, unsafe_statement, integrity_failed (rolled back), database or driver_unavailable.

The subpath imports nothing from the postgres-url adapter, the control plane, the kernel or the runtime, and the adapter’s own export and import use the same manifest schema, hash and statement code, so the format cannot drift between them.

Limits

The tools read one table object into memory at a time. Export refuses a backend that holds Postgres large objects, because the format does not carry them: move that data into tables first. The format has no signature, no encryption and no compression; apply those to the files around it.