Data Analytics

A local governed quality check of supplied tabular snapshots. The data:check read tool parses bounded CSV text or JSON rows, profiles each column, evaluates explicit structural checks and returns an inspectable report bound to source and configuration digests. It never fetches data, connects to a warehouse, writes to a provider or judges whether data is correct. A passing report only states that the configured structural checks passed.

The ESM package exports checkDataset, DataConfigSchema and DataInputSchema; the check subpath exposes the check function. The build bundles the handler and its dependencies, including the pinned csv-parse 7.0.3 parser, into one artifact for the kernel sandbox, which exposes only the verified handler file. The upstream MIT notice and provenance ship in THIRD_PARTY_NOTICES.txt. The bundled host explicitly registers trusted-host-adapter:data-analytics. The installed runtime supplies canonical pack_config; caller input and metadata cannot set policy. No pack-specific CLI command is declared.

Installation and dispatch

Install the licensed CLI using the licence setup guide, initialize a workspace, then use the supported pack and configuration commands:

pnpm exec pack-install --help
pnpm exec pack-install --id data-analytics --source builtin --version 1.0.7
pnpm exec config-set --help
pnpm exec config-set --key data_analytics.required_columns --json-value '["id","email"]'
pnpm exec config-set --key data_analytics.numeric_ranges --json-value '[{"column":"score","min":0,"max":100}]'
pnpm exec config-get --key data_analytics.required_columns
pnpm exec config-unset --key data_analytics.numeric_ranges

Run each command’s help before first use. Repository scripts expose the same commands as pnpm pack:install and pnpm config:set/get/unset. Always configure through these commands; the pack pin and consumer settings have separate owners.

A host binds a claimed task/run/session and dispatches data:check through its governed tool interface, for example caller.executeToolWithReceipt('data:check', input, selector). Enroll network posture off in the complete workspace security and selected lanes roots through config-set --json-value, preserving existing scopes and deny rules, and declare it on the persisted task. All four workspace/lane/task/tool scope sets must have a nonempty intersection; an empty set denies execution. The installed host resolves the pinned CLI bundle, freezes canonical workspace configuration, and passes it to the subprocess worker. A local execution host binds exactly one persisted caller batch, so use one host per claimed run and close it afterwards. Reports and execution receipts are separate evidence. The pack adds no standalone check CLI command. See the tool reference, the event reference and the consumer configuration guide.

Consumer policy

Configure data_analytics with supported config:set, config:get and config:unset commands after pack:install --id data-analytics --source builtin --version 1.0.7. The shipped strict schema accepts only:

Key	Default	Meaning
`max_rows`	`10000`	Data rows per source, from 1 to 100000
`max_bytes`	`1048576`	UTF-8 bytes of the serialized input before parsing, from 1 to 4194304
`max_columns`	`100`	Columns per source, from 1 to 1000
`required_columns`	`[]`	Columns every source must contain
`not_null_columns`	`[]`	Columns with no null cell in any row
`unique_columns`	`[]`	Columns where no non-null value repeats
`numeric_ranges`	`[]`	`{column, min?, max?}` finite bounds, one entry per column
`max_reported_failures`	`100`	Failures listed per check, from 1 to 1000, within the 1000-failure whole-report budget; counts and outcome stay full

Effective defaults and sorted unique column lists are returned with a SHA-256 configuration digest. Unknown settings, reserved column names (__proto__, constructor, prototype), a range without min or max, min above max, and invalid bounds are refused.

Supplied input

{
  "sources": [
    { "source_id": "orders", "format": "csv", "content": "id,score\n1,10\n2,20\n" },
    { "source_id": "events", "format": "json", "rows": [{ "id": 1, "score": 5 }] }
  ]
}

Source IDs are unique, 1–128 characters, start with an ASCII letter or digit and then contain letters, digits, dot, underscore, colon or hyphen. At most 16 sources are accepted. The envelope and every source object are strict; unknown fields, including pack_config and metadata, are refused. JSON row objects are free-form but must be flat scalar cells with no reserved keys.

Whole-report limits are fixed and not configurable: at most 1000 failures are itemised across all checks, first-come in report order (reported_failure_limit). Each check takes up to max_reported_failures samples from what remains of that 1000-sample budget, so once earlier checks have listed 1000 samples in total (with the default of 100, after ten checks that each have 100 or more failures) every later check lists none. Every check keeps its full failure_count and a truncated flag, and every failing check carries first_failure_row (the earliest failing row number, or null for a column-level failure such as a missing column), which is never shed. A JSON snapshot may need at most 2000000 table cells (rows times distinct keys). The whole encoded report must be at most 900000 bytes so it fits the 1 MiB subprocess output limit. If it is larger, itemised samples are shed first: the limit is halved (1000, 500, 250, … down to 0, keeping the first samples in report order) until the report fits, so reported_failure_limit can drop below 1000. Only when the report still exceeds 900000 bytes with no samples at all (profile, check metadata and first_failure_row fields alone) is it refused with DATA_INPUT_LIMIT. A report of exactly 900000 bytes is accepted.

CSV (content): RFC 4180 text with a header row. A leading BOM, CRLF or LF line ends, quoted commas and newlines, and doubled-quote escapes are supported. Every record must have the header’s field count. Unterminated or stray quotes, empty, duplicate or reserved header names, and header-only files are refused. An empty field is null. A numeric cell is an optionally signed decimal with an optional exponent; nothing else is coerced.
JSON (rows): a nonempty array of flat objects whose values are strings, finite numbers, booleans or null. Columns are the union of keys in first-seen order; an absent key and null are null. Only JSON numbers are numeric.

Checks and evidence

Per source the report lists columns, a per-column profile (null_count, distinct_count), and one entry per configured check: required_column, not_null, unique (nulls are ignored; every row repeating an earlier value is listed with first_row), and numeric_range (null cells are skipped; other non-numeric cells fail). Failures carry the 1-based data row, column, supplied value (first 256 characters) and a reason, so each failure can be reproduced from the supplied snapshot alone. A configured column missing from a snapshot is one column-level missing_column failure.

Reports retain each source ID with a SHA-256 digest of the supplied content text or of JSON.stringify(rows) (key order matters). The effective policy and its configuration_digest are returned with the outcome. Output is deterministic and independent of check configuration order.

The manifest rejects malformed outer envelopes and unknown root fields before dispatch with INVALID_INPUT and preserves raw snapshots for the handler. Malformed snapshots return DATA_INPUT_INVALID, budget overflow (bytes, rows or columns) returns DATA_INPUT_LIMIT, and invalid canonical policy returns DATA_CONFIG_INVALID. Failures never silently scan a prefix.

Billing and marketplace charging remain future, unvalidated capabilities.