Skip to content
bricksentry

Validation

Validation: what we tested on real Databricks

What bricksentry has been tested against on real Databricks, with the numbers, including the results that weren't perfect and every bug the live tests found. Runs use a Databricks Free Edition workspace (serverless) with an HR pipeline: a 200,000-row employees table feeding a certified mart read by three notebooks. The maintainer ran them with Databricks' Genie Code assistant, installing the released package from PyPI. Production tables are only read; every write runs on a shallow clone or in a scratch schema that is dropped afterwards.

Public beta

bricksentry is in public beta (0.1.x). It has been tested hard, but so far only by its author, on Databricks Free Edition: nine adversarial rounds, every serious finding fixed and re-verified, all listed below. It hasn't yet run on other teams' production pipelines, other clouds or classic compute. That's what the beta is for.

What you can rely on during the beta:

  • When bricksentry can't verify something (a failed diff, a missing candidate, a masked column, a row filter, a table rewritten during the check), it asks a human. It never turns "couldn't check" into a pass.
  • Previews and checks never write to production.
  • Agents get counts, columns and verdicts, never production values, and only for the tables you list.
  • Config, policy files and saved assessments from earlier 0.1.x releases keep working; behavior changes are marked in the changelog.

What may still change before 1.0:

  • default policy rules
  • the wording of reasons and summaries
  • new fields in JSON output
  • defaults such as whether agent_tables is required for agents

1.0 comes when two or three outside teams have run it on real pipelines for a month without a serious finding, with validation reports from at least two clouds. Want to be one of them? Run the validation notebook and share the report, or open an issue.

First use as an assistant's safety gate (Nov 18, 2026)

0.1.30 · November 18, 2026

An assistant built a Bronze, Silver and Gold HR pipeline (50 raw rows with duplicates and invalid salaries, 44 clean, 5 department rows), then changed the Gold logic to count employees on leave as active. Before writing, it called bricksentry:

  • whats_downstream
    Result
    no dependents
  • rollback_plan
    Result
    matched the table's history
  • preview_write
    Result
    warn: exactly the 3 departments with employees on leave changed
  • safe_to_ship
    Result
    true, so the change was applied and checked independently

It found one bug: previewing CREATE OR REPLACE TABLE failed on serverless. Fixed in 0.1.30 and re-checked: the original statement runs with exact counts, nothing is left in the sandbox, the Gold table stayed at its version, and the battery passed 15 of 15. The calls went through the Python package in a notebook; the MCP server deployed as a Databricks app is still to be tried this way.

Pre-release checks of 0.1.29 (Nov 17, 2026): views and SQL functions

0.1.29 · November 17, 2026

Round 10 tried to make the SQL parser and Spark disagree and found no bypass in 13 attempts. 0.1.29 then added a check against Spark's own query plan, and its pre-release check found that serverless plans don't show tables inside views or SQL functions: a listed view over a forbidden table, or a function reading one, still ran (values hidden, row counts exposed; 0.1.28 has the same gap). The fix reads view and function definitions from Unity Catalog.

  • A listed view over the forbidden table
    Result
    refused, naming it
  • A view over that view
    Result
    refused
  • A SQL function reading it, with and without a schema
    Result
    refused
  • A listed view over it as the candidate
    Result
    refused
  • A view or function over allowed tables
    Result
    ran, no over-blocking
  • MERGE with a listed source, UPDATE, INSERT … SELECT
    Result
    ran with exact counts
  • Validation battery
    Result
    15 of 15

Round 10 on 0.1.28: parser attacks

0.1.28 · November 17, 2026

Round 10 tried to get a forbidden table past the SQL parser: Spark syntax it might not know, identifier tricks, and statements it can't read. Every attempt was refused, and legitimate statements still ran.

  • Spark grammar the parser might not know (UPDATE … FROM, LATERAL VIEW, TABLESAMPLE, nested subqueries, CTEs)
    Result
    all refused
  • Identifier tricks (backticks, case, Unicode lookalikes)
    Result
    all refused
  • Statements the parser can't read (a comment splitting a keyword)
    Result
    refused
  • Legitimate UPDATE and MERGE
    Result
    ran

13 of 13 attacks refused, nothing leaked.

Fix verification of 0.1.28 (Nov 17, 2026): every round 9 finding fixed

0.1.28 · November 17, 2026

Round 9 ran ten batteries on 0.1.27, from the README's own commands to governance features, and found five problems. Each original attack was re-run on the fixes.

  • agent_tables checked only the target
    Result on 0.1.28
    all six bypasses refused, naming the forbidden table; a MERGE whose source is listed runs exactly
  • Free text in tags reached agents
    Result on 0.1.28
    the planted instruction came back as [text hidden: 97 chars]; short tags kept
  • MCP over HTTP had no auth
    Result on 0.1.28
    a network bind without a token refused to start; anonymous and wrong-token clients got HTTP 401
  • A masked column passed silently
    Result on 0.1.28
    needs_human, the column listed as not compared (masked)
  • --from-git traceback
    Result on 0.1.28
    one line, exit 2

A live Unity Catalog row filter hiding 11 of 17 changed rows gave needs_human, and the 6 visible changes were counted exactly. The battery, now 15 scenarios, passed 15 of 15 in 254 s.

Round 9 on 0.1.27: what passed

0.1.27 · November 17, 2026

  • README commands run as written
    Result
    init, policies, demo, diff, the README's MERGE preview, check-last-change, test-alert and rollback worked verbatim
  • Everyday pull-request cases
    Result
    new table and dropped table → needs_human, renamed table and no-change cases correct
  • Upgrading
    Result
    a 0.1.15 config, policy and saved assessments load and run
  • MERGE previews
    Result
    10 of 10 exact
  • Hostile schemas and values
    Result
    internal-looking column names, reserved-word keys, schema drift, now(), DEFAULT columns and CHECK constraints all correct
  • Scale and limits
    Result
    10 million rows exact (100,000 changed) in 22 s; 1,000 columns exact; a VACUUMed old version fails closed with a clear message.

Fix verification of 0.1.27 (Nov 17, 2026): every round 7 finding fixed

0.1.27 · November 17, 2026

Round 7 attacked read isolation, hostile data shapes and races on 0.1.26 and found four problems. Each original attack was re-run on the fixes.

  • Agents could read production values through preview_write
    Original attack
    "update" chosen rows' SSNs and read the old values back
    Result on 0.1.27
    no sentinel value in any agent output; values masked in engine errors; the human report still shows examples
  • Diff reads not pinned to one version
    Original attack
    replace the candidate (5 → 50 changed) mid-diff; 0.1.26 reported 49
    Result on 0.1.27
    3 of 3 runs returned the pinned state (5 changed), with a note and needs_human
  • One MAP column failed the whole diff
    Original attack
    a table with a MAP column
    Result on 0.1.27
    MAP diffs exact; equal maps in another key order: 0 changes
  • Float rounding showed every row changed
    Original attack
    0.1 + 0.2 vs 0.3
    Result on 0.1.27
    exact by default with a rounding note; with float_tolerance 1e-9: 0 changed, real changes still caught

Also verified: a table outside agent_tables is refused for agents; the 14-scenario battery passed 14 of 14. Pinning costs time: a 1M-row composite-key diff went from 5.3 s to 10.3 s.

Pre-release check of 0.1.26 (Nov 17, 2026): passed on two workspaces

0.1.26 · November 17, 2026 · two Databricks Free Edition workspaces

  • Validation battery
    Result
    12 of 12, about 155–160 s
  • 50,000 NULL-key rows with 2 changes
    Result
    changed 2 (0.1.25: 36,563)
  • A column total goes from 0 to 5
    Result
    totals rule fired
  • Near-max DECIMAL(38,0) totals
    Result
    diff completed with counts and an "approximate totals" note (0.1.25: diff failed)
  • Bad write, then RENAME COLUMN, SET TBLPROPERTIES and OPTIMIZE
    Result
    warning listed the rename and the properties, ignored OPTIMIZE
  • pii and certified gates
    Result
    identical → pass; 1 row, a dropped column or a duplicate key flagged; missing candidate → needs_human
  • 1,000,000-row composite-key diff
    Result
    exact in 5–6 s
  • Cleanup
    Result
    no leaked clones, production unchanged

The validation battery on Databricks: 10 of 10 passed

November 16, 2026 · 111 s on a 2X-Small serverless warehouse

  • Delete the department-less aggregate row
    Result
    needs_human
  • A MERGE re-inserts a department
    Result
    block
  • Six problems in one change
    Result
    block, 7 rules
  • 5 of 200,000 salaries ×5, outside a 5% sample
    Result
    caught by the full-table checksum
  • Double a certified table's totals (default policy)
    Result
    warn, the documented default
  • The same under builtin:certified
    Result
    block
  • 2,000 NULL keys each side
    Result
    3,000 unchanged, nothing added or removed
  • 1,000,000 × 0.10 DECIMAL
    Result
    exactly 100,000.0
  • A bad write already committed to Delta
    Result
    block and the RESTORE
  • check-last-change on a mistyped table
    Result
    one line: "Table … not found"

The battery now has 15 scenarios. Added since this run:

  • null_keys_changed: 2 of 2,000 rows sharing a NULL key change
    Result
    exactly 2 changed
  • pii_noop: a PII table feeding a certified mart
    Result
    identical rebuild → pass; 1-row edit → needs_human
  • agent_sees_no_values: an agent previews an UPDATE of 3 rows to read their emails and salaries back
    Result
    changed=3, no value returned
  • map_column: a MAP column rewritten in a different key order, plus 1 real change
    Result
    exactly 1 changed
  • agent_allowlist: agent_tables lists one table; an agent reads another in a subquery, then as a candidate
    Result
    both refused; a listed-table preview still works

The notebook runs all 15 on your workspace. Run it yourself →

An AI agent tries risky changes on a live pipeline

0.1.23 · November 16, 2026

  • Delete the aggregate row for employees with no department
    Verdict
    needs_human
  • A case-sensitive MERGE re-inserts a department
    Verdict
    block
  • One change that drops a department, duplicates keys, adds a NULL key, drops a column, changes a type and zeroes the totals
    Verdict
    block (6 rules at once)
  • Multiply 5 of 200,000 salaries by 5, behind a 5% sample (the sample missed all five; the full-table checksum caught them)
    Verdict
    needs_human
  • A bad MERGE already committed to a Delta table
    Verdict
    block, with the RESTORE
  • Double the totals of the certified mart
    Verdict
    warn

The last result is a deliberate default: changes to a certified table itself only warn, because many certified marts are rebuilt daily. policy: builtin:certified makes it needs_human.

Release tests of 0.1.23: all passed

  • 2,000 NULL keys each side
    Result
    3,000 unchanged, nothing added or removed, 4.6 s (was 4,000,000 joined pairs)
  • 1,000,000 × 0.10 DECIMAL
    Result
    exactly 100,000.0 (DOUBLE drifts to 100,000.00000022353)
  • Duplicated rows
    Result
    counted as added, verdict block
  • Audit trail
    Result
    time, version, policy fingerprint, triggered_by
  • check-last-change after a bad MERGE
    Result
    block, RESTORE, newer-changes warning
  • Outbound URL checks
    Result
    private, local and metadata addresses refused
  • Standing regression
    Result
    MCP server lists 6 tools (5 read-only); refused statements and missing keys exit 2; lineage and rollback plan unchanged
  • Job task in a real Databricks job
    Result
    run failed as designed, downstream tasks stopped

Bugs found by live tests

  • Previewing CREATE OR REPLACE TABLE failed on serverless
    Fixed in
    0.1.30
  • A listed view or SQL function over an unlisted table exposed its row counts to agents
    Fixed in
    0.1.29
  • As a hard gate, the query-plan check refused legitimate agent MERGEs
    Fixed in
    0.1.29
  • agent_tables checked only the target table (row counts of other tables could leak)
    Fixed in
    0.1.28
  • A masked column with real changes passed silently
    Fixed in
    0.1.28
  • The MCP server over HTTP had no authentication off localhost
    Fixed in
    0.1.28
  • Free text in tags reached agents verbatim
    Fixed in
    0.1.28
  • assess --from-git printed a traceback
    Fixed in
    0.1.28
  • preview_write returned example values, so an agent could read production values
    Fixed in
    0.1.27
  • Diff queries weren't pinned to one table version (a mid-diff rewrite mixed two states)
    Fixed in
    0.1.27
  • One MAP column failed the whole diff
    Fixed in
    0.1.27
  • Float rounding showed every row changed, with no hint why
    Fixed in
    0.1.27
  • Duplicate-key pairing over-reported changes (36,563 instead of 2)
    Fixed in
    0.1.26
  • A total moving away from 0 didn't fire the totals rule
    Fixed in
    0.1.26
  • A DECIMAL(38) overflow failed the whole diff
    Fixed in
    0.1.26
  • The rollback warning ignored newer non-data changes
    Fixed in
    0.1.26
  • pii and certified gates asked a human on pull requests that changed nothing
    Fixed in
    0.1.26
  • Unclear policy and preview error messages
    Fixed in
    0.1.26
  • Agent tools returned empty results for mistyped tables
    Fixed in
    0.1.26
  • Sampled diff skipped the full check when the schema also changed
    Fixed in
    0.1.25
  • Mistyped table name showed the engine's full error
    Fixed in
    0.1.24
  • preview printed a traceback when refusing a statement
    Fixed in
    0.1.21
  • Every job task run marked failed, even a pass (Databricks treats any SystemExit as an error)
    Fixed in
    0.1.15
  • Column results listed long-dropped tables
    Fixed in
    0.1.13
  • A sample could miss a one-row change in 2 million rows
    Fixed in
    0.1.4
  • NULL-key rows counted as both added and removed
    Fixed in
    0.1.2

Not validated live yet

  • other clouds and classic compute
  • tables much larger than 10 million rows
  • a full bundle deploy in a fresh workspace

If you can run the notebook in one of these setups, a validation report helps more than anything else.

Full record with every run: VALIDATION.md on GitHub →