Validation
Validation: what we tested on real Databricks
What bricksentry has been tested against on real Databricks, with the numbers, including the results that weren't perfect and every bug the live tests found. Runs use a Databricks Free Edition workspace (serverless) with an HR pipeline: a 200,000-row employees table feeding a certified mart read by three notebooks. The maintainer ran them with Databricks' Genie Code assistant, installing the released package from PyPI. Production tables are only read; every write runs on a shallow clone or in a scratch schema that is dropped afterwards.
Public beta
bricksentry is in public beta (0.1.x). It has been tested hard, but so far only by its author, on Databricks Free Edition: nine adversarial rounds, every serious finding fixed and re-verified, all listed below. It hasn't yet run on other teams' production pipelines, other clouds or classic compute. That's what the beta is for.
What you can rely on during the beta:
- When bricksentry can't verify something (a failed diff, a missing candidate, a masked column, a row filter, a table rewritten during the check), it asks a human. It never turns "couldn't check" into a pass.
- Previews and checks never write to production.
- Agents get counts, columns and verdicts, never production values, and only for the tables you list.
- Config, policy files and saved assessments from earlier 0.1.x releases keep working; behavior changes are marked in the changelog.
What may still change before 1.0:
- default policy rules
- the wording of reasons and summaries
- new fields in JSON output
- defaults such as whether
agent_tablesis required for agents
1.0 comes when two or three outside teams have run it on real pipelines for a month without a serious finding, with validation reports from at least two clouds. Want to be one of them? Run the validation notebook and share the report, or open an issue.
First use as an assistant's safety gate (Nov 18, 2026)
0.1.30 · November 18, 2026
An assistant built a Bronze, Silver and Gold HR pipeline (50 raw rows with duplicates and invalid salaries, 44 clean, 5 department rows), then changed the Gold logic to count employees on leave as active. Before writing, it called bricksentry:
whats_downstreamResultno dependentsrollback_planResultmatched the table's historypreview_writeResultwarn: exactly the 3 departments with employees on leave changedsafe_to_shipResulttrue, so the change was applied and checked independently
It found one bug: previewing CREATE OR REPLACE TABLE failed on serverless. Fixed in 0.1.30 and re-checked: the original statement runs with exact counts, nothing is left in the sandbox, the Gold table stayed at its version, and the battery passed 15 of 15. The calls went through the Python package in a notebook; the MCP server deployed as a Databricks app is still to be tried this way.
Pre-release checks of 0.1.29 (Nov 17, 2026): views and SQL functions
0.1.29 · November 17, 2026
Round 10 tried to make the SQL parser and Spark disagree and found no bypass in 13 attempts. 0.1.29 then added a check against Spark's own query plan, and its pre-release check found that serverless plans don't show tables inside views or SQL functions: a listed view over a forbidden table, or a function reading one, still ran (values hidden, row counts exposed; 0.1.28 has the same gap). The fix reads view and function definitions from Unity Catalog.
- A listed view over the forbidden tableResultrefused, naming it
- A view over that viewResultrefused
- A SQL function reading it, with and without a schemaResultrefused
- A listed view over it as the candidateResultrefused
- A view or function over allowed tablesResultran, no over-blocking
- MERGE with a listed source, UPDATE, INSERT … SELECTResultran with exact counts
- Validation batteryResult15 of 15
Round 10 on 0.1.28: parser attacks
0.1.28 · November 17, 2026
Round 10 tried to get a forbidden table past the SQL parser: Spark syntax it might not know, identifier tricks, and statements it can't read. Every attempt was refused, and legitimate statements still ran.
- Spark grammar the parser might not know (UPDATE … FROM, LATERAL VIEW, TABLESAMPLE, nested subqueries, CTEs)Resultall refused
- Identifier tricks (backticks, case, Unicode lookalikes)Resultall refused
- Statements the parser can't read (a comment splitting a keyword)Resultrefused
- Legitimate UPDATE and MERGEResultran
13 of 13 attacks refused, nothing leaked.
Fix verification of 0.1.28 (Nov 17, 2026): every round 9 finding fixed
0.1.28 · November 17, 2026
Round 9 ran ten batteries on 0.1.27, from the README's own commands to governance features, and found five problems. Each original attack was re-run on the fixes.
- agent_tables checked only the targetResult on 0.1.28all six bypasses refused, naming the forbidden table; a MERGE whose source is listed runs exactly
- Free text in tags reached agentsResult on 0.1.28the planted instruction came back as [text hidden: 97 chars]; short tags kept
- MCP over HTTP had no authResult on 0.1.28a network bind without a token refused to start; anonymous and wrong-token clients got HTTP 401
- A masked column passed silentlyResult on 0.1.28needs_human, the column listed as not compared (masked)
- --from-git tracebackResult on 0.1.28one line, exit 2
A live Unity Catalog row filter hiding 11 of 17 changed rows gave needs_human, and the 6 visible changes were counted exactly. The battery, now 15 scenarios, passed 15 of 15 in 254 s.
Round 9 on 0.1.27: what passed
0.1.27 · November 17, 2026
- README commands run as writtenResultinit, policies, demo, diff, the README's MERGE preview, check-last-change, test-alert and rollback worked verbatim
- Everyday pull-request casesResultnew table and dropped table → needs_human, renamed table and no-change cases correct
- UpgradingResulta 0.1.15 config, policy and saved assessments load and run
- MERGE previewsResult10 of 10 exact
- Hostile schemas and valuesResultinternal-looking column names, reserved-word keys, schema drift, now(), DEFAULT columns and CHECK constraints all correct
- Scale and limitsResult10 million rows exact (100,000 changed) in 22 s; 1,000 columns exact; a VACUUMed old version fails closed with a clear message.
Fix verification of 0.1.27 (Nov 17, 2026): every round 7 finding fixed
0.1.27 · November 17, 2026
Round 7 attacked read isolation, hostile data shapes and races on 0.1.26 and found four problems. Each original attack was re-run on the fixes.
- Agents could read production values through preview_writeOriginal attack"update" chosen rows' SSNs and read the old values backResult on 0.1.27no sentinel value in any agent output; values masked in engine errors; the human report still shows examples
- Diff reads not pinned to one versionOriginal attackreplace the candidate (5 → 50 changed) mid-diff; 0.1.26 reported 49Result on 0.1.273 of 3 runs returned the pinned state (5 changed), with a note and needs_human
- One MAP column failed the whole diffOriginal attacka table with a MAP columnResult on 0.1.27MAP diffs exact; equal maps in another key order: 0 changes
- Float rounding showed every row changedOriginal attack0.1 + 0.2 vs 0.3Result on 0.1.27exact by default with a rounding note; with float_tolerance 1e-9: 0 changed, real changes still caught
Also verified: a table outside agent_tables is refused for agents; the 14-scenario battery passed 14 of 14. Pinning costs time: a 1M-row composite-key diff went from 5.3 s to 10.3 s.
Pre-release check of 0.1.26 (Nov 17, 2026): passed on two workspaces
0.1.26 · November 17, 2026 · two Databricks Free Edition workspaces
- Validation batteryResult12 of 12, about 155–160 s
- 50,000 NULL-key rows with 2 changesResultchanged 2 (0.1.25: 36,563)
- A column total goes from 0 to 5Resulttotals rule fired
- Near-max DECIMAL(38,0) totalsResultdiff completed with counts and an "approximate totals" note (0.1.25: diff failed)
- Bad write, then RENAME COLUMN, SET TBLPROPERTIES and OPTIMIZEResultwarning listed the rename and the properties, ignored OPTIMIZE
- pii and certified gatesResultidentical → pass; 1 row, a dropped column or a duplicate key flagged; missing candidate → needs_human
- 1,000,000-row composite-key diffResultexact in 5–6 s
- CleanupResultno leaked clones, production unchanged
The validation battery on Databricks: 10 of 10 passed
November 16, 2026 · 111 s on a 2X-Small serverless warehouse
- Delete the department-less aggregate rowResultneeds_human
- A MERGE re-inserts a departmentResultblock
- Six problems in one changeResultblock, 7 rules
- 5 of 200,000 salaries ×5, outside a 5% sampleResultcaught by the full-table checksum
- Double a certified table's totals (default policy)Resultwarn, the documented default
- The same under builtin:certifiedResultblock
- 2,000 NULL keys each sideResult3,000 unchanged, nothing added or removed
- 1,000,000 × 0.10 DECIMALResultexactly 100,000.0
- A bad write already committed to DeltaResultblock and the RESTORE
- check-last-change on a mistyped tableResultone line: "Table … not found"
The battery now has 15 scenarios. Added since this run:
null_keys_changed: 2 of 2,000 rows sharing a NULL key changeResultexactly 2 changedpii_noop: a PII table feeding a certified martResultidentical rebuild → pass; 1-row edit → needs_humanagent_sees_no_values: an agent previews an UPDATE of 3 rows to read their emails and salaries backResultchanged=3, no value returnedmap_column: a MAP column rewritten in a different key order, plus 1 real changeResultexactly 1 changedagent_allowlist: agent_tables lists one table; an agent reads another in a subquery, then as a candidateResultboth refused; a listed-table preview still works
The notebook runs all 15 on your workspace. Run it yourself →
An AI agent tries risky changes on a live pipeline
0.1.23 · November 16, 2026
- Delete the aggregate row for employees with no departmentVerdictneeds_human
- A case-sensitive MERGE re-inserts a departmentVerdictblock
- One change that drops a department, duplicates keys, adds a NULL key, drops a column, changes a type and zeroes the totalsVerdictblock (6 rules at once)
- Multiply 5 of 200,000 salaries by 5, behind a 5% sample (the sample missed all five; the full-table checksum caught them)Verdictneeds_human
- A bad MERGE already committed to a Delta tableVerdictblock, with the RESTORE
- Double the totals of the certified martVerdictwarn
The last result is a deliberate default: changes to a certified table itself only warn, because many certified marts are rebuilt daily. policy: builtin:certified makes it needs_human.
Release tests of 0.1.23: all passed
- 2,000 NULL keys each sideResult3,000 unchanged, nothing added or removed, 4.6 s (was 4,000,000 joined pairs)
- 1,000,000 × 0.10 DECIMALResultexactly 100,000.0 (DOUBLE drifts to 100,000.00000022353)
- Duplicated rowsResultcounted as added, verdict block
- Audit trailResulttime, version, policy fingerprint, triggered_by
- check-last-change after a bad MERGEResultblock, RESTORE, newer-changes warning
- Outbound URL checksResultprivate, local and metadata addresses refused
- Standing regressionResultMCP server lists 6 tools (5 read-only); refused statements and missing keys exit 2; lineage and rollback plan unchanged
- Job task in a real Databricks jobResultrun failed as designed, downstream tasks stopped
Bugs found by live tests
- Previewing CREATE OR REPLACE TABLE failed on serverlessFixed in0.1.30
- A listed view or SQL function over an unlisted table exposed its row counts to agentsFixed in0.1.29
- As a hard gate, the query-plan check refused legitimate agent MERGEsFixed in0.1.29
- agent_tables checked only the target table (row counts of other tables could leak)Fixed in0.1.28
- A masked column with real changes passed silentlyFixed in0.1.28
- The MCP server over HTTP had no authentication off localhostFixed in0.1.28
- Free text in tags reached agents verbatimFixed in0.1.28
- assess --from-git printed a tracebackFixed in0.1.28
- preview_write returned example values, so an agent could read production valuesFixed in0.1.27
- Diff queries weren't pinned to one table version (a mid-diff rewrite mixed two states)Fixed in0.1.27
- One MAP column failed the whole diffFixed in0.1.27
- Float rounding showed every row changed, with no hint whyFixed in0.1.27
- Duplicate-key pairing over-reported changes (36,563 instead of 2)Fixed in0.1.26
- A total moving away from 0 didn't fire the totals ruleFixed in0.1.26
- A DECIMAL(38) overflow failed the whole diffFixed in0.1.26
- The rollback warning ignored newer non-data changesFixed in0.1.26
- pii and certified gates asked a human on pull requests that changed nothingFixed in0.1.26
- Unclear policy and preview error messagesFixed in0.1.26
- Agent tools returned empty results for mistyped tablesFixed in0.1.26
- Sampled diff skipped the full check when the schema also changedFixed in0.1.25
- Mistyped table name showed the engine's full errorFixed in0.1.24
- preview printed a traceback when refusing a statementFixed in0.1.21
- Every job task run marked failed, even a pass (Databricks treats any SystemExit as an error)Fixed in0.1.15
- Column results listed long-dropped tablesFixed in0.1.13
- A sample could miss a one-row change in 2 million rowsFixed in0.1.4
- NULL-key rows counted as both added and removedFixed in0.1.2
Not validated live yet
- other clouds and classic compute
- tables much larger than 10 million rows
- a full bundle deploy in a fresh workspace
If you can run the notebook in one of these setups, a validation report helps more than anything else.
Full record with every run: VALIDATION.md on GitHub →