Blog · Release

Introducing bricksentry: change-safety for Databricks pipelines and AI agents

By Brahma · November 10, 2026

Over the last year, AI agents and coding assistants started writing a large share of the pipeline code I review. They are good at it. The code compiles, the tests pass, the pull request reads cleanly. And then a dashboard is off by a few percent on Monday and nobody can say which merge did it.

The problem is not the agents. It is that our review process still only reads code. A reviewer looks at a diff of Python or SQL and has to imagine what it does to twenty thousand rows and to everything that reads them. That was already hard when people wrote every line. It does not scale when an agent opens ten pull requests before lunch.

So I built bricksentry, a change-safety check for Databricks pipelines and the agents that edit them. It looks at the data your change produced, not just the code, and gives a verdict.

Three failures it catches

The package ships with a demo built on a 20,000-row sales.orders table. Downstream of it sit a certified sales.daily_revenue table owned by finance-analytics, a month-close table, a Genie space, an hourly job and the Exec Revenue dashboard. Three changes, three outcomes:

A silent revenue shift. A pull request applies a discount to APAC orders. The code is two lines and looks reasonable. bricksentry reports 6,666 rows changed, sum(amount) down 2.67%, and 100% of the affected rows in region APAC. Because a certified table sits downstream, the verdict is needs_human, and the owners from Unity Catalog, finance-analytics and sales-data-eng, are tagged for review. Finance hears about it before the number moves, not after.

A fan-out join. A second version of the same change joins to a lookup table that has duplicate keys. Row count goes up by 40. That is 0.20%, small enough to miss by eye. bricksentry finds 40 duplicate keys on order_id where production had none, and returns block. CI fails. The fix happens before the merge.

An agent's DELETE. An agent decides to remove refunded orders with DELETE FROM sales.orders WHERE status <> 'paid'. Instead of running it, it calls preview_write. bricksentry runs the statement on a disposable shallow clone and reports 2,000 rows removed, 10.01% of the amount total, spread evenly across regions. Production still has all 20,000 rows. The verdict is needs_human, and the agent's next_step tells it to open a pull request and ask the owners.

Four checks, in order

• Diff. Match production and the candidate on key columns. Count added, removed and changed rows, schema changes, column totals and duplicate keys, then find the segment where the change is concentrated. It all runs as SQL on your warehouse.
• Blast radius. Walk Unity Catalog lineage downstream and attach owners and tags.
• Policy. Apply the rules in policy.yml. The worst matching rule wins.
• Rollback. Read Delta history and hand back the exact RESTORE command before anything is applied.

Why rules, not an LLM

It would have been easy to ask a model whether a change looks safe. I chose not to. A safety gate has to give the same answer to the same change every time, and it has to be something a team can read and argue about in a pull request. A YAML rule like "certified downstream means needs_human" meets both bars. An agent's LLM can read the explanation and act on it, but it never changes the verdict.

Why runtime lineage

Most Databricks work is not dbt. It is PySpark notebooks, Jobs and Lakeflow pipelines, and the readers are dashboards, Genie spaces and ad hoc queries. Parsing SQL files misses most of that. Unity Catalog records lineage at runtime, from what actually ran, so those notebooks and dashboards show up in the blast radius.

What's next

v0.2 adds AWS support through Athena and Iceberg on the Glue Data Catalog, and an OpenLineage provider for Airflow, dbt and Spark. v0.3 brings column-level blast radius from system.access.column_lineage and an aggregate diff for tables without a stable key. The full list is on the roadmap.

Try it

Install it from PyPI. To run the demo without a Databricks workspace:

pip install "bricksentry[duckdb]"
bricksentry demo

Then read the quickstart to point it at a real warehouse. This is v0.1, and I'd like to know where it breaks on your pipelines. Open an issue on GitHub with what it caught, what it missed, and what you'd want it to check next.

Brahma