Is anyone using Great Expectations or dbt tests to validate EHR extracts at scale? I’m standardizing 30-day readmission trend monitoring across two systems and need a repeatable way to flag integrity shifts (like a >5% DRG mix change week over week) before dashboards update. Pointers to templates, sample rules, or a concise guide would be hugely helpful.
I’ve had good luck in dbt by aggregating weekly DRG proportions per system and a custom generic test that fails when the week-over-week change exceeds 5% for any DRG; the prior week’s proportions live in a tiny seed that we update post-run so it gates the next refresh. If you want a quick start, dbt-expectations has solid patterns for distribution checks and works well in CI: GitHub - calogica/dbt-expectations: Port(ish) of Great Expectations to dbt test macros. Backfills can false-flag, so we run it on a 3-day lag — want the macro/schema.yml I use?
Lean on Great Expectations’ KL divergence to catch your ‘DRG mix >5%’ drift: materialize weekly DRG proportions by system and compare this week to last with expect_column_kl_divergence_to_be_less_than in a pre-dashboard checkpoint. I also gate on a denominator sanity check for 30‑day readmissions (e.g., ±3%) to cut noise. @alexis_g90 does this fit your setup, and can you run a small Python GE step or must it stay pure SQL?
I keep a dbt snapshot of last week’s extract and run a chi‑square on DRG categories; if p<0.01 or any category shifts >5% (your threshold), I fail the job and a tiny ‘stop‑the‑world’ flag table blocks the dashboard refresh. Small caveat: align both systems’ week definitions and code set updates, or you’ll chase false drift. Want a lightweight example model?