Has anyone implemented a custom metric vault on Snowflake for logging execution history - timings, records inserted etc? If so, any recommendations for going about - if streams and tasks are the basis for loading the raw vault from staging?
cheers,
John
not a metric vault, but a framework
I extended the framework here: What is the Shape of your Data?. … and why is it important. | by Patrick Cuba | The Modern Scientist | Medium
I find that the dbt-artifacts package by broolyn data does a good job of tracking most of the metrics I need.
Hi @patrickcuba - looking at your sample framework code, I see the reconciliation of staging to target (reconcile_hub_reconciliation_errors) is querying the source staging table directly - but elsewhere it is advocated to create a view with a stream on top of that view. Presuming the staging table is persistent and the stream is indicating the delta to load to target, should this reconciliation be against just the delta too, to prevent longer running test execution and additional compute being consumed against a entire staging table?
If so, given the staging view stream gets consumed by the load into the target, is it recommended to simply create an additional stream against the staging table view and use that for reconciliation? There is a danger the two staging view streams could be out of synch if new data lands into the staging table before the data reconciliation stream gets consumed. What is your recommendation here?
cheers,
John
Totally up to you, I created that as a framework which you can expand on.
In here I show the full ingestion and consumption, Zero to Data Vault on Snowflake - Snowflake
Of course the demo I show is non-parameterised, that part I leave up to you.