dbt model versions and DV history

AutomateDV (formerly known as dbtvault)
1

Hi,

we are starting to ramp up AutomateDV, dbt, data contracts and model versioning. I was wondering how model versioning will work with incremental models for satellites if business logic evolves.

For instance:

  • Stage has columns A,B,C as a payload along with effectivity information
  • This data is stored into a link effectivity satellite in the Raw Vault (MyRVSat)
  • A business rule creates a column MyResult as A+B which also creates an effectivity satellite (MyBVSat)
  • all models are stored as version 1 in dbt and dbt will create tables MyRVSat_v1, MyBVSat_v1
  • Records are initially loaded and the data vault captures multiple rows for the MyRVSat and MyBVSat
  • The business rule changes to A+b+C=Result, a version 2 of the MyBVSat model is created
  • dbt will now create a second table MyBVSat_v2 (which lacks history and will re-load all records from MyRVSat again and produce different Result records for the old data

My questions: Are my assumptions correct? How can I ensure MyBVSat_v2 keeps the history of v1 and writes new records for the new business rule from the day the change is introduced? Is there any out-of-the-box support by the AutomateDV macros?

Thanks for your help!

2

This is an interesting thought. I’ve not had a chance to mess around with this feature yet but my naiive guess would be that you’re right. I wonder if dbt clones the current version into the new and then continues to increment it or if it just starts from a clean slate each time?

Thanks for the brain scratcher
Frankie