What’s Happening
We’re using vault_insert_by_rank
to load various raw_vault tables. We have created our DBTVAULT_RANK column in our staging layers. During the base load, only RANK 1 is being inserted into the raw_table. In order to load the other ranked rows, dbt run
needs to be called again, indicating that subsequent loads are working ok. Is this expected?
Right now, we loose records with RANK > 1 when there is a base load.
Expected Behavior
I expect all ranks to be loaded during the base load. This is important so that we don’t loose records from our up-stream processes that incrementally stage data for loading.
dbtvault’s documentation explains the vault_insert_by_rank materialization to working like this:
The
vault_insert_by_rank
custom materialisation provides the means
to iteratively load raw vault structures from an arbitrary rank
column, created in the staging layer.
Why would the base load only load the 1st rank?
Environment
dbt version: 1.4.1
dbtvault Snowflake: 1.4.0