Hello, dear community!
I’ve been figuring out for my self the best approach for loading pattern of the DV core objects from staging tables.
Lets us having N staging tables from M source systems.
What is the best approach to load hubs, link and stas regarding options I can think about:
-
Treat each stage table as and atominc pipeline, the one that is responsible to load all the core objects that depend on this stage table. So the processes are decoupled and can be sceduled at any rate of stage table refrsh.
-
Move data layer by layer. From landing to staging and then to core ONLY WHEN ALL the staging is done. In such case we have piplines per hub, sat or link that may address several stage tables.
p.1 seems better for me as we can have different load rates depending on busness area (domain), as opposed to p.2 where we need wait for all domain data to arrive.
(for my project I see different business areas refreh at different rates)
or may be I’m looking a wrong way completely?
Thank You!