Change Details

This introduces a more_data_to_fetch which allows to not write immediately the snapshot if we need some more data to fetch in other iterations of the loop. Prior to this commit, it was implied that the store_data could only be called once. It's a limitation that needs to change for some ongoing optimizations in the loader git. Related to T3625

This introduces a `create_partial_visit` flag to the `store_data` method call which allows to write partial visit with the current snapshot built during one iteration of the ingestion loop. The final loop of the visit will not create a partial visit. It will create the last visit with status "full" targeting the final snapshot as before. The main difference is that an ingestion is actually more verbose in terms of origin_visit_status. Which in turn allows to be more incremental in subsequent visit of the same origin. Related to T3625 j

This introduces a more_data_to_fetch which allows`create_partial_visit` flag to not write immediately the snapshotthe `store_data` method call which if we need some more data to fetch in other iterations of the loop. Prior to thisallows to write partial visit with the current snapshot built during one iteration of commit, it was implied thatthe ingestion loop. The final loop of the store_data could only be called oncevisit will not create a partial visit. It's a limitationIt will create the last that needs to change for some ongoing optimizations in the loader gitvisit with status "full" targeting the final snapshot as before. The main difference is that an ingestion is actually more verbose in terms of origin_visit_status. Which in turn allows to be more incremental in subsequent visit of the same origin. Related to T3625 j