Change Details

This introduces a `create_partial_visit` flag to the `store_data` method call which allows to write partial visit with the current snapshot built during one iteration of the ingestion loop. The final loop of the visit will not create a partial visit. It will create the last visit with status "full" targeting the final snapshot as before. The main difference is that an ingestion is actually more verbose in terms of origin_visit_status. Which in turn allows to be more incremental in subsequent visit of the same origin. This is required to allow performance improvments on the loader git [1]. [1] D6386 Related to T3625

This introduces a `create_partial_snapshot` parameter to the base loader constructor. When activated, during each call of the `store_data` method, if there are more data to fetch, this will create a partial snapshot. The final loop behaves as before, create the last visit with status 'full' targeting the snapshot. The main difference between the 2 behavior is that an ingestion with that parameter on is more verbose in terms of origin_visit_status. This, in turn, allows to be incremental in subsequent visits for the same origin. This may especially be interesting for cases when loading fail due to out of hand resources issues (e.g. large svn or git repositories). This is required to allow performance improvments on the loader git [1]. [1] D6386 Related to T3625

This introduces a `create_partial_visit` flagsnapshot` parameter to the `store_data` method call whichbase loader constructor. allows to write partial visit withWhen activated, during each call of the current snapshot built during one iteration of`store_data` method, if there are more data to the ingestion loopfetch, this will create a partial snapshot. The final loop of the visit will not create a partial visit.behaves as before, It will create the lastcreate the last visit with status 'full' targeting the visit with status "full" targeting the final snapshot as before. The main difference is that an ingestionbetween the 2 behavior is actually more verbose in terms ofthat an ingestion with that parameter on is more verbose in terms of origin_visit_status. This, in turn, Which in turn allows to be more incremental in subsequent visit of the same originin subsequent visits for the same origin. This may especially be interesting for cases when loading fail due to out of hand resources issues (e.g. large svn or git repositories). This is required to allow performance improvments on the loader git [1]. [1] D6386 Related to T3625