Page MenuHomeSoftware Heritage

bitbucket: Allow incremental lister to start properly the first time
ClosedPublic

Authored by ardumont on Jun 21 2019, 7:46 PM.

Details

Summary

When there is no data in db, prior to this commit, it insisted on listing
starting from around now. In the end, listed only a few repositories.

This reduces the scope of this diff to this commit only

Related T1826
Depends on D1635

Test Plan

tox

Diff Detail

Repository
rDLS Listers
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 6447
Build 8959: tox-on-jenkinsJenkins
Build 8958: arc lint + arc unit

Event Timeline

swh/lister/bitbucket/lister.py
55–56

Otherwise, we are always doing nothing when starting an incremental lister from nothing (since we get a now() date as starting point)

  • Fix edge case on workaround which resulting in dead infinite loop.
  • bitbucket.lister: Allow to define flush packet db size
ardumont added projects: Lister, Bitbucket lister.
ardumont edited the test plan for this revision. (Show Details)

Will also split this into 1 diff per commit.

Allow incremental lister to actually start properly the first time

When there is no data in db, prior to this commit, it insisted on listing
starting from around now. In the end, listed only a few repositories.

This reduces the scope of this diff to this commit only

Related T1826

ardumont edited the test plan for this revision. (Show Details)
ardumont retitled this revision from bitbucket: Work around pagination issue with bitbucket api to bitbucket: Allow incremental lister to start properly the first time.
ardumont edited the summary of this revision. (Show Details)

This will be amended.

ardumont edited the test plan for this revision. (Show Details)

relister: Fix consistently the behavior for the first time relisting

If nothing has been done prior to a full relisting, there is actually nothing
to list. So the relister in question does nothing.

Related T1826
Related e129e48

This revision is now accepted and ready to land.Jun 24 2019, 4:26 PM

Maybe add a line in the commit message about the behavior change of db_partition_indices() (returns an empty list instead of raising a ValueError)

Update commit message about method behavior change

This revision was automatically updated to reflect the committed changes.