Page MenuHomeSoftware Heritage
Differential D1632

bitbucket: Allow incremental lister to start properly the first time
ClosedPublic
Actions

Authored by ardumont on Jun 21 2019, 7:46 PM.

Details

Reviewers
douardda
olasd
Group Reviewers
Reviewers
Commits
rDLSb99617f97615: relister: Fix consistently the behavior for the first time relisting
Summary

When there is no data in db, prior to this commit, it insisted on listing
starting from around now. In the end, listed only a few repositories.

This reduces the scope of this diff to this commit only

Related T1826
Depends on D1635

Test Plan

tox

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Jun 21 2019, 7:46 PM
swh-public-ci added a comment.Jun 21 2019, 7:47 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/223/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/223/console

Harbormaster failed remote builds in B6420: Diff 5433!Jun 21 2019, 7:47 PM
ardumont updated this revision to Diff 5434.Jun 21 2019, 7:52 PM

Fix pep8 violations

ardumont added inline comments.Jun 21 2019, 7:53 PM
swh/lister/bitbucket/lister.py
58–59

Otherwise, we are always doing nothing when starting an incremental lister from nothing (since we get a now() date as starting point)

swh-public-ci added a comment.Jun 21 2019, 7:55 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/224/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/224/console

Harbormaster failed remote builds in B6421: Diff 5434!Jun 21 2019, 7:55 PM
ardumont updated this revision to Diff 5437.Jun 22 2019, 12:21 AM
  • Fix edge case on workaround which resulting in dead infinite loop.
  • bitbucket.lister: Allow to define flush packet db size
ardumont edited the test plan for this revision. (Show Details)Jun 22 2019, 12:22 AM
ardumont added projects: Lister, Bitbucket lister.
ardumont edited the test plan for this revision. (Show Details)
swh-public-ci added a comment.Jun 22 2019, 12:25 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/225/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/225/console

Harbormaster failed remote builds in B6424: Diff 5437!Jun 22 2019, 12:25 AM
ardumont edited the test plan for this revision. (Show Details)Jun 22 2019, 7:00 AM
ardumont planned changes to this revision.Jun 23 2019, 8:29 AM

Will also split this into 1 diff per commit.

ardumont updated this revision to Diff 5438.Jun 23 2019, 9:30 AM

Allow incremental lister to actually start properly the first time

When there is no data in db, prior to this commit, it insisted on listing
starting from around now. In the end, listed only a few repositories.

This reduces the scope of this diff to this commit only

Related T1826

ardumont edited the summary of this revision. (Show Details)Jun 23 2019, 9:30 AM
ardumont edited the test plan for this revision. (Show Details)
ardumont retitled this revision from bitbucket: Work around pagination issue with bitbucket api to bitbucket: Allow incremental lister to start properly the first time.
ardumont edited the summary of this revision. (Show Details)
ardumont added a child revision: D1633: bitbucket: Unify logging instructions.Jun 23 2019, 9:32 AM
swh-public-ci added a comment.Jun 23 2019, 9:34 AM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/226/ for more details.

Harbormaster completed remote builds in B6426: Diff 5438.Jun 23 2019, 9:34 AM
ardumont planned changes to this revision.Jun 24 2019, 10:51 AM

This will be amended.

ardumont updated this revision to Diff 5456.Jun 24 2019, 4:18 PM
ardumont edited the test plan for this revision. (Show Details)

relister: Fix consistently the behavior for the first time relisting

If nothing has been done prior to a full relisting, there is actually nothing
to list. So the relister in question does nothing.

Related T1826
Related e129e48

swh-public-ci added a comment.Jun 24 2019, 4:19 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/237/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/237/console

Harbormaster failed remote builds in B6443: Diff 5456!Jun 24 2019, 4:19 PM
ardumont updated this revision to Diff 5457.Jun 24 2019, 4:22 PM

Fix git amend mess

douardda accepted this revision.Jun 24 2019, 4:26 PM

LGTM but I'm biased

This revision is now accepted and ready to land.Jun 24 2019, 4:26 PM
swh-public-ci added a comment.Jun 24 2019, 4:26 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/238/ for more details.

Harbormaster completed remote builds in B6444: Diff 5457.Jun 24 2019, 4:26 PM
douardda added a comment.Jun 24 2019, 4:27 PM

Maybe add a line in the commit message about the behavior change of db_partition_indices() (returns an empty list instead of raising a ValueError)

ardumont updated this revision to Diff 5460.Jun 24 2019, 4:31 PM

Update commit message about method behavior change

swh-public-ci added a comment.Jun 24 2019, 4:39 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/241/ for more details.

Harbormaster completed remote builds in B6447: Diff 5460.Jun 24 2019, 4:39 PM
ardumont updated this revision to Diff 5484.Jun 25 2019, 2:34 PM

Rebase

Harbormaster failed remote builds in B6474: Diff 5484!Jun 25 2019, 2:35 PM
swh-public-ci added a comment.Jun 25 2019, 2:35 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/260/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/260/console

ardumont removed a child revision: D1633: bitbucket: Unify logging instructions.Jun 25 2019, 2:36 PM
ardumont edited the summary of this revision. (Show Details)
ardumont added a parent revision: D1635: indexing-lister: Allow to define flush packet size.
ardumont added a child revision: D1634: lister: Type correctly the 'indexable' column.Jun 25 2019, 2:39 PM
swh-public-ci added a comment.Jun 25 2019, 2:53 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/264/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/264/console

ardumont updated this revision to Diff 5488.Jun 25 2019, 2:58 PM

Rebase

swh-public-ci added a comment.Jun 25 2019, 3:09 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/268/ for more details.

Harbormaster completed remote builds in B6478: Diff 5488.Jun 25 2019, 3:09 PM
Closed by commit rDLSb99617f97615: relister: Fix consistently the behavior for the first time relisting (authored by ardumont). · Explain WhyJun 26 2019, 11:19 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
swh/
lister/
bitbucket/
19 lines
9 lines
core/
17 lines
github/
10 lines
gitlab/
5 lines
phabricator/
23 lines

Diff 5503

View Options

swh/lister/bitbucket/lister.py

Loading...
View Options

swh/lister/bitbucket/tasks.py

Loading...
View Options

swh/lister/core/indexing_lister.py

Loading...
View Options

swh/lister/github/tasks.py

Loading...
View Options

swh/lister/gitlab/tasks.py

Loading...
View Options

swh/lister/phabricator/lister.py

Loading...
About · Developer information · Legal