Page MenuHomeSoftware Heritage
Differential D7059

Add incremental support for the bzr loader
ClosedPublic
Actions

Authored by Alphare on Feb 2 2022, 10:39 AM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDLDBZRfc1f8e738405: Add incremental support for the bzr loader

Diff Detail

Repository
rDLDBZR BZR loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Alphare created this revision.Feb 2 2022, 10:39 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 2 2022, 10:39 AM
Alphare added a parent revision: D6344: Non-incremental, non-delta Bazaar/Breezy loader.Feb 2 2022, 10:39 AM
swh-public-ci added a comment.Feb 2 2022, 10:44 AM

Build is green

Patch application report for D7059 (id=25609)

Could not rebase; Attempt merge onto 347c7b8375...

Updating 347c7b8..bc832e0
Fast-forward
 .gitignore                                         |  14 +
 .pre-commit-config.yaml                            |   1 -
 swh/foo/loader.py => conftest.py                   |  10 +-
 mypy.ini                                           |   6 +
 requirements-swh.txt                               |   6 +-
 requirements-test.txt                              |   4 +
 requirements.txt                                   |   1 +
 setup.py                                           |  13 +-
 swh/foo/tests/__init__.py                          |   0
 swh/foo/tests/test_loader.py                       |   3 -
 swh/loader/__init__.py                             |   3 +
 swh/loader/bzr/__init__.py                         |  16 +
 swh/loader/bzr/loader.py                           | 610 +++++++++++++++++++++
 swh/{foo => loader/bzr}/py.typed                   |   0
 swh/{foo => loader/bzr/tests}/__init__.py          |   0
 swh/loader/bzr/tests/conftest.py                   |  39 ++
 swh/loader/bzr/tests/data/broken-tags.sh           |   9 +
 swh/loader/bzr/tests/data/broken-tags.tgz          | Bin 0 -> 980 bytes
 swh/loader/bzr/tests/data/empty.sh                 |   6 +
 swh/loader/bzr/tests/data/empty.tgz                | Bin 0 -> 962 bytes
 swh/loader/bzr/tests/data/ghosts.py                |  13 +
 swh/loader/bzr/tests/data/ghosts.tgz               | Bin 0 -> 2451 bytes
 .../bzr/tests/data/metadata-and-type-changes.sh    |  38 ++
 .../bzr/tests/data/metadata-and-type-changes.tgz   | Bin 0 -> 13831 bytes
 swh/loader/bzr/tests/data/needs-upgrade.sh         |   7 +
 swh/loader/bzr/tests/data/needs-upgrade.tgz        | Bin 0 -> 880 bytes
 swh/loader/bzr/tests/data/no-branch.sh             |  10 +
 swh/loader/bzr/tests/data/no-branch.tgz            | Bin 0 -> 882 bytes
 swh/loader/bzr/tests/data/nominal.sh               |  36 ++
 swh/loader/bzr/tests/data/nominal.tgz              | Bin 0 -> 11366 bytes
 swh/loader/bzr/tests/data/renames.sh               |  17 +
 swh/loader/bzr/tests/data/renames.tgz              | Bin 0 -> 5068 bytes
 swh/loader/bzr/tests/test_loader.py                | 424 ++++++++++++++
 tox.ini                                            |   4 +-
 34 files changed, 1274 insertions(+), 16 deletions(-)
 create mode 100644 .gitignore
 rename swh/foo/loader.py => conftest.py (52%)
 delete mode 100644 swh/foo/tests/__init__.py
 delete mode 100644 swh/foo/tests/test_loader.py
 create mode 100644 swh/loader/__init__.py
 create mode 100644 swh/loader/bzr/__init__.py
 create mode 100644 swh/loader/bzr/loader.py
 rename swh/{foo => loader/bzr}/py.typed (100%)
 rename swh/{foo => loader/bzr/tests}/__init__.py (100%)
 create mode 100644 swh/loader/bzr/tests/conftest.py
 create mode 100644 swh/loader/bzr/tests/data/broken-tags.sh
 create mode 100644 swh/loader/bzr/tests/data/broken-tags.tgz
 create mode 100644 swh/loader/bzr/tests/data/empty.sh
 create mode 100644 swh/loader/bzr/tests/data/empty.tgz
 create mode 100644 swh/loader/bzr/tests/data/ghosts.py
 create mode 100644 swh/loader/bzr/tests/data/ghosts.tgz
 create mode 100644 swh/loader/bzr/tests/data/metadata-and-type-changes.sh
 create mode 100644 swh/loader/bzr/tests/data/metadata-and-type-changes.tgz
 create mode 100644 swh/loader/bzr/tests/data/needs-upgrade.sh
 create mode 100644 swh/loader/bzr/tests/data/needs-upgrade.tgz
 create mode 100644 swh/loader/bzr/tests/data/no-branch.sh
 create mode 100644 swh/loader/bzr/tests/data/no-branch.tgz
 create mode 100644 swh/loader/bzr/tests/data/nominal.sh
 create mode 100644 swh/loader/bzr/tests/data/nominal.tgz
 create mode 100644 swh/loader/bzr/tests/data/renames.sh
 create mode 100644 swh/loader/bzr/tests/data/renames.tgz
 create mode 100644 swh/loader/bzr/tests/test_loader.py
Changes applied before test
commit bc832e0cc16042551a19f6362edc0956b9aee977
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Feb 2 10:31:57 2022 +0100

    Add incremental support for the bzr loader

commit 0114e255c83190063bc2b6e46e8ccc77fa0048a8
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Fri Sep 24 17:08:27 2021 +0200

    Add a non-optimized non-incremental Bazaar loader
    
    The work for optimizing the loader has been paved with issues, in
    the mean time, this works and seems robust enough on a test run
    on all launchpad projects.
    
    The incremental logic will be added in a future patch.
    
    This also updates the requirements file to more explicitly match newer
    versions of the swh dependencies.

See https://jenkins.softwareheritage.org/job/DLDBZR/job/tests-on-diff/16/ for more details.

Harbormaster completed remote builds in B26437: Diff 25609.Feb 2 2022, 10:44 AM
Alphare requested review of this revision.Feb 2 2022, 10:44 AM
vlorentz added a subscriber: vlorentz.Feb 2 2022, 12:57 PM

Looks good, thanks

swh/loader/bzr/loader.py
567–570

you should find a way to batch requests to this endpoint eventually, but it's not a blocker for this diff.

568–571

why not this?

Alphare updated this revision to Diff 25615.Feb 2 2022, 2:10 PM

Remove useless list comprehension

swh-public-ci added a comment.Feb 2 2022, 2:15 PM

Build is green

Patch application report for D7059 (id=25615)

Could not rebase; Attempt merge onto 347c7b8375...

Updating 347c7b8..2a0df80
Fast-forward
 .gitignore                                         |  14 +
 .pre-commit-config.yaml                            |   1 -
 swh/foo/loader.py => conftest.py                   |  10 +-
 mypy.ini                                           |   6 +
 requirements-swh.txt                               |   6 +-
 requirements-test.txt                              |   4 +
 requirements.txt                                   |   1 +
 setup.py                                           |  13 +-
 swh/foo/tests/__init__.py                          |   0
 swh/foo/tests/test_loader.py                       |   3 -
 swh/loader/__init__.py                             |   3 +
 swh/loader/bzr/__init__.py                         |  16 +
 swh/loader/bzr/loader.py                           | 609 +++++++++++++++++++++
 swh/{foo => loader/bzr}/py.typed                   |   0
 swh/{foo => loader/bzr/tests}/__init__.py          |   0
 swh/loader/bzr/tests/conftest.py                   |  39 ++
 swh/loader/bzr/tests/data/broken-tags.sh           |   9 +
 swh/loader/bzr/tests/data/broken-tags.tgz          | Bin 0 -> 980 bytes
 swh/loader/bzr/tests/data/empty.sh                 |   6 +
 swh/loader/bzr/tests/data/empty.tgz                | Bin 0 -> 962 bytes
 swh/loader/bzr/tests/data/ghosts.py                |  13 +
 swh/loader/bzr/tests/data/ghosts.tgz               | Bin 0 -> 2451 bytes
 .../bzr/tests/data/metadata-and-type-changes.sh    |  38 ++
 .../bzr/tests/data/metadata-and-type-changes.tgz   | Bin 0 -> 13831 bytes
 swh/loader/bzr/tests/data/needs-upgrade.sh         |   7 +
 swh/loader/bzr/tests/data/needs-upgrade.tgz        | Bin 0 -> 880 bytes
 swh/loader/bzr/tests/data/no-branch.sh             |  10 +
 swh/loader/bzr/tests/data/no-branch.tgz            | Bin 0 -> 882 bytes
 swh/loader/bzr/tests/data/nominal.sh               |  36 ++
 swh/loader/bzr/tests/data/nominal.tgz              | Bin 0 -> 11366 bytes
 swh/loader/bzr/tests/data/renames.sh               |  17 +
 swh/loader/bzr/tests/data/renames.tgz              | Bin 0 -> 5068 bytes
 swh/loader/bzr/tests/test_loader.py                | 424 ++++++++++++++
 tox.ini                                            |   4 +-
 34 files changed, 1273 insertions(+), 16 deletions(-)
 create mode 100644 .gitignore
 rename swh/foo/loader.py => conftest.py (52%)
 delete mode 100644 swh/foo/tests/__init__.py
 delete mode 100644 swh/foo/tests/test_loader.py
 create mode 100644 swh/loader/__init__.py
 create mode 100644 swh/loader/bzr/__init__.py
 create mode 100644 swh/loader/bzr/loader.py
 rename swh/{foo => loader/bzr}/py.typed (100%)
 rename swh/{foo => loader/bzr/tests}/__init__.py (100%)
 create mode 100644 swh/loader/bzr/tests/conftest.py
 create mode 100644 swh/loader/bzr/tests/data/broken-tags.sh
 create mode 100644 swh/loader/bzr/tests/data/broken-tags.tgz
 create mode 100644 swh/loader/bzr/tests/data/empty.sh
 create mode 100644 swh/loader/bzr/tests/data/empty.tgz
 create mode 100644 swh/loader/bzr/tests/data/ghosts.py
 create mode 100644 swh/loader/bzr/tests/data/ghosts.tgz
 create mode 100644 swh/loader/bzr/tests/data/metadata-and-type-changes.sh
 create mode 100644 swh/loader/bzr/tests/data/metadata-and-type-changes.tgz
 create mode 100644 swh/loader/bzr/tests/data/needs-upgrade.sh
 create mode 100644 swh/loader/bzr/tests/data/needs-upgrade.tgz
 create mode 100644 swh/loader/bzr/tests/data/no-branch.sh
 create mode 100644 swh/loader/bzr/tests/data/no-branch.tgz
 create mode 100644 swh/loader/bzr/tests/data/nominal.sh
 create mode 100644 swh/loader/bzr/tests/data/nominal.tgz
 create mode 100644 swh/loader/bzr/tests/data/renames.sh
 create mode 100644 swh/loader/bzr/tests/data/renames.tgz
 create mode 100644 swh/loader/bzr/tests/test_loader.py
Changes applied before test
commit 2a0df804049af95f5a0230dd12c0803471ca6c15
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Feb 2 10:31:57 2022 +0100

    Add incremental support for the bzr loader

commit 0114e255c83190063bc2b6e46e8ccc77fa0048a8
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Fri Sep 24 17:08:27 2021 +0200

    Add a non-optimized non-incremental Bazaar loader
    
    The work for optimizing the loader has been paved with issues, in
    the mean time, this works and seems robust enough on a test run
    on all launchpad projects.
    
    The incremental logic will be added in a future patch.
    
    This also updates the requirements file to more explicitly match newer
    versions of the swh dependencies.

See https://jenkins.softwareheritage.org/job/DLDBZR/job/tests-on-diff/17/ for more details.

Harbormaster completed remote builds in B26445: Diff 25615.Feb 2 2022, 2:15 PM
Alphare marked an inline comment as done.Feb 2 2022, 2:27 PM
Alphare added inline comments.
swh/loader/bzr/loader.py
567–570

I'm not sure what the improvement would be. The endpoint is rarely accessed, only once or twice when doing an incremental run or with broken tags. Maybe I'm missing your point. :)

568–571

I actually have no idea why this is the way it is. I possibly needed to filter something and forgot about it later. Thanks for spotting it

ardumont accepted this revision.Feb 3 2022, 9:24 AM
ardumont added a subscriber: ardumont.

lgtm

couple of suggestions inline.

swh/loader/bzr/loader.py
489

do we need to continue iterating over the list if we found what we were looking for?

swh/loader/bzr/tests/test_loader.py
355–381

(or something)

as the last time in the hg loader ;)

423

same suggestion with the dict comparison.

This revision is now accepted and ready to land.Feb 3 2022, 9:24 AM
Alphare updated this revision to Diff 25647.Feb 3 2022, 9:37 AM
Alphare marked 3 inline comments as done.

Improve tests readability

swh/loader/bzr/loader.py
489

We do, since we're only appending to new_revisions after we've found the previous head.

swh-public-ci added a comment.Feb 3 2022, 9:42 AM

Build is green

Patch application report for D7059 (id=25647)

Could not rebase; Attempt merge onto 347c7b8375...

Updating 347c7b8..090756f
Fast-forward
 .gitignore                                         |  14 +
 .pre-commit-config.yaml                            |   1 -
 swh/foo/loader.py => conftest.py                   |  10 +-
 mypy.ini                                           |   6 +
 requirements-swh.txt                               |   6 +-
 requirements-test.txt                              |   4 +
 requirements.txt                                   |   1 +
 setup.py                                           |  13 +-
 swh/foo/tests/__init__.py                          |   0
 swh/foo/tests/test_loader.py                       |   3 -
 swh/loader/__init__.py                             |   3 +
 swh/loader/bzr/__init__.py                         |  16 +
 swh/loader/bzr/loader.py                           | 609 +++++++++++++++++++++
 swh/{foo => loader/bzr}/py.typed                   |   0
 swh/{foo => loader/bzr/tests}/__init__.py          |   0
 swh/loader/bzr/tests/conftest.py                   |  39 ++
 swh/loader/bzr/tests/data/broken-tags.sh           |   9 +
 swh/loader/bzr/tests/data/broken-tags.tgz          | Bin 0 -> 980 bytes
 swh/loader/bzr/tests/data/empty.sh                 |   6 +
 swh/loader/bzr/tests/data/empty.tgz                | Bin 0 -> 962 bytes
 swh/loader/bzr/tests/data/ghosts.py                |  13 +
 swh/loader/bzr/tests/data/ghosts.tgz               | Bin 0 -> 2451 bytes
 .../bzr/tests/data/metadata-and-type-changes.sh    |  38 ++
 .../bzr/tests/data/metadata-and-type-changes.tgz   | Bin 0 -> 13831 bytes
 swh/loader/bzr/tests/data/needs-upgrade.sh         |   7 +
 swh/loader/bzr/tests/data/needs-upgrade.tgz        | Bin 0 -> 880 bytes
 swh/loader/bzr/tests/data/no-branch.sh             |  10 +
 swh/loader/bzr/tests/data/no-branch.tgz            | Bin 0 -> 882 bytes
 swh/loader/bzr/tests/data/nominal.sh               |  36 ++
 swh/loader/bzr/tests/data/nominal.tgz              | Bin 0 -> 11366 bytes
 swh/loader/bzr/tests/data/renames.sh               |  17 +
 swh/loader/bzr/tests/data/renames.tgz              | Bin 0 -> 5068 bytes
 swh/loader/bzr/tests/test_loader.py                | 410 ++++++++++++++
 tox.ini                                            |   4 +-
 34 files changed, 1259 insertions(+), 16 deletions(-)
 create mode 100644 .gitignore
 rename swh/foo/loader.py => conftest.py (52%)
 delete mode 100644 swh/foo/tests/__init__.py
 delete mode 100644 swh/foo/tests/test_loader.py
 create mode 100644 swh/loader/__init__.py
 create mode 100644 swh/loader/bzr/__init__.py
 create mode 100644 swh/loader/bzr/loader.py
 rename swh/{foo => loader/bzr}/py.typed (100%)
 rename swh/{foo => loader/bzr/tests}/__init__.py (100%)
 create mode 100644 swh/loader/bzr/tests/conftest.py
 create mode 100644 swh/loader/bzr/tests/data/broken-tags.sh
 create mode 100644 swh/loader/bzr/tests/data/broken-tags.tgz
 create mode 100644 swh/loader/bzr/tests/data/empty.sh
 create mode 100644 swh/loader/bzr/tests/data/empty.tgz
 create mode 100644 swh/loader/bzr/tests/data/ghosts.py
 create mode 100644 swh/loader/bzr/tests/data/ghosts.tgz
 create mode 100644 swh/loader/bzr/tests/data/metadata-and-type-changes.sh
 create mode 100644 swh/loader/bzr/tests/data/metadata-and-type-changes.tgz
 create mode 100644 swh/loader/bzr/tests/data/needs-upgrade.sh
 create mode 100644 swh/loader/bzr/tests/data/needs-upgrade.tgz
 create mode 100644 swh/loader/bzr/tests/data/no-branch.sh
 create mode 100644 swh/loader/bzr/tests/data/no-branch.tgz
 create mode 100644 swh/loader/bzr/tests/data/nominal.sh
 create mode 100644 swh/loader/bzr/tests/data/nominal.tgz
 create mode 100644 swh/loader/bzr/tests/data/renames.sh
 create mode 100644 swh/loader/bzr/tests/data/renames.tgz
 create mode 100644 swh/loader/bzr/tests/test_loader.py
Changes applied before test
commit 090756f4f2f691963f88db037adf1a31350f5ddb
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Feb 2 10:31:57 2022 +0100

    Add incremental support for the bzr loader

commit 0114e255c83190063bc2b6e46e8ccc77fa0048a8
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Fri Sep 24 17:08:27 2021 +0200

    Add a non-optimized non-incremental Bazaar loader
    
    The work for optimizing the loader has been paved with issues, in
    the mean time, this works and seems robust enough on a test run
    on all launchpad projects.
    
    The incremental logic will be added in a future patch.
    
    This also updates the requirements file to more explicitly match newer
    versions of the swh dependencies.

See https://jenkins.softwareheritage.org/job/DLDBZR/job/tests-on-diff/18/ for more details.

Harbormaster completed remote builds in B26478: Diff 25647.Feb 3 2022, 9:42 AM
Alphare updated this revision to Diff 25659.Feb 3 2022, 12:08 PM

Rebase

swh-public-ci added a comment.Feb 3 2022, 12:11 PM

Build has FAILED

Patch application report for D7059 (id=25659)

Could not rebase; Attempt merge onto 347c7b8375...

Updating 347c7b8..fc1f8e7
Fast-forward
 .gitignore                                         |  14 +
 .pre-commit-config.yaml                            |   1 -
 swh/foo/loader.py => conftest.py                   |  10 +-
 mypy.ini                                           |   3 +
 requirements-swh.txt                               |   6 +-
 requirements-test.txt                              |   4 +
 requirements.txt                                   |   1 +
 setup.py                                           |  13 +-
 swh/foo/tests/__init__.py                          |   0
 swh/foo/tests/test_loader.py                       |   3 -
 swh/loader/__init__.py                             |   3 +
 swh/loader/bzr/__init__.py                         |  16 +
 swh/loader/bzr/loader.py                           | 609 +++++++++++++++++++++
 swh/{foo => loader/bzr}/py.typed                   |   0
 swh/{foo => loader/bzr/tests}/__init__.py          |   0
 swh/loader/bzr/tests/conftest.py                   |  39 ++
 swh/loader/bzr/tests/data/broken-tags.sh           |   9 +
 swh/loader/bzr/tests/data/broken-tags.tgz          | Bin 0 -> 980 bytes
 swh/loader/bzr/tests/data/empty.sh                 |   6 +
 swh/loader/bzr/tests/data/empty.tgz                | Bin 0 -> 962 bytes
 swh/loader/bzr/tests/data/ghosts.py                |  13 +
 swh/loader/bzr/tests/data/ghosts.tgz               | Bin 0 -> 2451 bytes
 .../bzr/tests/data/metadata-and-type-changes.sh    |  38 ++
 .../bzr/tests/data/metadata-and-type-changes.tgz   | Bin 0 -> 13831 bytes
 swh/loader/bzr/tests/data/needs-upgrade.sh         |   7 +
 swh/loader/bzr/tests/data/needs-upgrade.tgz        | Bin 0 -> 880 bytes
 swh/loader/bzr/tests/data/no-branch.sh             |  10 +
 swh/loader/bzr/tests/data/no-branch.tgz            | Bin 0 -> 882 bytes
 swh/loader/bzr/tests/data/nominal.sh               |  36 ++
 swh/loader/bzr/tests/data/nominal.tgz              | Bin 0 -> 11366 bytes
 swh/loader/bzr/tests/data/renames.sh               |  17 +
 swh/loader/bzr/tests/data/renames.tgz              | Bin 0 -> 5068 bytes
 swh/loader/bzr/tests/py.typed                      |   1 +
 swh/loader/bzr/tests/test_loader.py                | 410 ++++++++++++++
 tox.ini                                            |   4 +-
 35 files changed, 1257 insertions(+), 16 deletions(-)
 create mode 100644 .gitignore
 rename swh/foo/loader.py => conftest.py (52%)
 delete mode 100644 swh/foo/tests/__init__.py
 delete mode 100644 swh/foo/tests/test_loader.py
 create mode 100644 swh/loader/__init__.py
 create mode 100644 swh/loader/bzr/__init__.py
 create mode 100644 swh/loader/bzr/loader.py
 rename swh/{foo => loader/bzr}/py.typed (100%)
 rename swh/{foo => loader/bzr/tests}/__init__.py (100%)
 create mode 100644 swh/loader/bzr/tests/conftest.py
 create mode 100644 swh/loader/bzr/tests/data/broken-tags.sh
 create mode 100644 swh/loader/bzr/tests/data/broken-tags.tgz
 create mode 100644 swh/loader/bzr/tests/data/empty.sh
 create mode 100644 swh/loader/bzr/tests/data/empty.tgz
 create mode 100644 swh/loader/bzr/tests/data/ghosts.py
 create mode 100644 swh/loader/bzr/tests/data/ghosts.tgz
 create mode 100644 swh/loader/bzr/tests/data/metadata-and-type-changes.sh
 create mode 100644 swh/loader/bzr/tests/data/metadata-and-type-changes.tgz
 create mode 100644 swh/loader/bzr/tests/data/needs-upgrade.sh
 create mode 100644 swh/loader/bzr/tests/data/needs-upgrade.tgz
 create mode 100644 swh/loader/bzr/tests/data/no-branch.sh
 create mode 100644 swh/loader/bzr/tests/data/no-branch.tgz
 create mode 100644 swh/loader/bzr/tests/data/nominal.sh
 create mode 100644 swh/loader/bzr/tests/data/nominal.tgz
 create mode 100644 swh/loader/bzr/tests/data/renames.sh
 create mode 100644 swh/loader/bzr/tests/data/renames.tgz
 create mode 100644 swh/loader/bzr/tests/py.typed
 create mode 100644 swh/loader/bzr/tests/test_loader.py
Changes applied before test
commit fc1f8e738405e9cc2bfa5bb26439e17e4de53353
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Feb 2 10:31:57 2022 +0100

    Add incremental support for the bzr loader

commit 04b9a34cf7419e1d51bb05952f44713d420c18c9
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Fri Sep 24 17:08:27 2021 +0200

    Add a non-optimized non-incremental Bazaar loader
    
    The work for optimizing the loader has been paved with issues, in
    the mean time, this works and seems robust enough on a test run
    on all launchpad projects.
    
    The incremental logic will be added in a future patch.
    
    This also updates the requirements file to more explicitly match newer
    versions of the swh dependencies.

Link to build: https://jenkins.softwareheritage.org/job/DLDBZR/job/tests-on-diff/22/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBZR/job/tests-on-diff/22/console

Harbormaster failed remote builds in B26489: Diff 25659!Feb 3 2022, 12:11 PM
Alphare updated this revision to Diff 25662.Feb 3 2022, 3:04 PM

Re-run CI

swh-public-ci added a comment.Feb 3 2022, 3:09 PM

Build is green

Patch application report for D7059 (id=25662)

Could not rebase; Attempt merge onto 347c7b8375...

Updating 347c7b8..fc1f8e7
Fast-forward
 .gitignore                                         |  14 +
 .pre-commit-config.yaml                            |   1 -
 swh/foo/loader.py => conftest.py                   |  10 +-
 mypy.ini                                           |   3 +
 requirements-swh.txt                               |   6 +-
 requirements-test.txt                              |   4 +
 requirements.txt                                   |   1 +
 setup.py                                           |  13 +-
 swh/foo/tests/__init__.py                          |   0
 swh/foo/tests/test_loader.py                       |   3 -
 swh/loader/__init__.py                             |   3 +
 swh/loader/bzr/__init__.py                         |  16 +
 swh/loader/bzr/loader.py                           | 609 +++++++++++++++++++++
 swh/{foo => loader/bzr}/py.typed                   |   0
 swh/{foo => loader/bzr/tests}/__init__.py          |   0
 swh/loader/bzr/tests/conftest.py                   |  39 ++
 swh/loader/bzr/tests/data/broken-tags.sh           |   9 +
 swh/loader/bzr/tests/data/broken-tags.tgz          | Bin 0 -> 980 bytes
 swh/loader/bzr/tests/data/empty.sh                 |   6 +
 swh/loader/bzr/tests/data/empty.tgz                | Bin 0 -> 962 bytes
 swh/loader/bzr/tests/data/ghosts.py                |  13 +
 swh/loader/bzr/tests/data/ghosts.tgz               | Bin 0 -> 2451 bytes
 .../bzr/tests/data/metadata-and-type-changes.sh    |  38 ++
 .../bzr/tests/data/metadata-and-type-changes.tgz   | Bin 0 -> 13831 bytes
 swh/loader/bzr/tests/data/needs-upgrade.sh         |   7 +
 swh/loader/bzr/tests/data/needs-upgrade.tgz        | Bin 0 -> 880 bytes
 swh/loader/bzr/tests/data/no-branch.sh             |  10 +
 swh/loader/bzr/tests/data/no-branch.tgz            | Bin 0 -> 882 bytes
 swh/loader/bzr/tests/data/nominal.sh               |  36 ++
 swh/loader/bzr/tests/data/nominal.tgz              | Bin 0 -> 11366 bytes
 swh/loader/bzr/tests/data/renames.sh               |  17 +
 swh/loader/bzr/tests/data/renames.tgz              | Bin 0 -> 5068 bytes
 swh/loader/bzr/tests/py.typed                      |   1 +
 swh/loader/bzr/tests/test_loader.py                | 410 ++++++++++++++
 tox.ini                                            |   4 +-
 35 files changed, 1257 insertions(+), 16 deletions(-)
 create mode 100644 .gitignore
 rename swh/foo/loader.py => conftest.py (52%)
 delete mode 100644 swh/foo/tests/__init__.py
 delete mode 100644 swh/foo/tests/test_loader.py
 create mode 100644 swh/loader/__init__.py
 create mode 100644 swh/loader/bzr/__init__.py
 create mode 100644 swh/loader/bzr/loader.py
 rename swh/{foo => loader/bzr}/py.typed (100%)
 rename swh/{foo => loader/bzr/tests}/__init__.py (100%)
 create mode 100644 swh/loader/bzr/tests/conftest.py
 create mode 100644 swh/loader/bzr/tests/data/broken-tags.sh
 create mode 100644 swh/loader/bzr/tests/data/broken-tags.tgz
 create mode 100644 swh/loader/bzr/tests/data/empty.sh
 create mode 100644 swh/loader/bzr/tests/data/empty.tgz
 create mode 100644 swh/loader/bzr/tests/data/ghosts.py
 create mode 100644 swh/loader/bzr/tests/data/ghosts.tgz
 create mode 100644 swh/loader/bzr/tests/data/metadata-and-type-changes.sh
 create mode 100644 swh/loader/bzr/tests/data/metadata-and-type-changes.tgz
 create mode 100644 swh/loader/bzr/tests/data/needs-upgrade.sh
 create mode 100644 swh/loader/bzr/tests/data/needs-upgrade.tgz
 create mode 100644 swh/loader/bzr/tests/data/no-branch.sh
 create mode 100644 swh/loader/bzr/tests/data/no-branch.tgz
 create mode 100644 swh/loader/bzr/tests/data/nominal.sh
 create mode 100644 swh/loader/bzr/tests/data/nominal.tgz
 create mode 100644 swh/loader/bzr/tests/data/renames.sh
 create mode 100644 swh/loader/bzr/tests/data/renames.tgz
 create mode 100644 swh/loader/bzr/tests/py.typed
 create mode 100644 swh/loader/bzr/tests/test_loader.py
Changes applied before test
commit fc1f8e738405e9cc2bfa5bb26439e17e4de53353
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Wed Feb 2 10:31:57 2022 +0100

    Add incremental support for the bzr loader

commit 04b9a34cf7419e1d51bb05952f44713d420c18c9
Author: Raphaël Gomès <rgomes@octobus.net>
Date:   Fri Sep 24 17:08:27 2021 +0200

    Add a non-optimized non-incremental Bazaar loader
    
    The work for optimizing the loader has been paved with issues, in
    the mean time, this works and seems robust enough on a test run
    on all launchpad projects.
    
    The incremental logic will be added in a future patch.
    
    This also updates the requirements file to more explicitly match newer
    versions of the swh dependencies.

See https://jenkins.softwareheritage.org/job/DLDBZR/job/tests-on-diff/23/ for more details.

Harbormaster completed remote builds in B26493: Diff 25662.Feb 3 2022, 3:10 PM
Closed by commit rDLDBZRfc1f8e738405: Add incremental support for the bzr loader (authored by Alphare). · Explain WhyFeb 7 2022, 3:18 PM
This revision was automatically updated to reflect the committed changes.
Alphare added a commit: rDLDBZRfc1f8e738405: Add incremental support for the bzr loader.

Revision Contents
Changeset List

PathSize
swh/
loader/
bzr/
76 lines
tests/
113 lines

Diff 25752

View Options

swh/loader/bzr/loader.py

Loading...
View Options

swh/loader/bzr/tests/test_loader.py

Loading...
About · Developer information · Legal