Page MenuHomeSoftware Heritage

Make branch filtering a common behavior between the disk and remote loaders
ClosedPublic

Authored by olasd on May 28 2020, 11:12 AM.

Details

Summary

A further step would probably be making the branch filtering policy properly
configurable, rather than hard-coded in these functions. That way, we can decide
that we do want to archive all the branches from an archived repo, rather than
just the "live" branches that we'd commonly archive from remote repositories.

Depends on D3188.

Test Plan

existing tox test moved to the common test scaffold

Diff Detail

Repository
rDLDG Git loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D3189 (id=11331)

Could not rebase; Attempt merge onto dfd5be7b14...

Updating dfd5be7..cf82abf
Fast-forward
 swh/loader/git/from_disk.py            | 17 +++++++-
 swh/loader/git/loader.py               | 38 +++++++---------
 swh/loader/git/tests/test_from_disk.py | 79 +++++++++++++++++++++++++++++++++-
 swh/loader/git/tests/test_loader.py    | 49 +--------------------
 swh/loader/git/utils.py                | 46 ++++++++++++++++++++
 5 files changed, 155 insertions(+), 74 deletions(-)
Changes applied before test
commit cf82abfc83db590fbbb1638e9a31641a7bdbd17d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu May 28 11:08:21 2020 +0200

    Make branch filtering a common behavior between the disk and remote loaders
    
    A further step would probably be making the branch filtering policy properly
    configurable, rather than hard-coded in these functions. That way, we can decide
    that we do want to archive all the branches from an archived repo, rather than
    just the "live" branches that we'd commonly archive from remote repositories.

commit 0c31a06b94f6f7cccd1db85317afce2032229461
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu May 28 10:31:14 2020 +0200

    Add a default target for symbolic reference branches
    
    Some repositories are wrongly configured and they have symbolic references (e.g.
    a HEAD) that points to a non-existent branch. In the general case, we still want
    to load these origins; Create a dangling branch for these symbolic references to
    point to.

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/13/ for more details.

This revision is now accepted and ready to land.May 28 2020, 11:44 AM

Build is green

Patch application report for D3189 (id=11336)

Could not rebase; Attempt merge onto dfd5be7b14...

Updating dfd5be7..b5467f9
Fast-forward
 swh/loader/git/from_disk.py            | 17 +++++++-
 swh/loader/git/loader.py               | 38 +++++++---------
 swh/loader/git/tests/test_from_disk.py | 79 +++++++++++++++++++++++++++++++++-
 swh/loader/git/tests/test_loader.py    | 51 +---------------------
 swh/loader/git/utils.py                | 48 ++++++++++++++++++++-
 5 files changed, 157 insertions(+), 76 deletions(-)
Changes applied before test
commit b5467f9c3fae85e41c3b74e965d658e9448f7709
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu May 28 11:08:21 2020 +0200

    Make branch filtering a common behavior between the disk and remote loaders
    
    A further step would probably be making the branch filtering policy properly
    configurable, rather than hard-coded in these functions. That way, we can decide
    that we do want to archive all the branches from an archived repo, rather than
    just the "live" branches that we'd commonly archive from remote repositories.

commit ee22325a193dbc2b52b37496f3374c6bf0c1d1e1
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu May 28 10:31:14 2020 +0200

    Add a default target for symbolic reference branches
    
    Some repositories are wrongly configured and they have symbolic references (e.g.
    a HEAD) that points to a non-existent branch. In the general case, we still want
    to load these origins; Create a dangling branch for these symbolic references to
    point to.

See https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/15/ for more details.