Page MenuHomeSoftware Heritage
Differential D4312

Add `Hg20BundleLoader` tests from json files
ClosedPublic
Actions

Authored by acezar on Oct 19 2020, 10:42 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Commits
rDLDHGbc32e1280cfd: Add `Hg20BundleLoader` tests from json files
Summary

Generated json files with swh/loader/mercurial/tests/data/build.py for
existing repositories and added them to Hg20BundleLoader tests.

Introduce LoaderChecker as a standardized way to test repositories
against json files.

Diff Detail

Repository
rDLDHG Mercurial loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

acezar created this revision.Oct 19 2020, 10:42 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 19 2020, 10:42 PM
swh-public-ci added a comment.Oct 19 2020, 10:43 PM

Build has FAILED

Patch application report for D4312 (id=15241)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..db9a43d
Fast-forward
 setup.py                                          |   2 +
 swh/loader/mercurial/identify.py                  | 494 ++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py          | 187 ++++++++
 swh/loader/mercurial/tests/data/hello.json        |   1 +
 swh/loader/mercurial/tests/data/the-sandbox.json  |   1 +
 swh/loader/mercurial/tests/data/transplant.json   |   1 +
 swh/loader/mercurial/tests/dvcs_loader_checker.py | 132 ++++++
 swh/loader/mercurial/tests/test_identify.py       |  88 ++++
 swh/loader/mercurial/tests/test_loader.py         |  12 +
 9 files changed, 918 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/data/hello.json
 create mode 100644 swh/loader/mercurial/tests/data/the-sandbox.json
 create mode 100644 swh/loader/mercurial/tests/data/transplant.json
 create mode 100644 swh/loader/mercurial/tests/dvcs_loader_checker.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit db9a43d8c0b532a06c07fb357af81726e2f957a7
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.

commit 4a92800d8ab7cd108ad621b6d9da1533c09886d3
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder

commit b543125f433ab3022cc80c5b76e3a2eab0162b61
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/63/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/63/console

Harbormaster failed remote builds in B16473: Diff 15241!Oct 19 2020, 10:43 PM
acezar added a parent revision: D4311: Add testing repository builder.Oct 19 2020, 10:47 PM
acezar added a child revision: D4313: Add new example repository generated from script.
douardda added a subscriber: douardda.Oct 22 2020, 11:52 AM

Would be nice to have a README file in tests/data explaining what these json files are and how to produce them.

douardda accepted this revision.Oct 22 2020, 12:32 PM

globally ok, but please add a README file as suggested in the previous comment

This revision is now accepted and ready to land.Oct 22 2020, 12:32 PM
acezar updated this revision to Diff 15407.Oct 27 2020, 10:19 AM
acezar edited the summary of this revision. (Show Details)

Updated commit message

swh-public-ci added a comment.Oct 27 2020, 10:20 AM

Build has FAILED

Patch application report for D4312 (id=15407)

Rebasing onto bd914dec39...

Current branch diff-target is up to date.
Changes applied before test
commit f5a7a0cdf21fdf9823cff33895883638bb495415
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.
    
    Introduce `DVCSLoaderChecker` as a standardized way to test repositories
    against json files.

commit 5b0256683349ad892ffed749e7dc45d8b768e5fb
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder

commit c3d959fe1cf5bee461ff7366697f907a578d9288
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/69/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/69/console

Harbormaster failed remote builds in B16636: Diff 15407!Oct 27 2020, 10:20 AM
acezar updated this revision to Diff 15417.Oct 27 2020, 11:06 AM

Fighting arc

swh-public-ci added a comment.Oct 27 2020, 11:07 AM

Build has FAILED

Patch application report for D4312 (id=15417)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..f5a7a0c
Fast-forward
 setup.py                                          |   2 +
 swh/loader/mercurial/identify.py                  | 532 ++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py          | 187 ++++++++
 swh/loader/mercurial/tests/data/hello.json        |   1 +
 swh/loader/mercurial/tests/data/the-sandbox.json  |   1 +
 swh/loader/mercurial/tests/data/transplant.json   |   1 +
 swh/loader/mercurial/tests/dvcs_loader_checker.py | 132 ++++++
 swh/loader/mercurial/tests/test_identify.py       |  88 ++++
 swh/loader/mercurial/tests/test_loader.py         |  12 +
 9 files changed, 956 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/data/hello.json
 create mode 100644 swh/loader/mercurial/tests/data/the-sandbox.json
 create mode 100644 swh/loader/mercurial/tests/data/transplant.json
 create mode 100644 swh/loader/mercurial/tests/dvcs_loader_checker.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit f5a7a0cdf21fdf9823cff33895883638bb495415
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.
    
    Introduce `DVCSLoaderChecker` as a standardized way to test repositories
    against json files.

commit 5b0256683349ad892ffed749e7dc45d8b768e5fb
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder

commit c3d959fe1cf5bee461ff7366697f907a578d9288
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/73/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/73/console

Harbormaster failed remote builds in B16644: Diff 15417!Oct 27 2020, 11:08 AM
acezar updated this revision to Diff 15439.Oct 27 2020, 3:58 PM

failing at arc again

swh-public-ci added a comment.Oct 27 2020, 3:59 PM

Build has FAILED

Patch application report for D4312 (id=15439)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..175611c
Fast-forward
 setup.py                                          |   2 +
 swh/loader/mercurial/identify.py                  | 532 ++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py          | 265 +++++++++++
 swh/loader/mercurial/tests/data/hello.json        |   1 +
 swh/loader/mercurial/tests/data/the-sandbox.json  |   1 +
 swh/loader/mercurial/tests/data/transplant.json   |   1 +
 swh/loader/mercurial/tests/dvcs_loader_checker.py | 139 ++++++
 swh/loader/mercurial/tests/test_identify.py       |  88 ++++
 swh/loader/mercurial/tests/test_loader.py         |  12 +
 9 files changed, 1041 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/data/hello.json
 create mode 100644 swh/loader/mercurial/tests/data/the-sandbox.json
 create mode 100644 swh/loader/mercurial/tests/data/transplant.json
 create mode 100644 swh/loader/mercurial/tests/dvcs_loader_checker.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit 175611cfa1d8cc15a5131a8db920193def3cd4fd
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.
    
    Introduce `DVCSLoaderChecker` as a standardized way to test repositories
    against json files.

commit 838149f98d435fce7d809fbe1c324c4c42dc2ba7
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit c3d959fe1cf5bee461ff7366697f907a578d9288
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/78/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/78/console

Harbormaster failed remote builds in B16667: Diff 15439!Oct 27 2020, 3:59 PM
acezar updated this revision to Diff 15453.Oct 27 2020, 4:50 PM

Update

swh-public-ci added a comment.Oct 27 2020, 4:53 PM

Build has FAILED

Patch application report for D4312 (id=15453)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..c0d13d2
Fast-forward
 setup.py                                          |   2 +
 swh/loader/mercurial/identify.py                  | 532 ++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py          | 265 +++++++++++
 swh/loader/mercurial/tests/data/hello.json        |   1 +
 swh/loader/mercurial/tests/data/the-sandbox.json  |   1 +
 swh/loader/mercurial/tests/data/transplant.json   |   1 +
 swh/loader/mercurial/tests/dvcs_loader_checker.py | 144 ++++++
 swh/loader/mercurial/tests/test_identify.py       |  93 ++++
 swh/loader/mercurial/tests/test_loader.py         |  12 +
 9 files changed, 1051 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/data/hello.json
 create mode 100644 swh/loader/mercurial/tests/data/the-sandbox.json
 create mode 100644 swh/loader/mercurial/tests/data/transplant.json
 create mode 100644 swh/loader/mercurial/tests/dvcs_loader_checker.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit c0d13d25aeaf4658a10ef6c1330208332328205f
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.
    
    Introduce `DVCSLoaderChecker` as a standardized way to test repositories
    against json files.

commit b4e053fdcdd4cd7cdaef915de79c89829186b067
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit e75b99287ba58bdbc6cb95703f11303c561a8f34
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/83/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/83/console

Harbormaster failed remote builds in B16681: Diff 15453!Oct 27 2020, 4:53 PM
acezar updated this revision to Diff 15786.Nov 10 2020, 2:53 PM

Update to make ci pass

swh-public-ci added a comment.Nov 10 2020, 2:55 PM

Build is green

Patch application report for D4312 (id=15786)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..e4eb5c2
Fast-forward
 requirements.txt                                  |   1 +
 setup.py                                          |   2 +
 swh/loader/mercurial/identify.py                  | 533 ++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py          | 265 +++++++++++
 swh/loader/mercurial/tests/data/hello.json        |   1 +
 swh/loader/mercurial/tests/data/the-sandbox.json  |   1 +
 swh/loader/mercurial/tests/data/transplant.json   |   1 +
 swh/loader/mercurial/tests/dvcs_loader_checker.py | 144 ++++++
 swh/loader/mercurial/tests/test_identify.py       |  93 ++++
 swh/loader/mercurial/tests/test_loader.py         |  12 +
 10 files changed, 1053 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/data/hello.json
 create mode 100644 swh/loader/mercurial/tests/data/the-sandbox.json
 create mode 100644 swh/loader/mercurial/tests/data/transplant.json
 create mode 100644 swh/loader/mercurial/tests/dvcs_loader_checker.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit e4eb5c21afc064812c692b6b36fbbbf1801b3552
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.
    
    Introduce `DVCSLoaderChecker` as a standardized way to test repositories
    against json files.

commit ea7cc149edbee8778121fc37a099edaedc584f60
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit 00b709bae8c01854fa9eb37e2b5d9989fa68a17d
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/90/ for more details.

Harbormaster completed remote builds in B16997: Diff 15786.Nov 10 2020, 2:55 PM
acezar updated this revision to Diff 15862.Nov 12 2020, 11:52 PM

Change DVCSLoaderChecker to LoaderChecker.

As HgLoaderFromDisk will stop inherit from DVCSLoader intercepting
objects from the interface to make assertions as things progress is not feasible
anymore. LoaderChecker check hashes from storage instead.

swh-public-ci added a comment.Nov 12 2020, 11:53 PM

Build is green

Patch application report for D4312 (id=15862)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..bc32e12
Fast-forward
 requirements.txt                                 |   1 +
 setup.py                                         |   2 +
 swh/loader/mercurial/identify.py                 | 541 +++++++++++++++++++++++
 swh/loader/mercurial/tests/data/build.py         | 265 +++++++++++
 swh/loader/mercurial/tests/data/hello.json       |   1 +
 swh/loader/mercurial/tests/data/the-sandbox.json |   1 +
 swh/loader/mercurial/tests/data/transplant.json  |   1 +
 swh/loader/mercurial/tests/loader_checker.py     |  74 ++++
 swh/loader/mercurial/tests/test_identify.py      |  74 ++++
 swh/loader/mercurial/tests/test_loader.py        |  12 +
 10 files changed, 972 insertions(+)
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/data/hello.json
 create mode 100644 swh/loader/mercurial/tests/data/the-sandbox.json
 create mode 100644 swh/loader/mercurial/tests/data/transplant.json
 create mode 100644 swh/loader/mercurial/tests/loader_checker.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
Changes applied before test
commit bc32e1280cfd6a59df595cdcbcc2c2b51b3618aa
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.
    
    Introduce `LoaderChecker` as a standardized way to test repositories
    against json files.

commit ff11f77f1b493bd1c8ed257e790ded8da276101c
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit a2e9cf16919a5f81a06f955a533a254a9b3c9689
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/93/ for more details.

Harbormaster completed remote builds in B17071: Diff 15862.Nov 12 2020, 11:53 PM
acezar edited the summary of this revision. (Show Details)Nov 12 2020, 11:56 PM
Closed by commit rDLDHGbc32e1280cfd: Add `Hg20BundleLoader` tests from json files (authored by acezar). · Explain WhyNov 27 2020, 5:29 PM
This revision was automatically updated to reflect the committed changes.
acezar added a commit: rDLDHGbc32e1280cfd: Add `Hg20BundleLoader` tests from json files.

Revision Contents
Changeset List

PathSize
swh/
loader/
mercurial/
tests/
data/
1 line
1 line
1 line
74 lines
12 lines

Diff 16418

View Options

swh/loader/mercurial/tests/data/hello.json

Loading...
View Options

swh/loader/mercurial/tests/data/the-sandbox.json

Loading...
View Options

swh/loader/mercurial/tests/data/transplant.json

Loading...
View Options

swh/loader/mercurial/tests/loader_checker.py

Loading...
View Options

swh/loader/mercurial/tests/test_loader.py

Loading...
About · Developer information · Legal