Page MenuHomeSoftware Heritage
Differential D4541

Add content lru cache to HgLoaderFromDisk
ClosedPublic
Actions

Authored by acezar on Nov 20 2020, 11:15 AM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Commits
rDLDHGd3885c7f6e7a: Add content lru cache to HgLoaderFromDisk
Summary

Avoid recalculation of unchanged content hash between revisions

Diff Detail

Repository
rDLDHG Mercurial loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

acezar created this revision.Nov 20 2020, 11:15 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 20 2020, 11:15 AM
swh-public-ci added a comment.Nov 20 2020, 11:16 AM

Build is green

Patch application report for D4541 (id=16105)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..bd64586
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   2 +
 swh/loader/mercurial/__init__.py                   |   4 +-
 swh/loader/mercurial/cli.py                        |   6 +-
 swh/loader/mercurial/from_bundle.py                | 641 ++++++++++++++++++++
 swh/loader/mercurial/from_disk.py                  | 467 +++++++++++++++
 swh/loader/mercurial/hgutil.py                     |  66 +++
 swh/loader/mercurial/identify.py                   | 541 +++++++++++++++++
 swh/loader/mercurial/loader.py                     | 645 +--------------------
 swh/loader/mercurial/tasks.py                      |   8 +-
 swh/loader/mercurial/tests/data/build.py           | 265 +++++++++
 swh/loader/mercurial/tests/data/example.json       |   1 +
 swh/loader/mercurial/tests/data/example.sh         |  59 ++
 swh/loader/mercurial/tests/data/example.tgz        | Bin 0 -> 51200 bytes
 swh/loader/mercurial/tests/data/hello.json         |   1 +
 swh/loader/mercurial/tests/data/the-sandbox.json   |   1 +
 swh/loader/mercurial/tests/data/transplant.json    |   1 +
 swh/loader/mercurial/tests/loader_checker.py       |  74 +++
 .../tests/{test_loader.py => test_from_bundle.py}  |  14 +-
 swh/loader/mercurial/tests/test_from_disk.py       | 209 +++++++
 swh/loader/mercurial/tests/test_hgutil.py          |  46 ++
 swh/loader/mercurial/tests/test_identify.py        |  74 +++
 swh/loader/mercurial/tests/test_loader.org         | 121 ----
 swh/loader/mercurial/tests/test_tasks.py           |   6 +-
 24 files changed, 2477 insertions(+), 776 deletions(-)
 create mode 100644 swh/loader/mercurial/from_bundle.py
 create mode 100644 swh/loader/mercurial/from_disk.py
 create mode 100644 swh/loader/mercurial/hgutil.py
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/data/example.json
 create mode 100644 swh/loader/mercurial/tests/data/example.sh
 create mode 100644 swh/loader/mercurial/tests/data/example.tgz
 create mode 100644 swh/loader/mercurial/tests/data/hello.json
 create mode 100644 swh/loader/mercurial/tests/data/the-sandbox.json
 create mode 100644 swh/loader/mercurial/tests/data/transplant.json
 create mode 100644 swh/loader/mercurial/tests/loader_checker.py
 rename swh/loader/mercurial/tests/{test_loader.py => test_from_bundle.py} (93%)
 create mode 100644 swh/loader/mercurial/tests/test_from_disk.py
 create mode 100644 swh/loader/mercurial/tests/test_hgutil.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
 delete mode 100644 swh/loader/mercurial/tests/test_loader.org
Changes applied before test
commit bd6458613df54e232bddd760967a2799dff84d39
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Nov 20 11:11:39 2020 +0100

    Add content lru cache to HgLoaderFromDisk
    
    Avoid recalculation of unchanged content hash between revisions

commit ee3c158d22394a8a4a989b55b3c92890838085bf
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Wed Oct 28 11:33:17 2020 +0100

    Add mercurial.from_disk.HgLoaderFromDisk
    
    Rather than relying on mercurial bundles this loader expect a local repository.

commit c8c91ab674a9ade49caacd63a5b507bab67df9dc
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:42:21 2020 +0200

    Add new example repository generated from script
    
    First updatable example repository documented by its generation script.

commit bc32e1280cfd6a59df595cdcbcc2c2b51b3618aa
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.
    
    Introduce `LoaderChecker` as a standardized way to test repositories
    against json files.

commit ff11f77f1b493bd1c8ed257e790ded8da276101c
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit a2e9cf16919a5f81a06f955a533a254a9b3c9689
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/102/ for more details.

Harbormaster completed remote builds in B17303: Diff 16105.Nov 20 2020, 11:16 AM
acezar added a parent revision: D3435: Add mercurial.from_disk.HgLoaderFromDisk.Nov 20 2020, 11:16 AM
acezar added a child revision: D4540: Add tree diffing in HgLoaderFromDisk.Nov 20 2020, 11:44 AM
marmoute added a subscriber: marmoute.Nov 20 2020, 12:08 PM

I have two small feedback. this looks good otherwise.

swh/loader/mercurial/from_disk.py
13

I am not super happy to import some internal Mercurial class, but there don't seems to be an easily accessible lru cache implementation otherwise.

Could we move this in hgutil to avoid importing Mercurial directly from here ?

365

This should be file_ctx.filenode(), should it not ?

douardda accepted this revision.Nov 20 2020, 3:37 PM
douardda added a subscriber: douardda.

Besides @marmoute 's comments, lgtm

This revision is now accepted and ready to land.Nov 20 2020, 3:37 PM
acezar updated this revision to Diff 16174.Nov 23 2020, 12:17 PM
acezar marked 2 inline comments as done.

Followup

swh-public-ci added a comment.Nov 23 2020, 12:19 PM

Build is green

Patch application report for D4541 (id=16174)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..ff6c586
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   2 +
 swh/loader/mercurial/__init__.py                   |   4 +-
 swh/loader/mercurial/cli.py                        |   6 +-
 swh/loader/mercurial/from_bundle.py                | 641 ++++++++++++++++++++
 swh/loader/mercurial/from_disk.py                  | 471 +++++++++++++++
 swh/loader/mercurial/hgutil.py                     |  78 +++
 swh/loader/mercurial/identify.py                   | 541 +++++++++++++++++
 swh/loader/mercurial/loader.py                     | 645 +--------------------
 swh/loader/mercurial/tasks.py                      |   8 +-
 swh/loader/mercurial/tests/data/build.py           | 265 +++++++++
 swh/loader/mercurial/tests/data/example.json       |   1 +
 swh/loader/mercurial/tests/data/example.sh         |  59 ++
 swh/loader/mercurial/tests/data/example.tgz        | Bin 0 -> 51200 bytes
 swh/loader/mercurial/tests/data/hello.json         |   1 +
 swh/loader/mercurial/tests/data/the-sandbox.json   |   1 +
 swh/loader/mercurial/tests/data/transplant.json    |   1 +
 swh/loader/mercurial/tests/loader_checker.py       |  74 +++
 .../tests/{test_loader.py => test_from_bundle.py}  |  14 +-
 swh/loader/mercurial/tests/test_from_disk.py       | 199 +++++++
 swh/loader/mercurial/tests/test_hgutil.py          |  46 ++
 swh/loader/mercurial/tests/test_identify.py        |  74 +++
 swh/loader/mercurial/tests/test_loader.org         | 121 ----
 swh/loader/mercurial/tests/test_tasks.py           |   6 +-
 24 files changed, 2485 insertions(+), 774 deletions(-)
 create mode 100644 swh/loader/mercurial/from_bundle.py
 create mode 100644 swh/loader/mercurial/from_disk.py
 create mode 100644 swh/loader/mercurial/hgutil.py
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/data/example.json
 create mode 100644 swh/loader/mercurial/tests/data/example.sh
 create mode 100644 swh/loader/mercurial/tests/data/example.tgz
 create mode 100644 swh/loader/mercurial/tests/data/hello.json
 create mode 100644 swh/loader/mercurial/tests/data/the-sandbox.json
 create mode 100644 swh/loader/mercurial/tests/data/transplant.json
 create mode 100644 swh/loader/mercurial/tests/loader_checker.py
 rename swh/loader/mercurial/tests/{test_loader.py => test_from_bundle.py} (93%)
 create mode 100644 swh/loader/mercurial/tests/test_from_disk.py
 create mode 100644 swh/loader/mercurial/tests/test_hgutil.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
 delete mode 100644 swh/loader/mercurial/tests/test_loader.org
Changes applied before test
commit ff6c5860ffe8becb8a6bc9389b1f1119e2c54254
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Nov 20 11:11:39 2020 +0100

    Add content lru cache to HgLoaderFromDisk
    
    Summary: Avoid recalculation of unchanged content hash between revisions
    
    Reviewers: #reviewers
    
    Differential Revision: https://forge.softwareheritage.org/D4541

commit 6499cb88483cedd834267e91b01a5d14b2c4920e
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Wed Oct 28 11:33:17 2020 +0100

    Add mercurial.from_disk.HgLoaderFromDisk
    
    Rather than relying on mercurial bundles this loader expect a local repository.
    
    Differential Revision: https://forge.softwareheritage.org/D3435

commit c8c91ab674a9ade49caacd63a5b507bab67df9dc
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:42:21 2020 +0200

    Add new example repository generated from script
    
    First updatable example repository documented by its generation script.

commit bc32e1280cfd6a59df595cdcbcc2c2b51b3618aa
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.
    
    Introduce `LoaderChecker` as a standardized way to test repositories
    against json files.

commit ff11f77f1b493bd1c8ed257e790ded8da276101c
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit a2e9cf16919a5f81a06f955a533a254a9b3c9689
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/106/ for more details.

Harbormaster completed remote builds in B17374: Diff 16174.Nov 23 2020, 12:19 PM
acezar updated this revision to Diff 16203.Nov 23 2020, 4:15 PM

Followup

swh-public-ci added a comment.Nov 23 2020, 4:17 PM

Build is green

Patch application report for D4541 (id=16203)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..9074bd9
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   2 +
 swh/loader/mercurial/__init__.py                   |   4 +-
 swh/loader/mercurial/cli.py                        |   6 +-
 swh/loader/mercurial/from_bundle.py                | 641 ++++++++++++++++++++
 swh/loader/mercurial/from_disk.py                  | 471 +++++++++++++++
 swh/loader/mercurial/hgutil.py                     |  78 +++
 swh/loader/mercurial/identify.py                   | 541 +++++++++++++++++
 swh/loader/mercurial/loader.py                     | 645 +--------------------
 swh/loader/mercurial/tasks.py                      |   8 +-
 swh/loader/mercurial/tests/data/build.py           | 265 +++++++++
 swh/loader/mercurial/tests/data/example.json       |   1 +
 swh/loader/mercurial/tests/data/example.sh         |  59 ++
 swh/loader/mercurial/tests/data/example.tgz        | Bin 0 -> 51200 bytes
 swh/loader/mercurial/tests/data/hello.json         |   1 +
 swh/loader/mercurial/tests/data/the-sandbox.json   |   1 +
 swh/loader/mercurial/tests/data/transplant.json    |   1 +
 swh/loader/mercurial/tests/loader_checker.py       |  74 +++
 .../tests/{test_loader.py => test_from_bundle.py}  |  14 +-
 swh/loader/mercurial/tests/test_from_disk.py       | 199 +++++++
 swh/loader/mercurial/tests/test_hgutil.py          |  46 ++
 swh/loader/mercurial/tests/test_identify.py        |  74 +++
 swh/loader/mercurial/tests/test_loader.org         | 121 ----
 swh/loader/mercurial/tests/test_tasks.py           |   6 +-
 24 files changed, 2485 insertions(+), 774 deletions(-)
 create mode 100644 swh/loader/mercurial/from_bundle.py
 create mode 100644 swh/loader/mercurial/from_disk.py
 create mode 100644 swh/loader/mercurial/hgutil.py
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/data/example.json
 create mode 100644 swh/loader/mercurial/tests/data/example.sh
 create mode 100644 swh/loader/mercurial/tests/data/example.tgz
 create mode 100644 swh/loader/mercurial/tests/data/hello.json
 create mode 100644 swh/loader/mercurial/tests/data/the-sandbox.json
 create mode 100644 swh/loader/mercurial/tests/data/transplant.json
 create mode 100644 swh/loader/mercurial/tests/loader_checker.py
 rename swh/loader/mercurial/tests/{test_loader.py => test_from_bundle.py} (93%)
 create mode 100644 swh/loader/mercurial/tests/test_from_disk.py
 create mode 100644 swh/loader/mercurial/tests/test_hgutil.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
 delete mode 100644 swh/loader/mercurial/tests/test_loader.org
Changes applied before test
commit 9074bd977debf9b5412fcad025c98f5630d05666
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Nov 20 11:11:39 2020 +0100

    Add content lru cache to HgLoaderFromDisk
    
    Summary: Avoid recalculation of unchanged content hash between revisions
    
    Reviewers: #reviewers
    
    Differential Revision: https://forge.softwareheritage.org/D4541

commit b35536071623338213dcf22352c3a8f332b32344
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Wed Oct 28 11:33:17 2020 +0100

    Add mercurial.from_disk.HgLoaderFromDisk
    
    Rather than relying on mercurial bundles this loader expect a local repository.
    
    Differential Revision: https://forge.softwareheritage.org/D3435

commit c8c91ab674a9ade49caacd63a5b507bab67df9dc
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:42:21 2020 +0200

    Add new example repository generated from script
    
    First updatable example repository documented by its generation script.

commit bc32e1280cfd6a59df595cdcbcc2c2b51b3618aa
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.
    
    Introduce `LoaderChecker` as a standardized way to test repositories
    against json files.

commit ff11f77f1b493bd1c8ed257e790ded8da276101c
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit a2e9cf16919a5f81a06f955a533a254a9b3c9689
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/108/ for more details.

Harbormaster completed remote builds in B17403: Diff 16203.Nov 23 2020, 4:17 PM
acezar updated this revision to Diff 16258.Nov 24 2020, 2:10 PM

Followup

swh-public-ci added a comment.Nov 24 2020, 2:12 PM

Build is green

Patch application report for D4541 (id=16258)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..908349f
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   3 +
 swh/loader/mercurial/__init__.py                   |  10 +
 swh/loader/mercurial/from_disk.py                  | 485 ++++++++++++++++++
 swh/loader/mercurial/hgutil.py                     |  78 +++
 swh/loader/mercurial/identify.py                   | 541 +++++++++++++++++++++
 swh/loader/mercurial/tasks_from_disk.py            |  33 ++
 swh/loader/mercurial/tests/data/build.py           | 265 ++++++++++
 swh/loader/mercurial/tests/data/example.json       |   1 +
 swh/loader/mercurial/tests/data/example.sh         |  59 +++
 swh/loader/mercurial/tests/data/example.tgz        | Bin 0 -> 51200 bytes
 swh/loader/mercurial/tests/data/hello.json         |   1 +
 swh/loader/mercurial/tests/data/the-sandbox.json   |   1 +
 swh/loader/mercurial/tests/data/transplant.json    |   1 +
 swh/loader/mercurial/tests/loader_checker.py       |  74 +++
 swh/loader/mercurial/tests/test_from_disk.py       | 199 ++++++++
 swh/loader/mercurial/tests/test_hgutil.py          |  46 ++
 swh/loader/mercurial/tests/test_identify.py        |  74 +++
 swh/loader/mercurial/tests/test_loader.org         | 121 -----
 swh/loader/mercurial/tests/test_tasks_from_disk.py |  47 ++
 20 files changed, 1919 insertions(+), 121 deletions(-)
 create mode 100644 swh/loader/mercurial/from_disk.py
 create mode 100644 swh/loader/mercurial/hgutil.py
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100644 swh/loader/mercurial/tasks_from_disk.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/data/example.json
 create mode 100644 swh/loader/mercurial/tests/data/example.sh
 create mode 100644 swh/loader/mercurial/tests/data/example.tgz
 create mode 100644 swh/loader/mercurial/tests/data/hello.json
 create mode 100644 swh/loader/mercurial/tests/data/the-sandbox.json
 create mode 100644 swh/loader/mercurial/tests/data/transplant.json
 create mode 100644 swh/loader/mercurial/tests/loader_checker.py
 create mode 100644 swh/loader/mercurial/tests/test_from_disk.py
 create mode 100644 swh/loader/mercurial/tests/test_hgutil.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
 delete mode 100644 swh/loader/mercurial/tests/test_loader.org
 create mode 100644 swh/loader/mercurial/tests/test_tasks_from_disk.py
Changes applied before test
commit 908349f1155e8f4abe939e6ae7b0d53419545f58
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Nov 20 11:11:39 2020 +0100

    Add content lru cache to HgLoaderFromDisk
    
    Summary: Avoid recalculation of unchanged content hash between revisions
    
    Reviewers: #reviewers
    
    Differential Revision: https://forge.softwareheritage.org/D4541

commit ede3e31d7b8b654c81607a967982e7330d88c98a
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Wed Oct 28 11:33:17 2020 +0100

    Add mercurial.from_disk.HgLoaderFromDisk
    
    Rather than relying on mercurial bundles this loader expect a local repository.
    
    Differential Revision: https://forge.softwareheritage.org/D3435

commit c8c91ab674a9ade49caacd63a5b507bab67df9dc
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:42:21 2020 +0200

    Add new example repository generated from script
    
    First updatable example repository documented by its generation script.

commit bc32e1280cfd6a59df595cdcbcc2c2b51b3618aa
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.
    
    Introduce `LoaderChecker` as a standardized way to test repositories
    against json files.

commit ff11f77f1b493bd1c8ed257e790ded8da276101c
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit a2e9cf16919a5f81a06f955a533a254a9b3c9689
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/112/ for more details.

Harbormaster completed remote builds in B17451: Diff 16258.Nov 24 2020, 2:12 PM
acezar updated this revision to Diff 16278.Nov 24 2020, 7:20 PM

Followup

swh-public-ci added a comment.Nov 24 2020, 7:22 PM

Build is green

Patch application report for D4541 (id=16278)

Could not rebase; Attempt merge onto bd914dec39...

Updating bd914de..bd98bad
Fast-forward
 requirements.txt                                   |   1 +
 setup.py                                           |   3 +
 swh/loader/mercurial/__init__.py                   |  10 +
 swh/loader/mercurial/from_disk.py                  | 492 +++++++++++++++++++
 swh/loader/mercurial/hgutil.py                     |  78 +++
 swh/loader/mercurial/identify.py                   | 541 +++++++++++++++++++++
 swh/loader/mercurial/tasks_from_disk.py            |  33 ++
 swh/loader/mercurial/tests/data/build.py           | 265 ++++++++++
 swh/loader/mercurial/tests/data/example.json       |   1 +
 swh/loader/mercurial/tests/data/example.sh         |  59 +++
 swh/loader/mercurial/tests/data/example.tgz        | Bin 0 -> 51200 bytes
 swh/loader/mercurial/tests/data/hello.json         |   1 +
 swh/loader/mercurial/tests/data/the-sandbox.json   |   1 +
 swh/loader/mercurial/tests/data/transplant.json    |   1 +
 swh/loader/mercurial/tests/loader_checker.py       |  74 +++
 swh/loader/mercurial/tests/test_from_disk.py       | 199 ++++++++
 swh/loader/mercurial/tests/test_hgutil.py          |  46 ++
 swh/loader/mercurial/tests/test_identify.py        |  74 +++
 swh/loader/mercurial/tests/test_loader.org         | 121 -----
 swh/loader/mercurial/tests/test_tasks_from_disk.py |  47 ++
 20 files changed, 1926 insertions(+), 121 deletions(-)
 create mode 100644 swh/loader/mercurial/from_disk.py
 create mode 100644 swh/loader/mercurial/hgutil.py
 create mode 100644 swh/loader/mercurial/identify.py
 create mode 100644 swh/loader/mercurial/tasks_from_disk.py
 create mode 100755 swh/loader/mercurial/tests/data/build.py
 create mode 100644 swh/loader/mercurial/tests/data/example.json
 create mode 100644 swh/loader/mercurial/tests/data/example.sh
 create mode 100644 swh/loader/mercurial/tests/data/example.tgz
 create mode 100644 swh/loader/mercurial/tests/data/hello.json
 create mode 100644 swh/loader/mercurial/tests/data/the-sandbox.json
 create mode 100644 swh/loader/mercurial/tests/data/transplant.json
 create mode 100644 swh/loader/mercurial/tests/loader_checker.py
 create mode 100644 swh/loader/mercurial/tests/test_from_disk.py
 create mode 100644 swh/loader/mercurial/tests/test_hgutil.py
 create mode 100644 swh/loader/mercurial/tests/test_identify.py
 delete mode 100644 swh/loader/mercurial/tests/test_loader.org
 create mode 100644 swh/loader/mercurial/tests/test_tasks_from_disk.py
Changes applied before test
commit bd98badaa4e3c25363a905f17f91ecc06a42af29
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Nov 20 11:11:39 2020 +0100

    Add content lru cache to HgLoaderFromDisk
    
    Avoid recalculation of unchanged content hash between revisions

commit f14a65f97b272db087b2ad823dbdeb44fda768e0
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Wed Oct 28 11:33:17 2020 +0100

    Add mercurial.from_disk.HgLoaderFromDisk
    
    Rather than relying on mercurial bundles this loader expect a local repository.

commit c8c91ab674a9ade49caacd63a5b507bab67df9dc
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:42:21 2020 +0200

    Add new example repository generated from script
    
    First updatable example repository documented by its generation script.

commit bc32e1280cfd6a59df595cdcbcc2c2b51b3618aa
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Oct 19 16:22:07 2020 +0200

    Add `Hg20BundleLoader` tests from json files
    
    Generated json files with `swh/loader/mercurial/tests/data/build.py` for
    existing repositories and added them to `Hg20BundleLoader` tests.
    
    Introduce `LoaderChecker` as a standardized way to test repositories
    against json files.

commit ff11f77f1b493bd1c8ed257e790ded8da276101c
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Oct 16 11:28:35 2020 +0200

    Add testing repository builder
    
    This build script purpose is to create example repositories from bash scripts
    and extract assertion data from them into json files.
    
    Advantages:
    
        - the bash script documents the repository creation
        - automating creation allow easy repository update
        - automation extraction allow easier update of assertion data

commit a2e9cf16919a5f81a06f955a533a254a9b3c9689
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Oct 8 18:07:50 2020 +0200

    add swh-hg-identify a cli to identify hg objects

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/115/ for more details.

Harbormaster completed remote builds in B17471: Diff 16278.Nov 24 2020, 7:22 PM
acezar mentioned this in D3435: Add mercurial.from_disk.HgLoaderFromDisk.Nov 25 2020, 2:52 PM
douardda accepted this revision.Dec 1 2020, 5:33 PM
acezar updated this revision to Diff 16467.Dec 1 2020, 5:50 PM

Update commit hash

swh-public-ci added a comment.Dec 1 2020, 5:51 PM

Build is green

Patch application report for D4541 (id=16467)

Rebasing onto a1c8afa5e4...

Current branch diff-target is up to date.
Changes applied before test
commit d3885c7f6e7a5ab19bb226576cb656ab7501106e
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Fri Nov 20 11:11:39 2020 +0100

    Add content lru cache to HgLoaderFromDisk
    
    Avoid recalculation of unchanged content hash between revisions

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/131/ for more details.

Harbormaster completed remote builds in B17653: Diff 16467.Dec 1 2020, 5:51 PM
acezar added a comment.Dec 1 2020, 5:53 PM

I'm really bad at fabricator... yet some small un pushed changes mainly doc and renaming vars

Closed by commit rDLDHGd3885c7f6e7a: Add content lru cache to HgLoaderFromDisk (authored by acezar). · Explain WhyDec 1 2020, 5:54 PM
This revision was automatically updated to reflect the committed changes.
acezar added a commit: rDLDHGd3885c7f6e7a: Add content lru cache to HgLoaderFromDisk.

Revision Contents
Changeset List

PathSize
swh/
loader/
mercurial/
19 lines
3 lines

Diff 16468

View Options

swh/loader/mercurial/from_disk.py

Loading...
View Options

swh/loader/mercurial/hgutil.py

Loading...
About · Developer information · Legal