Page MenuHomeSoftware Heritage

HgLoaderFromDisk: uneventful load when unchanged
ClosedPublic

Authored by Alphare on Dec 2 2020, 11:54 AM.

Details

Summary

By looking at the previous snapshot heads, loading of an unchanged repository
will be uneventful.

Diff Detail

Repository
rDLDHG Mercurial loader
Branch
D4643
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17748
Build 27435: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 27434: arc lint + arc unit

Unit TestsFailed

TimeTest
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_tasks::test_archive_loader
swh_scheduler_celery_app = <Celery celery.tests at 0x7fd9dd768278> swh_scheduler_celery_includes = ['swh.scheduler.tests.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.mercurial.tasks_from_disk', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', ...]
11,817 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_tasks::test_loader
swh_scheduler_celery_app = <Celery celery.tests at 0x7fd9dd768278> swh_scheduler_celery_includes = ['swh.scheduler.tests.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.mercurial.tasks_from_disk', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', ...]
140 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_tasks_from_disk::test_archive_loader
swh_scheduler_celery_app = <Celery celery.tests at 0x7fd9dd768278> swh_scheduler_celery_includes = ['swh.scheduler.tests.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.mercurial.tasks_from_disk', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', ...]
0 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_tasks_from_disk::test_loader
swh_scheduler_celery_app = <Celery celery.tests at 0x7fd9dd768278> swh_scheduler_celery_includes = ['swh.scheduler.tests.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.mercurial.tasks_from_disk', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', ...]
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_converters.TestParseAuthorConverters::test_parse_author_2
View Full Test Results (4 Failed · 29 Passed)

Event Timeline

Build is green

Patch application report for D4643 (id=16475)

Rebasing onto 4bf91cff72...

Current branch diff-target is up to date.
Changes applied before test
commit b9af2e3ea95319e2a28eb6c535dd9c9bc291a71a
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Nov 30 14:29:47 2020 +0100

    Make loading an unchanged repository uneventful
    
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/132/ for more details.

acezar retitled this revision from Make loading an unchanged repository uneventful to HgLoaderFromDisk: uneventful load when unchanged.Dec 2 2020, 3:13 PM
acezar edited the summary of this revision. (Show Details)
douardda requested changes to this revision.Dec 4 2020, 2:26 PM
douardda added a subscriber: douardda.

overall lgtm but see comments

swh/loader/mercurial/from_disk.py
253

should add type annotation

282

nit: why not if not revs: ?

This revision now requires changes to proceed.Dec 4 2020, 2:26 PM
acezar marked 2 inline comments as done.
acezar edited the summary of this revision. (Show Details)

Followup

swh/loader/mercurial/from_disk.py
282

Was not sure __bool__ / __nonzero__ was implemented by the class

Build was aborted

Patch application report for D4643 (id=16565)

Rebasing onto 4bf91cff72...

Current branch diff-target is up to date.
Changes applied before test
commit 7af11ca8dcaaec1b65250bfbdb9a9d43d3c588cf
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Nov 30 14:29:47 2020 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/134/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/134/console

This revision is now accepted and ready to land.Dec 4 2020, 5:04 PM
Alphare added a subscriber: Alphare.

Fix broken test + rebase (taking over from Antoine's work)

Build has FAILED

Patch application report for D4643 (id=18418)

Rebasing onto d8572187e9...

Current branch diff-target is up to date.
Changes applied before test
commit 7a8328ef77cd9ba179e6767602dfc69bfef395ab
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Feb 25 17:35:32 2021 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    Summary:
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.
    
    Reviewers: #reviewers, douardda
    
    Reviewed By: #reviewers, douardda
    
    Subscribers: douardda
    
    Differential Revision: https://forge.softwareheritage.org/D4643

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/168/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/168/console

Build is green

Patch application report for D4643 (id=18426)

Rebasing onto d8572187e9...

Current branch diff-target is up to date.
Changes applied before test
commit d6a7b44fa33ad769d5670f79d7aa53691f43f415
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Feb 25 17:35:32 2021 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    Summary:
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.
    
    Reviewers: #reviewers, douardda
    
    Reviewed By: #reviewers, douardda
    
    Subscribers: douardda
    
    Differential Revision: https://forge.softwareheritage.org/D4643

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/169/ for more details.

Alphare added a reviewer: acezar.
ardumont added inline comments.
swh/loader/mercurial/tests/test_from_disk.py
246

swh_config is no longer needed since you inject the swh_storage instance into the loader now.

for the sake of my question, remark ;)

swh/loader/mercurial/from_disk.py
223

can't we make that one for comprehension instead of 2?

self._latest_heads = [
    hash_to_bytes(revision.metadata["node"])
    for revision in self.storage.revision_get(snapshot_branches)
    if revision and revision.metadata
]
This revision now requires changes to proceed.Feb 26 2021, 10:53 AM
swh/loader/mercurial/from_disk.py
214–223

same proposal with the diff format...

(neat trick) the phabricator suggestion edit.

Improve heads handling + remove useless fixture call

Alphare added inline comments.
swh/loader/mercurial/from_disk.py
223

Much better, thanks. I saw the patch was accepted and just rebased it, I should have paid more attention. :)

Build is green

Patch application report for D4643 (id=18427)

Rebasing onto d8572187e9...

Current branch diff-target is up to date.
Changes applied before test
commit 00b221eaa5aa4f326150ccb3a29259dfafd0eb6f
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Feb 25 17:35:32 2021 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    Summary:
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.
    
    Reviewers: #reviewers, douardda
    
    Reviewed By: #reviewers, douardda
    
    Subscribers: douardda
    
    Differential Revision: https://forge.softwareheritage.org/D4643

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/170/ for more details.

Build is green

Patch application report for D4643 (id=18430)

Rebasing onto 1dbb8474ad...

Current branch diff-target is up to date.
Changes applied before test
commit 47c712ecd233611547e64c19783727c6e302bda1
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Feb 25 17:35:32 2021 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    Summary:
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.
    
    Reviewers: #reviewers, douardda
    
    Reviewed By: #reviewers, douardda
    
    Subscribers: douardda
    
    Differential Revision: https://forge.softwareheritage.org/D4643

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/172/ for more details.

I've added 2 more non-blocking remarks inline (i missed earlier sorry) that you might want to have a look ;)

But otherwise, lgtm.

swh/loader/mercurial/from_disk.py
255
262
This revision is now accepted and ready to land.Feb 26 2021, 12:02 PM

Add suggested assertion and fix comment typo

Build is green

Patch application report for D4643 (id=18434)

Rebasing onto 1dbb8474ad...

Current branch diff-target is up to date.
Changes applied before test
commit 04a1213f3679e7bb1cb3f8cfcc1e7573a60b2419
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Feb 25 17:35:32 2021 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    Summary:
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.
    
    Reviewers: #reviewers, douardda
    
    Reviewed By: #reviewers, douardda
    
    Subscribers: douardda
    
    Differential Revision: https://forge.softwareheritage.org/D4643

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/174/ for more details.