Page MenuHomeSoftware Heritage
Differential D4643

HgLoaderFromDisk: uneventful load when unchanged
ClosedPublic
Actions

Authored by Alphare on Dec 2 2020, 11:54 AM.

Details

Reviewers
douardda
acezar
ardumont
Group Reviewers
Reviewers
Commits
rDLDHG04a1213f3679: HgLoaderFromDisk: uneventful load when unchanged
Summary

By looking at the previous snapshot heads, loading of an unchanged repository
will be uneventful.

Diff Detail

Repository
rDLDHG Mercurial loader
Branch
D4643
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17748
Build 27435: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 27434: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_tasks::test_archive_loader
swh_scheduler_celery_app = <Celery celery.tests at 0x7fd9dd768278> swh_scheduler_celery_includes = ['swh.scheduler.tests.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.mercurial.tasks_from_disk', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', ...]
11,817 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_tasks::test_loader
swh_scheduler_celery_app = <Celery celery.tests at 0x7fd9dd768278> swh_scheduler_celery_includes = ['swh.scheduler.tests.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.mercurial.tasks_from_disk', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', ...]
140 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_tasks_from_disk::test_archive_loader
swh_scheduler_celery_app = <Celery celery.tests at 0x7fd9dd768278> swh_scheduler_celery_includes = ['swh.scheduler.tests.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.mercurial.tasks_from_disk', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', ...]
0 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_tasks_from_disk::test_loader
swh_scheduler_celery_app = <Celery celery.tests at 0x7fd9dd768278> swh_scheduler_celery_includes = ['swh.scheduler.tests.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.mercurial.tasks_from_disk', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', ...]
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.mercurial.tests.test_converters.TestParseAuthorConverters::test_parse_author_2
View Full Test Results (4 Failed · 29 Passed)

Event Timeline

acezar created this revision.Dec 2 2020, 11:54 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptDec 2 2020, 11:54 AM
swh-public-ci added a comment.Dec 2 2020, 11:56 AM

Build is green

Patch application report for D4643 (id=16475)

Rebasing onto 4bf91cff72...

Current branch diff-target is up to date.
Changes applied before test
commit b9af2e3ea95319e2a28eb6c535dd9c9bc291a71a
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Nov 30 14:29:47 2020 +0100

    Make loading an unchanged repository uneventful
    
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/132/ for more details.

Harbormaster completed remote builds in B17661: Diff 16475.Dec 2 2020, 11:56 AM
acezar retitled this revision from Make loading an unchanged repository uneventful to HgLoaderFromDisk: uneventful load when unchanged.Dec 2 2020, 3:13 PM
acezar edited the summary of this revision. (Show Details)
acezar added a child revision: D4649: HgLoaderFromDisk: Only load new commits.
douardda requested changes to this revision.Dec 4 2020, 2:26 PM
douardda added a subscriber: douardda.

overall lgtm but see comments

swh/loader/mercurial/from_disk.py
253

should add type annotation

282

nit: why not if not revs: ?

This revision now requires changes to proceed.Dec 4 2020, 2:26 PM
acezar updated this revision to Diff 16565.Dec 4 2020, 4:41 PM
acezar marked 2 inline comments as done.
acezar edited the summary of this revision. (Show Details)

Followup

swh/loader/mercurial/from_disk.py
282

Was not sure __bool__ / __nonzero__ was implemented by the class

swh-public-ci added a comment.Dec 4 2020, 5:02 PM

Build was aborted

Patch application report for D4643 (id=16565)

Rebasing onto 4bf91cff72...

Current branch diff-target is up to date.
Changes applied before test
commit 7af11ca8dcaaec1b65250bfbdb9a9d43d3c588cf
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Mon Nov 30 14:29:47 2020 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/134/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/134/console

Harbormaster failed remote builds in B17748: Diff 16565!Dec 4 2020, 5:02 PM
douardda accepted this revision.Dec 4 2020, 5:04 PM
This revision is now accepted and ready to land.Dec 4 2020, 5:04 PM
Alphare updated this revision to Diff 18418.Feb 25 2021, 6:21 PM
Alphare added a subscriber: Alphare.

Fix broken test + rebase (taking over from Antoine's work)

swh-public-ci added a comment.Feb 25 2021, 6:22 PM

Build has FAILED

Patch application report for D4643 (id=18418)

Rebasing onto d8572187e9...

Current branch diff-target is up to date.
Changes applied before test
commit 7a8328ef77cd9ba179e6767602dfc69bfef395ab
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Feb 25 17:35:32 2021 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    Summary:
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.
    
    Reviewers: #reviewers, douardda
    
    Reviewed By: #reviewers, douardda
    
    Subscribers: douardda
    
    Differential Revision: https://forge.softwareheritage.org/D4643

Link to build: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/168/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/168/console

Harbormaster failed remote builds in B19496: Diff 18418!Feb 25 2021, 6:22 PM
Alphare updated this revision to Diff 18426.Feb 26 2021, 10:43 AM

Fix flake8

swh-public-ci added a comment.Feb 26 2021, 10:45 AM

Build is green

Patch application report for D4643 (id=18426)

Rebasing onto d8572187e9...

Current branch diff-target is up to date.
Changes applied before test
commit d6a7b44fa33ad769d5670f79d7aa53691f43f415
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Feb 25 17:35:32 2021 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    Summary:
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.
    
    Reviewers: #reviewers, douardda
    
    Reviewed By: #reviewers, douardda
    
    Subscribers: douardda
    
    Differential Revision: https://forge.softwareheritage.org/D4643

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/169/ for more details.

Harbormaster completed remote builds in B19505: Diff 18426.Feb 26 2021, 10:45 AM
Alphare commandeered this revision.Feb 26 2021, 10:45 AM
Alphare added a reviewer: acezar.
ardumont added a subscriber: ardumont.Feb 26 2021, 10:50 AM
ardumont added inline comments.
swh/loader/mercurial/tests/test_from_disk.py
246

swh_config is no longer needed since you inject the swh_storage instance into the loader now.

ardumont requested changes to this revision.Feb 26 2021, 10:53 AM

for the sake of my question, remark ;)

swh/loader/mercurial/from_disk.py
223

can't we make that one for comprehension instead of 2?

self._latest_heads = [
    hash_to_bytes(revision.metadata["node"])
    for revision in self.storage.revision_get(snapshot_branches)
    if revision and revision.metadata
]
This revision now requires changes to proceed.Feb 26 2021, 10:53 AM
ardumont added inline comments.Feb 26 2021, 10:55 AM
swh/loader/mercurial/from_disk.py
214–223

same proposal with the diff format...

(neat trick) the phabricator suggestion edit.

Alphare updated this revision to Diff 18427.Feb 26 2021, 11:07 AM

Improve heads handling + remove useless fixture call

Alphare marked an inline comment as done.Feb 26 2021, 11:08 AM
Alphare added inline comments.
swh/loader/mercurial/from_disk.py
223

Much better, thanks. I saw the patch was accepted and just rebased it, I should have paid more attention. :)

swh-public-ci added a comment.Feb 26 2021, 11:09 AM

Build is green

Patch application report for D4643 (id=18427)

Rebasing onto d8572187e9...

Current branch diff-target is up to date.
Changes applied before test
commit 00b221eaa5aa4f326150ccb3a29259dfafd0eb6f
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Feb 25 17:35:32 2021 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    Summary:
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.
    
    Reviewers: #reviewers, douardda
    
    Reviewed By: #reviewers, douardda
    
    Subscribers: douardda
    
    Differential Revision: https://forge.softwareheritage.org/D4643

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/170/ for more details.

Harbormaster completed remote builds in B19506: Diff 18427.Feb 26 2021, 11:09 AM
Alphare updated this revision to Diff 18430.Feb 26 2021, 11:20 AM

Rebase

swh-public-ci added a comment.Feb 26 2021, 11:22 AM

Build is green

Patch application report for D4643 (id=18430)

Rebasing onto 1dbb8474ad...

Current branch diff-target is up to date.
Changes applied before test
commit 47c712ecd233611547e64c19783727c6e302bda1
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Feb 25 17:35:32 2021 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    Summary:
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.
    
    Reviewers: #reviewers, douardda
    
    Reviewed By: #reviewers, douardda
    
    Subscribers: douardda
    
    Differential Revision: https://forge.softwareheritage.org/D4643

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/172/ for more details.

Harbormaster completed remote builds in B19509: Diff 18430.Feb 26 2021, 11:22 AM
ardumont accepted this revision.Feb 26 2021, 12:02 PM

I've added 2 more non-blocking remarks inline (i missed earlier sorry) that you might want to have a look ;)

But otherwise, lgtm.

swh/loader/mercurial/from_disk.py
255
262
This revision is now accepted and ready to land.Feb 26 2021, 12:02 PM
Alphare updated this revision to Diff 18434.Feb 26 2021, 12:07 PM

Add suggested assertion and fix comment typo

swh-public-ci added a comment.Feb 26 2021, 12:09 PM

Build is green

Patch application report for D4643 (id=18434)

Rebasing onto 1dbb8474ad...

Current branch diff-target is up to date.
Changes applied before test
commit 04a1213f3679e7bb1cb3f8cfcc1e7573a60b2419
Author: Antoine Cezar <antoine.cezar@octobus.net>
Date:   Thu Feb 25 17:35:32 2021 +0100

    HgLoaderFromDisk: uneventful load when unchanged
    
    Summary:
    By looking at the previous snapshot heads, loading of an unchanged repository
    will be uneventful.
    
    Reviewers: #reviewers, douardda
    
    Reviewed By: #reviewers, douardda
    
    Subscribers: douardda
    
    Differential Revision: https://forge.softwareheritage.org/D4643

See https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/174/ for more details.

Harbormaster completed remote builds in B19513: Diff 18434.Feb 26 2021, 12:09 PM
Closed by commit rDLDHG04a1213f3679: HgLoaderFromDisk: uneventful load when unchanged (authored by acezar, committed by Alphare). · Explain WhyFeb 26 2021, 12:18 PM
This revision was automatically updated to reflect the committed changes.
Alphare added a commit: rDLDHG04a1213f3679: HgLoaderFromDisk: uneventful load when unchanged.

Revision Contents
Changeset List

PathSize
swh/
loader/
mercurial/
79 lines
4 lines
tests/
34 lines

Diff 16565

View Options

swh/loader/mercurial/from_disk.py

Loading...
View Options

swh/loader/mercurial/hgutil.py

Loading...
View Options

swh/loader/mercurial/tests/test_from_disk.py

Loading...
About · Developer information · Legal