Page MenuHomeSoftware Heritage
Differential D8023

Install `swh provenance origin from-journal` cli and tests
ClosedPublic
Actions

Authored by ardumont on Jun 22 2022, 3:48 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDPROVa721125c2a0a: Explicitly manage the provenance storage connection in journal client
rDPROVaf616497e108: Isolate journal tests
rDPROVa23c22827dfa: Install `swh provenance origin from-journal` cli and tests
rDPROVdb7be64ef1d5: Install `swh provenance origin from-journal` cli and tests
Summary

As a consequence, this introduces the use of swh.journal in the provenance swh
requirements.

This also:

  • deprecates the swh provenance iter-origins
  • in favor of the new swh provenance origin from-csv cli
  • Add tests around those CLIs ^

Co-authored with @douardda (author of the initial commit)

Test Plan

tox

Diff Detail

Repository
rDPROV Provenance database
Branch
wip_ardumont
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 29985
Build 46879: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 46878: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
swh-public-ci added a comment.Jun 22 2022, 4:49 PM

Build has FAILED

Patch application report for D8023 (id=28911)

Rebasing onto 80434e3b21...

Current branch diff-target is up to date.
Changes applied before test
commit db7be64ef1d574441ecb475597f617f8af5d7c31
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/607/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/607/console

Harbormaster failed remote builds in B30005: Diff 28911!Jun 22 2022, 4:49 PM
ardumont added a comment.Jun 22 2022, 5:08 PM

Build has FAILED

lolwat?

swh-public-ci added a comment.Jun 22 2022, 5:16 PM

Build is green

Patch application report for D8023 (id=28911)

Rebasing onto 80434e3b21...

Current branch diff-target is up to date.
Changes applied before test
commit db7be64ef1d574441ecb475597f617f8af5d7c31
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/608/ for more details.

Harbormaster completed remote builds in B30005: Diff 28911.Jun 22 2022, 5:16 PM
ardumont updated this revision to Diff 28966.Jun 28 2022, 10:24 AM

Rebase

swh-public-ci added a comment.Jun 28 2022, 10:35 AM

Build was aborted

Patch application report for D8023 (id=28966)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/614/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/614/console

Harbormaster failed remote builds in B30054: Diff 28966!Jun 28 2022, 10:36 AM
swh-public-ci added a comment.Jun 28 2022, 11:25 AM

Build was aborted

Patch application report for D8023 (id=28966)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/615/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/615/console

swh-public-ci added a comment.Jun 28 2022, 11:54 AM

Build was aborted

Patch application report for D8023 (id=28966)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/616/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/616/console

swh-public-ci added a comment.Jun 28 2022, 2:46 PM

Build was aborted

Patch application report for D8023 (id=28966)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/617/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/617/console

ardumont added a comment.Jun 28 2022, 4:51 PM

Build was aborted

It was fine prior to the rebase (and master build is fine).
I've no idea what's happening nor the patience to unstuck it currently.

swh-public-ci added a comment.Jun 28 2022, 6:52 PM

Build was aborted

Patch application report for D8023 (id=28966)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/618/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/618/console

ardumont updated this revision to Diff 28985.Jun 29 2022, 11:19 AM

Trigger build again

swh-public-ci added a comment.Jun 29 2022, 11:35 AM

Build was aborted

Patch application report for D8023 (id=28985)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/619/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/619/console

Harbormaster failed remote builds in B30072: Diff 28985!Jun 29 2022, 11:35 AM
ardumont updated this revision to Diff 28988.Jun 29 2022, 11:45 AM

wip tryout (drop rabbitmq from fixture) to check if that still hangs or not

ardumont mentioned this in T4361: Unstuck provenance diff build hanging and then aborted.Jun 29 2022, 11:49 AM
swh-public-ci added a comment.Jun 29 2022, 11:50 AM

Build is green

Patch application report for D8023 (id=28988)

Could not rebase; Attempt merge onto f5f7413663...

Updating f5f7413..89caa89
Fast-forward
 mypy.ini                                    |  4 +-
 requirements-swh.txt                        |  1 +
 requirements-test.txt                       |  1 +
 requirements.txt                            |  1 +
 swh/provenance/cli.py                       | 77 ++++++++++++++++++++++++++-
 swh/provenance/journal_client.py            | 22 ++++++++
 swh/provenance/postgresql/provenance.py     |  5 ++
 swh/provenance/tests/conftest.py            |  2 +-
 swh/provenance/tests/data/origins.csv       |  1 +
 swh/provenance/tests/test_cli.py            | 59 ++++++++++++++++++++-
 swh/provenance/tests/test_journal_client.py | 81 +++++++++++++++++++++++++++++
 swh/provenance/tests/test_utils.py          | 31 +++++++++++
 12 files changed, 280 insertions(+), 5 deletions(-)
 create mode 100644 swh/provenance/journal_client.py
 create mode 100644 swh/provenance/tests/data/origins.csv
 create mode 100644 swh/provenance/tests/test_journal_client.py
 create mode 100644 swh/provenance/tests/test_utils.py
Changes applied before test
commit 89caa898182ccf8316225b51df9b56a9110a152b
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 11:45:09 2022 +0200

    wip workaround build

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/620/ for more details.

Harbormaster completed remote builds in B30075: Diff 28988.Jun 29 2022, 11:50 AM
ardumont updated this revision to Diff 28989.Jun 29 2022, 11:53 AM

including the right amount of commits

swh-public-ci added a comment.Jun 29 2022, 11:58 AM

Build is green

Patch application report for D8023 (id=28989)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit 89caa898182ccf8316225b51df9b56a9110a152b
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 11:45:09 2022 +0200

    wip workaround build

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/621/ for more details.

Harbormaster completed remote builds in B30076: Diff 28989.Jun 29 2022, 11:58 AM
ardumont updated this revision to Diff 29046.Jun 29 2022, 4:37 PM
  • Tentatively work around the rabbitmq initialization.
vlorentz accepted this revision.Jun 29 2022, 4:42 PM
This revision is now accepted and ready to land.Jun 29 2022, 4:42 PM
swh-public-ci added a comment.Jun 29 2022, 4:48 PM

Build has FAILED

Patch application report for D8023 (id=29046)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit 4d272aec6d8cb72dd87a69855f2b8f5f14c5f295
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 15:59:51 2022 +0200

    Tentatively work around the rabbitmq initialization.

commit 89caa898182ccf8316225b51df9b56a9110a152b
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 11:45:09 2022 +0200

    wip workaround build

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/622/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/622/console

Harbormaster failed remote builds in B30132: Diff 29046!Jun 29 2022, 4:48 PM
swh-public-ci added a comment.Jun 29 2022, 5:06 PM

Build was aborted

Patch application report for D8023 (id=29046)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit 4d272aec6d8cb72dd87a69855f2b8f5f14c5f295
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 15:59:51 2022 +0200

    Tentatively work around the rabbitmq initialization.

commit 89caa898182ccf8316225b51df9b56a9110a152b
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 11:45:09 2022 +0200

    wip workaround build

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/623/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/623/console

ardumont updated this revision to Diff 29047.Jun 29 2022, 5:11 PM
  • Tentatively work around the rabbitmq initialization 3/n
swh-public-ci added a comment.Jun 29 2022, 5:28 PM

Build was aborted

Patch application report for D8023 (id=29047)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit 297d7621398e8df90efeafa4d901cbb36f0cfa0a
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 15:59:51 2022 +0200

    Tentatively work around the rabbitmq initialization.

commit 89caa898182ccf8316225b51df9b56a9110a152b
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 11:45:09 2022 +0200

    wip workaround build

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/624/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/624/console

Harbormaster failed remote builds in B30133: Diff 29047!Jun 29 2022, 5:28 PM
ardumont updated this revision to Diff 29050.Jun 29 2022, 5:38 PM
  • Tentatively work around the rabbitmq tests build failures 4/n
swh-public-ci added a comment.Jun 29 2022, 5:46 PM

Build was aborted

Patch application report for D8023 (id=29050)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit 972e4645a5e57f9543a65d36af049e20451c5c57
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 15:59:51 2022 +0200

    Tentatively work around the rabbitmq tests build failures

commit 89caa898182ccf8316225b51df9b56a9110a152b
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 11:45:09 2022 +0200

    wip workaround build

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/625/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/625/console

Harbormaster failed remote builds in B30136: Diff 29050!Jun 29 2022, 5:47 PM
ardumont updated this revision to Diff 29051.Jun 29 2022, 5:47 PM
  • Tentatively work around the rabbitmq tests build failures 5/n
swh-public-ci added a comment.Jun 29 2022, 5:54 PM

Build was aborted

Patch application report for D8023 (id=29051)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit 709a55c3ed638d4ebf608435275b11b488efaffa
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 17:44:36 2022 +0200

    Tentatively work around the rabbitmq tests build failures 5/n

commit 972e4645a5e57f9543a65d36af049e20451c5c57
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 15:59:51 2022 +0200

    Tentatively work around the rabbitmq tests build failures

commit 89caa898182ccf8316225b51df9b56a9110a152b
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 11:45:09 2022 +0200

    wip workaround build

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/626/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/626/console

Harbormaster failed remote builds in B30137: Diff 29051!Jun 29 2022, 5:54 PM
ardumont updated this revision to Diff 29053.Jun 29 2022, 6:07 PM

Revert to original commit

ardumont added a comment.Jun 29 2022, 6:16 PM

I'm of a mind of just push it and check whether that's not some discrepancy with the master build somehow.
Master build is fine and local build as well.

swh-public-ci added a comment.Jun 29 2022, 6:23 PM

Build was aborted

Patch application report for D8023 (id=29053)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/627/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/627/console

Harbormaster failed remote builds in B30139: Diff 29053!Jun 29 2022, 6:23 PM
ardumont added a subscriber: olasd.Jun 29 2022, 6:40 PM
ardumont added inline comments.
swh/provenance/postgresql/provenance.py
79

@olasd mentions this could be the source of the failing build.

ardumont updated this revision to Diff 29055.Jun 29 2022, 6:43 PM
  • Drop self.conn shenanigans (tests no longer stuck but journal client tests are not happy)
olasd updated this revision to Diff 29056.Jun 29 2022, 6:46 PM
  • Explicitly manage the provenance storage connection in journal client
ardumont added inline comments.Jun 29 2022, 6:47 PM
swh/provenance/postgresql/provenance.py
79

and instead i should wrap the client call in a with provenance: (which explicitly opens/closes the connection to the backend storage).

swh-public-ci added a comment.Jun 29 2022, 6:48 PM

Build was aborted

Patch application report for D8023 (id=29055)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit fa30351e8b60da52f58d4faf73b312f62c9f8048
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Wed Jun 29 18:41:25 2022 +0200

    Drop self.conn shenanigans
    
    I recalled that without it, db transactions in (journal?) tests would fail...

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/628/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/628/console

Harbormaster failed remote builds in B30141: Diff 29055!Jun 29 2022, 6:48 PM
swh-public-ci added a comment.Jun 29 2022, 6:52 PM

Build was aborted
See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/629/ for more details.

Harbormaster failed remote builds in B30142: Diff 29056!Jun 29 2022, 6:52 PM
swh-public-ci added a comment.Jun 29 2022, 7:10 PM

Build was aborted

Patch application report for D8023 (id=29056)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit a721125c2a0a3bb9c400fc9fe31d7b9b82243a9c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 18:44:57 2022 +0200

    Explicitly manage the provenance storage connection in journal client
    
    Doing it implicitly makes pytest fixture teardown impossible because of
    object references that are held for too long, preventing the postgresql
    server from shutting down.

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/630/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/630/console

olasd updated this revision to Diff 29057.Jun 29 2022, 7:20 PM
  • Force rabbitmq server termination in tests
swh-public-ci added a comment.Jun 29 2022, 7:37 PM

Build was aborted

Patch application report for D8023 (id=29057)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit 0cc365f242b325c34b88f3c79ecad18b4693d56c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 19:20:04 2022 +0200

    Force rabbitmq server termination in tests

commit a721125c2a0a3bb9c400fc9fe31d7b9b82243a9c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 18:44:57 2022 +0200

    Explicitly manage the provenance storage connection in journal client
    
    Doing it implicitly makes pytest fixture teardown impossible because of
    object references that are held for too long, preventing the postgresql
    server from shutting down.

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/631/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/631/console

Harbormaster failed remote builds in B30143: Diff 29057!Jun 29 2022, 7:37 PM
ardumont added inline comments.Jun 29 2022, 7:51 PM
swh/provenance/tests/test_journal_client.py
67

I explicitely drop that part as i noticed discrepancies in storage instances when using that ^ in initial runs.
(which then fails the assertion below since it's not the same storages).

ardumont added inline comments.Jun 30 2022, 8:10 AM
swh/provenance/tests/test_journal_client.py
67

as a heads up ^

olasd updated this revision to Diff 29060.Jun 30 2022, 10:53 AM
  • Isolate journal tests
swh-public-ci added a comment.Jun 30 2022, 11:00 AM

Build is green

Patch application report for D8023 (id=29060)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit 8c44dcb5280e253000bc5536387d30551b495327
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jun 30 10:50:20 2022 +0200

    Isolate journal tests

commit 0cc365f242b325c34b88f3c79ecad18b4693d56c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 19:20:04 2022 +0200

    Force rabbitmq server termination in tests

commit a721125c2a0a3bb9c400fc9fe31d7b9b82243a9c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 18:44:57 2022 +0200

    Explicitly manage the provenance storage connection in journal client
    
    Doing it implicitly makes pytest fixture teardown impossible because of
    object references that are held for too long, preventing the postgresql
    server from shutting down.

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/632/ for more details.

Harbormaster completed remote builds in B30146: Diff 29060.Jun 30 2022, 11:01 AM
olasd updated this revision to Diff 29065.Jun 30 2022, 11:02 AM

Pass {posargs} to both pytest runs

vlorentz added inline comments.Jun 30 2022, 11:08 AM
tox.ini
10–16 ↗(On Diff #29065)

please add a comment to explain it's necessary because of the rdkafka teardown issues

swh-public-ci added a comment.Jun 30 2022, 11:11 AM

Build is green

Patch application report for D8023 (id=29065)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit 0facfc3fcd58f4320e547adb818e54925729d618
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jun 30 10:50:20 2022 +0200

    Isolate journal tests

commit 0cc365f242b325c34b88f3c79ecad18b4693d56c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 19:20:04 2022 +0200

    Force rabbitmq server termination in tests

commit a721125c2a0a3bb9c400fc9fe31d7b9b82243a9c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 18:44:57 2022 +0200

    Explicitly manage the provenance storage connection in journal client
    
    Doing it implicitly makes pytest fixture teardown impossible because of
    object references that are held for too long, preventing the postgresql
    server from shutting down.

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/633/ for more details.

Harbormaster completed remote builds in B30151: Diff 29065.Jun 30 2022, 11:11 AM
olasd updated this revision to Diff 29066.Jun 30 2022, 11:23 AM

Add explanatory comment for the tests split in tox.ini

swh-public-ci added a comment.Jun 30 2022, 11:31 AM

Build is green

Patch application report for D8023 (id=29066)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit 0b38286f53013491f3d00c8747595977412d4f6e
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jun 30 10:50:20 2022 +0200

    Isolate journal tests

commit 0cc365f242b325c34b88f3c79ecad18b4693d56c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 19:20:04 2022 +0200

    Force rabbitmq server termination in tests

commit a721125c2a0a3bb9c400fc9fe31d7b9b82243a9c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 18:44:57 2022 +0200

    Explicitly manage the provenance storage connection in journal client
    
    Doing it implicitly makes pytest fixture teardown impossible because of
    object references that are held for too long, preventing the postgresql
    server from shutting down.

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/634/ for more details.

Harbormaster completed remote builds in B30152: Diff 29066.Jun 30 2022, 11:31 AM
olasd updated this revision to Diff 29068.Jun 30 2022, 11:34 AM

Ensure coverage results are kept incrementally

swh-public-ci added a comment.Jun 30 2022, 11:41 AM

Build is green

Patch application report for D8023 (id=29068)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit 1934605d037722db70199f25a9a2fd2b50f97f4e
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jun 30 10:50:20 2022 +0200

    Isolate journal tests

commit 0cc365f242b325c34b88f3c79ecad18b4693d56c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 19:20:04 2022 +0200

    Force rabbitmq server termination in tests

commit a721125c2a0a3bb9c400fc9fe31d7b9b82243a9c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 18:44:57 2022 +0200

    Explicitly manage the provenance storage connection in journal client
    
    Doing it implicitly makes pytest fixture teardown impossible because of
    object references that are held for too long, preventing the postgresql
    server from shutting down.

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/635/ for more details.

Harbormaster completed remote builds in B30154: Diff 29068.Jun 30 2022, 11:41 AM
olasd updated this revision to Diff 29069.Jun 30 2022, 11:44 AM

Drop useless rabbitmq server change

swh-public-ci added a comment.Jun 30 2022, 11:51 AM

Build is green

Patch application report for D8023 (id=29069)

Rebasing onto f5f7413663...

Current branch diff-target is up to date.
Changes applied before test
commit af616497e108810a1a8f8a93753710e4c6d087ca
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jun 30 10:50:20 2022 +0200

    Isolate journal tests

commit a721125c2a0a3bb9c400fc9fe31d7b9b82243a9c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jun 29 18:44:57 2022 +0200

    Explicitly manage the provenance storage connection in journal client
    
    Doing it implicitly makes pytest fixture teardown impossible because of
    object references that are held for too long, preventing the postgresql
    server from shutting down.

commit a23c22827dfa262f89d0f300d39c954e9d703a03
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 22 12:40:38 2022 +0200

    Install `swh provenance origin from-journal` cli and tests
    
    As a consequence, this introduces the use of swh.journal in the provenance swh
    requirements.
    
    This also:
    - deprecates the `swh provenance iter-origins`
    - in favor of the new `swh provenance origin from-csv` cli
    - Add tests around those CLIs ^

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/636/ for more details.

Harbormaster completed remote builds in B30155: Diff 29069.Jun 30 2022, 11:51 AM
Closed by commit rDPROVa23c22827dfa: Install `swh provenance origin from-journal` cli and tests (authored by douardda, committed by ardumont). · Explain WhyJun 30 2022, 2:10 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDPROVa23c22827dfa: Install `swh provenance origin from-journal` cli and tests.
olasd added a commit: rDPROVa721125c2a0a: Explicitly manage the provenance storage connection in journal client.Jun 30 2022, 2:10 PM
olasd added a commit: rDPROVaf616497e108: Isolate journal tests.

Revision Contents
Changeset List

PathSize
1 line
swh/
provenance/
77 lines
25 lines
postgresql/
5 lines
tests/
data/
1 line
59 lines
91 lines
31 lines

Diff 28894

View Options

requirements-swh.txt

# Copyright (C) 2018-2020 The Software Heritage developers # Copyright (C) 2018-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import copy
import logging import logging
import os import os
import time import time
from typing import Any, Dict
from unittest.mock import patch
import hglib import hglib
import pytest import pytest
from swh.model import hashutil from swh.model import hashutil
from swh.model.model import RevisionType from swh.model.model import RevisionType
from swh.loader.core.tests import BaseLoaderTest
from swh.storage.algos.snapshot import snapshot_get_latest from swh.storage.algos.snapshot import snapshot_get_latest
from swh.loader.tests import ( from swh.loader.tests import (
assert_last_visit_matches, assert_last_visit_matches,
check_snapshot, check_snapshot,
get_stats, get_stats,
prepare_repository_from_archive, prepare_repository_from_archive,
) )
from .common import HgLoaderMemoryStorage, HgArchiveLoaderMemoryStorage from ..loader import HgBundle20Loader, HgArchiveBundle20Loader, CloneTimeoutError
from ..loader import HgBundle20Loader, CloneTimeoutError
class BaseHgLoaderTest(BaseLoaderTest):
"""Mixin base loader test to prepare the mercurial
repository to uncompress, load and test the results.
"""
def setUp(
self,
archive_name="the-sandbox.tgz",
filename="the-sandbox",
uncompress_archive=True,
):
super().setUp(
archive_name=archive_name,
filename=filename,
prefix_tmp_folder_name="swh.loader.mercurial.",
start_path=os.path.dirname(__file__),
uncompress_archive=uncompress_archive,
)
def test_loader_hg_new_visit(swh_config, datadir, tmp_path): def test_loader_hg_new_visit_no_release(swh_config, datadir, tmp_path):
"""Eventful visit should yield 1 snapshot""" """Eventful visit should yield 1 snapshot"""
archive_name = "the-sandbox" archive_name = "the-sandbox"
archive_path = os.path.join(datadir, f"{archive_name}.tgz") archive_path = os.path.join(datadir, f"{archive_name}.tgz")
repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path) repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
loader = HgBundle20Loader(repo_url) loader = HgBundle20Loader(repo_url)
assert loader.load() == {"status": "eventful"} assert loader.load() == {"status": "eventful"}
Show All 27 Lines expected_snapshot = {
"develop": {"target": tip_revision_develop, "target_type": "revision"}, "develop": {"target": tip_revision_develop, "target_type": "revision"},
"default": {"target": tip_revision_default, "target_type": "revision"}, "default": {"target": tip_revision_default, "target_type": "revision"},
"HEAD": {"target": "develop", "target_type": "alias",}, "HEAD": {"target": "develop", "target_type": "alias",},
}, },
} }
check_snapshot(expected_snapshot, loader.storage) check_snapshot(expected_snapshot, loader.storage)
# Ensure archive loader yields the same snapshot
loader2 = HgArchiveBundle20Loader(
url=archive_path,
archive_path=archive_path,
visit_date="2016-05-03 15:16:32+00",
)
actual_load_status = loader2.load()
assert actual_load_status == {"status": "eventful"}
stats2 = get_stats(loader2.storage)
expected_stats = copy.deepcopy(stats)
expected_stats["origin"] += 1
expected_stats["origin_visit"] += 1
assert stats2 == expected_stats
# That visit yields the same snapshot
assert_last_visit_matches(
loader2.storage,
archive_path,
status="full",
type="hg",
snapshot=hashutil.hash_to_bytes("3b8fe58e467deb7597b12a5fd3b2c096b8c02028"),
)
def test_loader_hg_new_visit_with_release(swh_config, datadir, tmp_path):
"""Eventful visit with release should yield 1 snapshot"""
archive_name = "hello"
archive_path = os.path.join(datadir, f"{archive_name}.tgz")
repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
loader = HgBundle20Loader(url=repo_url, visit_date="2016-05-03 15:16:32+00",)
actual_load_status = loader.load()
assert actual_load_status == {"status": "eventful"}
class CommonHgLoaderData:
def assert_data_ok(self, actual_load_status: Dict[str, Any]):
# then # then
self.assertCountContents(3) # type: ignore stats = get_stats(loader.storage)
self.assertCountDirectories(3) # type: ignore assert stats == {
self.assertCountReleases(1) # type: ignore "content": 3,
self.assertCountRevisions(3) # type: ignore "directory": 3,
"origin": 1,
"origin_visit": 1,
"person": 3,
"release": 1,
"revision": 3,
"skipped_content": 0,
"snapshot": 1,
}
# cf. test_loader.org for explaining from where those hashes
tip_release = "515c4d72e089404356d0f4b39d60f948b8999140" tip_release = "515c4d72e089404356d0f4b39d60f948b8999140"
self.assertReleasesContain([tip_release]) # type: ignore release = loader.storage.release_get([hashutil.hash_to_bytes(tip_release)])
assert release is not None
tip_revision_default = "c3dbe4fbeaaa98dd961834e4007edb3efb0e2a27" tip_revision_default = "c3dbe4fbeaaa98dd961834e4007edb3efb0e2a27"
# cf. test_loader.org for explaining from where those hashes revision = loader.storage.revision_get(
# come from [hashutil.hash_to_bytes(tip_revision_default)]
expected_revisions = { )
# revision hash | directory hash # noqa assert revision is not None
"93b48d515580522a05f389bec93227fc8e43d940": "43d727f2f3f2f7cb3b098ddad1d7038464a4cee2", # noqa
"8dd3db5d5519e4947f035d141581d304565372d2": "b3f85f210ff86d334575f64cb01c5bf49895b63e", # noqa
tip_revision_default: "8f2be433c945384c85920a8e60f2a68d2c0f20fb",
}
self.assertRevisionsContain(expected_revisions) # type: ignore
self.assertCountSnapshots(1) # type: ignore
expected_snapshot_id = "d35668e02e2ba4321dc951cd308cf883786f918a"
expected_snapshot = { expected_snapshot = {
"id": "d35668e02e2ba4321dc951cd308cf883786f918a", "id": expected_snapshot_id,
"branches": { "branches": {
"default": {"target": tip_revision_default, "target_type": "revision"}, "default": {"target": tip_revision_default, "target_type": "revision"},
"0.1": {"target": tip_release, "target_type": "release"}, "0.1": {"target": tip_release, "target_type": "release"},
"HEAD": {"target": "default", "target_type": "alias",}, "HEAD": {"target": "default", "target_type": "alias",},
}, },
} }
self.assertSnapshotEqual(expected_snapshot) # type: ignore check_snapshot(expected_snapshot, loader.storage)
assert actual_load_status == {"status": "eventful"}
assert_last_visit_matches( assert_last_visit_matches(
self.storage, # type: ignore loader.storage, repo_url, type=RevisionType.MERCURIAL.value, status="full",
self.repo_url, # type: ignore
type=RevisionType.MERCURIAL.value,
status="full",
) )
# Ensure archive loader yields the same snapshot
class WithReleaseLoaderTest(BaseHgLoaderTest, CommonHgLoaderData): loader2 = HgArchiveBundle20Loader(
"""Load a mercurial repository with release url=archive_path,
archive_path=archive_path,
"""
def setUp(self):
super().setUp(archive_name="hello.tgz", filename="hello")
self.loader = HgLoaderMemoryStorage(
url=self.repo_url,
visit_date="2016-05-03 15:16:32+00", visit_date="2016-05-03 15:16:32+00",
directory=self.destination_path,
) )
self.storage = self.loader.storage
def test_load(self):
"""Load a repository with tags results in 1 snapshot
""" actual_load_status = loader2.load()
# when assert actual_load_status == {"status": "eventful"}
actual_load_status = self.loader.load()
self.assert_data_ok(actual_load_status)
class ArchiveLoaderTest(BaseHgLoaderTest, CommonHgLoaderData):
"""Load a mercurial repository archive with release
""" stats2 = get_stats(loader2.storage)
expected_stats = copy.deepcopy(stats)
expected_stats["origin"] += 1
expected_stats["origin_visit"] += 1
assert stats2 == expected_stats
def setUp(self): # That visit yields the same snapshot
super().setUp( assert_last_visit_matches(
archive_name="hello.tgz", filename="hello", uncompress_archive=False loader2.storage,
) archive_path,
self.loader = HgArchiveLoaderMemoryStorage( status="full",
url=self.repo_url, type="hg",
visit_date="2016-05-03 15:16:32+00", snapshot=hashutil.hash_to_bytes(expected_snapshot_id),
archive_path=self.destination_path,
) )
self.storage = self.loader.storage
def test_load(self):
"""Load a mercurial repository archive with tags results in 1 snapshot
"""
# when
actual_load_status = self.loader.load()
self.assert_data_ok(actual_load_status)
@patch("swh.loader.mercurial.archive_extract.patoolib") def test_visit_with_archive_decompression_failure(swh_config, mocker, datadir):
def test_load_with_failure(self, mock_patoo): """Failure to decompress should fail early, no data is ingested"""
mock_patoo = mocker.patch("swh.loader.mercurial.archive_extract.patoolib")
mock_patoo.side_effect = ValueError mock_patoo.side_effect = ValueError
# when archive_name = "hello"
r = self.loader.load() archive_path = os.path.join(datadir, f"{archive_name}.tgz")
# Ensure archive loader yields the same snapshot
loader = HgArchiveBundle20Loader(
url=archive_path, visit_date="2016-05-03 15:16:32+00",
)
self.assertEqual(r, {"status": "failed"}) actual_load_status = loader.load()
self.assertCountContents(0) assert actual_load_status == {"status": "failed"}
self.assertCountDirectories(0)
self.assertCountRevisions(0) stats = get_stats(loader.storage)
self.assertCountReleases(0) assert stats == {
self.assertCountSnapshots(0) "content": 0,
"directory": 0,
"origin": 1,
"origin_visit": 1,
"person": 0,
"release": 0,
"revision": 0,
"skipped_content": 0,
"snapshot": 0,
}
# That visit yields the same snapshot
assert_last_visit_matches(
loader.storage, archive_path, status="partial", type="hg", snapshot=None
)
class WithTransplantLoaderTest(BaseHgLoaderTest): def test_visit_repository_with_transplant_operations(swh_config, datadir, tmp_path):
"""Load a mercurial repository where transplant operations """Visit a mercurial repository visit transplant operations within should yield a
have been used. snapshot as well.
ardumontAuthorUnsubmitted
Done
Inline Actions

I don't recall exactly what exactly was the bug about but that's the scenario to ensure we don't fail on it again [1]

[1] https://www.mercurial-scm.org/wiki/TransplantExtension

ardumont: I don't recall exactly what exactly was the bug about but that's the scenario to ensure we…
anlambertUnsubmitted
Not Done
Inline Actions

Test context: T1729

anlambert: Test context: T1729
ardumontAuthorUnsubmitted
Done
Inline Actions

Thank you, sir ;)

ardumont: Thank you, sir ;)
""" """
def setUp(self): archive_name = "transplant"
super().setUp(archive_name="transplant.tgz", filename="transplant") archive_path = os.path.join(datadir, f"{archive_name}.tgz")
self.loader = HgLoaderMemoryStorage( repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
url=self.repo_url, loader = HgBundle20Loader(url=repo_url, visit_date="2019-05-23 12:06:00+00",)
visit_date="2019-05-23 12:06:00+00",
directory=self.destination_path,
)
self.storage = self.loader.storage
def test_load(self):
# load hg repository # load hg repository
actual_load_status = self.loader.load() actual_load_status = loader.load()
assert actual_load_status == {"status": "eventful"} assert actual_load_status == {"status": "eventful"}
# collect swh revisions # collect swh revisions
origin_url = self.storage.origin_get([{"type": "hg", "url": self.repo_url}])[0][
"url"
]
assert_last_visit_matches( assert_last_visit_matches(
self.storage, origin_url, type=RevisionType.MERCURIAL.value, status="full" loader.storage, repo_url, type=RevisionType.MERCURIAL.value, status="full"
) )
revisions = [] revisions = []
snapshot = snapshot_get_latest(self.storage, origin_url) snapshot = snapshot_get_latest(loader.storage, repo_url)
for branch in snapshot.branches.values(): for branch in snapshot.branches.values():
if branch.target_type.value != "revision": if branch.target_type.value != "revision":
continue continue
revisions.append(branch.target) revisions.append(branch.target)
# extract original changesets info and the transplant sources # extract original changesets info and the transplant sources
hg_changesets = set() hg_changesets = set()
transplant_sources = set() transplant_sources = set()
for rev in self.storage.revision_log(revisions): for rev in loader.storage.revision_log(revisions):
hg_changesets.add(rev["metadata"]["node"]) hg_changesets.add(rev["metadata"]["node"])
for k, v in rev["metadata"]["extra_headers"]: for k, v in rev["metadata"]["extra_headers"]:
if k == "transplant_source": if k == "transplant_source":
transplant_sources.add(v.decode("ascii")) transplant_sources.add(v.decode("ascii"))
# check extracted data are valid # check extracted data are valid
self.assertTrue(len(hg_changesets) > 0) assert len(hg_changesets) > 0
self.assertTrue(len(transplant_sources) > 0) assert len(transplant_sources) > 0
self.assertTrue(transplant_sources.issubset(hg_changesets)) assert transplant_sources.issubset(hg_changesets)
def test_clone_with_timeout_timeout(caplog, tmp_path, monkeypatch): def test_clone_with_timeout_timeout(caplog, tmp_path, monkeypatch):
log = logging.getLogger("test_clone_with_timeout") log = logging.getLogger("test_clone_with_timeout")
def clone_timeout(source, dest): def clone_timeout(source, dest):
time.sleep(60) time.sleep(60)
Show All 39 Lines
View Options

swh/provenance/cli.py

# Copyright (C) 2018-2020 The Software Heritage developers # Copyright (C) 2018-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import copy
import logging import logging
import os import os
import time import time
from typing import Any, Dict
from unittest.mock import patch
import hglib import hglib
import pytest import pytest
from swh.model import hashutil from swh.model import hashutil
from swh.model.model import RevisionType from swh.model.model import RevisionType
from swh.loader.core.tests import BaseLoaderTest
from swh.storage.algos.snapshot import snapshot_get_latest from swh.storage.algos.snapshot import snapshot_get_latest
from swh.loader.tests import ( from swh.loader.tests import (
assert_last_visit_matches, assert_last_visit_matches,
check_snapshot, check_snapshot,
get_stats, get_stats,
prepare_repository_from_archive, prepare_repository_from_archive,
) )
from .common import HgLoaderMemoryStorage, HgArchiveLoaderMemoryStorage from ..loader import HgBundle20Loader, HgArchiveBundle20Loader, CloneTimeoutError
from ..loader import HgBundle20Loader, CloneTimeoutError
class BaseHgLoaderTest(BaseLoaderTest):
"""Mixin base loader test to prepare the mercurial
repository to uncompress, load and test the results.
"""
def setUp(
self,
archive_name="the-sandbox.tgz",
filename="the-sandbox",
uncompress_archive=True,
):
super().setUp(
archive_name=archive_name,
filename=filename,
prefix_tmp_folder_name="swh.loader.mercurial.",
start_path=os.path.dirname(__file__),
uncompress_archive=uncompress_archive,
)
def test_loader_hg_new_visit(swh_config, datadir, tmp_path): def test_loader_hg_new_visit_no_release(swh_config, datadir, tmp_path):
"""Eventful visit should yield 1 snapshot""" """Eventful visit should yield 1 snapshot"""
archive_name = "the-sandbox" archive_name = "the-sandbox"
archive_path = os.path.join(datadir, f"{archive_name}.tgz") archive_path = os.path.join(datadir, f"{archive_name}.tgz")
repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path) repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
loader = HgBundle20Loader(repo_url) loader = HgBundle20Loader(repo_url)
assert loader.load() == {"status": "eventful"} assert loader.load() == {"status": "eventful"}
Show All 27 Lines expected_snapshot = {
"develop": {"target": tip_revision_develop, "target_type": "revision"}, "develop": {"target": tip_revision_develop, "target_type": "revision"},
"default": {"target": tip_revision_default, "target_type": "revision"}, "default": {"target": tip_revision_default, "target_type": "revision"},
"HEAD": {"target": "develop", "target_type": "alias",}, "HEAD": {"target": "develop", "target_type": "alias",},
}, },
} }
check_snapshot(expected_snapshot, loader.storage) check_snapshot(expected_snapshot, loader.storage)
# Ensure archive loader yields the same snapshot
loader2 = HgArchiveBundle20Loader(
url=archive_path,
archive_path=archive_path,
visit_date="2016-05-03 15:16:32+00",
)
actual_load_status = loader2.load()
assert actual_load_status == {"status": "eventful"}
stats2 = get_stats(loader2.storage)
expected_stats = copy.deepcopy(stats)
expected_stats["origin"] += 1
expected_stats["origin_visit"] += 1
assert stats2 == expected_stats
# That visit yields the same snapshot
assert_last_visit_matches(
loader2.storage,
archive_path,
status="full",
type="hg",
snapshot=hashutil.hash_to_bytes("3b8fe58e467deb7597b12a5fd3b2c096b8c02028"),
)
def test_loader_hg_new_visit_with_release(swh_config, datadir, tmp_path):
"""Eventful visit with release should yield 1 snapshot"""
archive_name = "hello"
archive_path = os.path.join(datadir, f"{archive_name}.tgz")
repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
loader = HgBundle20Loader(url=repo_url, visit_date="2016-05-03 15:16:32+00",)
actual_load_status = loader.load()
assert actual_load_status == {"status": "eventful"}
class CommonHgLoaderData:
def assert_data_ok(self, actual_load_status: Dict[str, Any]):
# then # then
self.assertCountContents(3) # type: ignore stats = get_stats(loader.storage)
self.assertCountDirectories(3) # type: ignore assert stats == {
self.assertCountReleases(1) # type: ignore "content": 3,
self.assertCountRevisions(3) # type: ignore "directory": 3,
"origin": 1,
"origin_visit": 1,
"person": 3,
"release": 1,
"revision": 3,
"skipped_content": 0,
"snapshot": 1,
}
# cf. test_loader.org for explaining from where those hashes
tip_release = "515c4d72e089404356d0f4b39d60f948b8999140" tip_release = "515c4d72e089404356d0f4b39d60f948b8999140"
self.assertReleasesContain([tip_release]) # type: ignore release = loader.storage.release_get([hashutil.hash_to_bytes(tip_release)])
assert release is not None
tip_revision_default = "c3dbe4fbeaaa98dd961834e4007edb3efb0e2a27" tip_revision_default = "c3dbe4fbeaaa98dd961834e4007edb3efb0e2a27"
# cf. test_loader.org for explaining from where those hashes revision = loader.storage.revision_get(
# come from [hashutil.hash_to_bytes(tip_revision_default)]
expected_revisions = { )
# revision hash | directory hash # noqa assert revision is not None
"93b48d515580522a05f389bec93227fc8e43d940": "43d727f2f3f2f7cb3b098ddad1d7038464a4cee2", # noqa
"8dd3db5d5519e4947f035d141581d304565372d2": "b3f85f210ff86d334575f64cb01c5bf49895b63e", # noqa
tip_revision_default: "8f2be433c945384c85920a8e60f2a68d2c0f20fb",
}
self.assertRevisionsContain(expected_revisions) # type: ignore
self.assertCountSnapshots(1) # type: ignore
expected_snapshot_id = "d35668e02e2ba4321dc951cd308cf883786f918a"
expected_snapshot = { expected_snapshot = {
"id": "d35668e02e2ba4321dc951cd308cf883786f918a", "id": expected_snapshot_id,
"branches": { "branches": {
"default": {"target": tip_revision_default, "target_type": "revision"}, "default": {"target": tip_revision_default, "target_type": "revision"},
"0.1": {"target": tip_release, "target_type": "release"}, "0.1": {"target": tip_release, "target_type": "release"},
"HEAD": {"target": "default", "target_type": "alias",}, "HEAD": {"target": "default", "target_type": "alias",},
}, },
} }
self.assertSnapshotEqual(expected_snapshot) # type: ignore check_snapshot(expected_snapshot, loader.storage)
assert actual_load_status == {"status": "eventful"}
assert_last_visit_matches( assert_last_visit_matches(
self.storage, # type: ignore loader.storage, repo_url, type=RevisionType.MERCURIAL.value, status="full",
self.repo_url, # type: ignore
type=RevisionType.MERCURIAL.value,
status="full",
) )
# Ensure archive loader yields the same snapshot
class WithReleaseLoaderTest(BaseHgLoaderTest, CommonHgLoaderData): loader2 = HgArchiveBundle20Loader(
"""Load a mercurial repository with release url=archive_path,
archive_path=archive_path,
"""
def setUp(self):
super().setUp(archive_name="hello.tgz", filename="hello")
self.loader = HgLoaderMemoryStorage(
url=self.repo_url,
visit_date="2016-05-03 15:16:32+00", visit_date="2016-05-03 15:16:32+00",
directory=self.destination_path,
) )
self.storage = self.loader.storage
def test_load(self):
"""Load a repository with tags results in 1 snapshot
""" actual_load_status = loader2.load()
# when assert actual_load_status == {"status": "eventful"}
actual_load_status = self.loader.load()
self.assert_data_ok(actual_load_status)
class ArchiveLoaderTest(BaseHgLoaderTest, CommonHgLoaderData):
"""Load a mercurial repository archive with release
""" stats2 = get_stats(loader2.storage)
expected_stats = copy.deepcopy(stats)
expected_stats["origin"] += 1
expected_stats["origin_visit"] += 1
assert stats2 == expected_stats
def setUp(self): # That visit yields the same snapshot
super().setUp( assert_last_visit_matches(
archive_name="hello.tgz", filename="hello", uncompress_archive=False loader2.storage,
) archive_path,
self.loader = HgArchiveLoaderMemoryStorage( status="full",
url=self.repo_url, type="hg",
visit_date="2016-05-03 15:16:32+00", snapshot=hashutil.hash_to_bytes(expected_snapshot_id),
archive_path=self.destination_path,
) )
self.storage = self.loader.storage
def test_load(self):
"""Load a mercurial repository archive with tags results in 1 snapshot
"""
# when
actual_load_status = self.loader.load()
self.assert_data_ok(actual_load_status)
@patch("swh.loader.mercurial.archive_extract.patoolib") def test_visit_with_archive_decompression_failure(swh_config, mocker, datadir):
def test_load_with_failure(self, mock_patoo): """Failure to decompress should fail early, no data is ingested"""
mock_patoo = mocker.patch("swh.loader.mercurial.archive_extract.patoolib")
mock_patoo.side_effect = ValueError mock_patoo.side_effect = ValueError
# when archive_name = "hello"
r = self.loader.load() archive_path = os.path.join(datadir, f"{archive_name}.tgz")
# Ensure archive loader yields the same snapshot
loader = HgArchiveBundle20Loader(
url=archive_path, visit_date="2016-05-03 15:16:32+00",
)
self.assertEqual(r, {"status": "failed"}) actual_load_status = loader.load()
self.assertCountContents(0) assert actual_load_status == {"status": "failed"}
self.assertCountDirectories(0)
self.assertCountRevisions(0) stats = get_stats(loader.storage)
self.assertCountReleases(0) assert stats == {
self.assertCountSnapshots(0) "content": 0,
"directory": 0,
"origin": 1,
"origin_visit": 1,
"person": 0,
"release": 0,
"revision": 0,
"skipped_content": 0,
"snapshot": 0,
}
# That visit yields the same snapshot
assert_last_visit_matches(
loader.storage, archive_path, status="partial", type="hg", snapshot=None
)
class WithTransplantLoaderTest(BaseHgLoaderTest): def test_visit_repository_with_transplant_operations(swh_config, datadir, tmp_path):
"""Load a mercurial repository where transplant operations """Visit a mercurial repository visit transplant operations within should yield a
have been used. snapshot as well.
ardumontAuthorUnsubmitted
Done
Inline Actions

I don't recall exactly what exactly was the bug about but that's the scenario to ensure we don't fail on it again [1]

[1] https://www.mercurial-scm.org/wiki/TransplantExtension

ardumont: I don't recall exactly what exactly was the bug about but that's the scenario to ensure we…
anlambertUnsubmitted
Not Done
Inline Actions

Test context: T1729

anlambert: Test context: T1729
ardumontAuthorUnsubmitted
Done
Inline Actions

Thank you, sir ;)

ardumont: Thank you, sir ;)
""" """
def setUp(self): archive_name = "transplant"
super().setUp(archive_name="transplant.tgz", filename="transplant") archive_path = os.path.join(datadir, f"{archive_name}.tgz")
self.loader = HgLoaderMemoryStorage( repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
url=self.repo_url, loader = HgBundle20Loader(url=repo_url, visit_date="2019-05-23 12:06:00+00",)
visit_date="2019-05-23 12:06:00+00",
directory=self.destination_path,
)
self.storage = self.loader.storage
def test_load(self):
# load hg repository # load hg repository
actual_load_status = self.loader.load() actual_load_status = loader.load()
assert actual_load_status == {"status": "eventful"} assert actual_load_status == {"status": "eventful"}
# collect swh revisions # collect swh revisions
origin_url = self.storage.origin_get([{"type": "hg", "url": self.repo_url}])[0][
"url"
]
assert_last_visit_matches( assert_last_visit_matches(
self.storage, origin_url, type=RevisionType.MERCURIAL.value, status="full" loader.storage, repo_url, type=RevisionType.MERCURIAL.value, status="full"
) )
revisions = [] revisions = []
snapshot = snapshot_get_latest(self.storage, origin_url) snapshot = snapshot_get_latest(loader.storage, repo_url)
for branch in snapshot.branches.values(): for branch in snapshot.branches.values():
if branch.target_type.value != "revision": if branch.target_type.value != "revision":
continue continue
revisions.append(branch.target) revisions.append(branch.target)
# extract original changesets info and the transplant sources # extract original changesets info and the transplant sources
hg_changesets = set() hg_changesets = set()
transplant_sources = set() transplant_sources = set()
for rev in self.storage.revision_log(revisions): for rev in loader.storage.revision_log(revisions):
hg_changesets.add(rev["metadata"]["node"]) hg_changesets.add(rev["metadata"]["node"])
for k, v in rev["metadata"]["extra_headers"]: for k, v in rev["metadata"]["extra_headers"]:
if k == "transplant_source": if k == "transplant_source":
transplant_sources.add(v.decode("ascii")) transplant_sources.add(v.decode("ascii"))
# check extracted data are valid # check extracted data are valid
self.assertTrue(len(hg_changesets) > 0) assert len(hg_changesets) > 0
self.assertTrue(len(transplant_sources) > 0) assert len(transplant_sources) > 0
self.assertTrue(transplant_sources.issubset(hg_changesets)) assert transplant_sources.issubset(hg_changesets)
def test_clone_with_timeout_timeout(caplog, tmp_path, monkeypatch): def test_clone_with_timeout_timeout(caplog, tmp_path, monkeypatch):
log = logging.getLogger("test_clone_with_timeout") log = logging.getLogger("test_clone_with_timeout")
def clone_timeout(source, dest): def clone_timeout(source, dest):
time.sleep(60) time.sleep(60)
Show All 39 Lines
View Options

swh/provenance/journal_client.py

# Copyright (C) 2018-2020 The Software Heritage developers # Copyright (C) 2018-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import copy
import logging import logging
import os import os
import time import time
from typing import Any, Dict
from unittest.mock import patch
import hglib import hglib
import pytest import pytest
from swh.model import hashutil from swh.model import hashutil
from swh.model.model import RevisionType from swh.model.model import RevisionType
from swh.loader.core.tests import BaseLoaderTest
from swh.storage.algos.snapshot import snapshot_get_latest from swh.storage.algos.snapshot import snapshot_get_latest
from swh.loader.tests import ( from swh.loader.tests import (
assert_last_visit_matches, assert_last_visit_matches,
check_snapshot, check_snapshot,
get_stats, get_stats,
prepare_repository_from_archive, prepare_repository_from_archive,
) )
from .common import HgLoaderMemoryStorage, HgArchiveLoaderMemoryStorage from ..loader import HgBundle20Loader, HgArchiveBundle20Loader, CloneTimeoutError
from ..loader import HgBundle20Loader, CloneTimeoutError
class BaseHgLoaderTest(BaseLoaderTest):
"""Mixin base loader test to prepare the mercurial
repository to uncompress, load and test the results.
"""
def setUp(
self,
archive_name="the-sandbox.tgz",
filename="the-sandbox",
uncompress_archive=True,
):
super().setUp(
archive_name=archive_name,
filename=filename,
prefix_tmp_folder_name="swh.loader.mercurial.",
start_path=os.path.dirname(__file__),
uncompress_archive=uncompress_archive,
)
def test_loader_hg_new_visit(swh_config, datadir, tmp_path): def test_loader_hg_new_visit_no_release(swh_config, datadir, tmp_path):
"""Eventful visit should yield 1 snapshot""" """Eventful visit should yield 1 snapshot"""
archive_name = "the-sandbox" archive_name = "the-sandbox"
archive_path = os.path.join(datadir, f"{archive_name}.tgz") archive_path = os.path.join(datadir, f"{archive_name}.tgz")
repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path) repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
loader = HgBundle20Loader(repo_url) loader = HgBundle20Loader(repo_url)
assert loader.load() == {"status": "eventful"} assert loader.load() == {"status": "eventful"}
Show All 27 Lines expected_snapshot = {
"develop": {"target": tip_revision_develop, "target_type": "revision"}, "develop": {"target": tip_revision_develop, "target_type": "revision"},
"default": {"target": tip_revision_default, "target_type": "revision"}, "default": {"target": tip_revision_default, "target_type": "revision"},
"HEAD": {"target": "develop", "target_type": "alias",}, "HEAD": {"target": "develop", "target_type": "alias",},
}, },
} }
check_snapshot(expected_snapshot, loader.storage) check_snapshot(expected_snapshot, loader.storage)
# Ensure archive loader yields the same snapshot
loader2 = HgArchiveBundle20Loader(
url=archive_path,
archive_path=archive_path,
visit_date="2016-05-03 15:16:32+00",
)
actual_load_status = loader2.load()
assert actual_load_status == {"status": "eventful"}
stats2 = get_stats(loader2.storage)
expected_stats = copy.deepcopy(stats)
expected_stats["origin"] += 1
expected_stats["origin_visit"] += 1
assert stats2 == expected_stats
# That visit yields the same snapshot
assert_last_visit_matches(
loader2.storage,
archive_path,
status="full",
type="hg",
snapshot=hashutil.hash_to_bytes("3b8fe58e467deb7597b12a5fd3b2c096b8c02028"),
)
def test_loader_hg_new_visit_with_release(swh_config, datadir, tmp_path):
"""Eventful visit with release should yield 1 snapshot"""
archive_name = "hello"
archive_path = os.path.join(datadir, f"{archive_name}.tgz")
repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
loader = HgBundle20Loader(url=repo_url, visit_date="2016-05-03 15:16:32+00",)
actual_load_status = loader.load()
assert actual_load_status == {"status": "eventful"}
class CommonHgLoaderData:
def assert_data_ok(self, actual_load_status: Dict[str, Any]):
# then # then
self.assertCountContents(3) # type: ignore stats = get_stats(loader.storage)
self.assertCountDirectories(3) # type: ignore assert stats == {
self.assertCountReleases(1) # type: ignore "content": 3,
self.assertCountRevisions(3) # type: ignore "directory": 3,
"origin": 1,
"origin_visit": 1,
"person": 3,
"release": 1,
"revision": 3,
"skipped_content": 0,
"snapshot": 1,
}
# cf. test_loader.org for explaining from where those hashes
tip_release = "515c4d72e089404356d0f4b39d60f948b8999140" tip_release = "515c4d72e089404356d0f4b39d60f948b8999140"
self.assertReleasesContain([tip_release]) # type: ignore release = loader.storage.release_get([hashutil.hash_to_bytes(tip_release)])
assert release is not None
tip_revision_default = "c3dbe4fbeaaa98dd961834e4007edb3efb0e2a27" tip_revision_default = "c3dbe4fbeaaa98dd961834e4007edb3efb0e2a27"
# cf. test_loader.org for explaining from where those hashes revision = loader.storage.revision_get(
# come from [hashutil.hash_to_bytes(tip_revision_default)]
expected_revisions = { )
# revision hash | directory hash # noqa assert revision is not None
"93b48d515580522a05f389bec93227fc8e43d940": "43d727f2f3f2f7cb3b098ddad1d7038464a4cee2", # noqa
"8dd3db5d5519e4947f035d141581d304565372d2": "b3f85f210ff86d334575f64cb01c5bf49895b63e", # noqa
tip_revision_default: "8f2be433c945384c85920a8e60f2a68d2c0f20fb",
}
self.assertRevisionsContain(expected_revisions) # type: ignore
self.assertCountSnapshots(1) # type: ignore
expected_snapshot_id = "d35668e02e2ba4321dc951cd308cf883786f918a"
expected_snapshot = { expected_snapshot = {
"id": "d35668e02e2ba4321dc951cd308cf883786f918a", "id": expected_snapshot_id,
"branches": { "branches": {
"default": {"target": tip_revision_default, "target_type": "revision"}, "default": {"target": tip_revision_default, "target_type": "revision"},
"0.1": {"target": tip_release, "target_type": "release"}, "0.1": {"target": tip_release, "target_type": "release"},
"HEAD": {"target": "default", "target_type": "alias",}, "HEAD": {"target": "default", "target_type": "alias",},
}, },
} }
self.assertSnapshotEqual(expected_snapshot) # type: ignore check_snapshot(expected_snapshot, loader.storage)
assert actual_load_status == {"status": "eventful"}
assert_last_visit_matches( assert_last_visit_matches(
self.storage, # type: ignore loader.storage, repo_url, type=RevisionType.MERCURIAL.value, status="full",
self.repo_url, # type: ignore
type=RevisionType.MERCURIAL.value,
status="full",
) )
# Ensure archive loader yields the same snapshot
class WithReleaseLoaderTest(BaseHgLoaderTest, CommonHgLoaderData): loader2 = HgArchiveBundle20Loader(
"""Load a mercurial repository with release url=archive_path,
archive_path=archive_path,
"""
def setUp(self):
super().setUp(archive_name="hello.tgz", filename="hello")
self.loader = HgLoaderMemoryStorage(
url=self.repo_url,
visit_date="2016-05-03 15:16:32+00", visit_date="2016-05-03 15:16:32+00",
directory=self.destination_path,
) )
self.storage = self.loader.storage
def test_load(self):
"""Load a repository with tags results in 1 snapshot
""" actual_load_status = loader2.load()
# when assert actual_load_status == {"status": "eventful"}
actual_load_status = self.loader.load()
self.assert_data_ok(actual_load_status)
class ArchiveLoaderTest(BaseHgLoaderTest, CommonHgLoaderData):
"""Load a mercurial repository archive with release
""" stats2 = get_stats(loader2.storage)
expected_stats = copy.deepcopy(stats)
expected_stats["origin"] += 1
expected_stats["origin_visit"] += 1
assert stats2 == expected_stats
def setUp(self): # That visit yields the same snapshot
super().setUp( assert_last_visit_matches(
archive_name="hello.tgz", filename="hello", uncompress_archive=False loader2.storage,
) archive_path,
self.loader = HgArchiveLoaderMemoryStorage( status="full",
url=self.repo_url, type="hg",
visit_date="2016-05-03 15:16:32+00", snapshot=hashutil.hash_to_bytes(expected_snapshot_id),
archive_path=self.destination_path,
) )
self.storage = self.loader.storage
def test_load(self):
"""Load a mercurial repository archive with tags results in 1 snapshot
"""
# when
actual_load_status = self.loader.load()
self.assert_data_ok(actual_load_status)
@patch("swh.loader.mercurial.archive_extract.patoolib") def test_visit_with_archive_decompression_failure(swh_config, mocker, datadir):
def test_load_with_failure(self, mock_patoo): """Failure to decompress should fail early, no data is ingested"""
mock_patoo = mocker.patch("swh.loader.mercurial.archive_extract.patoolib")
mock_patoo.side_effect = ValueError mock_patoo.side_effect = ValueError
# when archive_name = "hello"
r = self.loader.load() archive_path = os.path.join(datadir, f"{archive_name}.tgz")
# Ensure archive loader yields the same snapshot
loader = HgArchiveBundle20Loader(
url=archive_path, visit_date="2016-05-03 15:16:32+00",
)
self.assertEqual(r, {"status": "failed"}) actual_load_status = loader.load()
self.assertCountContents(0) assert actual_load_status == {"status": "failed"}
self.assertCountDirectories(0)
self.assertCountRevisions(0) stats = get_stats(loader.storage)
self.assertCountReleases(0) assert stats == {
self.assertCountSnapshots(0) "content": 0,
"directory": 0,
"origin": 1,
"origin_visit": 1,
"person": 0,
"release": 0,
"revision": 0,
"skipped_content": 0,
"snapshot": 0,
}
# That visit yields the same snapshot
assert_last_visit_matches(
loader.storage, archive_path, status="partial", type="hg", snapshot=None
)
class WithTransplantLoaderTest(BaseHgLoaderTest): def test_visit_repository_with_transplant_operations(swh_config, datadir, tmp_path):
"""Load a mercurial repository where transplant operations """Visit a mercurial repository visit transplant operations within should yield a
have been used. snapshot as well.
ardumontAuthorUnsubmitted
Done
Inline Actions

I don't recall exactly what exactly was the bug about but that's the scenario to ensure we don't fail on it again [1]

[1] https://www.mercurial-scm.org/wiki/TransplantExtension

ardumont: I don't recall exactly what exactly was the bug about but that's the scenario to ensure we…
anlambertUnsubmitted
Not Done
Inline Actions

Test context: T1729

anlambert: Test context: T1729
ardumontAuthorUnsubmitted
Done
Inline Actions

Thank you, sir ;)

ardumont: Thank you, sir ;)
""" """
def setUp(self): archive_name = "transplant"
super().setUp(archive_name="transplant.tgz", filename="transplant") archive_path = os.path.join(datadir, f"{archive_name}.tgz")
self.loader = HgLoaderMemoryStorage( repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
url=self.repo_url, loader = HgBundle20Loader(url=repo_url, visit_date="2019-05-23 12:06:00+00",)
visit_date="2019-05-23 12:06:00+00",
directory=self.destination_path,
)
self.storage = self.loader.storage
def test_load(self):
# load hg repository # load hg repository
actual_load_status = self.loader.load() actual_load_status = loader.load()
assert actual_load_status == {"status": "eventful"} assert actual_load_status == {"status": "eventful"}
# collect swh revisions # collect swh revisions
origin_url = self.storage.origin_get([{"type": "hg", "url": self.repo_url}])[0][
"url"
]
assert_last_visit_matches( assert_last_visit_matches(
self.storage, origin_url, type=RevisionType.MERCURIAL.value, status="full" loader.storage, repo_url, type=RevisionType.MERCURIAL.value, status="full"
) )
revisions = [] revisions = []
snapshot = snapshot_get_latest(self.storage, origin_url) snapshot = snapshot_get_latest(loader.storage, repo_url)
for branch in snapshot.branches.values(): for branch in snapshot.branches.values():
if branch.target_type.value != "revision": if branch.target_type.value != "revision":
continue continue
revisions.append(branch.target) revisions.append(branch.target)
# extract original changesets info and the transplant sources # extract original changesets info and the transplant sources
hg_changesets = set() hg_changesets = set()
transplant_sources = set() transplant_sources = set()
for rev in self.storage.revision_log(revisions): for rev in loader.storage.revision_log(revisions):
hg_changesets.add(rev["metadata"]["node"]) hg_changesets.add(rev["metadata"]["node"])
for k, v in rev["metadata"]["extra_headers"]: for k, v in rev["metadata"]["extra_headers"]:
if k == "transplant_source": if k == "transplant_source":
transplant_sources.add(v.decode("ascii")) transplant_sources.add(v.decode("ascii"))
# check extracted data are valid # check extracted data are valid
self.assertTrue(len(hg_changesets) > 0) assert len(hg_changesets) > 0
self.assertTrue(len(transplant_sources) > 0) assert len(transplant_sources) > 0
self.assertTrue(transplant_sources.issubset(hg_changesets)) assert transplant_sources.issubset(hg_changesets)
def test_clone_with_timeout_timeout(caplog, tmp_path, monkeypatch): def test_clone_with_timeout_timeout(caplog, tmp_path, monkeypatch):
log = logging.getLogger("test_clone_with_timeout") log = logging.getLogger("test_clone_with_timeout")
def clone_timeout(source, dest): def clone_timeout(source, dest):
time.sleep(60) time.sleep(60)
Show All 39 Lines
View Options

swh/provenance/postgresql/provenance.py

# Copyright (C) 2018-2020 The Software Heritage developers # Copyright (C) 2018-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import copy
import logging import logging
import os import os
import time import time
from typing import Any, Dict
from unittest.mock import patch
import hglib import hglib
import pytest import pytest
from swh.model import hashutil from swh.model import hashutil
from swh.model.model import RevisionType from swh.model.model import RevisionType
from swh.loader.core.tests import BaseLoaderTest
from swh.storage.algos.snapshot import snapshot_get_latest from swh.storage.algos.snapshot import snapshot_get_latest
from swh.loader.tests import ( from swh.loader.tests import (
assert_last_visit_matches, assert_last_visit_matches,
check_snapshot, check_snapshot,
get_stats, get_stats,
prepare_repository_from_archive, prepare_repository_from_archive,
) )
from .common import HgLoaderMemoryStorage, HgArchiveLoaderMemoryStorage from ..loader import HgBundle20Loader, HgArchiveBundle20Loader, CloneTimeoutError
from ..loader import HgBundle20Loader, CloneTimeoutError
class BaseHgLoaderTest(BaseLoaderTest):
"""Mixin base loader test to prepare the mercurial
repository to uncompress, load and test the results.
"""
def setUp(
self,
archive_name="the-sandbox.tgz",
filename="the-sandbox",
uncompress_archive=True,
):
super().setUp(
archive_name=archive_name,
filename=filename,
prefix_tmp_folder_name="swh.loader.mercurial.",
start_path=os.path.dirname(__file__),
uncompress_archive=uncompress_archive,
)
def test_loader_hg_new_visit(swh_config, datadir, tmp_path): def test_loader_hg_new_visit_no_release(swh_config, datadir, tmp_path):
"""Eventful visit should yield 1 snapshot""" """Eventful visit should yield 1 snapshot"""
archive_name = "the-sandbox" archive_name = "the-sandbox"
archive_path = os.path.join(datadir, f"{archive_name}.tgz") archive_path = os.path.join(datadir, f"{archive_name}.tgz")
repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path) repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
loader = HgBundle20Loader(repo_url) loader = HgBundle20Loader(repo_url)
assert loader.load() == {"status": "eventful"} assert loader.load() == {"status": "eventful"}
Show All 27 Lines expected_snapshot = {
"develop": {"target": tip_revision_develop, "target_type": "revision"}, "develop": {"target": tip_revision_develop, "target_type": "revision"},
"default": {"target": tip_revision_default, "target_type": "revision"}, "default": {"target": tip_revision_default, "target_type": "revision"},
"HEAD": {"target": "develop", "target_type": "alias",}, "HEAD": {"target": "develop", "target_type": "alias",},
}, },
} }
check_snapshot(expected_snapshot, loader.storage) check_snapshot(expected_snapshot, loader.storage)
# Ensure archive loader yields the same snapshot
loader2 = HgArchiveBundle20Loader(
url=archive_path,
archive_path=archive_path,
visit_date="2016-05-03 15:16:32+00",
)
actual_load_status = loader2.load()
assert actual_load_status == {"status": "eventful"}
stats2 = get_stats(loader2.storage)
expected_stats = copy.deepcopy(stats)
expected_stats["origin"] += 1
expected_stats["origin_visit"] += 1
assert stats2 == expected_stats
# That visit yields the same snapshot
assert_last_visit_matches(
loader2.storage,
archive_path,
status="full",
type="hg",
snapshot=hashutil.hash_to_bytes("3b8fe58e467deb7597b12a5fd3b2c096b8c02028"),
)
def test_loader_hg_new_visit_with_release(swh_config, datadir, tmp_path):
"""Eventful visit with release should yield 1 snapshot"""
archive_name = "hello"
archive_path = os.path.join(datadir, f"{archive_name}.tgz")
repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
loader = HgBundle20Loader(url=repo_url, visit_date="2016-05-03 15:16:32+00",)
actual_load_status = loader.load()
assert actual_load_status == {"status": "eventful"}
class CommonHgLoaderData:
def assert_data_ok(self, actual_load_status: Dict[str, Any]):
# then # then
self.assertCountContents(3) # type: ignore stats = get_stats(loader.storage)
self.assertCountDirectories(3) # type: ignore assert stats == {
self.assertCountReleases(1) # type: ignore "content": 3,
self.assertCountRevisions(3) # type: ignore "directory": 3,
"origin": 1,
"origin_visit": 1,
"person": 3,
"release": 1,
"revision": 3,
"skipped_content": 0,
"snapshot": 1,
}
# cf. test_loader.org for explaining from where those hashes
tip_release = "515c4d72e089404356d0f4b39d60f948b8999140" tip_release = "515c4d72e089404356d0f4b39d60f948b8999140"
self.assertReleasesContain([tip_release]) # type: ignore release = loader.storage.release_get([hashutil.hash_to_bytes(tip_release)])
assert release is not None
tip_revision_default = "c3dbe4fbeaaa98dd961834e4007edb3efb0e2a27" tip_revision_default = "c3dbe4fbeaaa98dd961834e4007edb3efb0e2a27"
# cf. test_loader.org for explaining from where those hashes revision = loader.storage.revision_get(
# come from [hashutil.hash_to_bytes(tip_revision_default)]
expected_revisions = { )
# revision hash | directory hash # noqa assert revision is not None
"93b48d515580522a05f389bec93227fc8e43d940": "43d727f2f3f2f7cb3b098ddad1d7038464a4cee2", # noqa
"8dd3db5d5519e4947f035d141581d304565372d2": "b3f85f210ff86d334575f64cb01c5bf49895b63e", # noqa
tip_revision_default: "8f2be433c945384c85920a8e60f2a68d2c0f20fb",
}
self.assertRevisionsContain(expected_revisions) # type: ignore
self.assertCountSnapshots(1) # type: ignore
expected_snapshot_id = "d35668e02e2ba4321dc951cd308cf883786f918a"
expected_snapshot = { expected_snapshot = {
"id": "d35668e02e2ba4321dc951cd308cf883786f918a", "id": expected_snapshot_id,
"branches": { "branches": {
"default": {"target": tip_revision_default, "target_type": "revision"}, "default": {"target": tip_revision_default, "target_type": "revision"},
"0.1": {"target": tip_release, "target_type": "release"}, "0.1": {"target": tip_release, "target_type": "release"},
"HEAD": {"target": "default", "target_type": "alias",}, "HEAD": {"target": "default", "target_type": "alias",},
}, },
} }
self.assertSnapshotEqual(expected_snapshot) # type: ignore check_snapshot(expected_snapshot, loader.storage)
assert actual_load_status == {"status": "eventful"}
assert_last_visit_matches( assert_last_visit_matches(
self.storage, # type: ignore loader.storage, repo_url, type=RevisionType.MERCURIAL.value, status="full",
self.repo_url, # type: ignore
type=RevisionType.MERCURIAL.value,
status="full",
) )
# Ensure archive loader yields the same snapshot
class WithReleaseLoaderTest(BaseHgLoaderTest, CommonHgLoaderData): loader2 = HgArchiveBundle20Loader(
"""Load a mercurial repository with release url=archive_path,
archive_path=archive_path,
"""
def setUp(self):
super().setUp(archive_name="hello.tgz", filename="hello")
self.loader = HgLoaderMemoryStorage(
url=self.repo_url,
visit_date="2016-05-03 15:16:32+00", visit_date="2016-05-03 15:16:32+00",
directory=self.destination_path,
) )
self.storage = self.loader.storage
def test_load(self):
"""Load a repository with tags results in 1 snapshot
""" actual_load_status = loader2.load()
# when assert actual_load_status == {"status": "eventful"}
actual_load_status = self.loader.load()
self.assert_data_ok(actual_load_status)
class ArchiveLoaderTest(BaseHgLoaderTest, CommonHgLoaderData):
"""Load a mercurial repository archive with release
""" stats2 = get_stats(loader2.storage)
expected_stats = copy.deepcopy(stats)
expected_stats["origin"] += 1
expected_stats["origin_visit"] += 1
assert stats2 == expected_stats
def setUp(self): # That visit yields the same snapshot
super().setUp( assert_last_visit_matches(
archive_name="hello.tgz", filename="hello", uncompress_archive=False loader2.storage,
) archive_path,
self.loader = HgArchiveLoaderMemoryStorage( status="full",
url=self.repo_url, type="hg",
visit_date="2016-05-03 15:16:32+00", snapshot=hashutil.hash_to_bytes(expected_snapshot_id),
archive_path=self.destination_path,
) )
self.storage = self.loader.storage
def test_load(self):
"""Load a mercurial repository archive with tags results in 1 snapshot
"""
# when
actual_load_status = self.loader.load()
self.assert_data_ok(actual_load_status)
@patch("swh.loader.mercurial.archive_extract.patoolib") def test_visit_with_archive_decompression_failure(swh_config, mocker, datadir):
def test_load_with_failure(self, mock_patoo): """Failure to decompress should fail early, no data is ingested"""
mock_patoo = mocker.patch("swh.loader.mercurial.archive_extract.patoolib")
mock_patoo.side_effect = ValueError mock_patoo.side_effect = ValueError
# when archive_name = "hello"
r = self.loader.load() archive_path = os.path.join(datadir, f"{archive_name}.tgz")
# Ensure archive loader yields the same snapshot
loader = HgArchiveBundle20Loader(
url=archive_path, visit_date="2016-05-03 15:16:32+00",
)
self.assertEqual(r, {"status": "failed"}) actual_load_status = loader.load()
self.assertCountContents(0) assert actual_load_status == {"status": "failed"}
self.assertCountDirectories(0)
self.assertCountRevisions(0) stats = get_stats(loader.storage)
self.assertCountReleases(0) assert stats == {
self.assertCountSnapshots(0) "content": 0,
"directory": 0,
"origin": 1,
"origin_visit": 1,
"person": 0,
"release": 0,
"revision": 0,
"skipped_content": 0,
"snapshot": 0,
}
# That visit yields the same snapshot
assert_last_visit_matches(
loader.storage, archive_path, status="partial", type="hg", snapshot=None
)
class WithTransplantLoaderTest(BaseHgLoaderTest): def test_visit_repository_with_transplant_operations(swh_config, datadir, tmp_path):
"""Load a mercurial repository where transplant operations """Visit a mercurial repository visit transplant operations within should yield a
have been used. snapshot as well.
ardumontAuthorUnsubmitted
Done
Inline Actions

I don't recall exactly what exactly was the bug about but that's the scenario to ensure we don't fail on it again [1]

[1] https://www.mercurial-scm.org/wiki/TransplantExtension

ardumont: I don't recall exactly what exactly was the bug about but that's the scenario to ensure we…
anlambertUnsubmitted
Not Done
Inline Actions

Test context: T1729

anlambert: Test context: T1729
ardumontAuthorUnsubmitted
Done
Inline Actions

Thank you, sir ;)

ardumont: Thank you, sir ;)
""" """
def setUp(self): archive_name = "transplant"
super().setUp(archive_name="transplant.tgz", filename="transplant") archive_path = os.path.join(datadir, f"{archive_name}.tgz")
self.loader = HgLoaderMemoryStorage( repo_url = prepare_repository_from_archive(archive_path, archive_name, tmp_path)
url=self.repo_url, loader = HgBundle20Loader(url=repo_url, visit_date="2019-05-23 12:06:00+00",)
visit_date="2019-05-23 12:06:00+00",
directory=self.destination_path,
)
self.storage = self.loader.storage
def test_load(self):
# load hg repository # load hg repository
actual_load_status = self.loader.load() actual_load_status = loader.load()
assert actual_load_status == {"status": "eventful"} assert actual_load_status == {"status": "eventful"}
# collect swh revisions # collect swh revisions
origin_url = self.storage.origin_get([{"type": "hg", "url": self.repo_url}])[0][
"url"
]
assert_last_visit_matches( assert_last_visit_matches(
self.storage, origin_url, type=RevisionType.MERCURIAL.value, status="full" loader.storage, repo_url, type=RevisionType.MERCURIAL.value, status="full"
) )
revisions = [] revisions = []
snapshot = snapshot_get_latest(self.storage, origin_url) snapshot = snapshot_get_latest(loader.storage, repo_url)
for branch in snapshot.branches.values(): for branch in snapshot.branches.values():
if branch.target_type.value != "revision": if branch.target_type.value != "revision":
continue continue
revisions.append(branch.target) revisions.append(branch.target)
# extract original changesets info and the transplant sources # extract original changesets info and the transplant sources
hg_changesets = set() hg_changesets = set()
transplant_sources = set() transplant_sources = set()
for rev in self.storage.revision_log(revisions): for rev in loader.storage.revision_log(revisions):
hg_changesets.add(rev["metadata"]["node"]) hg_changesets.add(rev["metadata"]["node"])
for k, v in rev["metadata"]["extra_headers"]: for k, v in rev["metadata"]["extra_headers"]:
if k == "transplant_source": if k == "transplant_source":
transplant_sources.add(v.decode("ascii")) transplant_sources.add(v.decode("ascii"))
# check extracted data are valid # check extracted data are valid
self.assertTrue(len(hg_changesets) > 0) assert len(hg_changesets) > 0
self.assertTrue(len(transplant_sources) > 0) assert len(transplant_sources) > 0
self.assertTrue(transplant_sources.issubset(hg_changesets)) assert transplant_sources.issubset(hg_changesets)
def test_clone_with_timeout_timeout(caplog, tmp_path, monkeypatch): def test_clone_with_timeout_timeout(caplog, tmp_path, monkeypatch):
log = logging.getLogger("test_clone_with_timeout") log = logging.getLogger("test_clone_with_timeout")
def clone_timeout(source, dest): def clone_timeout(source, dest):
time.sleep(60) time.sleep(60)
Show All 39 Lines
View Options

swh/provenance/tests/data/origins.csv

Loading...
View Options

swh/provenance/tests/test_cli.py

Loading...
View Options

swh/provenance/tests/test_journal_client.py

Loading...
View Options

swh/provenance/tests/test_utils.py

Loading...
About · Developer information · Legal