Page MenuHomeSoftware Heritage
Differential D8098

Add a `swh provenance revision from-journal` command
ClosedPublic
Actions

Authored by douardda on Jul 8 2022, 11:52 AM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Commits
rDPROVe30e9f68262f: Add a `swh provenance revision from-journal` command
rDPROV2492c8c97a25: Rename process_journal_objects as process_journal_origins
Summary

allowing to ingest revisions from the kafka journal.

Also provide a swh provenance revision from-csv command to replace the
swh provenance iter-revision command.

Also rename process_journal_objects as process_journal_origins
since it's only dealing with origins.

Diff Detail

Unit TestsFailed
View All

TimeTest
45,111 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.provenance.tests.test_journal_client::test_cli_revision_from_journal_client
swh_storage = <swh.storage.postgresql.storage.Storage object at 0x7f82dc7ec5f8> swh_storage_backend_config = {'check_config': {'check_write': True}, 'cls': 'postgresql', 'db': "user=postgres password=xxx dbname=tests host=127.0.0.1 port=15423 options=''", 'objstorage': {'cls': 'memory'}} kafka_prefix = 'wanrlzxbdh', kafka_server = '127.0.0.1:41567'
14,241 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.provenance.tests.test_journal_client::test_cli_origin_from_journal_client

Event Timeline

douardda created this revision.Jul 8 2022, 11:52 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 8 2022, 11:52 AM
douardda added a child revision: D8099: Remove unused mongo backend.Jul 8 2022, 11:53 AM
swh-public-ci added a comment.Jul 8 2022, 12:01 PM

Build is green

Patch application report for D8098 (id=29237)

Rebasing onto af616497e1...

Current branch diff-target is up to date.
Changes applied before test
commit 357917df50aec745ec024b3d46306ed870e87c20
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/637/ for more details.

Harbormaster completed remote builds in B30322: Diff 29237.Jul 8 2022, 12:01 PM
douardda requested review of this revision.Jul 8 2022, 12:01 PM
olasd added a subscriber: olasd.Jul 8 2022, 1:50 PM

Maybe a new cli test for the revision journal client would be nice

douardda retitled this revision from Rename process_journal_objects as process_journal_origins to Add a `swh provenance revision from-journal` command.Jul 8 2022, 2:27 PM
douardda edited the summary of this revision. (Show Details)
douardda updated this revision to Diff 29244.Jul 8 2022, 2:37 PM
douardda edited the summary of this revision. (Show Details)

add a test

olasd added inline comments.Jul 8 2022, 2:38 PM
swh/provenance/tests/test_journal_client.py
131

oops :-)

douardda added inline comments.Jul 8 2022, 2:39 PM
swh/provenance/tests/test_journal_client.py
131

grr (thx)

douardda updated this revision to Diff 29247.Jul 8 2022, 2:41 PM

grr

swh-public-ci added a comment.Jul 8 2022, 2:42 PM

Build has FAILED

Patch application report for D8098 (id=29244)

Rebasing onto af616497e1...

Current branch diff-target is up to date.
Changes applied before test
commit 545e14734f9df08759d4fcabb9621b05025a3048
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/641/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/641/console

Harbormaster failed remote builds in B30330: Diff 29244!Jul 8 2022, 2:42 PM
swh-public-ci added a comment.Jul 8 2022, 2:53 PM

Build is green

Patch application report for D8098 (id=29247)

Rebasing onto af616497e1...

Current branch diff-target is up to date.
Changes applied before test
commit 719a74d881415a5ff3601d7a4b57c342764dd2ce
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/644/ for more details.

Harbormaster completed remote builds in B30333: Diff 29247.Jul 8 2022, 2:53 PM
olasd requested changes to this revision.Jul 8 2022, 2:55 PM

Looks like some copy-and-waste happened here, sorry I hadn't noticed at first !

swh/provenance/cli.py
363–371

Looks like you're using the wrong function here

382

and the wrong object type here

This revision now requires changes to proceed.Jul 8 2022, 2:55 PM
douardda updated this revision to Diff 29253.Jul 8 2022, 5:12 PM

fix mistakes reported by olasd (thx a lot) and handle dates in process_journal_recisions

douardda marked 3 inline comments as done.Jul 8 2022, 5:13 PM
swh-public-ci added a comment.Jul 8 2022, 5:22 PM

Build is green

Patch application report for D8098 (id=29253)

Rebasing onto af616497e1...

Current branch diff-target is up to date.
Changes applied before test
commit e30e9f68262f079468dc7491c2c958530131af16
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/647/ for more details.

Harbormaster completed remote builds in B30339: Diff 29253.Jul 8 2022, 5:22 PM
olasd accepted this revision.Jul 8 2022, 5:51 PM
This revision is now accepted and ready to land.Jul 8 2022, 5:51 PM
Closed by commit rDPROV2492c8c97a25: Rename process_journal_objects as process_journal_origins (authored by douardda). · Explain WhyJul 9 2022, 10:02 AM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDPROV2492c8c97a25: Rename process_journal_objects as process_journal_origins.
douardda added a commit: rDPROVe30e9f68262f: Add a `swh provenance revision from-journal` command.

Revision Contents
Changeset List

PathSize
swh/
provenance/
184 lines
32 lines
tests/
53 lines

Diff 29244

View Options

swh/provenance/cli.py

Loading...
View Options

swh/provenance/journal_client.py

Loading...
View Options

swh/provenance/tests/test_journal_client.py

Loading...
About · Developer information · Legal