Details

Reviewers

olasd

Group Reviewers

Reviewers

Commits

rDPROV804c3a371eb5: Add a `swh provenance directory flatten` command

Summary

Depends on D8099

Diff Detail

Repository

rDPROV Provenance database

Branch

master

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 30332
Build 47422: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 47421: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	46,405 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.provenance.tests.test_journal_client::test_cli_revision_from_journal_client swh_storage = <swh.storage.postgresql.storage.Storage object at 0x7eff507436a0> swh_storage_backend_config = {'check_config': {'check_write': True}, 'cls': 'postgresql', 'db': "user=postgres password=xxx dbname=tests host=127.0.0.1 port=22108 options=''", 'objstorage': {'cls': 'memory'}} kafka_prefix = 'khhsyazvgo', kafka_server = '127.0.0.1:42141'
	14,246 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.provenance.tests.test_journal_client::test_cli_origin_from_journal_client

Event Timeline

douardda created this revision.Jul 8 2022, 11:53 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptJul 8 2022, 11:53 AM

Build is green

Patch application report for D8100 (id=29239)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..9dbc4bb
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  22 +
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  57 ++-
 swh/provenance/tests/test_provenance_storage.py |   5 -
 13 files changed, 357 insertions(+), 614 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py

Changes applied before test

commit 9dbc4bb45ae56252b0ea913b44e524dd0f8facce
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 6e445ea9d16c6f25565a809f8abb5b7ad9b85a83
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 357917df50aec745ec024b3d46306ed870e87c20
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/639/ for more details.

Harbormaster completed remote builds in B30324: Diff 29239.Jul 8 2022, 12:03 PM

douardda requested review of this revision.Jul 8 2022, 12:03 PM

olasd requested changes to this revision.Jul 8 2022, 2:00 PM

olasd added a subscriber: olasd.

olasd added inline comments.

swh/provenance/directory.py
43–65	this variable name looks wrong :-)
50
swh/provenance/interface.py
144
swh/provenance/tests/test_directory_flatten.py
89–118	This could probably move to a common initialization function for the directory flattening tests.

This revision now requires changes to proceed.Jul 8 2022, 2:00 PM

fixes/docstrings suggested by olasd

Build is green

Patch application report for D8100 (id=29243)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..9709276
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_provenance_storage.py |   5 -
 13 files changed, 340 insertions(+), 619 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py

Changes applied before test

commit 9709276cd1164d589f13fa1a8656322644965275
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 6e445ea9d16c6f25565a809f8abb5b7ad9b85a83
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 357917df50aec745ec024b3d46306ed870e87c20
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/640/ for more details.

Harbormaster completed remote builds in B30329: Diff 29243.Jul 8 2022, 2:33 PM

rebase

Nice, thanks!

This revision is now accepted and ready to land.Jul 8 2022, 2:38 PM

rebase

Build has FAILED

Patch application report for D8100 (id=29246)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..f8c643c
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_journal_client.py     |  53 +++
 swh/provenance/tests/test_provenance_storage.py |   5 -
 14 files changed, 393 insertions(+), 619 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py

Changes applied before test

commit f8c643c41ab5ca967cd64f054cbf7cc5154db07b
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 6786d040f446140160242bb9d64740ebf3002190
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 545e14734f9df08759d4fcabb9621b05025a3048
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/643/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/643/console

Harbormaster failed remote builds in B30332: Diff 29246!Jul 8 2022, 2:42 PM

Build is green

Patch application report for D8100 (id=29249)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..ce18e7a
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_journal_client.py     |  52 +++
 swh/provenance/tests/test_provenance_storage.py |   5 -
 14 files changed, 392 insertions(+), 619 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py

Changes applied before test

commit ce18e7a404f234916684f9fc0297f542cab869ea
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit e49a047cb0d19b4cbb735c30d13ad166eed44a3e
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 719a74d881415a5ff3601d7a4b57c342764dd2ce
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/646/ for more details.

Harbormaster completed remote builds in B30335: Diff 29249.Jul 8 2022, 2:53 PM

rebase

Build is green

Patch application report for D8100 (id=29255)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..804c3a3
Fast-forward
 mypy.ini                                        |   3 -
 pytest.ini                                      |   4 -
 requirements-test.txt                           |   1 -
 requirements.txt                                |   2 -
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  33 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_journal_client.py     |  55 ++-
 swh/provenance/tests/test_provenance_storage.py |   5 -
 18 files changed, 395 insertions(+), 630 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py

Changes applied before test

commit 804c3a371eb5181df6a33fd742f4576dd44fdd98
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 9d031913d29c59c5dc87991742bd4080c26ff73c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit e30e9f68262f079468dc7491c2c958530131af16
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/649/ for more details.

Harbormaster completed remote builds in B30341: Diff 29255.Jul 8 2022, 5:23 PM

Closed by commit rDPROV804c3a371eb5: Add a `swh provenance directory flatten` command (authored by douardda). · Explain WhyJul 9 2022, 10:02 AM

This revision was automatically updated to reflect the committed changes.

douardda added a commit: rDPROV804c3a371eb5: Add a `swh provenance directory flatten` command.

Add a `swh provenance directory flatten` command
ClosedPublic
Actions

Details

Diff Detail

Unit TestsFailed
View All

Event Timeline

Patch application report for D8100 (id=29239)

Changes applied before test

Patch application report for D8100 (id=29243)

Changes applied before test

Patch application report for D8100 (id=29246)

Changes applied before test

Patch application report for D8100 (id=29249)

Changes applied before test

Patch application report for D8100 (id=29255)

Changes applied before test

Revision Contents
Changeset List

Diff 29246

swh/provenance/cli.py

swh/provenance/directory.py

swh/provenance/interface.py

swh/provenance/postgresql/provenance.py

swh/provenance/sql/60-indexes.sql

swh/provenance/tests/test_directory_flatten.py

Add a `swh provenance directory flatten` commandClosedPublicActions

Details

Diff Detail

Unit TestsFailedView All

Event Timeline

Patch application report for D8100 (id=29239)

Changes applied before test

Patch application report for D8100 (id=29243)

Changes applied before test

Patch application report for D8100 (id=29246)

Changes applied before test

Patch application report for D8100 (id=29249)

Changes applied before test

Patch application report for D8100 (id=29255)

Changes applied before test

Revision ContentsChangeset List

Diff 29246

swh/provenance/cli.py

swh/provenance/directory.py

swh/provenance/interface.py

swh/provenance/postgresql/provenance.py

swh/provenance/sql/60-indexes.sql

swh/provenance/tests/test_directory_flatten.py

Add a `swh provenance directory flatten` command
ClosedPublic
Actions

Unit TestsFailed
View All

Revision Contents
Changeset List