Page MenuHomeSoftware Heritage

Add a `swh provenance directory flatten` command
ClosedPublic

Authored by douardda on Jul 8 2022, 11:53 AM.

Details

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D8100 (id=29239)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..9dbc4bb
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  22 +
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  57 ++-
 swh/provenance/tests/test_provenance_storage.py |   5 -
 13 files changed, 357 insertions(+), 614 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py
Changes applied before test
commit 9dbc4bb45ae56252b0ea913b44e524dd0f8facce
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 6e445ea9d16c6f25565a809f8abb5b7ad9b85a83
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 357917df50aec745ec024b3d46306ed870e87c20
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/639/ for more details.

olasd requested changes to this revision.Jul 8 2022, 2:00 PM
olasd added a subscriber: olasd.
olasd added inline comments.
swh/provenance/directory.py
43–65

this variable name looks wrong :-)

50
swh/provenance/interface.py
144
swh/provenance/tests/test_directory_flatten.py
89–118

This could probably move to a common initialization function for the directory flattening tests.

This revision now requires changes to proceed.Jul 8 2022, 2:00 PM
douardda marked 3 inline comments as done.

fixes/docstrings suggested by olasd

Build is green

Patch application report for D8100 (id=29243)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..9709276
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_provenance_storage.py |   5 -
 13 files changed, 340 insertions(+), 619 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py
Changes applied before test
commit 9709276cd1164d589f13fa1a8656322644965275
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 6e445ea9d16c6f25565a809f8abb5b7ad9b85a83
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 357917df50aec745ec024b3d46306ed870e87c20
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/640/ for more details.

This revision is now accepted and ready to land.Jul 8 2022, 2:38 PM

Build has FAILED

Patch application report for D8100 (id=29246)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..f8c643c
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_journal_client.py     |  53 +++
 swh/provenance/tests/test_provenance_storage.py |   5 -
 14 files changed, 393 insertions(+), 619 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py
Changes applied before test
commit f8c643c41ab5ca967cd64f054cbf7cc5154db07b
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 6786d040f446140160242bb9d64740ebf3002190
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 545e14734f9df08759d4fcabb9621b05025a3048
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/643/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/643/console

Build is green

Patch application report for D8100 (id=29249)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..ce18e7a
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_journal_client.py     |  52 +++
 swh/provenance/tests/test_provenance_storage.py |   5 -
 14 files changed, 392 insertions(+), 619 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py
Changes applied before test
commit ce18e7a404f234916684f9fc0297f542cab869ea
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit e49a047cb0d19b4cbb735c30d13ad166eed44a3e
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 719a74d881415a5ff3601d7a4b57c342764dd2ce
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/646/ for more details.

Build is green

Patch application report for D8100 (id=29255)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..804c3a3
Fast-forward
 mypy.ini                                        |   3 -
 pytest.ini                                      |   4 -
 requirements-test.txt                           |   1 -
 requirements.txt                                |   2 -
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  33 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_journal_client.py     |  55 ++-
 swh/provenance/tests/test_provenance_storage.py |   5 -
 18 files changed, 395 insertions(+), 630 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py
Changes applied before test
commit 804c3a371eb5181df6a33fd742f4576dd44fdd98
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 9d031913d29c59c5dc87991742bd4080c26ff73c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit e30e9f68262f079468dc7491c2c958530131af16
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/649/ for more details.