Page MenuHomeSoftware Heritage
Differential D8100

Add a `swh provenance directory flatten` command
ClosedPublic
Actions

Authored by douardda on Jul 8 2022, 11:53 AM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Commits
rDPROV804c3a371eb5: Add a `swh provenance directory flatten` command
Summary

Depends on D8099

Diff Detail

Event Timeline

douardda created this revision.Jul 8 2022, 11:53 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 8 2022, 11:53 AM
swh-public-ci added a comment.Jul 8 2022, 12:03 PM

Build is green

Patch application report for D8100 (id=29239)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..9dbc4bb
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  22 +
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  57 ++-
 swh/provenance/tests/test_provenance_storage.py |   5 -
 13 files changed, 357 insertions(+), 614 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py
Changes applied before test
commit 9dbc4bb45ae56252b0ea913b44e524dd0f8facce
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 6e445ea9d16c6f25565a809f8abb5b7ad9b85a83
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 357917df50aec745ec024b3d46306ed870e87c20
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/639/ for more details.

Harbormaster completed remote builds in B30324: Diff 29239.Jul 8 2022, 12:03 PM
douardda requested review of this revision.Jul 8 2022, 12:03 PM
olasd requested changes to this revision.Jul 8 2022, 2:00 PM
olasd added a subscriber: olasd.
olasd added inline comments.
swh/provenance/directory.py
43–65

this variable name looks wrong :-)

50
swh/provenance/interface.py
144
swh/provenance/tests/test_directory_flatten.py
89–118

This could probably move to a common initialization function for the directory flattening tests.

This revision now requires changes to proceed.Jul 8 2022, 2:00 PM
douardda updated this revision to Diff 29243.Jul 8 2022, 2:26 PM
douardda marked 3 inline comments as done.

fixes/docstrings suggested by olasd

swh-public-ci added a comment.Jul 8 2022, 2:33 PM

Build is green

Patch application report for D8100 (id=29243)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..9709276
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_provenance_storage.py |   5 -
 13 files changed, 340 insertions(+), 619 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py
Changes applied before test
commit 9709276cd1164d589f13fa1a8656322644965275
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 6e445ea9d16c6f25565a809f8abb5b7ad9b85a83
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 357917df50aec745ec024b3d46306ed870e87c20
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/640/ for more details.

Harbormaster completed remote builds in B30329: Diff 29243.Jul 8 2022, 2:33 PM
douardda updated this revision to Diff 29246.Jul 8 2022, 2:38 PM

rebase

olasd accepted this revision.Jul 8 2022, 2:38 PM

Nice, thanks!

This revision is now accepted and ready to land.Jul 8 2022, 2:38 PM
douardda updated this revision to Diff 29249.Jul 8 2022, 2:41 PM

rebase

swh-public-ci added a comment.Jul 8 2022, 2:42 PM

Build has FAILED

Patch application report for D8100 (id=29246)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..f8c643c
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_journal_client.py     |  53 +++
 swh/provenance/tests/test_provenance_storage.py |   5 -
 14 files changed, 393 insertions(+), 619 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py
Changes applied before test
commit f8c643c41ab5ca967cd64f054cbf7cc5154db07b
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 6786d040f446140160242bb9d64740ebf3002190
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 545e14734f9df08759d4fcabb9621b05025a3048
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/643/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/643/console

Harbormaster failed remote builds in B30332: Diff 29246!Jul 8 2022, 2:42 PM
swh-public-ci added a comment.Jul 8 2022, 2:53 PM

Build is green

Patch application report for D8100 (id=29249)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..ce18e7a
Fast-forward
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  32 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_journal_client.py     |  52 +++
 swh/provenance/tests/test_provenance_storage.py |   5 -
 14 files changed, 392 insertions(+), 619 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py
Changes applied before test
commit ce18e7a404f234916684f9fc0297f542cab869ea
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit e49a047cb0d19b4cbb735c30d13ad166eed44a3e
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit 719a74d881415a5ff3601d7a4b57c342764dd2ce
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/646/ for more details.

Harbormaster completed remote builds in B30335: Diff 29249.Jul 8 2022, 2:53 PM
douardda updated this revision to Diff 29255.Jul 8 2022, 5:13 PM

rebase

swh-public-ci added a comment.Jul 8 2022, 5:23 PM

Build is green

Patch application report for D8100 (id=29255)

Could not rebase; Attempt merge onto af616497e1...

Updating af61649..804c3a3
Fast-forward
 mypy.ini                                        |   3 -
 pytest.ini                                      |   4 -
 requirements-test.txt                           |   1 -
 requirements.txt                                |   2 -
 swh/provenance/__init__.py                      |   6 -
 swh/provenance/cli.py                           | 233 ++++++++++-
 swh/provenance/directory.py                     |  24 +-
 swh/provenance/interface.py                     |   9 +-
 swh/provenance/journal_client.py                |  33 +-
 swh/provenance/mongo/README.md                  |  44 --
 swh/provenance/mongo/__init__.py                |   0
 swh/provenance/mongo/backend.py                 | 529 ------------------------
 swh/provenance/postgresql/provenance.py         |  14 +
 swh/provenance/sql/60-indexes.sql               |   1 +
 swh/provenance/tests/conftest.py                |  19 +-
 swh/provenance/tests/test_directory_flatten.py  |  43 +-
 swh/provenance/tests/test_journal_client.py     |  55 ++-
 swh/provenance/tests/test_provenance_storage.py |   5 -
 18 files changed, 395 insertions(+), 630 deletions(-)
 delete mode 100644 swh/provenance/mongo/README.md
 delete mode 100644 swh/provenance/mongo/__init__.py
 delete mode 100644 swh/provenance/mongo/backend.py
Changes applied before test
commit 804c3a371eb5181df6a33fd742f4576dd44fdd98
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:51:01 2022 +0200

    Add a `swh provenance directory flatten` command

commit 9d031913d29c59c5dc87991742bd4080c26ff73c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:50:33 2022 +0200

    Remove unused mongo backend

commit e30e9f68262f079468dc7491c2c958530131af16
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:41:11 2022 +0200

    Add a `swh provenance revision from-journal` command
    
    allowing to ingest revisions from the kafka journal.
    
    Also provide a `swh provenance revision from-csv` command to replace the
    `swh provenance iter-revision` command.

commit 2492c8c97a2506b0aa2e1507af42f2c2fc37117f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jul 8 11:34:35 2022 +0200

    Rename process_journal_objects as process_journal_origins
    
    since it's only dealing with origins.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/649/ for more details.

Harbormaster completed remote builds in B30341: Diff 29255.Jul 8 2022, 5:23 PM
Closed by commit rDPROV804c3a371eb5: Add a `swh provenance directory flatten` command (authored by douardda). · Explain WhyJul 9 2022, 10:02 AM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDPROV804c3a371eb5: Add a `swh provenance directory flatten` command.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
provenance/
44 lines70%
Loading...
24 lines93%
Loading...
9 lines65%
Loading...
postgresql/
14 lines93%
Loading...
sql/
1 line--
tests/
43 lines100%
Loading...

Diff 29249

View Options

swh/provenance/cli.py

Loading...
View Options

swh/provenance/directory.py

Loading...
View Options

swh/provenance/interface.py

Loading...
View Options

swh/provenance/postgresql/provenance.py

Loading...
View Options

swh/provenance/sql/60-indexes.sql

Loading...
View Options

swh/provenance/tests/test_directory_flatten.py

Loading...
About · Developer information · Legal