Page MenuHomeSoftware Heritage

Reorganize the code
ClosedPublic

Authored by douardda on Sep 30 2022, 6:39 PM.

Details

Summary
  • move everything (swh)archive related in a archive/ submodule
  • move everything provenance storage related in a storage/ submodule (which remains a not ideal name, may be confusing with the general 'storage == swh-storage' acceptance in swh)
  • rename rabbitmq's backend from api/ to storage/rabbitmq
  • spit interface.py in 3 parts (one for each part, ProvenanceInterface, ProvenanceStorageInterface and ArchiveInterface).

Depends on D8592

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build was aborted

Patch application report for D8593 (id=31023)

Could not rebase; Attempt merge onto a2bcc8eace...

Updating a2bcc8e..9b08135
Fast-forward
 requirements-swh.txt                               |   4 +-
 swh/provenance/__init__.py                         | 100 +--
 swh/provenance/archive/__init__.py                 |  62 ++
 .../{archive.py => archive/interface.py}           |   0
 .../archive.py => archive/multiplexer.py}          |   0
 .../archive.py => archive/postgresql.py}           |   0
 .../{storage/archive.py => archive/storage.py}     |   0
 .../{swhgraph/archive.py => archive/swhgraph.py}   |   2 +-
 swh/provenance/cli.py                              |  18 +-
 swh/provenance/interface.py                        | 220 +------
 swh/provenance/multiplexer/__init__.py             |   0
 swh/provenance/postgresql/__init__.py              |   0
 swh/provenance/provenance.py                       |   6 +-
 swh/provenance/sql/30-schema.sql                   |  16 -
 swh/provenance/storage/__init__.py                 |  52 ++
 swh/provenance/storage/interface.py                | 231 +++++++
 .../provenance.py => storage/postgresql.py}        |   5 +-
 .../{api => storage/rabbitmq}/__init__.py          |   0
 swh/provenance/{api => storage/rabbitmq}/client.py |   8 +-
 .../{api => storage/rabbitmq}/serializers.py       |   0
 swh/provenance/{api => storage/rabbitmq}/server.py |  17 +-
 swh/provenance/swhgraph/__init__.py                |   0
 swh/provenance/tests/conftest.py                   | 110 ++--
 swh/provenance/tests/test_archive_interface.py     |   8 +-
 swh/provenance/tests/test_cli.py                   |   1 +
 swh/provenance/tests/test_conflict_resolution.py   |   6 +-
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_consistency.py           |   9 +-
 swh/provenance/tests/test_directory_flatten.py     |  10 +-
 swh/provenance/tests/test_directory_iterator.py    |   2 +-
 swh/provenance/tests/test_history_graph.py         |   3 +-
 swh/provenance/tests/test_init.py                  |   8 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_journal_client.py        |   9 +-
 swh/provenance/tests/test_origin_iterator.py       |   1 +
 swh/provenance/tests/test_origin_revision_layer.py |   6 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_storage.py    | 700 +++++++++++----------
 .../tests/test_provenance_storage_rabbitmq.py      |  43 ++
 ...st_provenance_storage_with_path_denormalized.py |  24 +
 .../tests/test_provenance_storage_without_path.py  |  24 +
 ...provenance_storage_without_path_denormalized.py |  24 +
 .../tests/test_revision_content_layer.py           |   5 +-
 swh/provenance/tests/test_revision_iterator.py     |   2 +-
 swh/provenance/tests/test_routing_keys.py          |   6 +-
 swh/provenance/tests/test_split_ranges.py          |   6 +-
 46 files changed, 969 insertions(+), 787 deletions(-)
 create mode 100644 swh/provenance/archive/__init__.py
 rename swh/provenance/{archive.py => archive/interface.py} (100%)
 rename swh/provenance/{multiplexer/archive.py => archive/multiplexer.py} (100%)
 rename swh/provenance/{postgresql/archive.py => archive/postgresql.py} (100%)
 rename swh/provenance/{storage/archive.py => archive/storage.py} (100%)
 rename swh/provenance/{swhgraph/archive.py => archive/swhgraph.py} (98%)
 delete mode 100644 swh/provenance/multiplexer/__init__.py
 delete mode 100644 swh/provenance/postgresql/__init__.py
 create mode 100644 swh/provenance/storage/interface.py
 rename swh/provenance/{postgresql/provenance.py => storage/postgresql.py} (99%)
 rename swh/provenance/{api => storage/rabbitmq}/__init__.py (100%)
 rename swh/provenance/{api => storage/rabbitmq}/client.py (99%)
 rename swh/provenance/{api => storage/rabbitmq}/serializers.py (100%)
 rename swh/provenance/{api => storage/rabbitmq}/server.py (98%)
 delete mode 100644 swh/provenance/swhgraph/__init__.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_rabbitmq.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_with_path_denormalized.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_without_path.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_without_path_denormalized.py
Changes applied before test
commit 9b081356d646d04ead03807111b73d943b265dcc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Sep 30 16:51:01 2022 +0200

    Reorganize the code
    
    - move everything (swh)archive related in a archive/ submodule
    - move everything provenance storage related in a storage/ submodule
      (which remains a not ideal name, may be confusing with the general
      'storage == swh-storage' acceptance in swh)
    - rename rabbitmq's backend from api/ to storage/rabbitmq
    - spit interface.py in 3 parts (one for each part, ProvenanceInterface,
      ProvenanceStorageInterface and ArchiveInterface).

commit 4f68403fc102adfbb93ff7ab1952e94a75afd116
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Sep 21 15:26:46 2022 +0200

    Mark origin layer tests as "origin_layer"
    
    so that one can easily run tests for the revision or origin
    layer only.

commit 4148bf1a6685028f725d589cac456aa2e5cd346f
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Sep 21 11:48:12 2022 +0200

    Adapt postgresql backend to swh.core.db >= 2.0

commit ffb7812a088598e3ddaeddc7f9625ce3e6e80846
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Sep 21 10:58:29 2022 +0200

    Upgrade to swh.graph 2.0

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/666/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/666/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 30 2022, 6:57 PM
Harbormaster failed remote builds in B31993: Diff 31023!

rebase and add a diff with more code reorg.

aka move ingestion related code for the 3 "layers" in an algos/ submodule.

Build has FAILED

Patch application report for D8593 (id=31037)

Could not rebase; Attempt merge onto a2bcc8eace...

Updating a2bcc8e..203c0cb
Fast-forward
 requirements-swh.txt                               |   4 +-
 swh/provenance/__init__.py                         | 100 +--
 swh/provenance/{ => algos}/directory.py            |   7 +-
 .../{graph.py => algos/isochrone_graph.py}         |  53 +-
 swh/provenance/{ => algos}/origin.py               |  56 +-
 swh/provenance/{ => algos}/revision.py             |  12 +-
 swh/provenance/archive/__init__.py                 |  62 ++
 .../{archive.py => archive/interface.py}           |   0
 .../archive.py => archive/multiplexer.py}          |   0
 .../archive.py => archive/postgresql.py}           |   0
 .../{storage/archive.py => archive/storage.py}     |   0
 .../{swhgraph/archive.py => archive/swhgraph.py}   |   2 +-
 swh/provenance/cli.py                              |  30 +-
 swh/provenance/interface.py                        | 220 +------
 swh/provenance/journal_client.py                   |   4 +-
 swh/provenance/multiplexer/__init__.py             |   0
 swh/provenance/postgresql/__init__.py              |   0
 swh/provenance/provenance.py                       |   6 +-
 swh/provenance/sql/30-schema.sql                   |  16 -
 swh/provenance/storage/__init__.py                 |  52 ++
 swh/provenance/storage/interface.py                | 231 +++++++
 .../provenance.py => storage/postgresql.py}        |   5 +-
 .../{api => storage/rabbitmq}/__init__.py          |   0
 swh/provenance/{api => storage/rabbitmq}/client.py |   8 +-
 .../{api => storage/rabbitmq}/serializers.py       |   0
 swh/provenance/{api => storage/rabbitmq}/server.py |  17 +-
 swh/provenance/swhgraph/__init__.py                |   0
 swh/provenance/tests/conftest.py                   | 110 ++--
 swh/provenance/tests/test_archive_interface.py     |   8 +-
 swh/provenance/tests/test_cli.py                   |   1 +
 swh/provenance/tests/test_conflict_resolution.py   |   6 +-
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_consistency.py           |  11 +-
 swh/provenance/tests/test_directory_flatten.py     |  12 +-
 swh/provenance/tests/test_directory_iterator.py    |   4 +-
 swh/provenance/tests/test_history_graph.py         |   6 +-
 swh/provenance/tests/test_init.py                  |   8 +-
 swh/provenance/tests/test_isochrone_graph.py       |  10 +-
 swh/provenance/tests/test_journal_client.py        |   9 +-
 swh/provenance/tests/test_origin_iterator.py       |   3 +-
 swh/provenance/tests/test_origin_revision_layer.py |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_storage.py    | 700 +++++++++++----------
 .../tests/test_provenance_storage_rabbitmq.py      |  43 ++
 ...st_provenance_storage_with_path_denormalized.py |  24 +
 .../tests/test_provenance_storage_without_path.py  |  24 +
 ...provenance_storage_without_path_denormalized.py |  24 +
 .../tests/test_revision_content_layer.py           |   9 +-
 swh/provenance/tests/test_revision_iterator.py     |   4 +-
 swh/provenance/tests/test_routing_keys.py          |   6 +-
 swh/provenance/tests/test_split_ranges.py          |   6 +-
 51 files changed, 1054 insertions(+), 873 deletions(-)
 rename swh/provenance/{ => algos}/directory.py (95%)
 rename swh/provenance/{graph.py => algos/isochrone_graph.py} (80%)
 rename swh/provenance/{ => algos}/origin.py (70%)
 rename swh/provenance/{ => algos}/revision.py (95%)
 create mode 100644 swh/provenance/archive/__init__.py
 rename swh/provenance/{archive.py => archive/interface.py} (100%)
 rename swh/provenance/{multiplexer/archive.py => archive/multiplexer.py} (100%)
 rename swh/provenance/{postgresql/archive.py => archive/postgresql.py} (100%)
 rename swh/provenance/{storage/archive.py => archive/storage.py} (100%)
 rename swh/provenance/{swhgraph/archive.py => archive/swhgraph.py} (98%)
 delete mode 100644 swh/provenance/multiplexer/__init__.py
 delete mode 100644 swh/provenance/postgresql/__init__.py
 create mode 100644 swh/provenance/storage/interface.py
 rename swh/provenance/{postgresql/provenance.py => storage/postgresql.py} (99%)
 rename swh/provenance/{api => storage/rabbitmq}/__init__.py (100%)
 rename swh/provenance/{api => storage/rabbitmq}/client.py (99%)
 rename swh/provenance/{api => storage/rabbitmq}/serializers.py (100%)
 rename swh/provenance/{api => storage/rabbitmq}/server.py (98%)
 delete mode 100644 swh/provenance/swhgraph/__init__.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_rabbitmq.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_with_path_denormalized.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_without_path.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_without_path_denormalized.py
Changes applied before test
commit 203c0cb92081ad9dd7975470bc633b4f9c800d52
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Sep 30 19:20:58 2022 +0200

    More core reorganization
    
    move ingestion related code for the 3 "layers" in an algos/ submodule.

commit 7c882f5716561851ff65182b5a0f9f68cf5bb0f6
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Sep 30 16:51:01 2022 +0200

    Reorganize the code
    
    - move everything (swh)archive related in a archive/ submodule
    - move everything provenance storage related in a storage/ submodule
      (which remains a not ideal name, may be confusing with the general
      'storage == swh-storage' acceptance in swh)
    - rename rabbitmq's backend from api/ to storage/rabbitmq
    - spit interface.py in 3 parts (one for each part, ProvenanceInterface,
      ProvenanceStorageInterface and ArchiveInterface).

commit 9a63bd8164afcf1d4f733736c60e6cc0012f65bb
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Sep 21 15:26:46 2022 +0200

    Mark origin layer tests as "origin_layer"
    
    so that one can easily run tests for the revision or origin
    layer only.

commit 67cd713b6aca7065e43efe68c66a3bae90fa99db
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Sep 21 11:48:12 2022 +0200

    Adapt postgresql backend to swh.core.db >= 2.0

commit bb0574c29f91471d39ab5f3d72959c6174c52738
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Sep 21 10:58:29 2022 +0200

    Upgrade to swh.graph 2.0

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/670/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/670/console

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 3 2022, 10:29 AM
Harbormaster failed remote builds in B32007: Diff 31037!

Build is green

Patch application report for D8593 (id=31038)

Could not rebase; Attempt merge onto a2bcc8eace...

Updating a2bcc8e..6f4a193
Fast-forward
 requirements-swh.txt                               |   4 +-
 swh/provenance/__init__.py                         | 100 +--
 swh/provenance/{api => algos}/__init__.py          |   0
 swh/provenance/{ => algos}/directory.py            |   7 +-
 .../{graph.py => algos/isochrone_graph.py}         |  53 +-
 swh/provenance/{ => algos}/origin.py               |  56 +-
 swh/provenance/{ => algos}/revision.py             |  12 +-
 swh/provenance/archive/__init__.py                 |  62 ++
 .../{archive.py => archive/interface.py}           |   0
 .../archive.py => archive/multiplexer.py}          |   0
 .../archive.py => archive/postgresql.py}           |   0
 .../{storage/archive.py => archive/storage.py}     |   0
 .../{swhgraph/archive.py => archive/swhgraph.py}   |   2 +-
 swh/provenance/cli.py                              |  30 +-
 swh/provenance/interface.py                        | 220 +------
 swh/provenance/journal_client.py                   |   4 +-
 swh/provenance/postgresql/__init__.py              |   0
 swh/provenance/provenance.py                       |   6 +-
 swh/provenance/sql/30-schema.sql                   |  16 -
 swh/provenance/storage/__init__.py                 |  52 ++
 swh/provenance/storage/interface.py                | 231 +++++++
 .../provenance.py => storage/postgresql.py}        |   5 +-
 .../{multiplexer => storage/rabbitmq}/__init__.py  |   0
 swh/provenance/{api => storage/rabbitmq}/client.py |   8 +-
 .../{api => storage/rabbitmq}/serializers.py       |   0
 swh/provenance/{api => storage/rabbitmq}/server.py |  17 +-
 swh/provenance/swhgraph/__init__.py                |   0
 swh/provenance/tests/conftest.py                   | 110 ++--
 swh/provenance/tests/test_archive_interface.py     |   8 +-
 swh/provenance/tests/test_cli.py                   |   1 +
 swh/provenance/tests/test_conflict_resolution.py   |   6 +-
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_consistency.py           |  11 +-
 swh/provenance/tests/test_directory_flatten.py     |  12 +-
 swh/provenance/tests/test_directory_iterator.py    |   4 +-
 swh/provenance/tests/test_history_graph.py         |   6 +-
 swh/provenance/tests/test_init.py                  |   8 +-
 swh/provenance/tests/test_isochrone_graph.py       |  10 +-
 swh/provenance/tests/test_journal_client.py        |   9 +-
 swh/provenance/tests/test_origin_iterator.py       |   3 +-
 swh/provenance/tests/test_origin_revision_layer.py |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_storage.py    | 700 +++++++++++----------
 .../tests/test_provenance_storage_rabbitmq.py      |  43 ++
 ...st_provenance_storage_with_path_denormalized.py |  24 +
 .../tests/test_provenance_storage_without_path.py  |  24 +
 ...provenance_storage_without_path_denormalized.py |  24 +
 .../tests/test_revision_content_layer.py           |   9 +-
 swh/provenance/tests/test_revision_iterator.py     |   4 +-
 swh/provenance/tests/test_routing_keys.py          |   6 +-
 swh/provenance/tests/test_split_ranges.py          |   6 +-
 51 files changed, 1054 insertions(+), 873 deletions(-)
 rename swh/provenance/{api => algos}/__init__.py (100%)
 rename swh/provenance/{ => algos}/directory.py (95%)
 rename swh/provenance/{graph.py => algos/isochrone_graph.py} (80%)
 rename swh/provenance/{ => algos}/origin.py (70%)
 rename swh/provenance/{ => algos}/revision.py (95%)
 create mode 100644 swh/provenance/archive/__init__.py
 rename swh/provenance/{archive.py => archive/interface.py} (100%)
 rename swh/provenance/{multiplexer/archive.py => archive/multiplexer.py} (100%)
 rename swh/provenance/{postgresql/archive.py => archive/postgresql.py} (100%)
 rename swh/provenance/{storage/archive.py => archive/storage.py} (100%)
 rename swh/provenance/{swhgraph/archive.py => archive/swhgraph.py} (98%)
 delete mode 100644 swh/provenance/postgresql/__init__.py
 create mode 100644 swh/provenance/storage/interface.py
 rename swh/provenance/{postgresql/provenance.py => storage/postgresql.py} (99%)
 rename swh/provenance/{multiplexer => storage/rabbitmq}/__init__.py (100%)
 rename swh/provenance/{api => storage/rabbitmq}/client.py (99%)
 rename swh/provenance/{api => storage/rabbitmq}/serializers.py (100%)
 rename swh/provenance/{api => storage/rabbitmq}/server.py (98%)
 delete mode 100644 swh/provenance/swhgraph/__init__.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_rabbitmq.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_with_path_denormalized.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_without_path.py
 create mode 100644 swh/provenance/tests/test_provenance_storage_without_path_denormalized.py
Changes applied before test
commit 6f4a193e9081e510d7e40d1854694dacdc901671
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Sep 30 19:20:58 2022 +0200

    More core reorganization
    
    move ingestion related code for the 3 "layers" in an algos/ submodule.

commit 7c882f5716561851ff65182b5a0f9f68cf5bb0f6
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Sep 30 16:51:01 2022 +0200

    Reorganize the code
    
    - move everything (swh)archive related in a archive/ submodule
    - move everything provenance storage related in a storage/ submodule
      (which remains a not ideal name, may be confusing with the general
      'storage == swh-storage' acceptance in swh)
    - rename rabbitmq's backend from api/ to storage/rabbitmq
    - spit interface.py in 3 parts (one for each part, ProvenanceInterface,
      ProvenanceStorageInterface and ArchiveInterface).

commit 9a63bd8164afcf1d4f733736c60e6cc0012f65bb
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Sep 21 15:26:46 2022 +0200

    Mark origin layer tests as "origin_layer"
    
    so that one can easily run tests for the revision or origin
    layer only.

commit 67cd713b6aca7065e43efe68c66a3bae90fa99db
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Sep 21 11:48:12 2022 +0200

    Adapt postgresql backend to swh.core.db >= 2.0

commit bb0574c29f91471d39ab5f3d72959c6174c52738
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Sep 21 10:58:29 2022 +0200

    Upgrade to swh.graph 2.0

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/672/ for more details.

olasd added a subscriber: olasd.

I think this makes sense, thanks!

This revision is now accepted and ready to land.Oct 3 2022, 11:17 AM
This revision was automatically updated to reflect the committed changes.