Page MenuHomeSoftware Heritage
Differential D7228

deposit: Explicit deposit type, either 'meta' or 'code' deposit
ClosedPublic
Actions

Authored by ardumont on Feb 23 2022, 11:19 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T3677: Separate origin-source-code and provenance-metadata in the deposit
Commits
rDDEP136cbaba04b0: deposit: Explicit deposit type, either 'meta' or 'code' deposit
Summary

This should ease deposit listing in whatever forms (backend db read or client consuming
deposit listing).

Deposit types stand for:

  • meta: metadata-only deposit
  • code: content deposit

This commit includes a migration schema script which adds a new column 'type'. The
script is also in charge of migration existing data with the right type values..

Related to T3677
Depends on D7227

Test Plan

tox

Diff Detail

Repository
rDDEP Push deposit
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Feb 23 2022, 11:19 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 23 2022, 11:19 AM
swh-public-ci added a comment.Feb 23 2022, 11:23 AM

Build has FAILED

Patch application report for D7228 (id=26197)

Could not rebase; Attempt merge onto 94f9aa2228...

Updating 94f9aa22..d6aa2e03
Fast-forward
 swh/deposit/api/common.py                          |  3 +-
 swh/deposit/api/private/deposit_list.py            | 23 ++++-
 swh/deposit/api/utils.py                           |  1 +
 swh/deposit/cli/client.py                          | 12 ++-
 .../migrations/0022_deposit_metadata_only.py       | 18 ++++
 swh/deposit/models.py                              |  9 ++
 swh/deposit/tests/api/test_deposit_private_list.py | 98 ++++++++++++++--------
 swh/deposit/tests/cli/test_client.py               | 27 +++++-
 .../data/atom/entry-data-with-swhid-no-prov.xml    | 13 +++
 .../tests/data/atom/entry-data-with-swhid.xml      |  4 +
 10 files changed, 165 insertions(+), 43 deletions(-)
 create mode 100644 swh/deposit/migrations/0022_deposit_metadata_only.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid-no-prov.xml
Changes applied before test
commit d6aa2e03abf96fc2a9d85b17a0e39480785ee657
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Feb 23 11:14:57 2022 +0100

    deposit: Explicit when a deposit is metadata only or not
    
    This should ease deposit listing in whatever form (backend read or client consuming
    deposit listing).
    
    Related to T3677

commit 07dc3fbe81a1b5567ef0ae95fb71f4f1782b8bcd
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Feb 23 10:23:48 2022 +0100

    cli: Warn when metadata-only deposit without metadata provenance
    
    Related to T3677

commit 5a7d59cb2cc7ddb5bae9ac30f8a31beb78c8dd5d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Feb 22 18:41:45 2022 +0100

    deposit-list: Allow deposit listing with their raw metadata if any
    
    This now lists the deposit with their associated raw metadata if any is present. This
    will allow adaptations in the moderation view [1] to display the metadata provenance
    url (provided it's parsed out of the raw metadata).
    
    [1] The moderation view consumes this internal api.
    
    Related to T3677

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/738/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/738/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 23 2022, 11:23 AM
Harbormaster failed remote builds in B27070: Diff 26197!
ardumont updated this revision to Diff 26212.Feb 23 2022, 6:11 PM

Fix

ardumont edited the test plan for this revision. (Show Details)Feb 23 2022, 6:12 PM
swh-public-ci added a comment.Feb 23 2022, 6:20 PM

Build has FAILED

Patch application report for D7228 (id=26212)

Could not rebase; Attempt merge onto 5a7d59cb2c...

Updating 5a7d59cb..ce720ceb
Fast-forward
 swh/deposit/api/common.py                          |  5 +-
 swh/deposit/cli/client.py                          | 12 ++++-
 .../0021_deposit_origin_url_20201124_1438.py       |  2 +-
 swh/deposit/migrations/0022_auto_20220223_1542.py  | 59 ++++++++++++++++++++++
 swh/deposit/models.py                              | 11 ++++
 swh/deposit/tests/cli/test_client.py               | 27 +++++++++-
 swh/deposit/tests/conftest.py                      |  1 +
 .../data/atom/entry-data-with-swhid-no-prov.xml    | 13 +++++
 .../tests/data/atom/entry-data-with-swhid.xml      |  4 ++
 swh/deposit/tests_migration/test_migrations.py     | 22 +++++++-
 10 files changed, 148 insertions(+), 8 deletions(-)
 create mode 100644 swh/deposit/migrations/0022_auto_20220223_1542.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid-no-prov.xml
Changes applied before test
commit ce720ceb0def34a8f034ee3393fb85ab46eba616
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Feb 23 11:14:57 2022 +0100

    deposit: Explicit deposit type, either 'meta' or 'code' deposit
    
    This should ease deposit listing in whatever forms (backend db read or client consuming
    deposit listing).
    
    Deposit types stand for:
    - meta: metadata-only deposit
    - code: content deposit
    
    Related to T3677

commit f9be97546ef95ec14ec2558b97853f9da67e8b31
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Feb 23 10:23:48 2022 +0100

    cli: Warn when metadata-only deposit without metadata provenance
    
    Related to T3677

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/742/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/742/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 23 2022, 6:20 PM
Harbormaster failed remote builds in B27085: Diff 26212!
ardumont updated this revision to Diff 26213.Feb 23 2022, 6:37 PM
  • Actually test the 'data' migration as well (only the schema was tested so far)
  • improve the listing to explicit the deposit 'type' is transmitted along
swh-public-ci added a comment.Feb 23 2022, 6:46 PM

Build has FAILED

Patch application report for D7228 (id=26213)

Could not rebase; Attempt merge onto 5a7d59cb2c...

Updating 5a7d59cb..4ed5d225
Fast-forward
 swh/deposit/api/common.py                          |  5 +-
 swh/deposit/cli/client.py                          | 12 ++-
 .../0021_deposit_origin_url_20201124_1438.py       |  2 +-
 swh/deposit/migrations/0022_auto_20220223_1542.py  | 58 +++++++++++++++
 swh/deposit/models.py                              | 15 +++-
 swh/deposit/tests/api/test_deposit_private_list.py |  7 +-
 swh/deposit/tests/cli/test_client.py               | 27 ++++++-
 swh/deposit/tests/conftest.py                      |  1 +
 .../data/atom/entry-data-with-swhid-no-prov.xml    | 13 ++++
 .../tests/data/atom/entry-data-with-swhid.xml      |  4 +
 swh/deposit/tests_migration/test_migrations.py     | 86 +++++++++++++++++++++-
 11 files changed, 220 insertions(+), 10 deletions(-)
 create mode 100644 swh/deposit/migrations/0022_auto_20220223_1542.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid-no-prov.xml
Changes applied before test
commit 4ed5d2257f09ded29260a61bf39c10b3f99e25c2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Feb 23 11:14:57 2022 +0100

    deposit: Explicit deposit type, either 'meta' or 'code' deposit
    
    This should ease deposit listing in whatever forms (backend db read or client consuming
    deposit listing).
    
    Deposit types stand for:
    - meta: metadata-only deposit
    - code: content deposit
    
    This commit includes a migration schema script which adds a new column 'type'. The
    script is also in charge of migration existing data with the right type values..
    
    Related to T3677

commit f9be97546ef95ec14ec2558b97853f9da67e8b31
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Feb 23 10:23:48 2022 +0100

    cli: Warn when metadata-only deposit without metadata provenance
    
    Related to T3677

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/743/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/743/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 23 2022, 6:46 PM
Harbormaster failed remote builds in B27086: Diff 26213!
ardumont retitled this revision from deposit: Explicit when a deposit is metadata only or not to deposit: Explicit deposit type, either 'meta' or 'code' deposit.Feb 23 2022, 6:54 PM
ardumont edited the summary of this revision. (Show Details)
ardumont mentioned this in D7233: Replace xmltodict with ElementTree to handle SWORD documents.Feb 24 2022, 9:44 AM
ardumont edited the summary of this revision. (Show Details)Feb 24 2022, 9:48 AM
ardumont added a parent revision: D7227: cli: Warn when metadata-only deposit without metadata provenance.
ardumont updated this revision to Diff 26228.Feb 24 2022, 10:05 AM

Rebase

swh-public-ci added a comment.Feb 24 2022, 10:06 AM

Build has FAILED

Patch application report for D7228 (id=26228)

Could not rebase; Attempt merge onto cf92b5f2fe...

Updating cf92b5f2..136cbaba
Fast-forward
 swh/deposit/api/common.py                          |  5 +-
 swh/deposit/cli/client.py                          | 12 ++-
 .../0021_deposit_origin_url_20201124_1438.py       |  2 +-
 swh/deposit/migrations/0022_auto_20220223_1542.py  | 58 +++++++++++++++
 swh/deposit/models.py                              | 15 +++-
 swh/deposit/tests/api/test_deposit_private_list.py |  7 +-
 swh/deposit/tests/cli/test_client.py               | 27 ++++++-
 swh/deposit/tests/conftest.py                      |  1 +
 .../data/atom/entry-data-with-swhid-no-prov.xml    | 13 ++++
 .../tests/data/atom/entry-data-with-swhid.xml      |  4 +
 swh/deposit/tests_migration/test_migrations.py     | 86 +++++++++++++++++++++-
 11 files changed, 220 insertions(+), 10 deletions(-)
 create mode 100644 swh/deposit/migrations/0022_auto_20220223_1542.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid-no-prov.xml
Changes applied before test
commit 136cbaba04b070d6cb62f0adcc3da5c316d17cbc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Feb 23 11:14:57 2022 +0100

    deposit: Explicit deposit type, either 'meta' or 'code' deposit
    
    This should ease deposit listing in whatever forms (backend db read or client consuming
    deposit listing).
    
    Deposit types stand for:
    - meta: metadata-only deposit
    - code: content deposit
    
    This commit includes a migration schema script which adds a new column 'type'. The
    script is also in charge of migration existing data with the right type values..
    
    Related to T3677

commit 163df3eefdac9b4b3d5ec33e3a11486fad02f682
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Feb 23 10:23:48 2022 +0100

    cli: Warn when metadata-only deposit without metadata provenance
    
    Related to T3677

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/752/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/752/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 24 2022, 10:06 AM
Harbormaster failed remote builds in B27101: Diff 26228!
swh-public-ci added a comment.Feb 24 2022, 10:17 AM

Build is green

Patch application report for D7228 (id=26228)

Could not rebase; Attempt merge onto cf92b5f2fe...

Updating cf92b5f2..136cbaba
Fast-forward
 swh/deposit/api/common.py                          |  5 +-
 swh/deposit/cli/client.py                          | 12 ++-
 .../0021_deposit_origin_url_20201124_1438.py       |  2 +-
 swh/deposit/migrations/0022_auto_20220223_1542.py  | 58 +++++++++++++++
 swh/deposit/models.py                              | 15 +++-
 swh/deposit/tests/api/test_deposit_private_list.py |  7 +-
 swh/deposit/tests/cli/test_client.py               | 27 ++++++-
 swh/deposit/tests/conftest.py                      |  1 +
 .../data/atom/entry-data-with-swhid-no-prov.xml    | 13 ++++
 .../tests/data/atom/entry-data-with-swhid.xml      |  4 +
 swh/deposit/tests_migration/test_migrations.py     | 86 +++++++++++++++++++++-
 11 files changed, 220 insertions(+), 10 deletions(-)
 create mode 100644 swh/deposit/migrations/0022_auto_20220223_1542.py
 create mode 100644 swh/deposit/tests/data/atom/entry-data-with-swhid-no-prov.xml
Changes applied before test
commit 136cbaba04b070d6cb62f0adcc3da5c316d17cbc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Feb 23 11:14:57 2022 +0100

    deposit: Explicit deposit type, either 'meta' or 'code' deposit
    
    This should ease deposit listing in whatever forms (backend db read or client consuming
    deposit listing).
    
    Deposit types stand for:
    - meta: metadata-only deposit
    - code: content deposit
    
    This commit includes a migration schema script which adds a new column 'type'. The
    script is also in charge of migration existing data with the right type values..
    
    Related to T3677

commit 163df3eefdac9b4b3d5ec33e3a11486fad02f682
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Feb 23 10:23:48 2022 +0100

    cli: Warn when metadata-only deposit without metadata provenance
    
    Related to T3677

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/753/ for more details.

Harbormaster completed remote builds in B27101: Diff 26228.Feb 24 2022, 10:17 AM
ardumont requested review of this revision.Feb 24 2022, 10:17 AM
vlorentz accepted this revision.Feb 24 2022, 10:20 AM
This revision is now accepted and ready to land.Feb 24 2022, 10:20 AM
ardumont edited the test plan for this revision. (Show Details)Feb 24 2022, 10:22 AM
Closed by commit rDDEP136cbaba04b0: deposit: Explicit deposit type, either 'meta' or 'code' deposit (authored by ardumont). · Explain WhyFeb 24 2022, 11:13 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDDEP136cbaba04b0: deposit: Explicit deposit type, either 'meta' or 'code' deposit.

Revision Contents
Changeset List

PathSize
swh/
deposit/
api/
5 lines
migrations/
2 lines
58 lines
15 lines
tests/
api/
7 lines
1 line
tests_migration/
86 lines

Diff 26237

View Options

swh/deposit/api/common.py

Loading...
View Options

swh/deposit/migrations/0021_deposit_origin_url_20201124_1438.py

Loading...
View Options

swh/deposit/migrations/0022_auto_20220223_1542.py

Loading...
View Options

swh/deposit/models.py

Loading...
View Options

swh/deposit/tests/api/test_deposit_private_list.py

Loading...
View Options

swh/deposit/tests/conftest.py

Loading...
View Options

swh/deposit/tests_migration/test_migrations.py

Loading...
About · Developer information · Legal