Page MenuHomeSoftware Heritage
Differential D4100

deposit: Transit raw metadata to the loader to unify with metadata update scenario
ClosedPublic
Actions

Authored by ardumont on Sep 30 2020, 2:06 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Maniphest Tasks
T2649: Modify deposit loader to send metadata from a content-deposit in raw format from to ERMDS
Commits
rDDEP2ecb65d4f534: Transit raw metadata to the loader to unify with metadata update scenario
Summary

The new update scenario now stores new metadata update to the metadata storage.
The loader does not, it currently stores the transformed xml (into json dict).
The loader passes by this deposit_read call to actually retrieve the data.

So prior to adapting the loader, the information returned by deposit_read need
to provide the raw metadata as well.

This also:

  • adds type to impacted methods along the way.
  • simplifies a bit the current deposit_read tests

In D4101, I refactor this deposit_read endpoint and add
some docs about it.

In D4105, I adapt the deposit loader to this new format change (so it sends
that raw information in the metadata storage instead of the current transformed
metadata)

Related to T2649

Test Plan

tox

Diff Detail

Repository
rDDEP Push deposit
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Sep 30 2020, 2:06 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 30 2020, 2:06 PM
swh-public-ci added a comment.Sep 30 2020, 2:08 PM

Build is green

Patch application report for D4100 (id=14449)

Rebasing onto 4d72d1be52...

Current branch diff-target is up to date.
Changes applied before test
commit d7e4ddc8e66744675ec0aaebc6614b5cc1b49e22
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Sep 30 14:06:21 2020 +0200

    deposit_read: Transit raw metadata to the loader
    
    Related to T2649

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/164/ for more details.

Harbormaster completed remote builds in B15727: Diff 14449.Sep 30 2020, 2:08 PM
ardumont edited the summary of this revision. (Show Details)Sep 30 2020, 2:19 PM
ardumont updated this revision to Diff 14451.Sep 30 2020, 2:30 PM
ardumont edited the summary of this revision. (Show Details)

Rework commit message to explicit the reasons.

ardumont retitled this revision from deposit_read: Transit raw metadata to the loader to deposit: Transit raw metadata to the loader to unify with metadata update scenario.Sep 30 2020, 2:31 PM
ardumont edited the summary of this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)Sep 30 2020, 2:33 PM
swh-public-ci added a comment.Sep 30 2020, 2:33 PM

Build is green

Patch application report for D4100 (id=14451)

Rebasing onto 4d72d1be52...

Current branch diff-target is up to date.
Changes applied before test
commit bc012e45ed8b5aa35afd7250c1d3c84204a20033
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Sep 30 14:06:21 2020 +0200

    Transit raw metadata to the loader to unify with metadata update scenario
    
    The new update scenario now stores new metadata update to the metadata storage.
    The loader does not, it currently stores the transformed xml (into json dict).
    The loader passes by this deposit_read call to actually retrieve the data.
    
    So prior to adapting the loader, this needs to happen.
    
    This also:
    - adds type to impacted methods along the way.
    - simplifies a bit the current deposit_read tests
    
    Related to T2649

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/165/ for more details.

Harbormaster completed remote builds in B15729: Diff 14451.Sep 30 2020, 2:33 PM
ardumont mentioned this in D4101: Simplify and document actual information returned to the loader.Sep 30 2020, 2:34 PM
anlambert accepted this revision.Sep 30 2020, 2:55 PM
anlambert added a subscriber: anlambert.

Looks good to me.

swh/deposit/api/private/__init__.py
44

s/1/one/

This revision is now accepted and ready to land.Sep 30 2020, 2:55 PM
ardumont updated this revision to Diff 14453.Sep 30 2020, 3:06 PM

Fix 1 into one

swh-public-ci added a comment.Sep 30 2020, 3:09 PM

Build is green

Patch application report for D4100 (id=14453)

Rebasing onto 4d72d1be52...

Current branch diff-target is up to date.
Changes applied before test
commit 05df4905ced239a205b5e7c8867354309330b31f
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Sep 30 14:06:21 2020 +0200

    Transit raw metadata to the loader to unify with metadata update scenario
    
    The new update scenario now stores new metadata update to the metadata storage.
    The loader does not, it currently stores the transformed xml (into json dict).
    The loader passes by this deposit_read call to actually retrieve the data.
    
    So prior to adapting the loader, this needs to happen.
    
    This also:
    - adds type to impacted methods along the way.
    - simplifies a bit the current deposit_read tests
    
    Related to T2649

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/167/ for more details.

Harbormaster completed remote builds in B15731: Diff 14453.Sep 30 2020, 3:09 PM
ardumont edited the summary of this revision. (Show Details)Sep 30 2020, 3:13 PM
ardumont mentioned this in D4105: deposit: Adapt loader to send extrinsic raw metadata to the metadata storage.Sep 30 2020, 5:59 PM
moranegg added a subscriber: moranegg.Oct 1 2020, 10:21 AM

I need a more clear distinction when data and metadata are used..
data and atom are not compatible things... atom is always metadata.
data should always refer to the software source code content.

swh/deposit/api/private/deposit_read.py
133–134

modify data to metadata entry.

data is the software content.

164

what's the data?

ardumont added inline comments.Oct 1 2020, 10:23 AM
swh/deposit/api/private/deposit_read.py
164

it's the dict response (transformed into json at some point) this api returns, so it's the response body.

vlorentz added a parent revision: D4101: Simplify and document actual information returned to the loader.Oct 1 2020, 11:15 AM
ardumont edited the summary of this revision. (Show Details)Oct 1 2020, 11:52 AM
ardumont updated this revision to Diff 14471.Oct 1 2020, 11:58 AM
ardumont edited the summary of this revision. (Show Details)
  • Rename raw_metadata into metadata_raw key
  • Add back the metadata (dict) under the key metadata_dict
  • Add some docstring to the deposit_read method (clarifying terms as asked by morane ;)
swh-public-ci added a comment.Oct 1 2020, 12:01 PM

Build is green

Patch application report for D4100 (id=14471)

Rebasing onto 4d72d1be52...

Current branch diff-target is up to date.
Changes applied before test
commit 2ecb65d4f534b2ee74d3bb55caa15ee111f8487b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Sep 30 14:06:21 2020 +0200

    Transit raw metadata to the loader to unify with metadata update scenario
    
    The new update scenario now stores new metadata update to the metadata storage.
    The loader does not, it currently stores the transformed xml (into json dict).
    The loader passes by this deposit_read call to actually retrieve the data.
    
    So prior to adapting the loader, the information returned by deposit_read need
    to provide the raw metadata as well.
    
    This also:
    - adds type to impacted methods along the way.
    - simplifies a bit the current deposit_read tests
    
    Related to T2649

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/169/ for more details.

Harbormaster completed remote builds in B15747: Diff 14471.Oct 1 2020, 12:01 PM
Closed by commit rDDEP2ecb65d4f534: Transit raw metadata to the loader to unify with metadata update scenario (authored by ardumont). · Explain WhyOct 1 2020, 2:42 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDDEP2ecb65d4f534: Transit raw metadata to the loader to unify with metadata update scenario.

Revision Contents
Changeset List

PathSize
swh/
deposit/
api/
private/
27 lines
11 lines
37 lines
tests/
api/
207 lines

Diff 14490

View Options

swh/deposit/api/private/__init__.py

Loading...
View Options

swh/deposit/api/private/deposit_check.py

Loading...
View Options

swh/deposit/api/private/deposit_read.py

Loading...
View Options

swh/deposit/tests/api/test_deposit_private_read_metadata.py

Loading...
About · Developer information · Legal