Page MenuHomeSoftware Heritage

pypi: write metadata on revisions instead of snapshots.
ClosedPublic

Authored by vlorentz on Oct 13 2020, 11:04 AM.

Details

Summary

Writing them on snapshot allowed us to write the raw metadata from the API,
but it causes a lot of duplication; after running for only a couple of months,
the metadata storage is already 700GB in size, mostly because of NPM
metadata, but also because of these (eg. many over 1MB each).

The metadata we wrote on snapshots was made of:

  • intrinsic metadata that PyPI extracted from the last upload
  • info on each file (sdist or otherwise)

The former we don't need to archive like this (as they are intrinsic),
and we keep loading the latter but only for source files and discard
extrinsic metadata for binary files, as they are not useful.

Closes T2667

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D4242 (id=14987)

Rebasing onto fe59ce84e5...

Current branch diff-target is up to date.
Changes applied before test
commit 0c766379f216c3c6174d6e749921f9effac235cf
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Oct 13 11:01:58 2020 +0200

    pypi: write metadata on revisions instead of snapshots.
    
    Writing them on snapshot allowed us to write the raw metadata from the API,
    but it causes a lot of duplication; after running for only a couple of months,
    the metadata storage is already 700GB in size, mostly because of NPM
    metadata, but also because of these (eg. many over 1MB each).
    
    The metadata we wrote on snapshots was made of:
    
    * intrinsic metadata that PyPI extracted from the last upload
    * info on each file (sdist or otherwise)
    
    The former we don't need to archive like this (as they are intrinsic),
    and we keep loading the latter but only for source files and discard
    extrinsic metadata for binary files, as they are not useful.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/327/ for more details.

ardumont added inline comments.
swh/loader/package/pypi/tests/test_pypi.py
355

why do you use the back and forth json conversion here?

lgtm

swh/loader/package/pypi/tests/test_pypi.py
355

nvm, got it.
_0805 is text representing json..., you want a subset of that part, ...

This revision is now accepted and ready to land.Oct 13 2020, 11:36 AM