Page MenuHomeSoftware Heritage

luigi: Remove copies of stamp files to/from S3
ClosedPublic

Authored by vlorentz on Nov 21 2022, 3:42 PM.

Details

Summary

They are only useful while exporting the dataset -- after the export is
finished, meta.json is good enough and stamp files only save a couple
of minutes when only some objects types are needed (ie. never in practice)

Diff Detail

Repository
rDDATASET Datasets
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32881
Build 51530: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 51529: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D8864 (id=31952)

Rebasing onto 23853dbfac...

Current branch diff-target is up to date.
Changes applied before test
commit ab4f22ef1494ad7496b57540f4123a72490555da
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Nov 21 15:42:33 2022 +0100

    luigi: Remove copies of stamp files to/from S3
    
    They are only useful while exporting the dataset -- after the export is
    finished, meta.json is good enough and stamp files only save a couple
    of minutes when only some objects types are needed (ie. never in practice)

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/164/ for more details.

Build is green

Patch application report for D8864 (id=31953)

Rebasing onto 23853dbfac...

Current branch diff-target is up to date.
Changes applied before test
commit b39436e38be5fefe16c92d6553845cd113bafd14
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Nov 21 15:42:33 2022 +0100

    luigi: Remove copies of stamp files to/from S3
    
    They are only useful while exporting the dataset -- after the export is
    finished, meta.json is good enough and stamp files only save a couple
    of minutes when only some objects types are needed (ie. never in practice)

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/165/ for more details.

This revision is now accepted and ready to land.Nov 22 2022, 5:05 PM