Page MenuHomeSoftware Heritage

luigi: Remove copies of stamp files to/from S3
ClosedPublic

Authored by vlorentz on Nov 21 2022, 3:42 PM.

Details

Summary

They are only useful while exporting the dataset -- after the export is
finished, meta.json is good enough and stamp files only save a couple
of minutes when only some objects types are needed (ie. never in practice)

Diff Detail

Repository
rDDATASET Datasets
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D8864 (id=31952)

Rebasing onto 23853dbfac...

Current branch diff-target is up to date.
Changes applied before test
commit ab4f22ef1494ad7496b57540f4123a72490555da
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Nov 21 15:42:33 2022 +0100

    luigi: Remove copies of stamp files to/from S3
    
    They are only useful while exporting the dataset -- after the export is
    finished, meta.json is good enough and stamp files only save a couple
    of minutes when only some objects types are needed (ie. never in practice)

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/164/ for more details.

Build is green

Patch application report for D8864 (id=31953)

Rebasing onto 23853dbfac...

Current branch diff-target is up to date.
Changes applied before test
commit b39436e38be5fefe16c92d6553845cd113bafd14
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Nov 21 15:42:33 2022 +0100

    luigi: Remove copies of stamp files to/from S3
    
    They are only useful while exporting the dataset -- after the export is
    finished, meta.json is good enough and stamp files only save a couple
    of minutes when only some objects types are needed (ie. never in practice)

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/165/ for more details.

This revision is now accepted and ready to land.Nov 22 2022, 5:05 PM