Page MenuHomeSoftware Heritage
Differential D8932

Replace RunAll with RunExportCompressUpload
ClosedPublic
Actions

Authored by vlorentz on Dec 7 2022, 12:36 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDGRPHcd69e48b5acc: Replace RunAll with RunExportCompressUpload
Summary

RunAll is not practical, because we need to mount the graph on a tmpfs
at some point. Let's keep that step manual for now, which means
RunExportCompressUpload and DeanonymizeOriginContributors will be
called separately.

Diff Detail

Event Timeline

vlorentz created this revision.Dec 7 2022, 12:36 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptDec 7 2022, 12:36 PM
swh-public-ci added a comment.Dec 7 2022, 12:40 PM

Build has FAILED

Patch application report for D8932 (id=32183)

Could not rebase; Attempt merge onto b8ddd6cead...

Updating b8ddd6c..c5061a7
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 +++++++-------
 swh/graph/luigi/misc_datasets.py            |  3 ++-
 swh/graph/luigi/origin_contributors.py      |  4 +++-
 swh/graph/luigi/utils.py                    | 10 ++++++++++
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 23 insertions(+), 10 deletions(-)
Changes applied before test
commit c5061a7076405090f38eefc87102030be2cc70cc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 16ec2214095dcd754d9c7e77ca6272caefb03d8d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 4f87b94a305a617ec3f8f66a7c3c6e2cef788442
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/324/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/324/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 7 2022, 12:40 PM
Harbormaster failed remote builds in B33136: Diff 32183!
vlorentz updated this revision to Diff 32186.Dec 7 2022, 12:50 PM

rebase

swh-public-ci added a comment.Dec 7 2022, 12:55 PM

Build is green

Patch application report for D8932 (id=32186)

Could not rebase; Attempt merge onto 66253a872d...

Updating 66253a8..31bd7b9
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 +++++++-------
 swh/graph/luigi/misc_datasets.py            |  3 ++-
 swh/graph/luigi/origin_contributors.py      |  4 +++-
 swh/graph/luigi/utils.py                    | 10 ++++++++++
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 23 insertions(+), 10 deletions(-)
Changes applied before test
commit 31bd7b91925871045083974869b033ef00b18526
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 634d0e4c56e219be20b6af112833df8d4945209a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 3f42f4fb0a4fd52291bc922de95dc09a86b4674d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/326/ for more details.

Harbormaster completed remote builds in B33140: Diff 32186.Dec 7 2022, 12:55 PM
vlorentz requested review of this revision.Dec 7 2022, 12:55 PM
ardumont accepted this revision.Dec 7 2022, 1:58 PM
This revision is now accepted and ready to land.Dec 7 2022, 1:58 PM
vlorentz updated this revision to Diff 32197.Dec 7 2022, 3:10 PM

rebase

vlorentz updated this revision to Diff 32200.Dec 7 2022, 3:18 PM

rebase

swh-public-ci added a comment.Dec 7 2022, 3:20 PM

Build is green

Patch application report for D8932 (id=32197)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..483b25a
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 +++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 61 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 4 files changed, 59 insertions(+), 19 deletions(-)
Changes applied before test
commit 483b25aad70d3ac8507871d28a139e55ff228f30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 2799da31ccb3e3165336cbd6e4c53a5df2f09bd3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/330/ for more details.

Harbormaster completed remote builds in B33151: Diff 32197.Dec 7 2022, 3:20 PM
vlorentz updated this revision to Diff 32204.Dec 7 2022, 3:24 PM

rebase

swh-public-ci added a comment.Dec 7 2022, 3:29 PM

Build is green

Patch application report for D8932 (id=32200)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..468c419
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 51 ++++++++++++++++++++++-------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 4 files changed, 49 insertions(+), 19 deletions(-)
Changes applied before test
commit 468c419ba0cf3f9a2053b1770c4954658047a5a9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 663a6976b789c004ed77082f052116115d891477
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/333/ for more details.

Harbormaster completed remote builds in B33154: Diff 32200.Dec 7 2022, 3:29 PM
swh-public-ci added a comment.Dec 7 2022, 3:36 PM

Build is green

Patch application report for D8932 (id=32204)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..cd69e48
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 52 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 4 files changed, 50 insertions(+), 19 deletions(-)
Changes applied before test
commit cd69e48b5acca41b5ea0a040372c4a953e511bd9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 233b0508395a1f8f6c3088db5b5dc938682620cb
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/336/ for more details.

Harbormaster completed remote builds in B33158: Diff 32204.Dec 7 2022, 3:36 PM
Closed by commit rDGRPHcd69e48b5acc: Replace RunAll with RunExportCompressUpload (authored by vlorentz). · Explain WhyDec 7 2022, 5:15 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDGRPHcd69e48b5acc: Replace RunAll with RunExportCompressUpload.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
graph/
luigi/
14 lines70%
Loading...

Diff 32200

View Options

swh/graph/luigi/__init__.py

Loading...
About · Developer information · Legal