Page MenuHomeSoftware Heritage
Differential D8932

Replace RunAll with RunExportCompressUpload
ClosedPublic
Actions

Authored by vlorentz on Dec 7 2022, 12:36 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDGRPHcd69e48b5acc: Replace RunAll with RunExportCompressUpload
Summary

RunAll is not practical, because we need to mount the graph on a tmpfs
at some point. Let's keep that step manual for now, which means
RunExportCompressUpload and DeanonymizeOriginContributors will be
called separately.

Diff Detail

Repository
rDGRPH Compressed graph representation
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Dec 7 2022, 12:36 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptDec 7 2022, 12:36 PM
swh-public-ci added a comment.Dec 7 2022, 12:40 PM

Build has FAILED

Patch application report for D8932 (id=32183)

Could not rebase; Attempt merge onto b8ddd6cead...

Updating b8ddd6c..c5061a7
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 +++++++-------
 swh/graph/luigi/misc_datasets.py            |  3 ++-
 swh/graph/luigi/origin_contributors.py      |  4 +++-
 swh/graph/luigi/utils.py                    | 10 ++++++++++
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 23 insertions(+), 10 deletions(-)
Changes applied before test
commit c5061a7076405090f38eefc87102030be2cc70cc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 16ec2214095dcd754d9c7e77ca6272caefb03d8d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 4f87b94a305a617ec3f8f66a7c3c6e2cef788442
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/324/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/324/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 7 2022, 12:40 PM
Harbormaster failed remote builds in B33136: Diff 32183!
vlorentz updated this revision to Diff 32186.Dec 7 2022, 12:50 PM

rebase

swh-public-ci added a comment.Dec 7 2022, 12:55 PM

Build is green

Patch application report for D8932 (id=32186)

Could not rebase; Attempt merge onto 66253a872d...

Updating 66253a8..31bd7b9
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 +++++++-------
 swh/graph/luigi/misc_datasets.py            |  3 ++-
 swh/graph/luigi/origin_contributors.py      |  4 +++-
 swh/graph/luigi/utils.py                    | 10 ++++++++++
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 23 insertions(+), 10 deletions(-)
Changes applied before test
commit 31bd7b91925871045083974869b033ef00b18526
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 634d0e4c56e219be20b6af112833df8d4945209a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 3f42f4fb0a4fd52291bc922de95dc09a86b4674d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/326/ for more details.

Harbormaster completed remote builds in B33140: Diff 32186.Dec 7 2022, 12:55 PM
vlorentz requested review of this revision.Dec 7 2022, 12:55 PM
ardumont accepted this revision.Dec 7 2022, 1:58 PM
This revision is now accepted and ready to land.Dec 7 2022, 1:58 PM
vlorentz updated this revision to Diff 32197.Dec 7 2022, 3:10 PM

rebase

vlorentz updated this revision to Diff 32200.Dec 7 2022, 3:18 PM

rebase

swh-public-ci added a comment.Dec 7 2022, 3:20 PM

Build is green

Patch application report for D8932 (id=32197)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..483b25a
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 +++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 61 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 4 files changed, 59 insertions(+), 19 deletions(-)
Changes applied before test
commit 483b25aad70d3ac8507871d28a139e55ff228f30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 2799da31ccb3e3165336cbd6e4c53a5df2f09bd3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/330/ for more details.

Harbormaster completed remote builds in B33151: Diff 32197.Dec 7 2022, 3:20 PM
vlorentz updated this revision to Diff 32204.Dec 7 2022, 3:24 PM

rebase

swh-public-ci added a comment.Dec 7 2022, 3:29 PM

Build is green

Patch application report for D8932 (id=32200)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..468c419
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 51 ++++++++++++++++++++++-------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 4 files changed, 49 insertions(+), 19 deletions(-)
Changes applied before test
commit 468c419ba0cf3f9a2053b1770c4954658047a5a9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 663a6976b789c004ed77082f052116115d891477
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/333/ for more details.

Harbormaster completed remote builds in B33154: Diff 32200.Dec 7 2022, 3:29 PM
swh-public-ci added a comment.Dec 7 2022, 3:36 PM

Build is green

Patch application report for D8932 (id=32204)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..cd69e48
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 52 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 4 files changed, 50 insertions(+), 19 deletions(-)
Changes applied before test
commit cd69e48b5acca41b5ea0a040372c4a953e511bd9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 233b0508395a1f8f6c3088db5b5dc938682620cb
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/336/ for more details.

Harbormaster completed remote builds in B33158: Diff 32204.Dec 7 2022, 3:36 PM
Closed by commit rDGRPHcd69e48b5acc: Replace RunAll with RunExportCompressUpload (authored by vlorentz). · Explain WhyDec 7 2022, 5:15 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDGRPHcd69e48b5acc: Replace RunAll with RunExportCompressUpload.

Revision Contents
Changeset List

PathSize
swh/
graph/
luigi/
14 lines

Diff 32215

View Options

swh/graph/luigi/__init__.py

Loading...
About · Developer information · Legal