Page MenuHomeSoftware Heritage

Replace RunAll with RunExportCompressUpload
ClosedPublic

Authored by vlorentz on Dec 7 2022, 12:36 PM.

Details

Summary

RunAll is not practical, because we need to mount the graph on a tmpfs
at some point. Let's keep that step manual for now, which means
RunExportCompressUpload and DeanonymizeOriginContributors will be
called separately.

Event Timeline

Build has FAILED

Patch application report for D8932 (id=32183)

Could not rebase; Attempt merge onto b8ddd6cead...

Updating b8ddd6c..c5061a7
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 +++++++-------
 swh/graph/luigi/misc_datasets.py            |  3 ++-
 swh/graph/luigi/origin_contributors.py      |  4 +++-
 swh/graph/luigi/utils.py                    | 10 ++++++++++
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 23 insertions(+), 10 deletions(-)
Changes applied before test
commit c5061a7076405090f38eefc87102030be2cc70cc
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 16ec2214095dcd754d9c7e77ca6272caefb03d8d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 4f87b94a305a617ec3f8f66a7c3c6e2cef788442
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/324/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/324/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 7 2022, 12:40 PM
Harbormaster failed remote builds in B33136: Diff 32183!

Build is green

Patch application report for D8932 (id=32186)

Could not rebase; Attempt merge onto 66253a872d...

Updating 66253a8..31bd7b9
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 +++++++-------
 swh/graph/luigi/misc_datasets.py            |  3 ++-
 swh/graph/luigi/origin_contributors.py      |  4 +++-
 swh/graph/luigi/utils.py                    | 10 ++++++++++
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 23 insertions(+), 10 deletions(-)
Changes applied before test
commit 31bd7b91925871045083974869b033ef00b18526
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 634d0e4c56e219be20b6af112833df8d4945209a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 3f42f4fb0a4fd52291bc922de95dc09a86b4674d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/326/ for more details.

This revision is now accepted and ready to land.Dec 7 2022, 1:58 PM

Build is green

Patch application report for D8932 (id=32197)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..483b25a
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 +++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 61 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 4 files changed, 59 insertions(+), 19 deletions(-)
Changes applied before test
commit 483b25aad70d3ac8507871d28a139e55ff228f30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 2799da31ccb3e3165336cbd6e4c53a5df2f09bd3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/330/ for more details.

Build is green

Patch application report for D8932 (id=32200)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..468c419
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 51 ++++++++++++++++++++++-------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 4 files changed, 49 insertions(+), 19 deletions(-)
Changes applied before test
commit 468c419ba0cf3f9a2053b1770c4954658047a5a9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 663a6976b789c004ed77082f052116115d891477
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/333/ for more details.

Build is green

Patch application report for D8932 (id=32204)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..cd69e48
Fast-forward
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 52 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 4 files changed, 50 insertions(+), 19 deletions(-)
Changes applied before test
commit cd69e48b5acca41b5ea0a040372c4a953e511bd9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 233b0508395a1f8f6c3088db5b5dc938682620cb
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/336/ for more details.