Differential D8935

Add dataset name to the export id
ClosedPublic
Actions

Authored by vlorentz on Dec 7 2022, 3:10 PM.

Details

Reviewers

douardda

Group Reviewers

Reviewers

Commits

rDGRPH94b1d2c14fe8: Add dataset name to the export id

Summary

It is easier to keep track of it in Grafana.

Diff Detail

Repository

rDGRPH Compressed graph representation

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Dec 7 2022, 3:10 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptDec 7 2022, 3:10 PM

rebase

Build is green

Patch application report for D8935 (id=32198)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..b02f4fc
Fast-forward
 swh/graph/cli.py                            |  2 +
 swh/graph/luigi/__init__.py                 | 14 +++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 61 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 61 insertions(+), 19 deletions(-)

Changes applied before test

commit b02f4fc42ed39f2e22964fe3751599647131fdc0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 13:01:37 2022 +0100

    Add dataset name to the export id
    
    It is easier to keep track of it in Grafana.

commit 483b25aad70d3ac8507871d28a139e55ff228f30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 2799da31ccb3e3165336cbd6e4c53a5df2f09bd3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/331/ for more details.

Harbormaster completed remote builds in B33152: Diff 32198.Dec 7 2022, 3:20 PM

rebase

Build is green

Patch application report for D8935 (id=32201)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..e108bfc
Fast-forward
 swh/graph/cli.py                            |  2 ++
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 51 ++++++++++++++++++++++-------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 51 insertions(+), 19 deletions(-)

Changes applied before test

commit e108bfc85513f7bbdecc60a320bd34fdcdb23a73
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 13:01:37 2022 +0100

    Add dataset name to the export id
    
    It is easier to keep track of it in Grafana.

commit 468c419ba0cf3f9a2053b1770c4954658047a5a9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 663a6976b789c004ed77082f052116115d891477
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/334/ for more details.

Harbormaster completed remote builds in B33155: Diff 32201.Dec 7 2022, 3:29 PM

Build has FAILED

Patch application report for D8935 (id=32205)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..94b1d2c
Fast-forward
 swh/graph/cli.py                            |  2 ++
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 52 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 52 insertions(+), 19 deletions(-)

Changes applied before test

commit 94b1d2c14fe80304e0dcac6547c4233211aa1de2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 13:01:37 2022 +0100

    Add dataset name to the export id
    
    It is easier to keep track of it in Grafana.

commit cd69e48b5acca41b5ea0a040372c4a953e511bd9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 233b0508395a1f8f6c3088db5b5dc938682620cb
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/337/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/337/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 7 2022, 3:36 PM

Harbormaster failed remote builds in B33159: Diff 32205!

Build is green

Patch application report for D8935 (id=32205)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..94b1d2c
Fast-forward
 swh/graph/cli.py                            |  2 ++
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 52 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 52 insertions(+), 19 deletions(-)

Changes applied before test

commit 94b1d2c14fe80304e0dcac6547c4233211aa1de2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 13:01:37 2022 +0100

    Add dataset name to the export id
    
    It is easier to keep track of it in Grafana.

commit cd69e48b5acca41b5ea0a040372c4a953e511bd9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 233b0508395a1f8f6c3088db5b5dc938682620cb
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/338/ for more details.

Harbormaster completed remote builds in B33159: Diff 32205.Dec 7 2022, 4:57 PM

vlorentz requested review of this revision.Dec 7 2022, 4:57 PM

douardda accepted this revision.Dec 8 2022, 11:26 AM

This revision is now accepted and ready to land.Dec 8 2022, 11:26 AM

Closed by commit rDGRPH94b1d2c14fe8: Add dataset name to the export id (authored by vlorentz). · Explain WhyDec 8 2022, 11:47 AM

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDGRPH94b1d2c14fe8: Add dataset name to the export id.