Page MenuHomeSoftware Heritage
Differential D8935

Add dataset name to the export id
ClosedPublic
Actions

Authored by vlorentz on Dec 7 2022, 3:10 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Commits
rDGRPH94b1d2c14fe8: Add dataset name to the export id
Summary

It is easier to keep track of it in Grafana.

Diff Detail

Event Timeline

vlorentz created this revision.Dec 7 2022, 3:10 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptDec 7 2022, 3:10 PM
vlorentz updated this revision to Diff 32201.Dec 7 2022, 3:18 PM

rebase

swh-public-ci added a comment.Dec 7 2022, 3:20 PM

Build is green

Patch application report for D8935 (id=32198)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..b02f4fc
Fast-forward
 swh/graph/cli.py                            |  2 +
 swh/graph/luigi/__init__.py                 | 14 +++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 61 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 61 insertions(+), 19 deletions(-)
Changes applied before test
commit b02f4fc42ed39f2e22964fe3751599647131fdc0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 13:01:37 2022 +0100

    Add dataset name to the export id
    
    It is easier to keep track of it in Grafana.

commit 483b25aad70d3ac8507871d28a139e55ff228f30
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 2799da31ccb3e3165336cbd6e4c53a5df2f09bd3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/331/ for more details.

Harbormaster completed remote builds in B33152: Diff 32198.Dec 7 2022, 3:20 PM
vlorentz updated this revision to Diff 32205.Dec 7 2022, 3:24 PM

rebase

swh-public-ci added a comment.Dec 7 2022, 3:29 PM

Build is green

Patch application report for D8935 (id=32201)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..e108bfc
Fast-forward
 swh/graph/cli.py                            |  2 ++
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 51 ++++++++++++++++++++++-------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 51 insertions(+), 19 deletions(-)
Changes applied before test
commit e108bfc85513f7bbdecc60a320bd34fdcdb23a73
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 13:01:37 2022 +0100

    Add dataset name to the export id
    
    It is easier to keep track of it in Grafana.

commit 468c419ba0cf3f9a2053b1770c4954658047a5a9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 663a6976b789c004ed77082f052116115d891477
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/334/ for more details.

Harbormaster completed remote builds in B33155: Diff 32201.Dec 7 2022, 3:29 PM
swh-public-ci added a comment.Dec 7 2022, 3:36 PM

Build has FAILED

Patch application report for D8935 (id=32205)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..94b1d2c
Fast-forward
 swh/graph/cli.py                            |  2 ++
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 52 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 52 insertions(+), 19 deletions(-)
Changes applied before test
commit 94b1d2c14fe80304e0dcac6547c4233211aa1de2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 13:01:37 2022 +0100

    Add dataset name to the export id
    
    It is easier to keep track of it in Grafana.

commit cd69e48b5acca41b5ea0a040372c4a953e511bd9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 233b0508395a1f8f6c3088db5b5dc938682620cb
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/337/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/337/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 7 2022, 3:36 PM
Harbormaster failed remote builds in B33159: Diff 32205!
swh-public-ci added a comment.Dec 7 2022, 4:57 PM

Build is green

Patch application report for D8935 (id=32205)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..94b1d2c
Fast-forward
 swh/graph/cli.py                            |  2 ++
 swh/graph/luigi/__init__.py                 | 14 ++++----
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 52 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 5 files changed, 52 insertions(+), 19 deletions(-)
Changes applied before test
commit 94b1d2c14fe80304e0dcac6547c4233211aa1de2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 13:01:37 2022 +0100

    Add dataset name to the export id
    
    It is easier to keep track of it in Grafana.

commit cd69e48b5acca41b5ea0a040372c4a953e511bd9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 12:35:35 2022 +0100

    Replace RunAll with RunExportCompressUpload
    
    RunAll is not practical, because we need to mount the graph on a tmpfs
    at some point. Let's keep that step manual for now, which means
    RunExportCompressUpload and DeanonymizeOriginContributors will be
    called separately.

commit 233b0508395a1f8f6c3088db5b5dc938682620cb
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/338/ for more details.

Harbormaster completed remote builds in B33159: Diff 32205.Dec 7 2022, 4:57 PM
vlorentz requested review of this revision.Dec 7 2022, 4:57 PM
douardda accepted this revision.Dec 8 2022, 11:26 AM
This revision is now accepted and ready to land.Dec 8 2022, 11:26 AM
Closed by commit rDGRPH94b1d2c14fe8: Add dataset name to the export id (authored by vlorentz). · Explain WhyDec 8 2022, 11:47 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDGRPH94b1d2c14fe8: Add dataset name to the export id.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
graph/
2 lines59%
Loading...

Diff 32205

View Options

swh/graph/cli.py

Loading...
About · Developer information · Legal