Page MenuHomeSoftware Heritage
Differential D8931

Prevent incorrect warning from being printed to output files
ClosedPublic
Actions

Authored by vlorentz on Dec 7 2022, 11:40 AM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Commits
rDGRPH233b0508395a: Prevent incorrect warning from being printed to output files
Summary

I cannot find how to make WebGraph log to stderr instead of stdout,
so it seems to be the only way.

Diff Detail

Unit TestsFailed
View All

TimeTest
109,719 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.graph.tests.test_cli::test_pipeline
def test_pipeline(): """run full compression pipeline""" # bare bone configuration, to allow testing the compression pipeline
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.graph.http_naive_client::swh.graph.http_naive_client.NaiveClient
57 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.graph.tests.test_cli::test_luigi[0]
6 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.graph.tests.test_cli::test_luigi[1]
61 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.graph.tests.test_grpc::test_leaves
View Full Test Results (1 Failed · 101 Passed · 9 Skipped)

Event Timeline

vlorentz created this revision.Dec 7 2022, 11:40 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptDec 7 2022, 11:40 AM
swh-public-ci added a comment.Dec 7 2022, 11:45 AM

Build has FAILED

Patch application report for D8931 (id=32180)

Could not rebase; Attempt merge onto b8ddd6cead...

Updating b8ddd6c..16ec221
Fast-forward
 swh/graph/luigi/misc_datasets.py            |  3 ++-
 swh/graph/luigi/origin_contributors.py      |  4 +++-
 swh/graph/luigi/utils.py                    | 10 ++++++++++
 swh/graph/tests/test_origin_contributors.py |  2 +-
 4 files changed, 16 insertions(+), 3 deletions(-)
Changes applied before test
commit 16ec2214095dcd754d9c7e77ca6272caefb03d8d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 4f87b94a305a617ec3f8f66a7c3c6e2cef788442
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/320/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/320/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 7 2022, 11:45 AM
Harbormaster failed remote builds in B33133: Diff 32180!
vlorentz updated this revision to Diff 32185.Dec 7 2022, 12:48 PM

rebase

swh-public-ci added a comment.Dec 7 2022, 12:54 PM

Build is green

Patch application report for D8931 (id=32185)

Could not rebase; Attempt merge onto 66253a872d...

Updating 66253a8..634d0e4
Fast-forward
 swh/graph/luigi/misc_datasets.py            |  3 ++-
 swh/graph/luigi/origin_contributors.py      |  4 +++-
 swh/graph/luigi/utils.py                    | 10 ++++++++++
 swh/graph/tests/test_origin_contributors.py |  2 +-
 4 files changed, 16 insertions(+), 3 deletions(-)
Changes applied before test
commit 634d0e4c56e219be20b6af112833df8d4945209a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 3f42f4fb0a4fd52291bc922de95dc09a86b4674d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

See https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/325/ for more details.

Harbormaster completed remote builds in B33139: Diff 32185.Dec 7 2022, 12:54 PM
vlorentz requested review of this revision.Dec 7 2022, 12:54 PM
olasd accepted this revision.Dec 7 2022, 1:37 PM
olasd added a subscriber: olasd.

Why not just touch all the files?

I believe that webgraph uses slf4j with the logback backend, so setting the logback.configurationFile property to point to a proper logback.xml config file might work?

This revision is now accepted and ready to land.Dec 7 2022, 1:37 PM
vlorentz added a comment.Dec 7 2022, 2:43 PM
In D8931#232231, @olasd wrote:

Why not just touch all the files?

Because it would not guarantee that mtimes for .obl files is greater or equal to mtimes .offset files

Aaaand I just realized it will fail when the compressed graph is not writeable (and it often won't), so this isn't an acceptable fix

I believe that webgraph uses slf4j with the logback backend, so setting the logback.configurationFile property to point to a proper logback.xml config file might work?

I'll look into this a bit more...

vlorentz planned changes to this revision.Dec 7 2022, 2:44 PM
vlorentz updated this revision to Diff 32196.Dec 7 2022, 3:09 PM

less awful fix

This revision is now accepted and ready to land.Dec 7 2022, 3:09 PM
vlorentz updated this revision to Diff 32199.Dec 7 2022, 3:18 PM

remove useless function

swh-public-ci added a comment.Dec 7 2022, 3:19 PM

Build has FAILED

Patch application report for D8931 (id=32196)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..2799da3
Fast-forward
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 61 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 3 files changed, 52 insertions(+), 12 deletions(-)
Changes applied before test
commit 2799da31ccb3e3165336cbd6e4c53a5df2f09bd3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/329/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/329/console

Harbormaster failed remote builds in B33150: Diff 32196!Dec 7 2022, 3:19 PM
vlorentz updated this revision to Diff 32203.Dec 7 2022, 3:24 PM

I'm tired

swh-public-ci added a comment.Dec 7 2022, 3:28 PM

Build has FAILED

Patch application report for D8931 (id=32199)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..663a697
Fast-forward
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 51 ++++++++++++++++++++++-------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 3 files changed, 42 insertions(+), 12 deletions(-)
Changes applied before test
commit 663a6976b789c004ed77082f052116115d891477
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/332/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/332/console

Harbormaster failed remote builds in B33153: Diff 32199!Dec 7 2022, 3:28 PM
swh-public-ci added a comment.Dec 7 2022, 3:35 PM

Build has FAILED

Patch application report for D8931 (id=32203)

Could not rebase; Attempt merge onto 100ccd1d1e...

Updating 100ccd1..233b050
Fast-forward
 swh/graph/luigi/origin_contributors.py      |  1 +
 swh/graph/luigi/utils.py                    | 52 +++++++++++++++++++++++------
 swh/graph/tests/test_origin_contributors.py |  2 +-
 3 files changed, 43 insertions(+), 12 deletions(-)
Changes applied before test
commit 233b0508395a1f8f6c3088db5b5dc938682620cb
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 11:40:23 2022 +0100

    Prevent incorrect warning from being printed to output files
    
    I cannot find how to make WebGraph log to stderr instead of stdout,
    so it seems to be the only way.

commit 042af3adf5b674d99733ba8738be82fcf41151a2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Dec 7 10:54:26 2022 +0100

    Fix crash when the sensitive dataset directory does not exist

Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/335/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tests-on-diff/335/console

Harbormaster failed remote builds in B33157: Diff 32203!Dec 7 2022, 3:35 PM
This revision was landed with ongoing or failed builds.Dec 7 2022, 5:15 PM
Closed by commit rDGRPH233b0508395a: Prevent incorrect warning from being printed to output files (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDGRPH233b0508395a: Prevent incorrect warning from being printed to output files.

Revision Contents
Changeset List

PathSize
swh/
graph/
luigi/
52 lines

Diff 32203

View Options

swh/graph/luigi/utils.py

Loading...
About · Developer information · Legal