Page MenuHomeSoftware Heritage
Differential D3764

package.loader: Split the warning message into multiple chunks
ClosedPublic
Actions

Authored by ardumont on Aug 12 2020, 11:07 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDLDBASE2423266fbb40: package.loader: Split the warning message into multiple chunks
Summary

Otherwise, when the list is too large, it cascades to a logging error...

Just to raise awareness, I notice this when extracting the issues on staging
(Sentry most probably references the issue, my network is too flaky to go and
browse there).

Please, don't hesitate to take this over and improve over it or completely
discard it with something better ;)

Error excerpt from worker0.internal.staging.swh.network journalctl -xef -u swh-worker@loader_nixguix:

Aug 11 22:29:51 worker0 python3[3081241]:   File "/usr/lib/python3.7/logging/__init__.py", line 905, in handle
Aug 11 22:29:51 worker0 python3[3081241]:     self.emit(record)                                                                                                                                                                                Aug 11 22:29:51 worker0 python3[3081241]:   File "/usr/lib/python3/dist-packages/swh/core/logger.py", line 117, in emit
Aug 11 22:29:51 worker0 python3[3081241]:     self.handleError(record)
Aug 11 22:29:51 worker0 python3[3081241]: Message: '%d failed branches: %s'
Aug 11 22:29:51 worker0 python3[3081241]: Arguments: (1776, ['https://www.abisource.com/downloads/abiword/3.0.4/source/abiword-3.0.4.tar.gz', 'http://www.mushware.com/files/adanaxisgpl-1.2.5.tar.gz', 'http://ftp.belnet.be/ftp.gnome.org/sou
rces/adwaita-icon-theme/3.34/adwaita-icon-theme-3.34.3.tar.xz', 'ftp://ftp.fau.de/gentoo/distfiles/agg-2.5.tar.gz', 'http://ftp.belnet.be/ftp.gnome.org/sources/aisleriot/3.22/aisleriot-3.22.9.tar.xz', 'ftp://ftp.alsa-project.org/pub/lib/al
sa-lib-1.2.2.tar.bz2', 'ftp://ftp.alsa-project.org/pub/plugins/alsa-plugins-1.2.2.tar.bz2', 'ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.2.2.tar.bz2', 'http://ftp.belnet.be/ftp.gnome.org/sources/amtk/5.0/amtk-5.0.2.tar.xz', 'http://w
Test Plan

tox

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Aug 12 2020, 11:07 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptAug 12 2020, 11:07 AM
swh-public-ci added a comment.Aug 12 2020, 11:09 AM

Build is green

Patch application report for D3764 (id=13245)

Rebasing onto 381ea57ad7...

First, rewinding head to replay your work on top of it...
Applying: package.loader: Split the warning message into multiple chunks
Changes applied before test
commit b7317fb51eb28633c138715836ae109e6ef6e762
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 12 10:57:37 2020 +0200

    package.loader: Split the warning message into multiple chunks
    
    Otherwise, when the list is too large, it fails to log.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/277/ for more details.

Harbormaster completed remote builds in B14508: Diff 13245.Aug 12 2020, 11:09 AM
ardumont edited the summary of this revision. (Show Details)Aug 12 2020, 11:09 AM
vlorentz accepted this revision.Aug 12 2020, 11:32 AM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/loader/package/loader.py
367–368

500 is a lot...

This revision is now accepted and ready to land.Aug 12 2020, 11:32 AM
ardumont added inline comments.Aug 12 2020, 11:34 AM
swh/loader/package/loader.py
367–368

yeah, but it seems to complain around 1000 ;)
whatdayasay? split in half 250?

vlorentz added inline comments.Aug 12 2020, 11:35 AM
swh/loader/package/loader.py
367–368

I don't know; 50 seems more reasonable, but whatever

ardumont updated this revision to Diff 13246.Aug 12 2020, 11:39 AM

chunks of 50

swh-public-ci added a comment.Aug 12 2020, 11:40 AM

Build is green

Patch application report for D3764 (id=13246)

Rebasing onto 381ea57ad7...

First, rewinding head to replay your work on top of it...
Applying: package.loader: Split the warning message into multiple chunks
Changes applied before test
commit 6e567a09ea27ad36a3e7647e731d6ba740996205
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 12 10:57:37 2020 +0200

    package.loader: Split the warning message into multiple chunks
    
    Otherwise, when the list is too large, it cascades to a logging error...

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/278/ for more details.

Harbormaster completed remote builds in B14509: Diff 13246.Aug 12 2020, 11:40 AM
ardumont updated this revision to Diff 13247.Aug 12 2020, 11:48 AM

Chunk of 50 and adapt logging message prefix

swh-public-ci added a comment.Aug 12 2020, 11:49 AM

Build is green

Patch application report for D3764 (id=13247)

Rebasing onto 381ea57ad7...

First, rewinding head to replay your work on top of it...
Applying: package.loader: Split the warning message into multiple chunks
Changes applied before test
commit 232c7b12ec975d28bc854b28f2396d52f5d681de
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 12 10:57:37 2020 +0200

    package.loader: Split the warning message into multiple chunks
    
    Otherwise, when the list is too large, it cascades to a logging error...

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/279/ for more details.

Harbormaster completed remote builds in B14510: Diff 13247.Aug 12 2020, 11:49 AM
ardumont updated this revision to Diff 13248.Aug 12 2020, 12:07 PM

Rebase

swh-public-ci added a comment.Aug 12 2020, 12:09 PM

Build is green

Patch application report for D3764 (id=13248)

Rebasing onto 381ea57ad7...

Current branch diff-target is up to date.
Changes applied before test
commit 2423266fbb40b020435d5862fb11f6bf6dff269e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 12 10:57:37 2020 +0200

    package.loader: Split the warning message into multiple chunks
    
    Otherwise, when the list is too large, it cascades to a logging error...

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/280/ for more details.

Harbormaster completed remote builds in B14511: Diff 13248.Aug 12 2020, 12:09 PM
Closed by commit rDLDBASE2423266fbb40: package.loader: Split the warning message into multiple chunks (authored by ardumont). · Explain WhyAug 12 2020, 12:09 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDLDBASE2423266fbb40: package.loader: Split the warning message into multiple chunks.

Revision Contents
Changeset List

PathSize
swh/
loader/
package/
10 lines

Diff 13249

View Options

swh/loader/package/loader.py

Loading...
About · Developer information · Legal