Page MenuHomeSoftware Heritage

package.loader: Split the warning message into multiple chunks
ClosedPublic

Authored by ardumont on Aug 12 2020, 11:07 AM.

Details

Summary

Otherwise, when the list is too large, it cascades to a logging error...

Just to raise awareness, I notice this when extracting the issues on staging
(Sentry most probably references the issue, my network is too flaky to go and
browse there).

Please, don't hesitate to take this over and improve over it or completely
discard it with something better ;)

Error excerpt from worker0.internal.staging.swh.network journalctl -xef -u swh-worker@loader_nixguix:

Aug 11 22:29:51 worker0 python3[3081241]:   File "/usr/lib/python3.7/logging/__init__.py", line 905, in handle
Aug 11 22:29:51 worker0 python3[3081241]:     self.emit(record)                                                                                                                                                                                Aug 11 22:29:51 worker0 python3[3081241]:   File "/usr/lib/python3/dist-packages/swh/core/logger.py", line 117, in emit
Aug 11 22:29:51 worker0 python3[3081241]:     self.handleError(record)
Aug 11 22:29:51 worker0 python3[3081241]: Message: '%d failed branches: %s'
Aug 11 22:29:51 worker0 python3[3081241]: Arguments: (1776, ['https://www.abisource.com/downloads/abiword/3.0.4/source/abiword-3.0.4.tar.gz', 'http://www.mushware.com/files/adanaxisgpl-1.2.5.tar.gz', 'http://ftp.belnet.be/ftp.gnome.org/sou
rces/adwaita-icon-theme/3.34/adwaita-icon-theme-3.34.3.tar.xz', 'ftp://ftp.fau.de/gentoo/distfiles/agg-2.5.tar.gz', 'http://ftp.belnet.be/ftp.gnome.org/sources/aisleriot/3.22/aisleriot-3.22.9.tar.xz', 'ftp://ftp.alsa-project.org/pub/lib/al
sa-lib-1.2.2.tar.bz2', 'ftp://ftp.alsa-project.org/pub/plugins/alsa-plugins-1.2.2.tar.bz2', 'ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.2.2.tar.bz2', 'http://ftp.belnet.be/ftp.gnome.org/sources/amtk/5.0/amtk-5.0.2.tar.xz', 'http://w
Test Plan

tox

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D3764 (id=13245)

Rebasing onto 381ea57ad7...

First, rewinding head to replay your work on top of it...
Applying: package.loader: Split the warning message into multiple chunks
Changes applied before test
commit b7317fb51eb28633c138715836ae109e6ef6e762
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 12 10:57:37 2020 +0200

    package.loader: Split the warning message into multiple chunks
    
    Otherwise, when the list is too large, it fails to log.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/277/ for more details.

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/loader/package/loader.py
367–368

500 is a lot...

This revision is now accepted and ready to land.Aug 12 2020, 11:32 AM
swh/loader/package/loader.py
367–368

yeah, but it seems to complain around 1000 ;)
whatdayasay? split in half 250?

swh/loader/package/loader.py
367–368

I don't know; 50 seems more reasonable, but whatever

Build is green

Patch application report for D3764 (id=13246)

Rebasing onto 381ea57ad7...

First, rewinding head to replay your work on top of it...
Applying: package.loader: Split the warning message into multiple chunks
Changes applied before test
commit 6e567a09ea27ad36a3e7647e731d6ba740996205
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 12 10:57:37 2020 +0200

    package.loader: Split the warning message into multiple chunks
    
    Otherwise, when the list is too large, it cascades to a logging error...

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/278/ for more details.

Chunk of 50 and adapt logging message prefix

Build is green

Patch application report for D3764 (id=13247)

Rebasing onto 381ea57ad7...

First, rewinding head to replay your work on top of it...
Applying: package.loader: Split the warning message into multiple chunks
Changes applied before test
commit 232c7b12ec975d28bc854b28f2396d52f5d681de
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 12 10:57:37 2020 +0200

    package.loader: Split the warning message into multiple chunks
    
    Otherwise, when the list is too large, it cascades to a logging error...

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/279/ for more details.

Build is green

Patch application report for D3764 (id=13248)

Rebasing onto 381ea57ad7...

Current branch diff-target is up to date.
Changes applied before test
commit 2423266fbb40b020435d5862fb11f6bf6dff269e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Aug 12 10:57:37 2020 +0200

    package.loader: Split the warning message into multiple chunks
    
    Otherwise, when the list is too large, it cascades to a logging error...

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/280/ for more details.