Page MenuHomeSoftware Heritage

Add some tracemalloc hooks
DraftPublic

Authored by olasd on Wed, Oct 6, 1:52 PM.
This is a draft revision that has not yet been submitted for review.

Details

Reviewers
None
Group Reviewers
Reviewers
Maniphest Tasks
T3625: Reduce git loader memory footprint
Summary

Use PYTHONTRACEMALLOC=25 as envvar to enable tracemalloc on start

And then:

export SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_git.yml
export PYTHONTRACEMALLOC=25
swh --log-level DEBUG loader run git <origin-url>

Don't forget the log-level flag otherwise, you won't see anything.

Related to T3625

Diff Detail

Repository
rDLDG Git loader
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 24249
Build 37849: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 37848: arc lint + arc unit

Unit TestsFailed

TimeTest
71 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack::test_load
self = <swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack object at 0x7f00621da278> swh_storage = <swh.storage.proxies.filter.FilteringProxyStorage object at 0x7f00621da710> datadir = '/var/lib/jenkins/workspace/DLDG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/git/tests/data'
68 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack::test_load_changed
self = <swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack object at 0x7f00620bcc50> swh_storage = <swh.storage.proxies.filter.FilteringProxyStorage object at 0x7f00620bc908> datadir = '/var/lib/jenkins/workspace/DLDG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/git/tests/data'
75 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack::test_load_dangling_symref
self = <swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack object at 0x7f00606b6c50> swh_storage = <swh.storage.proxies.filter.FilteringProxyStorage object at 0x7f00606b6a90> datadir = '/var/lib/jenkins/workspace/DLDG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/git/tests/data'
101 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack::test_load_despite_dulwich_exception[AttributeError]
self = <swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack object at 0x7f00624fd198> swh_storage = <swh.storage.proxies.filter.FilteringProxyStorage object at 0x7f0061788278> datadir = '/var/lib/jenkins/workspace/DLDG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/git/tests/data'
75 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack::test_load_despite_dulwich_exception[NotImplementedError]
self = <swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack object at 0x7f00605748d0> swh_storage = <swh.storage.proxies.filter.FilteringProxyStorage object at 0x7f0060574940> datadir = '/var/lib/jenkins/workspace/DLDG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/git/tests/data'
View Full Test Results (68 Failed · 39 Passed)

Event Timeline

Build was aborted

Patch application report for D6419 (id=23336)

Rebasing onto d3d6042142...

Current branch diff-target is up to date.
Changes applied before test
commit b5c75e7e86072f518eed745fefda8623cf583652
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Oct 6 13:52:17 2021 +0200

    Add some tracemalloc hooks

Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/142/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/142/console

(nothing to see but a typo)

swh/loader/git/loader.py
44

Drop some spurious logs at the same time.

175–177

as per irc suggestion because that takes time.