Differential D6419

Add some tracemalloc hooks
AbandonedPublicDraft
Actions

Authored by olasd on Oct 6 2021, 1:52 PM.

Details

Reviewers

None

Group Reviewers

Reviewers

Maniphest Tasks

T3625: Reduce git loader memory footprint

Summary

Use PYTHONTRACEMALLOC=25 as envvar to enable tracemalloc on start

And then:

export SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_git.yml
export PYTHONTRACEMALLOC=25
swh --log-level DEBUG loader run git <origin-url>

Don't forget the log-level flag otherwise, you won't see anything.

Related to T3625

Diff Detail

Repository

rDLDG Git loader

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 24249
Build 37849: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 37848: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	71 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack::test_load self = <swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack object at 0x7f00621da278> swh_storage = <swh.storage.proxies.filter.FilteringProxyStorage object at 0x7f00621da710> datadir = '/var/lib/jenkins/workspace/DLDG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/git/tests/data'
	68 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack::test_load_changed self = <swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack object at 0x7f00620bcc50> swh_storage = <swh.storage.proxies.filter.FilteringProxyStorage object at 0x7f00620bc908> datadir = '/var/lib/jenkins/workspace/DLDG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/git/tests/data'
	75 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack::test_load_dangling_symref self = <swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack object at 0x7f00606b6c50> swh_storage = <swh.storage.proxies.filter.FilteringProxyStorage object at 0x7f00606b6a90> datadir = '/var/lib/jenkins/workspace/DLDG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/git/tests/data'
	101 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack::test_load_despite_dulwich_exception[AttributeError] self = <swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack object at 0x7f00624fd198> swh_storage = <swh.storage.proxies.filter.FilteringProxyStorage object at 0x7f0061788278> datadir = '/var/lib/jenkins/workspace/DLDG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/git/tests/data'
	75 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack::test_load_despite_dulwich_exception[NotImplementedError] self = <swh.loader.git.tests.test_loader.TestDumbGitLoaderWithPack object at 0x7f00605748d0> swh_storage = <swh.storage.proxies.filter.FilteringProxyStorage object at 0x7f0060574940> datadir = '/var/lib/jenkins/workspace/DLDG/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/loader/git/tests/data'
		View Full Test Results (68 Failed · 39 Passed)

Event Timeline

olasd created this revision.Oct 6 2021, 1:52 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptOct 6 2021, 1:52 PM

Build was aborted

Patch application report for D6419 (id=23336)

Rebasing onto d3d6042142...

Current branch diff-target is up to date.

Changes applied before test

commit b5c75e7e86072f518eed745fefda8623cf583652
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Oct 6 13:52:17 2021 +0200

    Add some tracemalloc hooks

Link to build: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/142/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDG/job/tests-on-diff/142/console

Harbormaster failed remote builds in B24249: Diff 23336!Oct 6 2021, 2:13 PM

ardumont edited the summary of this revision. (Show Details)Oct 6 2021, 2:19 PM

ardumont added a task: T3625: Reduce git loader memory footprint.

ardumont edited the summary of this revision. (Show Details)Oct 6 2021, 2:24 PM