Page MenuHomeSoftware Heritage
Differential D8040

Limit the number of entries in the cache
ClosedPublic
Actions

Authored by vsellier on Jun 27 2022, 3:58 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T4313: [provenance] some process are oom killed
Commits
rDPROVf5f741366383: Limit the number of entries in the cache
Summary

The implementation test after each revision if there are
more than 100 000 entries of any kind in the cache. If yes,
it flush the content.
It's quite naive but the memory seems to stay around 5Go
max on big repositories like linux.

This can be still an issue with the current zeromq client because
the origin are sorted by repositories a lot of snapshots of big
repositories can be ingested in parallel.
With the journal implementation, it should be more distributed on
repositories of different sizes

Related to T4313

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vsellier created this revision.Jun 27 2022, 3:58 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 27 2022, 3:58 PM
vsellier updated this revision to Diff 28950.Jun 27 2022, 4:02 PM

update license headers

swh-public-ci added a comment.Jun 27 2022, 4:06 PM

Build is green

Patch application report for D8040 (id=28949)

Rebasing onto 80434e3b21...

Current branch diff-target is up to date.
Changes applied before test
commit 6bf00a395eca96eaa04c07a2168389a8d6ab85e6
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Mon Jun 27 15:48:47 2022 +0200

    Limit the number of entries in the cache
    
    The implementation test after each revision if there are
    more than 100 000 entries of any kind in the cache. If yes,
    it flush the content.
    It's quite naive but the memory seems to stay around 5Go
    max on big repositories like linux.
    
    This can be still an issue with the current zeromq client because
    the origin are sorted by repositories a lot of snapshots of big
    repositories can be ingested in parallel.
    With the journal implementation, it should be more distributed on
    repositories of different sizes
    
    Related to T4313

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/609/ for more details.

Harbormaster completed remote builds in B30039: Diff 28949.Jun 27 2022, 4:06 PM
swh-public-ci added a comment.Jun 27 2022, 4:10 PM

Build is green

Patch application report for D8040 (id=28950)

Rebasing onto 80434e3b21...

Current branch diff-target is up to date.
Changes applied before test
commit 571477be0498f2fbaf35fcca4b307447eeb430b5
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Mon Jun 27 15:48:47 2022 +0200

    Limit the number of entries in the cache
    
    The implementation test after each revision if there are
    more than 100 000 entries of any kind in the cache. If yes,
    it flush the content.
    It's quite naive but the memory seems to stay around 5Go
    max on big repositories like linux.
    
    This can be still an issue with the current zeromq client because
    the origin are sorted by repositories a lot of snapshots of big
    repositories can be ingested in parallel.
    With the journal implementation, it should be more distributed on
    repositories of different sizes
    
    Related to T4313

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/610/ for more details.

Harbormaster completed remote builds in B30040: Diff 28950.Jun 27 2022, 4:10 PM
vsellier requested review of this revision.Jun 27 2022, 4:10 PM
vlorentz added a subscriber: vlorentz.Jun 27 2022, 4:15 PM
vlorentz added inline comments.
swh/provenance/origin.py
107–111

saves a syscall when unnecessary

swh/provenance/provenance.py
101–113
116–128

would this work?

vlorentz added inline comments.Jun 27 2022, 4:16 PM
swh/provenance/origin.py
107–111

nvm, I misunderstood that the point was to time the function in the conditional

ardumont added a subscriber: ardumont.EditedJun 27 2022, 5:04 PM

lgtm, especially better if you attend to val's good suggestions ;)

A couple of docstring suggestions inline.

swh/provenance/provenance.py
124
ardumont added inline comments.Jun 27 2022, 5:07 PM
swh/provenance/interface.py
252–254
ardumont accepted this revision.Jun 27 2022, 5:10 PM
This revision is now accepted and ready to land.Jun 27 2022, 5:10 PM
vsellier updated this revision to Diff 28958.Jun 27 2022, 10:52 PM
vsellier marked 6 inline comments as done.

update according the reviews

  • simplify the cache management
  • fix the doc strings
swh/provenance/provenance.py
101–113

nice thanks

116–128

it works with a test on the data presence because of directory_flatten and revision_before_revision which are simple dict

swh-public-ci added a comment.Jun 27 2022, 10:54 PM

Build has FAILED

Patch application report for D8040 (id=28958)

Rebasing onto 80434e3b21...

Current branch diff-target is up to date.
Changes applied before test
commit b4f9226cffe69a211dbe0204541bd49e89daa8df
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Mon Jun 27 15:48:47 2022 +0200

    Limit the number of entries in the cache
    
    The implementation test after each revision if there are
    more than 100 000 entries of any kind in the cache. If yes,
    it flush the content.
    It's quite naive but the memory seems to stay around 5Go
    max on big repositories like linux.
    
    This can be still an issue with the current zeromq client because
    the origin are sorted by repositories a lot of snapshots of big
    repositories can be ingested in parallel.
    With the journal implementation, it should be more distributed on
    repositories of different sizes
    
    Related to T4313

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/611/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/611/console

Harbormaster failed remote builds in B30046: Diff 28958!Jun 27 2022, 10:54 PM
vsellier updated this revision to Diff 28959.Jun 27 2022, 10:59 PM

add missing parenthesis

swh-public-ci added a comment.Jun 27 2022, 11:01 PM

Build has FAILED

Patch application report for D8040 (id=28959)

Rebasing onto 80434e3b21...

Current branch diff-target is up to date.
Changes applied before test
commit 35c92963798c96475165aa0e132f5936be66e6f5
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Mon Jun 27 15:48:47 2022 +0200

    Limit the number of entries in the cache
    
    The implementation test after each revision if there are
    more than 100 000 entries of any kind in the cache. If yes,
    it flush the content.
    It's quite naive but the memory seems to stay around 5Go
    max on big repositories like linux.
    
    This can be still an issue with the current zeromq client because
    the origin are sorted by repositories a lot of snapshots of big
    repositories can be ingested in parallel.
    With the journal implementation, it should be more distributed on
    repositories of different sizes
    
    Related to T4313

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/612/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/612/console

Harbormaster failed remote builds in B30047: Diff 28959!Jun 27 2022, 11:01 PM
vsellier updated this revision to Diff 28962.Jun 28 2022, 9:38 AM

make mypy happy

swh-public-ci added a comment.Jun 28 2022, 9:46 AM

Build is green

Patch application report for D8040 (id=28962)

Rebasing onto 80434e3b21...

Current branch diff-target is up to date.
Changes applied before test
commit f5f741366383cff7e7f173a79f656e9c6e159602
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Mon Jun 27 15:48:47 2022 +0200

    Limit the number of entries in the cache
    
    The implementation test after each revision if there are
    more than 100 000 entries of any kind in the cache. If yes,
    it flush the content.
    It's quite naive but the memory seems to stay around 5Go
    max on big repositories like linux.
    
    This can be still an issue with the current zeromq client because
    the origin are sorted by repositories a lot of snapshots of big
    repositories can be ingested in parallel.
    With the journal implementation, it should be more distributed on
    repositories of different sizes
    
    Related to T4313

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/613/ for more details.

Harbormaster completed remote builds in B30050: Diff 28962.Jun 28 2022, 9:46 AM
Closed by commit rDPROVf5f741366383: Limit the number of entries in the cache (authored by vsellier). · Explain WhyJun 28 2022, 10:17 AM
This revision was automatically updated to reflect the committed changes.
vsellier added a commit: rDPROVf5f741366383: Limit the number of entries in the cache.

Revision Contents
Changeset List

PathSize
swh/
provenance/
9 lines
8 lines
24 lines

Diff 28964

View Options

swh/provenance/interface.py

Loading...
View Options

swh/provenance/origin.py

Loading...
View Options

swh/provenance/provenance.py

Loading...
About · Developer information · Legal