Page MenuHomeSoftware Heritage
Differential D5829

Refactor the cache handling in ProvenanceDB
ClosedPublic
Actions

Authored by douardda on Jun 9 2021, 10:39 AM.

Details

Reviewers
aeviso
Commits
rDPROVac8dc036bcc6: Refactor the cache handling in ProvenanceDB
Summary
  • use TypedDict structures to properly type the caches needed by the ProvenanceDB objects,
  • use only one cache plus a set of added (and eventually removed) ids of objects (within the cache) for revisisons, contents and directories.

Note: I am far from being satisfied by this cache handling, but I think it's better than before. I would prefer a more generic, lru-type, cache handling in which we do not scratch the (read) cache at each flush in the DB (aka "commit") and where relation caches (xxx_in_yyy) are also used in reading operations when possible.

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Jun 9 2021, 10:39 AM
swh-public-ci added a comment.Jun 9 2021, 10:40 AM

Build has FAILED

Patch application report for D5829 (id=20851)

Rebasing onto 4cd50e66bb...

Current branch diff-target is up to date.
Changes applied before test
commit d4e065ef534b1dcd72c6a72f067e75568e90018a
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 9 10:27:32 2021 +0200

    Refactor the cache handling in ProvenanceDB
    
    - use TypedDict structures to properly type the caches needed by the
      ProvenanceDB objects,
    - use only one cache plus a set of added (and eventually removed) ids of
      objects (within the cache) for revisisons, contents and directories.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/96/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/96/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 9 2021, 10:40 AM
Harbormaster failed remote builds in B21813: Diff 20851!
douardda updated this revision to Diff 20863.Jun 9 2021, 11:34 AM

rebase

swh-public-ci added a comment.Jun 9 2021, 11:35 AM

Build has FAILED

Patch application report for D5829 (id=20863)

Rebasing onto 6cdd424eba...

Current branch diff-target is up to date.
Changes applied before test
commit a025c7dc1c486159b8e38202da882dec7ecfcf69
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 9 10:27:32 2021 +0200

    Refactor the cache handling in ProvenanceDB
    
    - use TypedDict structures to properly type the caches needed by the
      ProvenanceDB objects,
    - use only one cache plus a set of added (and eventually removed) ids of
      objects (within the cache) for revisisons, contents and directories.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/98/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/98/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 9 2021, 11:35 AM
Harbormaster failed remote builds in B21825: Diff 20863!
douardda updated this revision to Diff 20867.Jun 9 2021, 11:48 AM

make it work for py 3.7

swh-public-ci added a comment.Jun 9 2021, 11:50 AM

Build is green

Patch application report for D5829 (id=20867)

Rebasing onto 6cdd424eba...

Current branch diff-target is up to date.
Changes applied before test
commit 98686698372951da6e5b325bae166ae8c7287f75
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 9 10:27:32 2021 +0200

    Refactor the cache handling in ProvenanceDB
    
    - use TypedDict structures to properly type the caches needed by the
      ProvenanceDB objects,
    - use only one cache plus a set of added (and eventually removed) ids of
      objects (within the cache) for revisisons, contents and directories.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/100/ for more details.

Harbormaster completed remote builds in B21829: Diff 20867.Jun 9 2021, 11:50 AM
douardda requested review of this revision.Jun 9 2021, 11:50 AM
douardda updated this revision to Diff 20868.Jun 9 2021, 11:53 AM

remove forgotten debugging stuff

swh-public-ci added a comment.Jun 9 2021, 11:57 AM

Build is green

Patch application report for D5829 (id=20868)

Rebasing onto 6cdd424eba...

Current branch diff-target is up to date.
Changes applied before test
commit e23832b21ad4ee7afcb56f98147e51f633b6c2d7
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 9 10:27:32 2021 +0200

    Refactor the cache handling in ProvenanceDB
    
    - use TypedDict structures to properly type the caches needed by the
      ProvenanceDB objects,
    - use only one cache plus a set of added (and eventually removed) ids of
      objects (within the cache) for revisisons, contents and directories.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/101/ for more details.

Harbormaster completed remote builds in B21830: Diff 20868.Jun 9 2021, 11:57 AM
douardda added a child revision: D5841: Remove the without-path flavor of ProvenanceDB.Jun 10 2021, 10:35 AM
douardda updated this revision to Diff 20921.Jun 10 2021, 2:52 PM

rebase

swh-public-ci added a comment.Jun 10 2021, 2:54 PM

Build is green

Patch application report for D5829 (id=20921)

Rebasing onto 075b0d6cd6...

Current branch diff-target is up to date.
Changes applied before test
commit 8aff35d251db39537a3a4bd14f98783dc06ebdc9
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 9 10:27:32 2021 +0200

    Refactor the cache handling in ProvenanceDB
    
    - use TypedDict structures to properly type the caches needed by the
      ProvenanceDB objects,
    - use only one cache plus a set of added (and eventually removed) ids of
      objects (within the cache) for revisisons, contents and directories.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/113/ for more details.

Harbormaster completed remote builds in B21903: Diff 20921.Jun 10 2021, 2:54 PM
douardda mentioned this in D5848: First stage of refactoring for the Provenance backend.Jun 10 2021, 4:11 PM
douardda updated this revision to Diff 21021.Jun 15 2021, 12:09 PM

rebase on current master

swh-public-ci added a comment.Jun 15 2021, 12:11 PM

Build is green

Patch application report for D5829 (id=21021)

Rebasing onto 206399eb8a...

Current branch diff-target is up to date.
Changes applied before test
commit 29673033e93e423ccd39cca46731c57faa22b02a
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 9 10:27:32 2021 +0200

    Refactor the cache handling in ProvenanceDB
    
    - use TypedDict structures to properly type the caches needed by the
      ProvenanceDB objects,
    - use only one (sha1, date) cache per entity, plus a set of added ids of
      objects (within the cache) (i.e. for revisisons, contents and directories).

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/132/ for more details.

Harbormaster completed remote builds in B21999: Diff 21021.Jun 15 2021, 12:11 PM
douardda added a child revision: D5842: Refactor the provenanceDB.insert_relation().Jun 15 2021, 12:12 PM
douardda edited the summary of this revision. (Show Details)Jun 15 2021, 12:17 PM
douardda updated this revision to Diff 21034.Jun 15 2021, 3:08 PM
douardda edited the summary of this revision. (Show Details)

rebase

swh-public-ci added a comment.Jun 15 2021, 3:10 PM

Build is green

Patch application report for D5829 (id=21034)

Rebasing onto 8c536b8d50...

Current branch diff-target is up to date.
Changes applied before test
commit ac8dc036bcc608ba39d65929569f13ad694ebd90
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Jun 9 10:27:32 2021 +0200

    Refactor the cache handling in ProvenanceDB
    
    - use TypedDict structures to properly type the caches needed by the
      ProvenanceDB objects,
    - use only one (sha1, date) cache per entity, plus a set of added ids of
      objects (within the cache) (i.e. for revisisons, contents and directories).

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/136/ for more details.

Harbormaster completed remote builds in B22027: Diff 21034.Jun 15 2021, 3:10 PM
aeviso accepted this revision.Jun 15 2021, 6:28 PM
This revision is now accepted and ready to land.Jun 15 2021, 6:28 PM
Closed by commit rDPROVac8dc036bcc6: Refactor the cache handling in ProvenanceDB (authored by douardda). · Explain WhyJun 16 2021, 9:06 AM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDPROVac8dc036bcc6: Refactor the cache handling in ProvenanceDB.

Revision Contents
Changeset List

PathSize
swh/
provenance/
postgresql/
12 lines
106 lines

Diff 21054

View Options

swh/provenance/postgresql/provenancedb_base.py

Loading...
View Options

swh/provenance/provenance.py

Loading...
About · Developer information · Legal