Page MenuHomeSoftware Heritage
Differential D4569

FUSE: cache: add 'date' column in metadata_cache for history/by-date
ClosedPublic
Actions

Authored by haltode on Nov 24 2020, 10:38 AM.

Details

Reviewers
seirl
Group Reviewers
Reviewers
Maniphest Tasks
T2803: FUSE history/by-{date,hash} views need an index to improve performances
Commits
rDFUSE1c778776d8bb: tests: extend max delay to populate history/by-date/
rDFUSE65e468780210: cache: add sqlite index in all tables
rDFUSE0a24b804f6f1: cache: add 'date' column in metadata_cache for history/by-date
Summary

Closes T2803.

Diff Detail

Repository
rDFUSE FUSE virtual file system
Branch
feature/add-by-date-cache
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17557
Build 27139: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 27138: arc lint + arc unit

Event Timeline

haltode created this revision.Nov 24 2020, 10:38 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 24 2020, 10:38 AM
swh-public-ci added a comment.Nov 24 2020, 10:41 AM

Build has FAILED

Patch application report for D4569 (id=16226)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/173/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/173/console

Harbormaster failed remote builds in B17425: Diff 16226!Nov 24 2020, 10:41 AM
haltode updated this revision to Diff 16229.Nov 24 2020, 11:18 AM

Make tox pass?

swh-public-ci added a comment.Nov 24 2020, 11:21 AM

Build has FAILED

Patch application report for D4569 (id=16229)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit 26533a8073283bcbbdb1a67b616deb50e2ce2f8b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/174/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/174/console

Harbormaster failed remote builds in B17428: Diff 16229!Nov 24 2020, 11:21 AM
haltode updated this revision to Diff 16230.Nov 24 2020, 11:34 AM

Extend delay again

swh-public-ci added a comment.Nov 24 2020, 11:37 AM

Build has FAILED

Patch application report for D4569 (id=16230)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit 5faa31d0905203a6cf69029b6af74cc1f4d64516
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/175/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/175/console

Harbormaster failed remote builds in B17429: Diff 16230!Nov 24 2020, 11:37 AM
haltode updated this revision to Diff 16235.Nov 24 2020, 11:54 AM

Try without fixed delay

swh-public-ci added a comment.Nov 24 2020, 12:02 PM

Build was aborted

Patch application report for D4569 (id=16235)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit bb4e5352816c6fa64069977b377bdad241d336b8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/176/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/176/console

Harbormaster failed remote builds in B17432: Diff 16235!Nov 24 2020, 12:02 PM
swh-public-ci added a comment.Nov 24 2020, 1:59 PM

Build was aborted

Patch application report for D4569 (id=16235)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit bb4e5352816c6fa64069977b377bdad241d336b8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/177/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/177/console

swh-public-ci added a comment.Nov 24 2020, 3:00 PM

Build is green

Patch application report for D4569 (id=16235)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit bb4e5352816c6fa64069977b377bdad241d336b8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/178/ for more details.

swh-public-ci added a comment.Nov 24 2020, 3:03 PM

Build is green

Patch application report for D4569 (id=16235)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit bb4e5352816c6fa64069977b377bdad241d336b8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/179/ for more details.

haltode added a reviewer: seirl.Nov 24 2020, 3:37 PM
haltode updated this revision to Diff 16327.Nov 26 2020, 9:51 AM
  • Rebasing on master
  • Date is now a new column in metadata_cache instead of new table
swh-public-ci added a comment.Nov 26 2020, 9:54 AM

Build has FAILED

Patch application report for D4569 (id=16327)

Rebasing onto b0c3d77e7d...

Current branch diff-target is up to date.
Changes applied before test
commit 3a5be8f8be0cbfd840acdcb9928c5b1048619cc5
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Nov 26 09:50:15 2020 +0100

    WIP: add 'date' as a new column instead of new table

commit e94782dfb6573897e6949c829dc1fc69958f0c95
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit ba98ab3b5cfb6ad40fcf8b7c7a131aa1f1592606
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit 141a3d46670af03ecd4ffd7ad80389d603a72201
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/189/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/189/console

Harbormaster failed remote builds in B17520: Diff 16327!Nov 26 2020, 9:54 AM
haltode retitled this revision from WIP: add date_cache sqlite table to speed up by-date/ to WIP: add date column in metadata cache to speed up by-date/.Nov 26 2020, 10:29 AM
haltode edited the summary of this revision. (Show Details)
haltode updated this revision to Diff 16365.Nov 27 2020, 9:31 AM

Rebasing on master

swh-public-ci added a comment.Nov 27 2020, 9:34 AM

Build is green

Patch application report for D4569 (id=16365)

Rebasing onto c73312c854...

Current branch diff-target is up to date.
Changes applied before test
commit 9efa111803292be87803e34ae7f303d8fc2ebd9c
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Nov 26 09:50:15 2020 +0100

    WIP: add 'date' as a new column instead of new table

commit 0539233244b1b84930d8c1de4a75226d1dcc4b2a
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit de158045b3ff67483efd8f2ca8105247f3c5d40b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit b920a31ba6a4aaf30c96584c8791f8c4d4810def
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/192/ for more details.

Harbormaster completed remote builds in B17557: Diff 16365.Nov 27 2020, 9:34 AM
seirl accepted this revision.Nov 27 2020, 2:16 PM
This revision is now accepted and ready to land.Nov 27 2020, 2:16 PM
haltode updated this revision to Diff 16400.Nov 27 2020, 2:34 PM

Rework commits history

haltode edited the summary of this revision. (Show Details)Nov 27 2020, 2:35 PM
swh-public-ci added a comment.Nov 27 2020, 2:37 PM

Build is green

Patch application report for D4569 (id=16400)

Rebasing onto 6ab2e73633...

Current branch diff-target is up to date.
Changes applied before test
commit 1c778776d8bbc9da9836d2b66950e88019ef2e47
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    tests: extend max delay to populate history/by-date/

commit 65e468780210ed749e2d991bc64bc6a4e3f4384b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    cache: add sqlite index in all tables

commit 0a24b804f6f1fd8431fe5f605c4d1bcb293a0b5e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    cache: add 'date' column in metadata_cache for history/by-date
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new column has been created to store a relation (rev swhid, date)
    meaning we only need to do one query to get all history with a specific
    date prefix.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/195/ for more details.

Harbormaster completed remote builds in B17592: Diff 16400.Nov 27 2020, 2:37 PM
haltode retitled this revision from WIP: add date column in metadata cache to speed up by-date/ to FUSE: cache: add 'date' column in metadata_cache for history/by-date.Nov 27 2020, 2:38 PM
Closed by commit rDFUSE0a24b804f6f1: cache: add 'date' column in metadata_cache for history/by-date (authored by haltode). · Explain WhyNov 27 2020, 2:49 PM
This revision was automatically updated to reflect the committed changes.
haltode added a commit: rDFUSE0a24b804f6f1: cache: add 'date' column in metadata_cache for history/by-date.
haltode added a commit: rDFUSE65e468780210: cache: add sqlite index in all tables.
haltode added a commit: rDFUSE1c778776d8bb: tests: extend max delay to populate history/by-date/.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
fuse/
106 lines89%
Loading...
3 lines61%
Loading...
fs/
16 lines96%
Loading...
tests/
6 lines92%
Loading...
4 lines98%
Loading...

Diff 16365

View Options

swh/fuse/cache.py

Loading...
View Options

swh/fuse/cli.py

Loading...
View Options

swh/fuse/fs/artifact.py

Loading...
View Options

swh/fuse/tests/conftest.py

Loading...
View Options

swh/fuse/tests/test_revision.py

Loading...
About · Developer information · Legal