Page MenuHomeSoftware Heritage

FUSE: cache: add 'date' column in metadata_cache for history/by-date
ClosedPublic

Authored by haltode on Nov 24 2020, 10:38 AM.

Diff Detail

Repository
rDFUSE FUSE virtual file system
Branch
feature/add-by-date-cache
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17557
Build 27139: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 27138: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D4569 (id=16226)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/173/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/173/console

Build has FAILED

Patch application report for D4569 (id=16229)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit 26533a8073283bcbbdb1a67b616deb50e2ce2f8b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/174/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/174/console

Build has FAILED

Patch application report for D4569 (id=16230)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit 5faa31d0905203a6cf69029b6af74cc1f4d64516
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/175/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/175/console

Build was aborted

Patch application report for D4569 (id=16235)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit bb4e5352816c6fa64069977b377bdad241d336b8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/176/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/176/console

Build was aborted

Patch application report for D4569 (id=16235)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit bb4e5352816c6fa64069977b377bdad241d336b8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/177/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/177/console

Build is green

Patch application report for D4569 (id=16235)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit bb4e5352816c6fa64069977b377bdad241d336b8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/178/ for more details.

Build is green

Patch application report for D4569 (id=16235)

Rebasing onto 0fe1497da4...

Current branch diff-target is up to date.
Changes applied before test
commit bb4e5352816c6fa64069977b377bdad241d336b8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit 6ba34f1acb8a2785ead4b859ac0868db832763da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit eb49564889e14d8ac47500f522c1bb9abb051b67
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/179/ for more details.

  • Rebasing on master
  • Date is now a new column in metadata_cache instead of new table

Build has FAILED

Patch application report for D4569 (id=16327)

Rebasing onto b0c3d77e7d...

Current branch diff-target is up to date.
Changes applied before test
commit 3a5be8f8be0cbfd840acdcb9928c5b1048619cc5
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Nov 26 09:50:15 2020 +0100

    WIP: add 'date' as a new column instead of new table

commit e94782dfb6573897e6949c829dc1fc69958f0c95
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit ba98ab3b5cfb6ad40fcf8b7c7a131aa1f1592606
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit 141a3d46670af03ecd4ffd7ad80389d603a72201
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/189/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/189/console

haltode retitled this revision from WIP: add date_cache sqlite table to speed up by-date/ to WIP: add date column in metadata cache to speed up by-date/.Nov 26 2020, 10:29 AM
haltode edited the summary of this revision. (Show Details)

Build is green

Patch application report for D4569 (id=16365)

Rebasing onto c73312c854...

Current branch diff-target is up to date.
Changes applied before test
commit 9efa111803292be87803e34ae7f303d8fc2ebd9c
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Nov 26 09:50:15 2020 +0100

    WIP: add 'date' as a new column instead of new table

commit 0539233244b1b84930d8c1de4a75226d1dcc4b2a
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    WIP: tests: extend delay

commit de158045b3ff67483efd8f2ca8105247f3c5d40b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    WIP: cache: add index in all tables

commit b920a31ba6a4aaf30c96584c8791f8c4d4810def
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    WIP: add date_cache sqlite table to speed up by-date/
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new table has been created to store a relation swhid -> date,
    meaning we only need to do one query to get all history with a specific
    date prefix.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/192/ for more details.

This revision is now accepted and ready to land.Nov 27 2020, 2:16 PM

Build is green

Patch application report for D4569 (id=16400)

Rebasing onto 6ab2e73633...

Current branch diff-target is up to date.
Changes applied before test
commit 1c778776d8bbc9da9836d2b66950e88019ef2e47
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 11:17:52 2020 +0100

    tests: extend max delay to populate history/by-date/

commit 65e468780210ed749e2d991bc64bc6a4e3f4384b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:36:05 2020 +0100

    cache: add sqlite index in all tables

commit 0a24b804f6f1fd8431fe5f605c4d1bcb293a0b5e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Nov 24 10:30:40 2020 +0100

    cache: add 'date' column in metadata_cache for history/by-date
    
    Closes T2803.
    
    Before, ls history/by-date/ took a few seconds and it was getting slower
    the more metadata were stored in the sqlite cache. This was because it
    would query the metadata cache for all SWHIDs in history.
    
    Now, a new column has been created to store a relation (rev swhid, date)
    meaning we only need to do one query to get all history with a specific
    date prefix.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/195/ for more details.

haltode retitled this revision from WIP: add date column in metadata cache to speed up by-date/ to FUSE: cache: add 'date' column in metadata_cache for history/by-date.Nov 27 2020, 2:38 PM