Page MenuHomeSoftware Heritage

fs: history: clean sharded dir implementation
ClosedPublic

Authored by haltode on Nov 18 2020, 3:26 PM.

Details

Summary
  • Remove manual direntry cache manipulation
  • Remove unnecessary by-hash/ sharding depth attribute
  • More extensive unit testing for by-date/

Diff Detail

Repository
rDFUSE FUSE virtual file system
Branch
feature/history-code-cleanup
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17218
Build 26582: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 26581: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D4509 (id=15991)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit b2cf00d0afc89f4cd1eaae3be4c76417714eb85f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/162/ for more details.

seirl added a subscriber: seirl.
seirl added inline comments.
swh/fuse/fs/artifact.py
359

Could you use None here?

This revision is now accepted and ready to land.Nov 18 2020, 5:57 PM

Build is green

Patch application report for D4509 (id=16011)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit b381f61806e68bea613138b0ed6c0eff892efaaf
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/163/ for more details.

Add time optimization for by-date/ partial results by utilizing the metadata cache

Build is green

Patch application report for D4509 (id=16023)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit 2fc39c5ba29c8705459ba51d81b391294242f117
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/164/ for more details.

Fix string formatting style

Build is green

Patch application report for D4509 (id=16024)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit 88ea17fd20114b729c03d6f9eada40487fe00dff
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/165/ for more details.

ensure_cached -> get_cached_subset

Build is green

Patch application report for D4509 (id=16037)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit 2ecfcf8bb4551ec5879255767d79d575d9aea779
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/166/ for more details.

swh/fuse/cache.py
136–137 ↗(On Diff #16024)

Use row/swhid to avoid reusing a variable name with a different type

swh/fuse/fs/artifact.py
309–310

Add parentheses to be easier to understand

haltode added inline comments.
swh/fuse/fs/artifact.py
309–310

black prevents parentheses here.

Build has FAILED

Patch application report for D4509 (id=16040)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit 9decc23931740a940b7e698d7c416644b2bc8a45
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/167/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/167/console

Build has FAILED

Patch application report for D4509 (id=16040)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit 9decc23931740a940b7e698d7c416644b2bc8a45
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/168/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/168/console

haltode marked an inline comment as done.

Rebasing on master

Build is green

Patch application report for D4509 (id=16087)

Rebasing onto b5956b0fa6...

Current branch diff-target is up to date.
Changes applied before test
commit c4dd1c367078c903b31ac40f639f7b5a60657737
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/170/ for more details.