Page MenuHomeSoftware Heritage
Differential D4509

fs: history: clean sharded dir implementation
ClosedPublic
Actions

Authored by haltode on Nov 18 2020, 3:26 PM.

Details

Reviewers
seirl
Group Reviewers
Reviewers
Commits
rDFUSEc4dd1c367078: fs: history: clean sharded dir implementation
Summary
  • Remove manual direntry cache manipulation
  • Remove unnecessary by-hash/ sharding depth attribute
  • More extensive unit testing for by-date/

Diff Detail

Repository
rDFUSE FUSE virtual file system
Branch
feature/history-code-cleanup
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17287
Build 26695: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 26694: arc lint + arc unit

Event Timeline

haltode created this revision.Nov 18 2020, 3:26 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 18 2020, 3:26 PM
swh-public-ci added a comment.Nov 18 2020, 3:29 PM

Build is green

Patch application report for D4509 (id=15991)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit b2cf00d0afc89f4cd1eaae3be4c76417714eb85f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/162/ for more details.

Harbormaster completed remote builds in B17194: Diff 15991.Nov 18 2020, 3:29 PM
seirl accepted this revision.Nov 18 2020, 5:57 PM
seirl added a subscriber: seirl.
seirl added inline comments.
swh/fuse/fs/artifact.py
355

Could you use None here?

This revision is now accepted and ready to land.Nov 18 2020, 5:57 PM
haltode updated this revision to Diff 16011.Nov 19 2020, 9:04 AM

Use None instead of -1.

swh-public-ci added a comment.Nov 19 2020, 9:06 AM

Build is green

Patch application report for D4509 (id=16011)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit b381f61806e68bea613138b0ed6c0eff892efaaf
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/163/ for more details.

Harbormaster completed remote builds in B17218: Diff 16011.Nov 19 2020, 9:06 AM
haltode updated this revision to Diff 16023.Nov 19 2020, 11:44 AM

Add time optimization for by-date/ partial results by utilizing the metadata cache

swh-public-ci added a comment.Nov 19 2020, 11:47 AM

Build is green

Patch application report for D4509 (id=16023)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit 2fc39c5ba29c8705459ba51d81b391294242f117
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/164/ for more details.

Harbormaster completed remote builds in B17230: Diff 16023.Nov 19 2020, 11:47 AM
haltode updated this revision to Diff 16024.Nov 19 2020, 11:53 AM

Fix string formatting style

swh-public-ci added a comment.Nov 19 2020, 11:55 AM

Build is green

Patch application report for D4509 (id=16024)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit 88ea17fd20114b729c03d6f9eada40487fe00dff
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/165/ for more details.

Harbormaster completed remote builds in B17231: Diff 16024.Nov 19 2020, 11:55 AM
haltode updated this revision to Diff 16037.Nov 19 2020, 2:31 PM

ensure_cached -> get_cached_subset

swh-public-ci added a comment.Nov 19 2020, 2:34 PM

Build is green

Patch application report for D4509 (id=16037)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit 2ecfcf8bb4551ec5879255767d79d575d9aea779
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/166/ for more details.

Harbormaster completed remote builds in B17243: Diff 16037.Nov 19 2020, 2:34 PM
seirl added inline comments.Nov 19 2020, 2:39 PM
swh/fuse/cache.py
135–136

Use row/swhid to avoid reusing a variable name with a different type

swh/fuse/fs/artifact.py
305–306

Add parentheses to be easier to understand

haltode marked an inline comment as done.Nov 19 2020, 2:42 PM
haltode updated this revision to Diff 16040.Nov 19 2020, 2:45 PM

Fix name variable style

haltode marked 2 inline comments as done.Nov 19 2020, 2:45 PM
haltode added inline comments.
swh/fuse/fs/artifact.py
305–306

black prevents parentheses here.

swh-public-ci added a comment.Nov 19 2020, 2:48 PM

Build has FAILED

Patch application report for D4509 (id=16040)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit 9decc23931740a940b7e698d7c416644b2bc8a45
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/167/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/167/console

Harbormaster failed remote builds in B17244: Diff 16040!Nov 19 2020, 2:48 PM
swh-public-ci added a comment.Nov 19 2020, 3:18 PM

Build has FAILED

Patch application report for D4509 (id=16040)

Rebasing onto 430fa0e249...

Current branch diff-target is up to date.
Changes applied before test
commit 9decc23931740a940b7e698d7c416644b2bc8a45
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/168/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/168/console

haltode updated this revision to Diff 16087.Nov 20 2020, 10:18 AM
haltode marked an inline comment as done.

Rebasing on master

swh-public-ci added a comment.Nov 20 2020, 10:21 AM

Build is green

Patch application report for D4509 (id=16087)

Rebasing onto b5956b0fa6...

Current branch diff-target is up to date.
Changes applied before test
commit c4dd1c367078c903b31ac40f639f7b5a60657737
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Nov 18 15:23:15 2020 +0100

    fs: history: clean sharded dir implementation
    
    - Remove manual direntry cache manipulation
    - Remove unnecessary by-hash/ sharding depth attribute
    - Time optimization for by-date/ partial results
    - More extensive unit testing for by-date/

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/170/ for more details.

Harbormaster completed remote builds in B17287: Diff 16087.Nov 20 2020, 10:21 AM
Closed by commit rDFUSEc4dd1c367078: fs: history: clean sharded dir implementation (authored by haltode). · Explain WhyNov 20 2020, 10:22 AM
This revision was automatically updated to reflect the committed changes.
haltode added a commit: rDFUSEc4dd1c367078: fs: history: clean sharded dir implementation.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
fuse/
21 lines88%
Loading...
1 line61%
Loading...
fs/
223 lines95%
Loading...
tests/
16 lines98%
Loading...

Diff 16087

View Options

swh/fuse/cache.py

Loading...
View Options

swh/fuse/cli.py

Loading...
View Options

swh/fuse/fs/artifact.py

Loading...
View Options

swh/fuse/tests/test_revision.py

Loading...
About · Developer information · Legal