Page MenuHomeSoftware Heritage

fs: history: add by-page/ sharded directory
ClosedPublic

Authored by haltode on Nov 16 2020, 11:09 AM.

Diff Detail

Repository
rDFUSE FUSE virtual file system
Branch
feature/add-history-by-page-sharding
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17100
Build 26393: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 26392: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D4476 (id=15881)

Rebasing onto 604665ab40...

Current branch diff-target is up to date.
Changes applied before test
commit 82d4081815c61b4f0724422ba4a8a6a30b8def27
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Nov 16 10:49:18 2020 +0100

    fs: history: add by-page/ sharded directory
    
    Closes T2772.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/150/ for more details.

swh/fuse/fs/artifact.py
313–336

A cleaner way would be to iterate on range(0, len(swhids), 10_000) but it would then require to slice the history list, hence we have the same copying problem we had before. However since it is done here only to prefill the cache, we might not really care about this memory overhead?

zack added inline comments.
swh/fuse/fs/artifact.py
315

minor: please move 10_000 to a constant somewhere, e.g., a PAGE_SIZE class attribute

323

bonus point: compute the number of needed leading zeros in the format string from PAGE_SIZE, otherwise it's for sure gonna break in the future

(or at least leave a TODO in the code about it)

Move magic number into constant class attributes

Build is green

Patch application report for D4476 (id=15888)

Rebasing onto 604665ab40...

Current branch diff-target is up to date.
Changes applied before test
commit 272de4b4f0fc35747afafe2055e4d7b81782dff1
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Nov 16 10:49:18 2020 +0100

    fs: history: add by-page/ sharded directory
    
    Closes T2772.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/151/ for more details.

Build is green

Patch application report for D4476 (id=15892)

Rebasing onto 1e09b9b1b2...

Current branch diff-target is up to date.
Changes applied before test
commit 05e2fe98e39d44c9f1828dc079dd8ca061a533af
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Nov 16 10:49:18 2020 +0100

    fs: history: add by-page/ sharded directory
    
    Closes T2772.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/153/ for more details.

seirl added a subscriber: seirl.
seirl added inline comments.
swh/fuse/fs/artifact.py
309

Please write this with a def, PEP8 discourages named lambdas.

This revision is now accepted and ready to land.Nov 16 2020, 2:27 PM

Use format() instead of lambda

Build is green

Patch application report for D4476 (id=15896)

Rebasing onto 1e09b9b1b2...

Current branch diff-target is up to date.
Changes applied before test
commit 5fdda3d1cf97b5d5a4e4f3e9988bc68586708733
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Nov 16 10:49:18 2020 +0100

    fs: history: add by-page/ sharded directory
    
    Closes T2772.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/154/ for more details.