Page MenuHomeSoftware Heritage

fs: history: add by-page/ sharded directory
ClosedPublic

Authored by haltode on Mon, Nov 16, 11:09 AM.

Diff Detail

Repository
rDFUSE FUSE virtual file system
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

haltode created this revision.Mon, Nov 16, 11:09 AM

Build is green

Patch application report for D4476 (id=15881)

Rebasing onto 604665ab40...

Current branch diff-target is up to date.
Changes applied before test
commit 82d4081815c61b4f0724422ba4a8a6a30b8def27
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Nov 16 10:49:18 2020 +0100

    fs: history: add by-page/ sharded directory
    
    Closes T2772.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/150/ for more details.

haltode added inline comments.Mon, Nov 16, 11:13 AM
swh/fuse/fs/artifact.py
314–337

A cleaner way would be to iterate on range(0, len(swhids), 10_000) but it would then require to slice the history list, hence we have the same copying problem we had before. However since it is done here only to prefill the cache, we might not really care about this memory overhead?

zack added a subscriber: zack.Mon, Nov 16, 11:39 AM
zack added inline comments.
swh/fuse/fs/artifact.py
316

minor: please move 10_000 to a constant somewhere, e.g., a PAGE_SIZE class attribute

324

bonus point: compute the number of needed leading zeros in the format string from PAGE_SIZE, otherwise it's for sure gonna break in the future

(or at least leave a TODO in the code about it)

haltode updated this revision to Diff 15888.Mon, Nov 16, 1:14 PM

Move magic number into constant class attributes

haltode marked 2 inline comments as done.Mon, Nov 16, 1:15 PM

Build is green

Patch application report for D4476 (id=15888)

Rebasing onto 604665ab40...

Current branch diff-target is up to date.
Changes applied before test
commit 272de4b4f0fc35747afafe2055e4d7b81782dff1
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Nov 16 10:49:18 2020 +0100

    fs: history: add by-page/ sharded directory
    
    Closes T2772.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/151/ for more details.

haltode updated this revision to Diff 15892.Mon, Nov 16, 1:55 PM

Rebase on master

Build is green

Patch application report for D4476 (id=15892)

Rebasing onto 1e09b9b1b2...

Current branch diff-target is up to date.
Changes applied before test
commit 05e2fe98e39d44c9f1828dc079dd8ca061a533af
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Nov 16 10:49:18 2020 +0100

    fs: history: add by-page/ sharded directory
    
    Closes T2772.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/153/ for more details.

seirl accepted this revision.Mon, Nov 16, 2:27 PM
seirl added a subscriber: seirl.
seirl added inline comments.
swh/fuse/fs/artifact.py
309

Please write this with a def, PEP8 discourages named lambdas.

This revision is now accepted and ready to land.Mon, Nov 16, 2:27 PM
haltode updated this revision to Diff 15896.Mon, Nov 16, 2:43 PM

Use format() instead of lambda

haltode marked an inline comment as done.Mon, Nov 16, 2:43 PM
seirl accepted this revision.Mon, Nov 16, 2:44 PM

Build is green

Patch application report for D4476 (id=15896)

Rebasing onto 1e09b9b1b2...

Current branch diff-target is up to date.
Changes applied before test
commit 5fdda3d1cf97b5d5a4e4f3e9988bc68586708733
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Nov 16 10:49:18 2020 +0100

    fs: history: add by-page/ sharded directory
    
    Closes T2772.

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/154/ for more details.

This revision was automatically updated to reflect the committed changes.