Page MenuHomeSoftware Heritage
Differential D6442

Extract the path slicing logic in a dedicated PathSlicer class
ClosedPublic
Actions

Authored by douardda on Oct 8 2021, 12:33 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDOBJS23b7f81c1483: Extract the path slicing logic in a dedicated PathSlicer class
Summary

so it can be reused in other backends, e.g. for cloud based storages,
one may want to organise the objects in a pathslicing hierarchy.

Diff Detail

Repository
rDOBJS Object storage
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Oct 8 2021, 12:33 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 8 2021, 12:33 PM
swh-public-ci added a comment.Oct 8 2021, 12:35 PM

Build has FAILED

Patch application report for D6442 (id=23394)

Rebasing onto c0bac6bc09...

Current branch diff-target is up to date.
Changes applied before test
commit 97deea02ac14f14973769b64dcf993bf40e40a97
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 8 12:30:22 2021 +0200

    Extract the path slicing logic in a dedicated PathSlicer class
    
    so it can be reused in other backends, e.g. for cloud based storages,
    one may want to organise the objects in a pathslicing hierarchy.

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/31/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/31/console

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 8 2021, 12:35 PM
Harbormaster failed remote builds in B24311: Diff 23394!
swh-public-ci added a comment.Oct 8 2021, 2:10 PM

Build is green

Patch application report for D6442 (id=23394)

Rebasing onto c0bac6bc09...

Current branch diff-target is up to date.
Changes applied before test
commit 97deea02ac14f14973769b64dcf993bf40e40a97
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 8 12:30:22 2021 +0200

    Extract the path slicing logic in a dedicated PathSlicer class
    
    so it can be reused in other backends, e.g. for cloud based storages,
    one may want to organise the objects in a pathslicing hierarchy.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/32/ for more details.

Harbormaster completed remote builds in B24311: Diff 23394.Oct 8 2021, 2:10 PM
douardda requested review of this revision.Oct 8 2021, 2:10 PM
douardda updated this revision to Diff 23398.Oct 8 2021, 2:12 PM

allow the pathslicer to be a noop (with an empty slicing)

swh-public-ci added a comment.Oct 8 2021, 2:13 PM

Build is green

Patch application report for D6442 (id=23398)

Rebasing onto c0bac6bc09...

Current branch diff-target is up to date.
Changes applied before test
commit 0492c28eb405f0559d38b55014b6930b06e1045b
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 8 12:30:22 2021 +0200

    Extract the path slicing logic in a dedicated PathSlicer class
    
    so it can be reused in other backends, e.g. for cloud based storages,
    one may want to organise the objects in a pathslicing hierarchy.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/33/ for more details.

Harbormaster completed remote builds in B24315: Diff 23398.Oct 8 2021, 2:14 PM
olasd added a subscriber: olasd.Oct 8 2021, 2:46 PM
olasd added inline comments.
swh/objstorage/backends/pathslicing.py
159–160

These args should probably be moved to the main class docstring, the rendering is weird otherwise.

swh/objstorage/tests/test_pathslicer.py
42–53

LOL. Let's hope noone actually uses this ever.

vlorentz added a subscriber: vlorentz.Oct 8 2021, 2:47 PM
vlorentz added inline comments.
swh/objstorage/backends/pathslicing.py
42

easier to read

44–52

more readable IMO

douardda added inline comments.Oct 8 2021, 3:53 PM
swh/objstorage/backends/pathslicing.py
42

agreed

44–52

sure, why not

159–160

ah yes sure! this came to my mind a some point, but vanished before anything happened...

douardda updated this revision to Diff 23416.Oct 8 2021, 4:19 PM

Better docstrings and kill a few map()

swh-public-ci added a comment.Oct 8 2021, 4:21 PM

Build is green

Patch application report for D6442 (id=23416)

Rebasing onto c0bac6bc09...

Current branch diff-target is up to date.
Changes applied before test
commit be4094fb296df666d778f5969a9954485d16c04f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 8 12:30:22 2021 +0200

    Extract the path slicing logic in a dedicated PathSlicer class
    
    so it can be reused in other backends, e.g. for cloud based storages,
    one may want to organise the objects in a pathslicing hierarchy.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/34/ for more details.

Harbormaster completed remote builds in B24332: Diff 23416.Oct 8 2021, 4:21 PM
vlorentz accepted this revision.Oct 12 2021, 11:44 AM
vlorentz added inline comments.
swh/objstorage/tests/test_pathslicer.py
50

forgot this one

This revision is now accepted and ready to land.Oct 12 2021, 11:44 AM
douardda added inline comments.Oct 12 2021, 5:50 PM
swh/objstorage/tests/test_pathslicer.py
50

thx a lot

douardda updated this revision to Diff 23479.Oct 12 2021, 5:52 PM

forgotten print statement...

swh-public-ci added a comment.Oct 12 2021, 5:54 PM

Build is green

Patch application report for D6442 (id=23479)

Rebasing onto c0bac6bc09...

Current branch diff-target is up to date.
Changes applied before test
commit 23b7f81c148383c788a0fffcf897fcf456980fa9
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 8 12:30:22 2021 +0200

    Extract the path slicing logic in a dedicated PathSlicer class
    
    so it can be reused in other backends, e.g. for cloud based storages,
    one may want to organise the objects in a pathslicing hierarchy.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/35/ for more details.

Harbormaster completed remote builds in B24396: Diff 23479.Oct 12 2021, 5:54 PM
Closed by commit rDOBJS23b7f81c1483: Extract the path slicing logic in a dedicated PathSlicer class (authored by douardda). · Explain WhyOct 13 2021, 3:19 PM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDOBJS23b7f81c1483: Extract the path slicing logic in a dedicated PathSlicer class.
douardda mentioned this in T3668: Improve the seaweedfs backend.Oct 18 2021, 3:33 PM

Revision Contents
Changeset List

PathSize
swh/
objstorage/
backends/
247 lines
tests/
2 lines
85 lines

Diff 23487

View Options

swh/objstorage/backends/pathslicing.py

Loading...
View Options

swh/objstorage/tests/test_objstorage_pathslicing.py

Loading...
View Options

swh/objstorage/tests/test_pathslicer.py

Loading...
About · Developer information · Legal