Page MenuHomeSoftware Heritage
Differential D6517

Improve tests of the seaweedfs backend
ClosedPublic
Actions

Authored by douardda on Oct 20 2021, 12:11 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDOBJS6269067ca7b3: Improve a bit the seaweedfs backend
rDOBJS55ff4b95d306: Improve tests of the seaweedfs backend
Summary

tests for this backend are now much more realistic,
mocking a seaweedfs Filer server instead of the WeedFiler directly.

And actually fix a bug in WeedFiler.get()...

Also remove tests for each compression algo, keeping only one
is enought.

Also improve a bit the seaweedfs backend

Diff Detail

Unit TestsFailed
View All

TimeTest
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.objstorage.tests.test_objstorage_multiplexer.TestMultiplexerObjStorage::test_restore_content
self = <swh.objstorage.tests.test_objstorage_multiplexer.TestMultiplexerObjStorage testMethod=test_restore_content> def test_restore_content(self):
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.objstorage.tests.test_multiplexer_filter.MixinTestReadFilter::test_can_check
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.objstorage.tests.test_multiplexer_filter.MixinTestReadFilter::test_can_contains
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.objstorage.tests.test_multiplexer_filter.MixinTestReadFilter::test_can_get
39 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.objstorage.tests.test_multiplexer_filter.MixinTestReadFilter::test_can_get_random
View Full Test Results (1 Failed · 679 Passed · 4 Skipped)

Event Timeline

douardda created this revision.Oct 20 2021, 12:11 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 20 2021, 12:11 PM
douardda added a child revision: D6492: Add support for pathslicing in seaweedfs backend.Oct 20 2021, 12:12 PM
swh-public-ci added a comment.Oct 20 2021, 12:13 PM

Build has FAILED

Patch application report for D6517 (id=23682)

Rebasing onto 23b7f81c14...

Current branch diff-target is up to date.
Changes applied before test
commit 78b1e82b8f87fbf7d6bb034204d5210ed71824cc
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 20 10:34:41 2021 +0200

    Improve a bit the seaweedfs backend

commit 55ff4b95d3062421a7be6465c6ca07f8cd4909ab
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 8 14:01:24 2021 +0200

    Improve tests of the seaweedfs backend
    
    tests for this backend are now much more realistic,
    mocking a seaweedfs Filer server instead of the `WeedFiler` directly.
    
    And actually fix a bug in WeedFiler.get()...
    
    Also remove tests for each compression algo, keeping only one
    is enought.

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/40/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/40/console

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 20 2021, 12:14 PM
Harbormaster failed remote builds in B24571: Diff 23682!
douardda mentioned this in D6492: Add support for pathslicing in seaweedfs backend.Oct 20 2021, 12:14 PM
douardda updated this revision to Diff 23684.Oct 20 2021, 12:33 PM

add a comment about DirMode value

swh-public-ci added a comment.Oct 20 2021, 12:35 PM

Build has FAILED

Patch application report for D6517 (id=23684)

Rebasing onto 23b7f81c14...

Current branch diff-target is up to date.
Changes applied before test
commit be26482e93e15a2a699f9f35ce63286ccbf737b5
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 20 10:34:41 2021 +0200

    Improve a bit the seaweedfs backend
    
    - use a persistent session object in WeedFiler,
    - better implementation of listing methods

commit 55ff4b95d3062421a7be6465c6ca07f8cd4909ab
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 8 14:01:24 2021 +0200

    Improve tests of the seaweedfs backend
    
    tests for this backend are now much more realistic,
    mocking a seaweedfs Filer server instead of the `WeedFiler` directly.
    
    And actually fix a bug in WeedFiler.get()...
    
    Also remove tests for each compression algo, keeping only one
    is enought.

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/42/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/42/console

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 20 2021, 12:35 PM
Harbormaster failed remote builds in B24573: Diff 23684!
swh-public-ci added a comment.Oct 20 2021, 12:40 PM

Build is green

Patch application report for D6517 (id=23682)

Rebasing onto 23b7f81c14...

Current branch diff-target is up to date.
Changes applied before test
commit 78b1e82b8f87fbf7d6bb034204d5210ed71824cc
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 20 10:34:41 2021 +0200

    Improve a bit the seaweedfs backend

commit 55ff4b95d3062421a7be6465c6ca07f8cd4909ab
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 8 14:01:24 2021 +0200

    Improve tests of the seaweedfs backend
    
    tests for this backend are now much more realistic,
    mocking a seaweedfs Filer server instead of the `WeedFiler` directly.
    
    And actually fix a bug in WeedFiler.get()...
    
    Also remove tests for each compression algo, keeping only one
    is enought.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/44/ for more details.

Harbormaster completed remote builds in B24571: Diff 23682.Oct 20 2021, 12:40 PM
douardda requested review of this revision.Oct 20 2021, 2:08 PM
douardda updated this revision to Diff 23689.Oct 20 2021, 2:12 PM

respawn jenkins

vlorentz accepted this revision.Oct 20 2021, 2:15 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/objstorage/backends/seaweed.py
74–114

would this work, as a minor simplification?

This revision is now accepted and ready to land.Oct 20 2021, 2:15 PM
swh-public-ci added a comment.Oct 20 2021, 2:15 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/45/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/45/console

douardda added inline comments.Oct 20 2021, 2:17 PM
swh/objstorage/backends/seaweed.py
74–114

need to check against actual seaweed filer

swh-public-ci added a comment.Oct 20 2021, 2:25 PM

Build is green

Patch application report for D6517 (id=23689)

Rebasing onto 23b7f81c14...

Current branch diff-target is up to date.
Changes applied before test
commit be26482e93e15a2a699f9f35ce63286ccbf737b5
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 20 10:34:41 2021 +0200

    Improve a bit the seaweedfs backend
    
    - use a persistent session object in WeedFiler,
    - better implementation of listing methods

commit 55ff4b95d3062421a7be6465c6ca07f8cd4909ab
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 8 14:01:24 2021 +0200

    Improve tests of the seaweedfs backend
    
    tests for this backend are now much more realistic,
    mocking a seaweedfs Filer server instead of the `WeedFiler` directly.
    
    And actually fix a bug in WeedFiler.get()...
    
    Also remove tests for each compression algo, keeping only one
    is enought.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/46/ for more details.

Harbormaster completed remote builds in B24578: Diff 23689.Oct 20 2021, 2:25 PM
douardda updated this revision to Diff 23691.Oct 20 2021, 2:30 PM

slight simplification as suggested by vlorentz

swh-public-ci added a comment.Oct 20 2021, 2:34 PM

Build has FAILED

Patch application report for D6517 (id=23691)

Rebasing onto 23b7f81c14...

Current branch diff-target is up to date.
Changes applied before test
commit 96eadebd2d05f06bc87fb5461892e80fa98ebefe
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 20 10:34:41 2021 +0200

    Improve a bit the seaweedfs backend
    
    - use a persistent session object in WeedFiler,
    - better implementation of listing methods

commit 55ff4b95d3062421a7be6465c6ca07f8cd4909ab
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 8 14:01:24 2021 +0200

    Improve tests of the seaweedfs backend
    
    tests for this backend are now much more realistic,
    mocking a seaweedfs Filer server instead of the `WeedFiler` directly.
    
    And actually fix a bug in WeedFiler.get()...
    
    Also remove tests for each compression algo, keeping only one
    is enought.

Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/48/
See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/48/console

Harbormaster failed remote builds in B24580: Diff 23691!Oct 20 2021, 2:34 PM
douardda updated this revision to Diff 23693.Oct 20 2021, 2:35 PM

remove a (comment) garbage line

swh-public-ci added a comment.Oct 20 2021, 2:38 PM

Build is green

Patch application report for D6517 (id=23693)

Rebasing onto 23b7f81c14...

Current branch diff-target is up to date.
Changes applied before test
commit 6269067ca7b3599ecc7cc2436ee72a4f284fdd85
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Oct 20 10:34:41 2021 +0200

    Improve a bit the seaweedfs backend
    
    - use a persistent session object in WeedFiler,
    - better implementation of listing methods

commit 55ff4b95d3062421a7be6465c6ca07f8cd4909ab
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 8 14:01:24 2021 +0200

    Improve tests of the seaweedfs backend
    
    tests for this backend are now much more realistic,
    mocking a seaweedfs Filer server instead of the `WeedFiler` directly.
    
    And actually fix a bug in WeedFiler.get()...
    
    Also remove tests for each compression algo, keeping only one
    is enought.

See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/50/ for more details.

Harbormaster completed remote builds in B24582: Diff 23693.Oct 20 2021, 2:38 PM
Closed by commit rDOBJS55ff4b95d306: Improve tests of the seaweedfs backend (authored by douardda). · Explain WhyOct 20 2021, 2:40 PM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDOBJS55ff4b95d306: Improve tests of the seaweedfs backend.
douardda added a commit: rDOBJS6269067ca7b3: Improve a bit the seaweedfs backend.
douardda mentioned this in T3668: Improve the seaweedfs backend.Oct 27 2021, 10:18 AM

Revision Contents
Changeset List

PathSize
3 lines
2 lines
1 line
swh/
objstorage/
backends/
104 lines
tests/
152 lines

Diff 23691

View Options

mypy.ini

Loading...
View Options

requirements-test.txt

Loading...
View Options

requirements.txt

Loading...
View Options

swh/objstorage/backends/seaweed.py

Loading...
View Options

swh/objstorage/tests/test_objstorage_seaweedfs.py

Loading...
About · Developer information · Legal