Page MenuHomeSoftware Heritage
Differential D6548

Instantiate a noop objstorage for testing purposes
ClosedPublic
Actions

Authored by ardumont on Oct 25 2021, 5:25 PM.

Details

Reviewers
olasd
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T3627: Consider dropping pull request references from the git loader ingestion
Commits
rDOBJS64acd020aa7c: Instantiate a noop objstorage for testing purposes
Summary

This also prevents the main server from running the noop objstorage in production.

Related to T3627

Test Plan

tox (happy)

Use docker-compose with override to allow using such objstorage without eating all disk
when running big tests on repositories.

Diff Detail

Repository
rDOBJS Object storage
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 24683
Build 38519: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 38518: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.objstorage.tests.test_server::test_load_and_check_config[empty-args-bw-compat]
tmpdir = local('/tmp/pytest-of-jenkins/pytest-0/test_load_and_check_config_emp0') config = {'client_max_size': '10', 'objstorage': {'args': {}, 'cls': 'memory'}}
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.objstorage.tests.test_server::test_load_and_check_config[empty-args]
tmpdir = local('/tmp/pytest-of-jenkins/pytest-0/test_load_and_check_config_emp1') config = {'client_max_size': '10', 'objstorage': {'cls': 'memory'}}
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.objstorage.tests.test_multiplexer_filter.MixinTestReadFilter::test_can_check
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.objstorage.tests.test_multiplexer_filter.MixinTestReadFilter::test_can_contains
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.objstorage.tests.test_multiplexer_filter.MixinTestReadFilter::test_can_get
View Full Test Results (2 Failed · 705 Passed · 4 Skipped)

Event Timeline

ardumont created this revision.Oct 25 2021, 5:25 PM
ardumont edited the summary of this revision. (Show Details)Oct 25 2021, 5:25 PM
swh-public-ci added a comment.Oct 25 2021, 5:26 PM

Build has FAILED

Patch application report for D6548 (id=23799)

Rebasing onto 8ed5f4ebc9...

First, rewinding head to replay your work on top of it...
Applying: Instantiate a noop objstorage for testing purposes
Using index info to reconstruct a base tree...
M	swh/objstorage/factory.py
Falling back to patching base and 3-way merge...
Auto-merging swh/objstorage/factory.py
CONFLICT (content): Merge conflict in swh/objstorage/factory.py
Patch failed at 0001 Instantiate a noop objstorage for testing purposes

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".

Rebase failed (ret=1)!




Could not rebase; Attempt merge onto 8ed5f4ebc9...



Already up to date.





Changes applied before test



commit ac0b08b12d15f29e3811091f39f3c508f795bfce
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 25 17:23:41 2021 +0200

    Instantiate a noop objstorage for testing purposes
    
    Related to T3627



Link to build: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/65/

See console output for more information: https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/65/console
Harbormaster returned this revision to the author for changes because remote builds failed.Oct 25 2021, 5:26 PM
Harbormaster failed remote builds in B24683: Diff 23799!
ardumont edited the summary of this revision. (Show Details)Oct 25 2021, 5:31 PM
ardumont updated this revision to Diff 23800.Oct 25 2021, 5:32 PM
ardumont edited the summary of this revision. (Show Details)
Focus only on preventing noop from running in production.
swh-public-ci added a comment.Oct 25 2021, 5:34 PM
Build is green



Patch application report for D6548 (id=23800)



Rebasing onto 8ed5f4ebc9...



First, rewinding head to replay your work on top of it...
Applying: Instantiate a noop objstorage for testing purposes
Using index info to reconstruct a base tree...
M	swh/objstorage/factory.py
Falling back to patching base and 3-way merge...
Auto-merging swh/objstorage/factory.py
CONFLICT (content): Merge conflict in swh/objstorage/factory.py
Patch failed at 0001 Instantiate a noop objstorage for testing purposes

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".



Rebase failed (ret=1)!






Could not rebase; Attempt merge onto 8ed5f4ebc9...



Already up to date.





Changes applied before test



commit ad1f2de3e9b26dd1b0ecf5486cbe4549ffb772a6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 25 17:23:41 2021 +0200

    Instantiate a noop objstorage for testing purposes
    
    Related to T3627



See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/66/ for more details.
Harbormaster completed remote builds in B24684: Diff 23800.Oct 25 2021, 5:34 PM
ardumont requested review of this revision.Oct 25 2021, 5:34 PM
ardumont added inline comments.Oct 25 2021, 5:38 PM
swh/objstorage/backends/noop.py


14
i have a better docstring locally:



Noop objstorage. Basic implementation which does no operations at all.

    Only intended for test purposes to avoid either memory or i/o operations. This
    allows swh clients to use the swh stack without having to deal with objstorage
    configuration. So users can concentrate on testing the remaining part of the stack
    without the objstorage.
vlorentz accepted this revision.Oct 27 2021, 12:44 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/objstorage/api/server.py


258
This revision is now accepted and ready to land.Oct 27 2021, 12:44 PM
ardumont updated this revision to Diff 23837.Oct 27 2021, 2:14 PM
Adapt according to suggestion
swh-public-ci added a comment.Oct 27 2021, 2:16 PM
Build is green



Patch application report for D6548 (id=23837)



Rebasing onto 8ed5f4ebc9...



First, rewinding head to replay your work on top of it...
Applying: Instantiate a noop objstorage for testing purposes
Using index info to reconstruct a base tree...
M	swh/objstorage/factory.py
Falling back to patching base and 3-way merge...
Auto-merging swh/objstorage/factory.py
CONFLICT (content): Merge conflict in swh/objstorage/factory.py
Patch failed at 0001 Instantiate a noop objstorage for testing purposes

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".



Rebase failed (ret=1)!






Could not rebase; Attempt merge onto 8ed5f4ebc9...



Already up to date.





Changes applied before test



commit 35afa820403f73beb809efce0aee0a4d7cc3dabc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 25 17:23:41 2021 +0200

    Instantiate a noop objstorage for testing purposes
    
    Related to T3627



See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/67/ for more details.
Harbormaster completed remote builds in B24720: Diff 23837.Oct 27 2021, 2:16 PM
ardumont updated this revision to Diff 23838.Oct 27 2021, 2:20 PM
Rebase
swh-public-ci added a comment.Oct 27 2021, 2:22 PM
Build is green



Patch application report for D6548 (id=23838)



Rebasing onto 8ed5f4ebc9...



Current branch diff-target is up to date.





Changes applied before test



commit 64acd020aa7c4d430b9f6b9c0b6c07438a6ebfb9
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Oct 25 17:23:41 2021 +0200

    Instantiate a noop objstorage for testing purposes
    
    Related to T3627



See https://jenkins.softwareheritage.org/job/DOBJS/job/tests-on-diff/68/ for more details.
Harbormaster completed remote builds in B24721: Diff 23838.Oct 27 2021, 2:23 PM
Closed by commit rDOBJS64acd020aa7c: Instantiate a noop objstorage for testing purposes (authored by ardumont).  ·  Explain WhyOct 27 2021, 2:23 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDOBJS64acd020aa7c: Instantiate a noop objstorage for testing purposes.
Revision Contents
Changeset List
PathSize
swh/
objstorage/
api/
5 lines
backends/
34 lines
4 lines
tests/
13 lines
14 lines
Diff 23799
View Options
swh/objstorage/api/server.py
Loading...
View Options
swh/objstorage/backends/noop.py
Loading...
View Options
swh/objstorage/factory.py
Loading...
View Options
swh/objstorage/tests/test_objstorage_noop.py
Loading...
View Options
swh/objstorage/tests/test_server.py
Loading...
About · Developer information · Legal