Page MenuHomeSoftware Heritage
Differential D317

Make the vault's scheduler configuration adjustable
ClosedPublic
Actions

Authored by ardumont on May 3 2018, 3:54 PM.

Details

Reviewers
seirl
Group Reviewers
Reviewers
Commits
rDVAU3688b7d627c2: api/server: Use remote service for default configuration storage
rDVAUc3a07c6b73cb: Make the vault's scheduler configuration adjustable
Summary

Use the scheduler's api instantiation

Not urgent

This can be merged at some point, when you are done with your current tasks for example ;)

Related T1035

Diff Detail

Repository
rDVAU Software Heritage Vault
Branch
vault-scheduler-configuration-adjustable
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 1352
Build 1696: arc lint + arc unit

Event Timeline

ardumont created this revision.May 3 2018, 3:54 PM
Harbormaster completed remote builds in B1213: Diff 1038.May 3 2018, 3:54 PM
seirl accepted this revision.May 4 2018, 2:22 PM

LGTM, I just have one nitpick on the default configuration that we might want to fix before merging this.

swh/vault/api/server.py
44–49

Aren't the default usually the remote services, and not the local ones?

This revision is now accepted and ready to land.May 4 2018, 2:22 PM
ardumont added inline comments.May 4 2018, 2:28 PM
swh/vault/api/server.py
44–49

I kept your default because i did not want to impose anything.
Running this by default, you would need to either run a scheduler service (or edit/add a ~/.config/swh/vault/server.yml with the local setup).

But sure, we can use the remote one.
That's what i did for the deposit [1].

[1] rDDEP7e535ab1c196fdeab8722ef610ca1d02ceadbc58

ardumont added inline comments.May 4 2018, 2:45 PM
swh/vault/api/server.py
44–49

Also, i'm matching the other default configuration of that very same file.
No other entry in the default configuration is using any remote service ;)

ardumont mentioned this in T1138: Migrate remaining modules to remote scheduler api use.Jul 18 2018, 11:42 AM
ardumont updated this revision to Diff 1229.Jul 18 2018, 2:50 PM

Amend the prior commit to fix and use the remote scheduler configuration

Herald added a reviewer: Reviewers. · View Herald TranscriptJul 18 2018, 2:50 PM
This revision now requires review to proceed.Jul 18 2018, 2:50 PM
Harbormaster completed remote builds in B1352: Diff 1229.Jul 18 2018, 2:51 PM
ardumont added inline comments.Jul 18 2018, 2:51 PM
swh/vault/api/server.py
44–49

Ok, so i had a remark about this on D372#inline-1828 that goes your way.
So fixing this.

Cheers,

ardumont marked 3 inline comments as done.Jul 18 2018, 2:51 PM
ardumont updated this revision to Diff 1230.Jul 18 2018, 2:55 PM
  • api/server: Use remote service for default configuration storage
Harbormaster completed remote builds in B1353: Diff 1230.Jul 18 2018, 2:55 PM
ardumont added a comment.Jul 18 2018, 2:55 PM

LGTM, I just have one nitpick on the default configuration that we might want to fix before merging this.

Fixed for the scheduler in the amended commit.
Fixed in the second commit for the storage part.

ardumont marked an inline comment as done.Jul 18 2018, 2:58 PM
This revision was not accepted when it landed; it landed in state Needs Review.Aug 1 2018, 3:23 PM
Closed by commit rDVAUc3a07c6b73cb: Make the vault's scheduler configuration adjustable (authored by ardumont). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
debian/
4 lines
2 lines
swh/
vault/
api/
9 lines
9 lines
tests/
2 lines

Diff 1229

View Options

debian/control

[tox] [tox]
envlist=black,flake8,mypy,py3-{core,db,server} envlist=black,flake8,mypy,py3-{core,db,github,server}
[testenv] [testenv]
passenv = PYTHONASYNCIODEBUG passenv = PYTHONASYNCIODEBUG
extras = extras =
testing-core testing-core
core: logging core: logging
db: db db: db
github: github
server: http server: http
deps = deps =
cover: pytest-cov cover: pytest-cov
commands = commands =
pytest --doctest-modules \ pytest --doctest-modules \
slow: --hypothesis-profile=slow \ slow: --hypothesis-profile=slow \
cover: --cov={envsitepackagesdir}/swh/core --cov-branch \ cover: --cov={envsitepackagesdir}/swh/core --cov-branch \
core: {envsitepackagesdir}/swh/core/tests \ core: {envsitepackagesdir}/swh/core/tests \
github: {envsitepackagesdir}/swh/core/github/tests \
db: {envsitepackagesdir}/swh/core/db/tests \ db: {envsitepackagesdir}/swh/core/db/tests \
server: {envsitepackagesdir}/swh/core/api/tests \ server: {envsitepackagesdir}/swh/core/api/tests \
{posargs} {posargs}
[testenv:py3] [testenv:py3]
skip_install = true skip_install = true
deps = tox deps = tox
commands = tox -e py3-core-db-server-slow-cover -- {posargs} commands = tox -e py3-core-db-server-slow-cover -- {posargs}
Show All 13 Lines
commands = commands =
{envpython} -m flake8 {envpython} -m flake8
[testenv:mypy] [testenv:mypy]
extras = extras =
testing-core testing-core
logging logging
db db
github
http http
deps = deps =
mypy==0.942 mypy==0.942
commands = commands =
mypy swh mypy swh
# build documentation outside swh-environment using the current # build documentation outside swh-environment using the current
# git HEAD of swh-docs, is executed on CI for each diff to prevent # git HEAD of swh-docs, is executed on CI for each diff to prevent
Show All 22 Lines
# of swh-docs package # of swh-docs package
[testenv:sphinx-dev] [testenv:sphinx-dev]
whitelist_externals = make whitelist_externals = make
usedevelop = true usedevelop = true
extras = extras =
testing-core testing-core
logging logging
db db
github
http http
deps = deps =
# install swh-docs in develop mode # install swh-docs in develop mode
-e ../swh-docs -e ../swh-docs
setenv = setenv =
SWH_PACKAGE_DOC_TOX_BUILD = 1 SWH_PACKAGE_DOC_TOX_BUILD = 1
# turn warnings into errors # turn warnings into errors
SPHINXOPTS = -W SPHINXOPTS = -W
commands = commands =
make -I ../.tox/sphinx-dev/src/swh-docs/swh/ -C docs make -I ../.tox/sphinx-dev/src/swh-docs/swh/ -C docs
View Options

requirements-swh.txt

[tox] [tox]
envlist=black,flake8,mypy,py3-{core,db,server} envlist=black,flake8,mypy,py3-{core,db,github,server}
[testenv] [testenv]
passenv = PYTHONASYNCIODEBUG passenv = PYTHONASYNCIODEBUG
extras = extras =
testing-core testing-core
core: logging core: logging
db: db db: db
github: github
server: http server: http
deps = deps =
cover: pytest-cov cover: pytest-cov
commands = commands =
pytest --doctest-modules \ pytest --doctest-modules \
slow: --hypothesis-profile=slow \ slow: --hypothesis-profile=slow \
cover: --cov={envsitepackagesdir}/swh/core --cov-branch \ cover: --cov={envsitepackagesdir}/swh/core --cov-branch \
core: {envsitepackagesdir}/swh/core/tests \ core: {envsitepackagesdir}/swh/core/tests \
github: {envsitepackagesdir}/swh/core/github/tests \
db: {envsitepackagesdir}/swh/core/db/tests \ db: {envsitepackagesdir}/swh/core/db/tests \
server: {envsitepackagesdir}/swh/core/api/tests \ server: {envsitepackagesdir}/swh/core/api/tests \
{posargs} {posargs}
[testenv:py3] [testenv:py3]
skip_install = true skip_install = true
deps = tox deps = tox
commands = tox -e py3-core-db-server-slow-cover -- {posargs} commands = tox -e py3-core-db-server-slow-cover -- {posargs}
Show All 13 Lines
commands = commands =
{envpython} -m flake8 {envpython} -m flake8
[testenv:mypy] [testenv:mypy]
extras = extras =
testing-core testing-core
logging logging
db db
github
http http
deps = deps =
mypy==0.942 mypy==0.942
commands = commands =
mypy swh mypy swh
# build documentation outside swh-environment using the current # build documentation outside swh-environment using the current
# git HEAD of swh-docs, is executed on CI for each diff to prevent # git HEAD of swh-docs, is executed on CI for each diff to prevent
Show All 22 Lines
# of swh-docs package # of swh-docs package
[testenv:sphinx-dev] [testenv:sphinx-dev]
whitelist_externals = make whitelist_externals = make
usedevelop = true usedevelop = true
extras = extras =
testing-core testing-core
logging logging
db db
github
http http
deps = deps =
# install swh-docs in develop mode # install swh-docs in develop mode
-e ../swh-docs -e ../swh-docs
setenv = setenv =
SWH_PACKAGE_DOC_TOX_BUILD = 1 SWH_PACKAGE_DOC_TOX_BUILD = 1
# turn warnings into errors # turn warnings into errors
SPHINXOPTS = -W SPHINXOPTS = -W
commands = commands =
make -I ../.tox/sphinx-dev/src/swh-docs/swh/ -C docs make -I ../.tox/sphinx-dev/src/swh-docs/swh/ -C docs
View Options

swh/vault/api/server.py

[tox] [tox]
envlist=black,flake8,mypy,py3-{core,db,server} envlist=black,flake8,mypy,py3-{core,db,github,server}
[testenv] [testenv]
passenv = PYTHONASYNCIODEBUG passenv = PYTHONASYNCIODEBUG
extras = extras =
testing-core testing-core
core: logging core: logging
db: db db: db
github: github
server: http server: http
deps = deps =
cover: pytest-cov cover: pytest-cov
commands = commands =
pytest --doctest-modules \ pytest --doctest-modules \
slow: --hypothesis-profile=slow \ slow: --hypothesis-profile=slow \
cover: --cov={envsitepackagesdir}/swh/core --cov-branch \ cover: --cov={envsitepackagesdir}/swh/core --cov-branch \
core: {envsitepackagesdir}/swh/core/tests \ core: {envsitepackagesdir}/swh/core/tests \
github: {envsitepackagesdir}/swh/core/github/tests \
db: {envsitepackagesdir}/swh/core/db/tests \ db: {envsitepackagesdir}/swh/core/db/tests \
server: {envsitepackagesdir}/swh/core/api/tests \ server: {envsitepackagesdir}/swh/core/api/tests \
{posargs} {posargs}
[testenv:py3] [testenv:py3]
skip_install = true skip_install = true
deps = tox deps = tox
commands = tox -e py3-core-db-server-slow-cover -- {posargs} commands = tox -e py3-core-db-server-slow-cover -- {posargs}
Show All 13 Lines
commands = commands =
{envpython} -m flake8 {envpython} -m flake8
[testenv:mypy] [testenv:mypy]
extras = extras =
testing-core testing-core
logging logging
db db
github
http http
deps = deps =
mypy==0.942 mypy==0.942
commands = commands =
mypy swh mypy swh
# build documentation outside swh-environment using the current # build documentation outside swh-environment using the current
# git HEAD of swh-docs, is executed on CI for each diff to prevent # git HEAD of swh-docs, is executed on CI for each diff to prevent
Show All 22 Lines
# of swh-docs package # of swh-docs package
[testenv:sphinx-dev] [testenv:sphinx-dev]
whitelist_externals = make whitelist_externals = make
usedevelop = true usedevelop = true
extras = extras =
testing-core testing-core
logging logging
db db
github
http http
deps = deps =
# install swh-docs in develop mode # install swh-docs in develop mode
-e ../swh-docs -e ../swh-docs
setenv = setenv =
SWH_PACKAGE_DOC_TOX_BUILD = 1 SWH_PACKAGE_DOC_TOX_BUILD = 1
# turn warnings into errors # turn warnings into errors
SPHINXOPTS = -W SPHINXOPTS = -W
commands = commands =
make -I ../.tox/sphinx-dev/src/swh-docs/swh/ -C docs make -I ../.tox/sphinx-dev/src/swh-docs/swh/ -C docs
View Options

swh/vault/backend.py

[tox] [tox]
envlist=black,flake8,mypy,py3-{core,db,server} envlist=black,flake8,mypy,py3-{core,db,github,server}
[testenv] [testenv]
passenv = PYTHONASYNCIODEBUG passenv = PYTHONASYNCIODEBUG
extras = extras =
testing-core testing-core
core: logging core: logging
db: db db: db
github: github
server: http server: http
deps = deps =
cover: pytest-cov cover: pytest-cov
commands = commands =
pytest --doctest-modules \ pytest --doctest-modules \
slow: --hypothesis-profile=slow \ slow: --hypothesis-profile=slow \
cover: --cov={envsitepackagesdir}/swh/core --cov-branch \ cover: --cov={envsitepackagesdir}/swh/core --cov-branch \
core: {envsitepackagesdir}/swh/core/tests \ core: {envsitepackagesdir}/swh/core/tests \
github: {envsitepackagesdir}/swh/core/github/tests \
db: {envsitepackagesdir}/swh/core/db/tests \ db: {envsitepackagesdir}/swh/core/db/tests \
server: {envsitepackagesdir}/swh/core/api/tests \ server: {envsitepackagesdir}/swh/core/api/tests \
{posargs} {posargs}
[testenv:py3] [testenv:py3]
skip_install = true skip_install = true
deps = tox deps = tox
commands = tox -e py3-core-db-server-slow-cover -- {posargs} commands = tox -e py3-core-db-server-slow-cover -- {posargs}
Show All 13 Lines
commands = commands =
{envpython} -m flake8 {envpython} -m flake8
[testenv:mypy] [testenv:mypy]
extras = extras =
testing-core testing-core
logging logging
db db
github
http http
deps = deps =
mypy==0.942 mypy==0.942
commands = commands =
mypy swh mypy swh
# build documentation outside swh-environment using the current # build documentation outside swh-environment using the current
# git HEAD of swh-docs, is executed on CI for each diff to prevent # git HEAD of swh-docs, is executed on CI for each diff to prevent
Show All 22 Lines
# of swh-docs package # of swh-docs package
[testenv:sphinx-dev] [testenv:sphinx-dev]
whitelist_externals = make whitelist_externals = make
usedevelop = true usedevelop = true
extras = extras =
testing-core testing-core
logging logging
db db
github
http http
deps = deps =
# install swh-docs in develop mode # install swh-docs in develop mode
-e ../swh-docs -e ../swh-docs
setenv = setenv =
SWH_PACKAGE_DOC_TOX_BUILD = 1 SWH_PACKAGE_DOC_TOX_BUILD = 1
# turn warnings into errors # turn warnings into errors
SPHINXOPTS = -W SPHINXOPTS = -W
commands = commands =
make -I ../.tox/sphinx-dev/src/swh-docs/swh/ -C docs make -I ../.tox/sphinx-dev/src/swh-docs/swh/ -C docs
View Options

swh/vault/tests/vault_testing.py

Loading...
About · Developer information · Legal