Page MenuHomeSoftware Heritage
Differential D631

Make Storage, Db, and tests more modular.
AbandonedPublic
Actions

Authored by vlorentz on Nov 7 2018, 11:26 AM.

Details

Reviewers
None
Group Reviewers
Reviewers
Summary

To prepare for a new backend that doesn't use postgresql. First step toward T1306.

Test Plan

'tox' still runs the same tests, and doesn't fail.

Diff Detail

Repository
rDSTO Storage manager
Branch
modular-pg
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 2255
Build 2711: tox-on-jenkinsJenkins
Build 2710: arc lint + arc unit

Event Timeline

vlorentz created this revision.Nov 7 2018, 11:26 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 7 2018, 11:26 AM
swh-public-ci added a subscriber: swh-public-ci.Nov 7 2018, 11:31 AM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/30/ for more details.

Harbormaster completed remote builds in B2247: Diff 1953.Nov 7 2018, 11:31 AM
vlorentz updated this revision to Diff 1954.Nov 7 2018, 11:34 AM
  • CommonTestStorage always needs DataTestStorage.
swh-public-ci added a comment.Nov 7 2018, 11:38 AM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/31/ for more details.

Harbormaster completed remote builds in B2248: Diff 1954.Nov 7 2018, 11:38 AM
vlorentz planned changes to this revision.Nov 7 2018, 11:57 AM
ardumont added a subscriber: ardumont.Nov 7 2018, 11:58 AM
ardumont added inline comments.
swh/storage/db.py
177

That will need a bump in storage and migration in other modules:

./swh-vault/.tox/py3/lib/python3.6/site-packages/swh/storage/tests/test_db.py:13:from swh.storage.db import Db
./swh-indexer/.tox/py3/lib/python3.6/site-packages/swh/storage/tests/test_db.py:13:from swh.storage.db import Db
./swh-storage/swh/storage/tests/test_db.py:13:from swh.storage.db import Db
./swh-lister/.tox/py3/lib/python3.6/site-packages/swh/storage/tests/test_db.py:13:from swh.storage.db import Db
./swh-journal/.tox/py3/lib/python3.6/site-packages/swh/storage/tests/test_db.py:13:from swh.storage.db import Db
swh/storage/storage.py
270

(e.g file whose size is too large)

swh/storage/tests/storage_testing.py
16

Same as for the db modules.
This implies bump and migration.

vlorentz updated this revision to Diff 1956.Nov 7 2018, 12:18 PM
  • Really move all pgsql-specific parts of Storage into PgStorage.
  • Apply @ardumont's comment
vlorentz marked an inline comment as done.Nov 7 2018, 12:19 PM
vlorentz added inline comments.
swh/storage/db.py
177

Do you want me to add an alias (with a DeprecationWarning)?

vlorentz updated this revision to Diff 1958.Nov 7 2018, 12:21 PM
  • Add missing metaclass.
swh-public-ci added a comment.Nov 7 2018, 12:22 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/32/ for more details.

Harbormaster completed remote builds in B2250: Diff 1956.Nov 7 2018, 12:22 PM
swh-public-ci added a comment.Nov 7 2018, 12:27 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/33/ for more details.

Harbormaster completed remote builds in B2252: Diff 1958.Nov 7 2018, 12:27 PM
vlorentz updated this revision to Diff 1959.Nov 7 2018, 12:31 PM
  • Move apply_options to BaseDb.
swh-public-ci added a comment.Nov 7 2018, 12:36 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/34/ for more details.

Harbormaster completed remote builds in B2253: Diff 1959.Nov 7 2018, 12:36 PM
ardumont added inline comments.Nov 7 2018, 12:56 PM
swh/storage/db.py
177

It was a jsyk basis.
It may be cleaner to do so though so let's say yes.

I expect the ci to go red fast when this hits master (maybe not though as we use pypi module now).

vlorentz updated this revision to Diff 1962.Nov 7 2018, 1:36 PM
  • Add aliases with a DeprecationWarning for compatibility.
swh-public-ci added a comment.Nov 7 2018, 1:41 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/35/ for more details.

Harbormaster completed remote builds in B2255: Diff 1962.Nov 7 2018, 1:41 PM
olasd mentioned this in D635: [WIP] In-memory storage implementation.Nov 7 2018, 5:35 PM
vlorentz abandoned this revision.Nov 7 2018, 6:49 PM

Abandonned in favor of D635.

Revision Contents
Changeset List

PathSize
swh/
storage/
6 lines
28 lines
157 lines
240 lines
231 lines
tests/
algos/
4 lines
12 lines
20 lines
4 lines
51 lines

Diff 1962

View Options

swh/storage/__init__.py

Loading...
View Options

swh/storage/common.py

Loading...
View Options

swh/storage/db.py

Loading...
View Options

swh/storage/pg_storage.py

Loading...
View Options

swh/storage/storage.py

Loading...
View Options

swh/storage/tests/algos/test_snapshot.py

Loading...
View Options

swh/storage/tests/storage_testing.py

Loading...
View Options

swh/storage/tests/test_api_client.py

Loading...
View Options

swh/storage/tests/test_db.py

Loading...
View Options

swh/storage/tests/test_storage.py

Loading...
About · Developer information · Legal