update indexer for storage 0.0.156
ClosedPublic
Actions

Authored by douardda on Oct 31 2019, 2:08 PM.

Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Commits

rDCIDXd27cd133ac34: update indexer for storage 0.0.156

Summary

this imply a refactoring of the db schema for origin_intrinsic_metadata, since
we do not have nor want numerical ids for origins, we use origin urls instead.

Note that IndexerStorage.origin_intrinsic_metadata_search_by_producer() still
have start/end arguments which are expected to be strings, thus uses
lexicographic comparisons between origin urls. This is far from
ideal, but a proper fix requires an (new?) endpoint that handle pagination
properly.

Depends on D2206

Diff Detail

Repository

rDCIDX Metadata indexer

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 8737
Build 12732: tox-on-jenkins	Jenkins
Build 12731: arc lint + arc unit

Event Timeline

douardda created this revision.Oct 31 2019, 2:08 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptOct 31 2019, 2:08 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/637/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/637/console

Harbormaster failed remote builds in B8737: Diff 7593!Oct 31 2019, 2:13 PM

vlorentz requested changes to this revision.Oct 31 2019, 3:35 PM

vlorentz added a subscriber: vlorentz.

vlorentz added inline comments.

swh/indexer/storage/__init__.py
695	singular (same for other occurences)
786	Make sure `swh indexer schedule reindex_origin_metadata` still works after this change. You should also probably explain quickly in the docstring how to paginate properly now.
792–793	must be updated
swh/indexer/storage/in_memory.py
716	forgot to change `id`
swh/indexer/tests/storage/test_storage.py
453–455	`self.origin_url_`
1556–1571	that doesn't test how a consumer of the API would use it. It needs to start with `start='url'`, then incrementally get new results using only the results from the previous call.

This revision now requires changes to proceed.Oct 31 2019, 3:35 PM

fix almost all vlorentz' comments + fix in_memory's SubStorage.get_all()

douardda marked 4 inline comments as done.Oct 31 2019, 4:42 PM

douardda added inline comments.

swh/indexer/storage/__init__.py
786	You should also probably explain quickly in the docstring how to paginate properly now. unfortunately I have no idea how to do such a thing.

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/638/ for more details.

Harbormaster completed remote builds in B8742: Diff 7594.Oct 31 2019, 4:43 PM

vlorentz requested changes to this revision.Nov 4 2019, 12:05 PM

vlorentz added inline comments.

swh/indexer/storage/__init__.py
786	Using the last result of the previous call as a `start`. It also means that this endpoint should return only results strictly greater than `start`.
swh/indexer/tests/storage/test_storage.py
1556–1571	Sorry, I meant it should start with `start=''`.

This revision now requires changes to proceed.Nov 4 2019, 12:05 PM

douardda added inline comments.Nov 4 2019, 4:18 PM

swh/indexer/tests/storage/test_storage.py
1556–1571	I agree, but I just replicated what the tests used to do. Once again, I believe the kind of modifications you ask for should be in a dedicated diff.