Details

Reviewers

Group Reviewers

Maniphest Tasks

T1095: indexer: Remove temporary table usage for read-only queries

Commits

Summary

To permit to deploy and use indexer storage service on azure (plugged to a replica db).

Closes T1095
Related T1094

Separated commits:
storage: content_mimetype_missing: Migrate away from temp table
storage: content_mimetype_get: Migrate away from temp table
storage: content_language_missing: Migrate away from temp table
storage: content_language_get: Migrate away from temp table
storage: content_language_get: Migrate away from temp table
storage: content_ctags_missing: Migrate away from temp table
storage: content_ctags_get: Migrate away from temp table
storage: c_fossology_license_get: Migrate away from temp table
storage: content_metadata_missing: Migrate away from temp table
storage: content_metadata_get: Migrate away from temp table
swh.indexer.storage: Update docstrings information and format
storage: revision_metadata_missing: Migrate away from temp table
storage: revision_metadata_get: Migrate away from temp table
swh.indexer.storage: Remove dead entry points origin_metadata*

Test Plan

Tests ok

Diff Detail

Repository

rDCIDX Metadata indexer

Branch

migrate-away-from-temp-table-in-read-only-queries

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 1250
Build 1594: arc lint + arc unit

Event Timeline

ardumont created this revision.Jun 11 2018, 2:31 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJun 11 2018, 2:31 PM

Harbormaster completed remote builds in B1246: Diff 1095.Jun 11 2018, 2:31 PM

ardumont retitled this revision from indexer: Migrate away from temporary table To permit to use the indexer storage service (on azure) plugged to a replica db. Closes T1095 Related T1094 to indexer: Migrate away from temporary table.Jun 11 2018, 2:34 PM

ardumont edited the summary of this revision. (Show Details)

ardumont added a task: T1095: indexer: Remove temporary table usage for read-only queries.

Remove no longer needed code

swh.indexer.storage: Remove usused import
swh.indexer.storage: Remove no longer used endpoints

Harbormaster completed remote builds in B1247: Diff 1096.Jun 11 2018, 2:46 PM

Refactoring

swh.indexer.storage.db: Refactor using a missing from list pattern
swh.indexer.storage.db: Refactor using a get from list pattern

Harbormaster completed remote builds in B1248: Diff 1097.Jun 11 2018, 3:17 PM

swh.indexer.storage: Fix typo in invariable name metadata

Harbormaster completed remote builds in B1249: Diff 1098.Jun 12 2018, 11:35 AM

I get a test failure in content_ctags_get (both remote and local). It seems that the order of returned lists of ctags has changed and the assertEqual should be an assertCountEqual (compare lists regardless of order). This might warrant a review of other tests.

In D343#6715, @olasd wrote:

I get a test failure in content_ctags_get (both remote and local). It seems that the order of returned lists of ctags has changed and the assertEqual should be an assertCountEqual (compare lists regardless of order).

You are right.

I see that an order by line got dropped in the transition (fortunately, i have independent commits for each stored procedure, so it's apparent).

What worries me is that i cannot reproduce that failure... No matter how many times i trigger the tests.

This might warrant a review of other tests.

Well, the commits first.

Fix and rebase work

storage: content_language_get: Migrate away from temp table
storage: content_ctags_missing: Migrate away from temp table
storage: content_ctags_get: Migrate away from temp table
storage: c_fossology_license_get: Migrate away from temp table
storage: content_metadata_missing: Migrate away from temp table
storage: content_metadata_get: Migrate away from temp table
swh.indexer.storage: Update docstrings information and format
storage: revision_metadata_missing: Migrate away from temp table
storage: revision_metadata_get: Migrate away from temp table
swh.indexer.storage: Remove dead entry points origin_metadata*
swh.indexer.storage: Remove usused import
swh.indexer.storage: Remove no longer used endpoints
swh.indexer.storage.db: Refactor using a missing from list pattern
swh.indexer.storage.db: Refactor using a get from list pattern
swh.indexer.storage: Fix typo in invariable name metadata

Harbormaster completed remote builds in B1250: Diff 1100.Jun 12 2018, 3:07 PM

I see that an order by line got dropped in the transition (fortunately, i have independent commits for each stored procedure, so it's apparent).

There, fixed.
Thanks for the heads up on this.

This might warrant a review of other tests.
Well, the commits first.

Done. Checked all endpoints (missing, get).
No other incongruities of the sort.

What worries me is that i cannot reproduce that failure... No matter how many times i trigger the tests.

I still don't get that though.

sql/swh-func.sql
361	This got dropped during the transition. It's now back.
swh/indexer/storage/db.py
168	Back where it belonged.

Because, as usual, as soon as i said "everything is fine", a typo (; wrongly placed in a query) slided into it...

storage: content_ctags_get: Migrate away from temp table
storage: c_fossology_license_get: Migrate away from temp table
storage: content_metadata_missing: Migrate away from temp table
storage: content_metadata_get: Migrate away from temp table
swh.indexer.storage: Update docstrings information and format
storage: revision_metadata_missing: Migrate away from temp table
storage: revision_metadata_get: Migrate away from temp table
swh.indexer.storage: Remove dead entry points origin_metadata*
swh.indexer.storage: Remove usused import
swh.indexer.storage: Remove no longer used endpoints
swh.indexer.storage.db: Refactor using a missing from list pattern
swh.indexer.storage.db: Refactor using a get from list pattern
swh.indexer.storage: Fix typo in invariable name metadata

Harbormaster completed remote builds in B1251: Diff 1101.Jun 12 2018, 3:17 PM

olasd accepted this revision.Jun 13 2018, 10:46 AM

This revision is now accepted and ready to land.Jun 13 2018, 10:46 AM

Closed by commit rDCIDX6311e37236f3: swh.indexer.storage: Fix typo in invariable name metadata (authored by ardumont). · Explain WhyJun 13 2018, 11:26 AM

This revision was automatically updated to reflect the committed changes.

indexer: Migrate away from temporary table
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 1100

sql/swh-func.sql

sql/swh-schema.sql

sql/upgrades/115.sql

swh/indexer/storage/init.py

swh/indexer/storage/api/client.py

swh/indexer/storage/db.py

swh/indexer/tests/storage/test_storage.py

indexer: Migrate away from temporary tableClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 1100

sql/swh-func.sql

sql/swh-schema.sql

sql/upgrades/115.sql

swh/indexer/storage/__init__.py

swh/indexer/storage/api/client.py

swh/indexer/storage/db.py

swh/indexer/tests/storage/test_storage.py

indexer: Migrate away from temporary table
ClosedPublic
Actions

Revision Contents
Changeset List

swh/indexer/storage/init.py