Maniphest T2652

Make the indexer-storage interface use attr classes instead of dicts
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	vlorentz
	Oct 1 2020, 12:09 PM

Tags

Subscribers

Description

This is needed to use the journal writer as it is currently designed.

Additionally, it goes toward our goal of typing the codebase.

Revisions and Commits

rDWAPPS Web applications
	D4190	rDWAPPS2630a6dd1121 Update to work with swh.storage.indexer 0.3.
rDCIDX Metadata indexer
	D4189	rDCIDXc6afd9d798a4 Properly define the type of index() methods of indexers.
	D4188	rDCIDX4a7292a8a816 tests: Clean up compatibility code with endpoints that return dicts.
	D4187	rDCIDX8c1b288b6298 use OriginIntrinsicMetadataRow in the storage interface instead of dicts.
	D4180	rDCIDXa822ada3a012 use ContentMetadataRow in the storage interface instead of dicts.
	D4184	rDCIDXf954853a4cd8 use RevisionIntrinsicMetadataRow in the storage interface instead of dicts.
	D4176	rDCIDXaf3c220b14e2 use ContentLanguageRow in the storage interface instead of dicts.
	D4177	rDCIDX7566fec49a46 use ContentCtagsRow in the storage interface instead of dicts.
	D4174	rDCIDXb4d084d675bb indexer.storage: Change return types from Iterable to List
	D4173	rDCIDX570816fd87a1 license: use ContentLicenseRow in the storage interface instead of dicts.
	D4172	rDCIDX7fe4a89dbcf2 base indexers: add type annotation for self.{storage,idx_storage}.
	D4171	rDCIDX486ee085f5ee indexer.storage: Update docstrings of mimetype-related endpoints.
	D4170	rDCIDX4ec112337909 indexer.storage: Change return type annotation from Iterator to Iterable.
	D4169	rDCIDXe8e94cf23747 tests: Enable type-checking on storage test functions.
	D4159	rDCIDX44cee8f213f2 Make base indexers generic, with the result of index() as their type parameter.
	D4158	rDCIDXc3caf3008300 mimetype: use ContentMimetypeRow in the storage interface instead of dicts.

Related Objects
Search...

		Status	Assigned	Task
		Open	None	T2213 Storage
		Work in Progress	vlorentz	T2214 Scale-out graph and database storage in production
		Open	vlorentz	T1892 Cassandra as a storage backend
		Open	None	T1523 Search tools on metadata
		Resolved	vlorentz	T1117 Origin search is slow when you look for very common words
		Resolved	vlorentz	T1910 Redesign origin search using a dedicated component (swh-search)
		Resolved	olasd	T2052 Publish swh-search on PyPI
		Resolved	ardumont	T2167 Deploy swh-search
		Duplicate	olasd	T2174 Add debian package for swh-search
		Resolved	vsellier	T2182 Switch production swh-web to use swh-search instead of postgresql search.
		Resolved	vlorentz	T2590 Finish the indexer -> swh-search pipeline
		Resolved	vlorentz	T2651 Make the indexer-storage publish its rows to Kafka
		Resolved	vlorentz	T1433 Refactor output of indexer storage's `get` methods.
		Resolved	vlorentz	T2652 Make the indexer-storage interface use attr classes instead of dicts

Event Timeline

vlorentz triaged this task as Normal priority.Oct 1 2020, 12:09 PM

vlorentz created this task.

vlorentz changed the task status from Open to Work in Progress.Oct 6 2020, 10:33 AM

vlorentz added a parent task: T1433: Refactor output of indexer storage's `get` methods..Oct 6 2020, 10:36 AM

vlorentz added revisions: D4190: Update to work with swh.storage.indexer 0.3., D4189: Properly define the type of index() methods of indexers., D4188: tests: Clean up compatibility code with endpoints that return dicts., D4187: use OriginIntrinsicMetadataRow in the storage interface instead of dicts., D4184: use RevisionIntrinsicMetadataRow in the storage interface instead of dicts., D4180: use ContentMetadataRow in the storage interface instead of dicts., D4177: use ContentCtagsRow in the storage interface instead of dicts., D4176: use ContentLanguageRow in the storage interface instead of dicts., D4174: indexer.storage: Change return types from Iterable to List, D4173: license: use ContentLicenseRow in the storage interface instead of dicts., D4172: base indexers: add type annotation for self.{storage,idx_storage}., D4171: indexer.storage: Update docstrings of mimetype-related endpoints., D4170: indexer.storage: Change return type annotation from Iterator to Iterable., D4169: tests: Enable type-checking on storage test functions., D4159: Make base indexers generic, with the result of index() as their type parameter., D4158: mimetype: use ContentMimetypeRow in the storage interface instead of dicts..Oct 7 2020, 5:58 PM

vlorentz added a commit: rDCIDXc3caf3008300: mimetype: use ContentMimetypeRow in the storage interface instead of dicts..Oct 8 2020, 1:01 PM

vlorentz added a commit: rDCIDX44cee8f213f2: Make base indexers generic, with the result of index() as their type parameter..

vlorentz added a commit: rDCIDXe8e94cf23747: tests: Enable type-checking on storage test functions..

vlorentz added a commit: rDCIDX4ec112337909: indexer.storage: Change return type annotation from Iterator to Iterable..

vlorentz added a commit: rDCIDX486ee085f5ee: indexer.storage: Update docstrings of mimetype-related endpoints..

vlorentz added a commit: rDCIDX7fe4a89dbcf2: base indexers: add type annotation for self.{storage,idx_storage}..

vlorentz added a commit: rDCIDX570816fd87a1: license: use ContentLicenseRow in the storage interface instead of dicts..

vlorentz added a commit: rDCIDXb4d084d675bb: indexer.storage: Change return types from Iterable to List.

vlorentz added a commit: rDCIDXaf3c220b14e2: use ContentLanguageRow in the storage interface instead of dicts..

vlorentz added a commit: rDCIDX7566fec49a46: use ContentCtagsRow in the storage interface instead of dicts..

vlorentz added a commit: rDCIDXf954853a4cd8: use RevisionIntrinsicMetadataRow in the storage interface instead of dicts..

vlorentz added a commit: rDCIDXa822ada3a012: use ContentMetadataRow in the storage interface instead of dicts..

vlorentz added a commit: rDCIDX8c1b288b6298: use OriginIntrinsicMetadataRow in the storage interface instead of dicts..

vlorentz added a commit: rDCIDX4a7292a8a816: tests: Clean up compatibility code with endpoints that return dicts..

vlorentz added a commit: rDCIDXc6afd9d798a4: Properly define the type of index() methods of indexers..

vlorentz added a commit: rDWAPPS2630a6dd1121: Update to work with swh.storage.indexer 0.3..Oct 8 2020, 1:36 PM

vlorentz closed this task as Resolved.Oct 9 2020, 8:41 PM

\o/