Refactor output of indexer storage's `get` methods.
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	vlorentz
	Dec 6 2018, 4:04 PM

Description

In the Indexer Storage API, most get methods (eg. content_ctags_get) yield items with this format:

{"id": sha1, "tool": TOOL, "ctags": ctags1}
{"id": sha1, "tool": TOOL, "ctags": ctags2}

Starting with T782/D301, content_fossology_license_get yields item with this format:

{sha1: {"tool": TOOL, "licenses": [license1, license2]}}

This task is twofold:

first, improve content_fossology_license_get's result to return a dictionary instead of yielding dictionaries each with a single key-value
secondly, refactor other _get methods to use the same format.

The files that should be edited are:

swh/indexer/tests/storage/test_storage.py: this are the test cases for both Indexer Storage implementations. It should be adapted to test for the new format.
swh/indexer/storage/in_memory.py: a fully in-memory implementation of the Indexer Storage. This is the easiest implementation to start with.
swh/indexer/storage/__init__.py and swh/indexer/storage/converters.py: an implementation of the Indexer Storage backed by postgresql. Look at D301 for examples of how to do it.

Related Objects
Search...

		Status	Assigned	Task
		Migrated	gitlab-migration	T1433 Refactor output of indexer storage's `get` methods.
		Migrated	gitlab-migration	T2652 Make the indexer-storage interface use attr classes instead of dicts

Event Timeline

vlorentz triaged this task as Low priority.Dec 6 2018, 4:04 PM

vlorentz created this task.

vlorentz updated the task description. (Show Details)Dec 6 2018, 4:07 PM

vlorentz updated the task description. (Show Details)

vlorentz edited projects, added Good first contribution; removed Easy hack.Dec 13 2018, 12:33 PM

vlorentz raised the priority of this task from Low to Normal.Dec 13 2018, 1:56 PM

vlorentz edited projects, added Easy hack; removed Good first contribution.Dec 13 2018, 3:07 PM

vlorentz mentioned this in D876: Make tests actually run..Dec 21 2018, 3:34 PM

Sowmya added a subscriber: Sowmya.Mar 9 2019, 3:43 AM

I am familiar with the web APIs and I went through the discussion in T782. When you say output a single dictionary, I believe you mean something like this

{
  sha1: [
    {tool: TOOL, licenses: [licences]},
    {tool: TOOL, licenses: [licences]}
  ],

  sha1: [
    {tool: TOOL, licenses: [licences]},
    {tool: TOOL, licenses: [licences]}
  ]
}

Following the setup guide I have hosted indexer locally and will start refactoring all the APIs one by one.

vlorentz updated the task description. (Show Details)Jul 8 2019, 1:25 PM

I went through all the tests in test_storage.py. It appears that only content_fossology_license_get needs to be refactored. All other storage methods return a dictionary or a list of dictionaries, where each dictionary has multiple keys.

twitu mentioned this in D1720: Modify API output and test.Jul 10 2019, 5:31 PM

vlorentz changed the task status from Open to Work in Progress.Oct 1 2020, 12:40 PM

vlorentz claimed this task.

vlorentz added a subtask: T2652: Make the indexer-storage interface use attr classes instead of dicts.Oct 6 2020, 10:36 AM

vlorentz closed this task as Resolved.Oct 9 2020, 8:41 PM

vlorentz closed subtask T2652: Make the indexer-storage interface use attr classes instead of dicts as Resolved.