Page MenuHomeSoftware Heritage

Migrate swh-storage api functions relative to indexers to swh-indexer
Closed, MigratedEdits Locked

Description

As swh-storage is the reflection of the db softwareheritage, this is no longer swh-storage's concern to open those api endpoints.
Those need to be migrated in swh-indexer.

As the endpoints migrate, it's important the tests continue to pass.
So we will move away the tests as well (local + remote storage).

In our daily developer live, we will also need to boostrap the swh-indexer db, first for the tests to pass, second for people to actually be able to run indexers locally.

As swh-web is currently reading indexed information on contents, this is also impacted.
We will add a new collaborator to the swh.web.service for accessing the swh-indexer's storage.

Also, there is also a need which is transverse to both storage and indexer, the tool part (indexer_configuration in the indexer).
This needs to still exist in the storage as it is used in the swh-loader-core.
So that too is impacted. The part that uses it in production is only the loader-deposit though for now (without dev, only a bump in dependency).

This Impacts a lot of repository:

  • swh-storage
  • swh-indexer
  • swh-storage-testdata
  • swh-environment
  • swh-web
  • swh-loader-core
  • swh-loader-deposit

At the end of it all, everything should be transparent for the rest of the team.

That is:

cd swh-environment
bin/update
make rebuild-testdata
make test

should result in the same result as a prior run.

Related Objects

Event Timeline

ardumont updated the task description. (Show Details)
ardumont changed the status of subtask T872: Deploy and restart indexers from Open to Work in Progress.