Page MenuHomeSoftware Heritage

Puppetize subset of indexers to start indexing computations
Closed, MigratedEdits Locked

Description

We can already start the indexing computations (language, mimetype) of the already copied 6M contents stored in azure.
We don't have to wait for to have the ad-hoc implementation (T575) which triggers the indexing computations once the data is copied to azure (via the archiver-azure)

This task is to leverage T327's work to ease deploying worker.
This concerns 3 task queues workers: orchestrator (T576), mimetype (T577), language (T578).

For now:

  • swh-storage is packaged ready (will need to upgrade the db to install new endpoint functions)
  • swh-indexer is packaged ready as well.
  • a priori, puppet manifests locally ready but i'm unsure of some stuff (so not pushed) - i need to test first (concat private/public config for one, factorizing by inheriting for another to avoid definition conflict...)

I intend to use worker01.azure worker for that test.