It'The current execution is running slowly for now as the current execution is. It's a simply ae loop over the mapping
file then execute the ingestion. We could make it go faster leveraging the
loader_oneshot and rabbitmq(one line is one origin) to trigger the ingestion per origin.
For this, we need to iterate over the mapping file and send tasks `swh.loader.mercurial.tasks.LoadMercurial`
with the proper parametersMake it go faster using the actual loader_oneshot.
And that should be enough as the loader_oneshot already consumes those tasks ^For this, we need to iterate over the mapping file and send tasks
`swh.loader.mercurial.tasks.LoadMercurial` with the proper parameters.
That will aalso have the follow to:ing pros:
- make the ingestion go faster (since the loader_oneshot is currently configured with a concurrency of 7).- no need to deploy another special worker
- have the logs of what actually happens/happened browsable [1systemctl logs pushed to elk, which make it browsable more easily (at least for staff)