Change Details

It's running slowly for now as the current execution is simply a loop over the mapping file then execute the ingestion. We could make it go faster leveraging the loader_oneshot and rabbitmq. For this, we need to iterate over the mapping file and send tasks `swh.loader.mercurial.tasks.LoadMercurial` with the proper parameters. And that should be enough as the loader_oneshot already consumes those tasks ^. That will allow to: - make the ingestion go faster (since the loader_oneshot is currently configured with a concurrency of 7). - have the logs of what actually happens/happened browsable [1] [1] http://kibana0.internal.softwareheritage.org:5601/goto/23e431d304850e45e602752e849a3f28

The current execution is running slowly for now. It's a simple loop over the mapping file (one line is one origin) to trigger the ingestion per origin. Make it go faster using the actual loader_oneshot. For this, we need to iterate over the mapping file and send tasks `swh.loader.mercurial.tasks.LoadMercurial` with the proper parameters. That also have the following pros: - no need to deploy another special worker - have the systemctl logs pushed to elk, which make it browsable more easily (at least for staff) [1] [1] http://kibana0.internal.softwareheritage.org:5601/goto/23e431d304850e45e602752e849a3f28

It'The current execution is running slowly for now as the current execution is. It's a simply ae loop over the mapping file then execute the ingestion. We could make it go faster leveraging the loader_oneshot and rabbitmq(one line is one origin) to trigger the ingestion per origin. For this, we need to iterate over the mapping file and send tasks `swh.loader.mercurial.tasks.LoadMercurial` with the proper parametersMake it go faster using the actual loader_oneshot. And that should be enough as the loader_oneshot already consumes those tasks ^For this, we need to iterate over the mapping file and send tasks `swh.loader.mercurial.tasks.LoadMercurial` with the proper parameters. That will aalso have the follow to:ing pros: - make the ingestion go faster (since the loader_oneshot is currently configured with a concurrency of 7).- no need to deploy another special worker - have the logs of what actually happens/happened browsable [1systemctl logs pushed to elk, which make it browsable more easily (at least for staff) [1] [1] http://kibana0.internal.softwareheritage.org:5601/goto/23e431d304850e45e602752e849a3f28