Page MenuHomeSoftware Heritage

staging: Migrate remaining static services to elastic workers
Started, Work in Progress, HighPublic

Description

  1. Creating venvs/docker images for remaining services:
  2. D8500: swh.loader.mercurial
  3. swh.loader.cvs
  4. swh.loader.svn
  5. swh.vault.cookers
  6. swh.deposit.checkers
  7. swh.indexer journal client
  1. Reference service in staging charts:
  2. D8501: mercurial
  3. loader-cvs
  4. loader-svn
  5. loader-archive
  6. loader-cran
  7. loader-debian
  8. loader-deposit
  9. loader-nixguix
  10. loader-opam
  11. remaining (staging) listers
  12. T4540#91856: vault cooker
  13. T4540#92071: deposit checker
  14. swh indexer journal client
  1. Decomission loader in staging static workers:
  2. loader-archive
  3. loader-cran
  4. loader-cvs
  5. loader-debian
  6. loader-deposit
  7. loader-svn
  8. loader-git
  9. loader-bzr
  10. loader-highpriority
  11. loader-maven
  12. loader-mercurial
  13. loader-npm
  14. loader-nixguix
  15. loader-opam
  16. loader-pypi
  17. remaining staging listers
  18. vault cookers
  19. deposit checker
  20. low hanging fruit indexers: swh indexer journal client
  21. migrate remaining fossology license indexer?

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ardumont changed the task status from Open to Work in Progress.Wed, Sep 21, 11:23 PM
ardumont moved this task from Backlog to in-progress on the System administration board.
ardumont renamed this task from staging: Migrate remaining static loader services to elastic workers to staging: Migrate remaining static loader/lister services to elastic workers.Thu, Sep 22, 9:45 AM

Remains a bunch of services not existing yet in the elastic stack (vault cooker, deposit checkers and indexer journal clients).
So the plan would be to integrate those in the elastic stack.

And then we could decomission all staging static workers altogether.

ardumont renamed this task from staging: Migrate remaining static loader/lister services to elastic workers to staging: Migrate remaining static services to elastic workers.Mon, Sep 26, 2:46 PM

Cookers migrated:

cookers [2022-09-26 14:48:17,948: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[722de41d-0f88-45f7-ad06-3fb65f3978e1] received
cookers [2022-09-26 14:48:54,924: INFO/ForkPoolWorker-1] Task swh.vault.cooking_tasks.SWHCookingTask[722de41d-0f88-45f7-ad06-3fb65f3978e1] succeeded in 36.83566682902165s: None
cookers [2022-09-26 14:54:28,340: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[0926cf58-6001-496c-99aa-d74f5c57175d] received
cookers [2022-09-26 14:54:30,130: INFO/ForkPoolWorker-2] Task swh.vault.cooking_tasks.SWHCookingTask[0926cf58-6001-496c-99aa-d74f5c57175d] succeeded in 1.7597111140494235s: None
cookers [2022-09-26 14:55:33,509: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[839c03c3-159f-46e9-9f19-56817866321a] received
cookers [2022-09-26 14:55:33,516: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[c6ad1f3c-31aa-4ef7-9f90-093d56011ddf] received
cookers [2022-09-26 14:56:34,268: INFO/ForkPoolWorker-3] Task swh.vault.cooking_tasks.SWHCookingTask[839c03c3-159f-46e9-9f19-56817866321a] succeeded in 60.72709982498782s: None
cookers [2022-09-26 14:56:41,961: INFO/ForkPoolWorker-4] Task swh.vault.cooking_tasks.SWHCookingTask[c6ad1f3c-31aa-4ef7-9f90-093d56011ddf] succeeded in 3.1700519190053456s: None

Deposit checkers migrated. Trigger icinga alert check on deposit, it went through the elastic workers [1] so done!

[1]

│ checker-deposit [2022-09-27 10:25:57,868: INFO/MainProcess] checker@checker-deposit-95b97647f-455jc ready.                                                                                                                                  │
│ checker-deposit [2022-09-27 10:26:40,799: INFO/MainProcess] Task swh.deposit.loader.tasks.ChecksDepositTsk[8dcc56b9-2ee8-48bc-a75f-126c83fdb871] received                                                                                   │
│ checker-deposit [2022-09-27 10:26:42,198: INFO/ForkPoolWorker-1] Task swh.deposit.loader.tasks.ChecksDepositTsk[8dcc56b9-2ee8-48bc-a75f-126c83fdb871] succeeded in 1.3463120240485296s: {'status': 'eventful'}                              │
│ checker-deposit [2022-09-27 10:27:46,933: INFO/MainProcess] Task swh.deposit.loader.tasks.ChecksDepositTsk[a6840e62-ae1d-4529-810d-c372f68c0fc4] received                                                                                   │
│ checker-deposit [2022-09-27 10:27:48,038: INFO/ForkPoolWorker-2] Task swh.deposit.loader.tasks.ChecksDepositTsk[a6840e62-ae1d-4529-810d-c372f68c0fc4] succeeded in 1.0527836499968544s: {'status': 'eventful'}
ardumont updated the task description. (Show Details)