Page MenuHomeSoftware Heritage
Create Task
Maniphest T4540

staging: Migrate remaining static services to elastic workers
Closed, MigratedEdits Locked
Actions

Description

  1. Creating venvs/docker images for remaining services:
  2. D8500: swh.loader.mercurial
  3. swh.loader.cvs
  4. swh.loader.svn
  5. swh.vault.cookers
  6. swh.deposit.checkers
  7. swh.indexer journal client
  1. Reference service in staging charts:
  2. D8501: mercurial
  3. loader-cvs
  4. loader-svn
  5. loader-archive
  6. loader-cran
  7. loader-debian
  8. loader-deposit
  9. loader-nixguix
  10. loader-opam
  11. remaining (staging) listers
  12. T4540#91856: vault cooker
  13. T4540#92071: deposit checker
  14. swh indexer journal client
  1. Decomission loader in staging static workers:
  2. loader-archive
  3. loader-cran
  4. loader-cvs
  5. loader-debian
  6. loader-deposit
  7. loader-svn
  8. loader-git
  9. loader-bzr
  10. loader-highpriority
  11. loader-maven
  12. loader-mercurial
  13. loader-npm
  14. loader-nixguix
  15. loader-opam
  16. loader-pypi
  17. remaining staging listers
  18. vault cookers
  19. deposit checker
  20. low hanging fruit indexers: swh indexer journal client
  21. migrate remaining fossology license indexer

Revisions and Commits

rDDEP Push deposit
D8545rDDEP730e81d8dd7e requirements-swh: Add missing dependency
R260 Helm charts for swh packages
D8501R260:6a7227c0f75d staging: Deploy loader mercurial worker
R259 Software Heritage virtual environment packaging manifests
D8500R259:e2a5c87156af swh-loader-mercurial: Generate new frozen requirements
D8500R259:ee655026e77b Add new swh-loader-mercurial application

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ardumont renamed this task from staging: Migrate remaining static loader services to elastic workers to staging: Migrate remaining static loader/lister services to elastic workers.Sep 22 2022, 9:45 AM
ardumont mentioned this in R260:a7c5b3412f9c: Activate ack late messages for add forge and save code now workers.Sep 22 2022, 10:07 AM
ardumont mentioned this in R260:87d45843abcf: keda: Switch to http connection.Sep 22 2022, 10:24 AM
ardumont mentioned this in R260:5371357ee50f: keda: Switch to http connection for lister as well.Sep 22 2022, 10:42 AM
ardumont mentioned this in rSPRE82a911b6558b: staging: Drop static worker3 node.Sep 22 2022, 11:38 AM
ardumont mentioned this in R260:17ef999bb77c: loaders: Ensure celery only receives one SIGTERM call.Sep 22 2022, 3:12 PM
ardumont mentioned this in R260:39a9fa5c10ab: loaders: Ensure celery only receives one SIGTERM call.Sep 22 2022, 3:27 PM
ardumont mentioned this in R260:69588b4a3f3e: loaders: Ensure celery only receives one SIGTERM signal.Sep 22 2022, 3:32 PM
ardumont mentioned this in R260:ba5d8c6ae212: loaders: Ensure celery only receives one SIGTERM signal.Sep 22 2022, 3:44 PM
ardumont mentioned this in R260:eb1c68907da4: Activate ack late messages for add forge and save code now workers.Sep 22 2022, 5:28 PM
ardumont mentioned this in R260:76e31254176e: keda: Switch to http connection.
ardumont mentioned this in R260:54830ad5bf88: keda: Switch to http connection for lister as well.
ardumont mentioned this in R260:27c05c8b290a: loaders: Ensure celery only receives one SIGTERM signal.
ardumont mentioned this in R260:c9540946d51d: staging: Decrease replica.Sep 22 2022, 5:30 PM
ardumont updated the task description. (Show Details)Sep 22 2022, 5:32 PM
ardumont mentioned this in R260:f5b96baa9290: loaders: Ensure celery only receives one SIGTERM signal.Sep 22 2022, 5:39 PM
ardumont mentioned this in R260:7a288f5d9975: staging: Decrease replica.
ardumont mentioned this in R260:005d3c0c37e3: listers: Ensure celery only receives one SIGTERM signal.
ardumont mentioned this in rSPREf00227d8965e: staging: Instantiate new elastic rancher-node-worker4 node.Sep 23 2022, 10:02 AM
ardumont mentioned this in rSPRE83dbdd20b841: staging: Match current staging workers migrated to pompidou.
ardumont mentioned this in R260:0e97c5b68f1f: staging: Decrease concurrency.Sep 23 2022, 3:52 PM
ardumont added a comment.Sep 23 2022, 5:50 PM

Remains a bunch of services not existing yet in the elastic stack (vault cooker, deposit checkers and indexer journal clients).
So the plan would be to integrate those in the elastic stack.

And then we could decomission all staging static workers altogether.

ardumont renamed this task from staging: Migrate remaining static loader/lister services to elastic workers to staging: Migrate remaining static services to elastic workers.Sep 26 2022, 2:46 PM
ardumont updated the task description. (Show Details)Sep 26 2022, 2:48 PM
ardumont mentioned this in R259:405168a76920: Add new swh-vault-cookers application.Sep 26 2022, 3:01 PM
ardumont mentioned this in R259:28c753452dec: swh.vault: Generate new frozen requirements from v1.8.
ardumont mentioned this in R260:e37a1c2dfcbd: staging: Deploy new vault cooker services.Sep 26 2022, 3:29 PM
ardumont mentioned this in rSPSITEc9ad6c20bf3a: staging: Migrate vault cookers to elastic workers.Sep 26 2022, 3:31 PM
ardumont mentioned this in R260:ed886a09cf0c: staging: Fix default vault cooker configuration.Sep 26 2022, 4:37 PM
ardumont mentioned this in R259:05e3ac003605: swh.vault: Fix docker image and frozen dependencies.Sep 26 2022, 4:47 PM
ardumont mentioned this in R260:91e3ec89bb37: staging: Fix default vault cooker configuration.
ardumont added a comment.EditedSep 26 2022, 4:50 PM

Cookers migrated:

cookers [2022-09-26 14:48:17,948: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[722de41d-0f88-45f7-ad06-3fb65f3978e1] received
cookers [2022-09-26 14:48:54,924: INFO/ForkPoolWorker-1] Task swh.vault.cooking_tasks.SWHCookingTask[722de41d-0f88-45f7-ad06-3fb65f3978e1] succeeded in 36.83566682902165s: None
cookers [2022-09-26 14:54:28,340: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[0926cf58-6001-496c-99aa-d74f5c57175d] received
cookers [2022-09-26 14:54:30,130: INFO/ForkPoolWorker-2] Task swh.vault.cooking_tasks.SWHCookingTask[0926cf58-6001-496c-99aa-d74f5c57175d] succeeded in 1.7597111140494235s: None
cookers [2022-09-26 14:55:33,509: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[839c03c3-159f-46e9-9f19-56817866321a] received
cookers [2022-09-26 14:55:33,516: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[c6ad1f3c-31aa-4ef7-9f90-093d56011ddf] received
cookers [2022-09-26 14:56:34,268: INFO/ForkPoolWorker-3] Task swh.vault.cooking_tasks.SWHCookingTask[839c03c3-159f-46e9-9f19-56817866321a] succeeded in 60.72709982498782s: None
cookers [2022-09-26 14:56:41,961: INFO/ForkPoolWorker-4] Task swh.vault.cooking_tasks.SWHCookingTask[c6ad1f3c-31aa-4ef7-9f90-093d56011ddf] succeeded in 3.1700519190053456s: None
ardumont updated the task description. (Show Details)Sep 26 2022, 4:50 PM
ardumont mentioned this in R259:fbc67439329e: swh-loader-svn: Bump release and fix entrypoint.Sep 26 2022, 5:09 PM
ardumont mentioned this in R260:f8e026df8df8: staging: Release swh-loader-svn image.
ardumont mentioned this in R259:960328f70fc4: swh-loader-cvs: Bump release and fix entrypoint.Sep 26 2022, 5:14 PM
ardumont mentioned this in R259:168682ae06cd: swh-loader-package: Bump release and fix entrypoint.
ardumont mentioned this in R259:0922d567d80f: swh-loader-bzr: Bump release and fix entrypoint.
ardumont mentioned this in R259:247b040d4bc0: swh-loader-git: Bump release and fix entrypoint.Sep 26 2022, 5:17 PM
ardumont mentioned this in R259:0b4dc94acd64: swh-loader-mercurial: Bump release and fix entrypoint.
ardumont mentioned this in R260:b0d7995c17c7: staging: Bump new swh-loader-package image.Sep 26 2022, 5:21 PM
ardumont mentioned this in R260:98347a15536a: staging: Bump new loader images.Sep 26 2022, 5:31 PM
ardumont mentioned this in R259:695eff4687ae: swh-loader-highpriority: Bump new release to match other images.Sep 26 2022, 5:41 PM
ardumont mentioned this in R260:406651fd6f13: staging: Bump new loader highpriority image.
ardumont mentioned this in R260:a46ad6ca6123: cookers: Fix typo.Sep 26 2022, 6:09 PM
ardumont mentioned this in R260:41ef21449685: staging: Deploy new deposit checker service.Sep 27 2022, 10:42 AM
ardumont mentioned this in R260:f01f2733b446: staging: Deploy new deposit checker service.Sep 27 2022, 10:50 AM
ardumont mentioned this in R259:166123f5fb3a: Add new swh-deposit-checkers application.Sep 27 2022, 10:55 AM
ardumont mentioned this in R259:13392eb3f765: swh.deposit.checkers: Generate new frozen requirements from v1.2.
ardumont added a revision: D8545: requirements-swh: Add missing dependency.Sep 27 2022, 11:31 AM
ardumont mentioned this in R259:cd9808b1f9eb: swh.deposit.checkers: Generate new frozen requirements from v1.2.Sep 27 2022, 11:44 AM
ardumont mentioned this in R260:0941eb6613ab: staging: Deploy fixed deposit checker image.
ardumont added a commit: rDDEP730e81d8dd7e: requirements-swh: Add missing dependency.Sep 27 2022, 11:46 AM
ardumont mentioned this in R260:9699a5e982c0: staging: Deploy fixed deposit checker image.Sep 27 2022, 11:55 AM
ardumont mentioned this in R260:bc4dcb56a8f8: swh/values: Disable new services by default.
ardumont mentioned this in rSPSITEc06913c4e48c: staging: Migrate deposit checker service to elastic workers.Sep 27 2022, 12:11 PM
ardumont mentioned this in R260:773cc7cbdc5e: staging: Fix checker_deposit configmap deposit host entry.Sep 27 2022, 12:19 PM
ardumont mentioned this in R260:4f33c608c49e: staging: Fix checker_deposit configmap secret mount.Sep 27 2022, 12:24 PM
ardumont mentioned this in R260:e5384408c683: staging: Fix checker_deposit configmap configuration.
ardumont updated the task description. (Show Details)EditedSep 27 2022, 12:31 PM

Deposit checkers migrated. Trigger icinga alert check on deposit, it went through the elastic workers [1] so done!

[1]

│ checker-deposit [2022-09-27 10:25:57,868: INFO/MainProcess] checker@checker-deposit-95b97647f-455jc ready.                                                                                                                                  │
│ checker-deposit [2022-09-27 10:26:40,799: INFO/MainProcess] Task swh.deposit.loader.tasks.ChecksDepositTsk[8dcc56b9-2ee8-48bc-a75f-126c83fdb871] received                                                                                   │
│ checker-deposit [2022-09-27 10:26:42,198: INFO/ForkPoolWorker-1] Task swh.deposit.loader.tasks.ChecksDepositTsk[8dcc56b9-2ee8-48bc-a75f-126c83fdb871] succeeded in 1.3463120240485296s: {'status': 'eventful'}                              │
│ checker-deposit [2022-09-27 10:27:46,933: INFO/MainProcess] Task swh.deposit.loader.tasks.ChecksDepositTsk[a6840e62-ae1d-4529-810d-c372f68c0fc4] received                                                                                   │
│ checker-deposit [2022-09-27 10:27:48,038: INFO/ForkPoolWorker-2] Task swh.deposit.loader.tasks.ChecksDepositTsk[a6840e62-ae1d-4529-810d-c372f68c0fc4] succeeded in 1.0527836499968544s: {'status': 'eventful'}
ardumont updated the task description. (Show Details)Sep 27 2022, 12:38 PM
ardumont updated the task description. (Show Details)
ardumont mentioned this in rSPREe1e530bf6d9d: staging: Decomission worker2.Sep 27 2022, 12:44 PM
ardumont mentioned this in rSPSITE461e9ea4edc0: staging: Decomission worker2.
ardumont mentioned this in R259:1e29474d9701: Add new indexer application.Sep 27 2022, 3:18 PM
ardumont mentioned this in R259:55223d622cb1: swh.deposit.checkers: Generate new frozen requirements from v2.6.
ardumont mentioned this in R260:accc6f6d6deb: staging: Fix loader deposit configmap configuration.Sep 27 2022, 3:41 PM
ardumont mentioned this in R260:5ac0fc14f538: staging: Deploy indexer journal client in elastic stack.Sep 27 2022, 3:45 PM
ardumont mentioned this in R260:3f327374bc98: staging: Deploy indexer journal client in elastic stack.Sep 27 2022, 3:49 PM
ardumont mentioned this in R260:af65cae682cc: staging: Deploy indexer journal client in elastic stack.Sep 27 2022, 4:02 PM
ardumont mentioned this in R260:c35b29d28de9: staging: Deploy indexer journal client in elastic stack.
ardumont mentioned this in R260:09472afe7c0e: staging: Deploy indexer journal client in elastic stack.Sep 27 2022, 4:06 PM
ardumont mentioned this in R259:5ce4d43fa4fc: swh.indexer: Fix default entrypoint for a journal client.Sep 27 2022, 4:12 PM
ardumont mentioned this in R260:5d425c8794da: staging: Deploy indexer journal client in elastic stack.Sep 27 2022, 4:53 PM
ardumont mentioned this in R260:47c886e2097e: staging: Deploy indexer journal client in elastic stack.Sep 27 2022, 5:00 PM
ardumont mentioned this in R259:9108900d9e05: swh.indexer: Fix again entrypoint.Sep 27 2022, 5:06 PM
ardumont mentioned this in R260:a973c3232a86: staging/indexer: Add missing scheduler configuration entry.Sep 27 2022, 7:07 PM
ardumont updated the task description. (Show Details)Sep 27 2022, 7:09 PM
ardumont mentioned this in rSPSITE2f67bfc5d310: staging: Migrate most indexer journal clients to elastic workers.Sep 27 2022, 7:15 PM
ardumont mentioned this in rSPRE91a8a005d5b7: staging: Decomission worker1.Sep 27 2022, 7:23 PM
ardumont updated the task description. (Show Details)Sep 27 2022, 7:31 PM
ardumont mentioned this in R259:52e66acec254: indexer: Add swh bullseye debian repo for runtime dependency.Sep 28 2022, 9:17 AM
ardumont mentioned this in R260:5d292b66596a: staging/indexer: Deploy content fossology license indexer.Sep 28 2022, 9:23 AM
ardumont mentioned this in rSPSITEb81df129d2c5: staging: Migrate last indexer journal client to elastic workers.Sep 28 2022, 10:10 AM
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.Sep 28 2022, 10:45 AM
ardumont closed this task as Resolved.Sep 28 2022, 7:16 PM
ardumont updated the task description. (Show Details)
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
ardumont mentioned this in rSPRE63d87f908275: staging: Decrease worker0 hardware.Sep 29 2022, 10:22 AM
ardumont mentioned this in rSPREf463e3e9bf4c: staging: Migrate elastic worker to pompidou.
ardumont mentioned this in T4385: Federate prometheus instances through thanos.Sep 30 2022, 9:52 AM
vsellier mentioned this in T4621: staging: Autoscaling of the indexers not properly configured.Oct 12 2022, 10:11 AM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:08 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal