Page MenuHomeSoftware Heritage
Feed Advanced Search

Sep 23 2022

ardumont added a comment to T4540: staging: Migrate remaining static services to elastic workers.

Remains a bunch of services not existing yet in the elastic stack (vault cooker, deposit checkers and indexer journal clients).
So the plan would be to integrate those in the elastic stack.

Sep 23 2022, 5:50 PM · meta-task, System administration, Roadmap 2022
ardumont updated the diff for D8341: Add nixguix lister.
  • nixguix: Keep only one url as origin and other urls as fallback ones
Sep 23 2022, 5:08 PM
ardumont updated the diff for D8341: Add nixguix lister.

Align file tarball listing on what we do for tarballs.

Sep 23 2022, 4:59 PM
ardumont committed R260:0e97c5b68f1f: staging: Decrease concurrency (authored by ardumont).
staging: Decrease concurrency
Sep 23 2022, 3:52 PM
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

Hum, for the 7 false, I have to check. For the 88 packages with no-origin, it is more
annoying. Well, some are metapackages as gcc-toolchain, so they can be skipped. Is it
ok for you to let this 'no-origin' type? For some others, I have to check if they are
covered elsewhere.

Sep 23 2022, 3:48 PM · Data Model, Nixguix loader
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

For ^, something like this would do [1]

Sep 23 2022, 3:38 PM · Data Model, Nixguix loader
ardumont updated the diff for D8341: Add nixguix lister.
  • nixguix: Keep only one tarball origin sending along fallback urls
Sep 23 2022, 3:37 PM
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

Thanks for all that ^! And great pointers!

Sep 23 2022, 3:09 PM · Data Model, Nixguix loader
ardumont added a comment to T3294: nixguix: Add support for pseudo-URLs with a missing schema.

I've moved around the parent task as the lister should be able to do something about this.

Sep 23 2022, 11:24 AM · Easy hack, Nixguix loader
ardumont removed a subtask for T2879: Finalize nixguix loader implementation: T3294: nixguix: Add support for pseudo-URLs with a missing schema.
Sep 23 2022, 11:23 AM · Nixguix loader, Package Loader
ardumont edited parent tasks for T3294: nixguix: Add support for pseudo-URLs with a missing schema, added: T3781: Replace the Nixguix loader with a lister; removed: T2879: Finalize nixguix loader implementation.
Sep 23 2022, 11:23 AM · Easy hack, Nixguix loader
ardumont added a subtask for T3781: Replace the Nixguix loader with a lister: T3294: nixguix: Add support for pseudo-URLs with a missing schema.
Sep 23 2022, 11:23 AM · Data Model, Nixguix loader
ardumont closed T2879: Finalize nixguix loader implementation as Wontfix.

This perimeter enters the scope of T3781.

Sep 23 2022, 11:22 AM · Nixguix loader, Package Loader
ardumont closed T2879: Finalize nixguix loader implementation, a subtask of T1991: Implement a Guix/Nix loader, as Wontfix.
Sep 23 2022, 11:22 AM · Package Loader
ardumont committed R260:b59599ea9bf4: staging: Deactivate debug logs for gogs listing (authored by ardumont).
staging: Deactivate debug logs for gogs listing
Sep 23 2022, 11:19 AM
ardumont updated the task description for T1721: Implementation of Gogs Lister.
Sep 23 2022, 11:18 AM · Gogs lister, Origin-Gitea/Gogs, Archive coverage, Lister
ardumont added a comment to T4478: Deploy Gogs lister to staging.

I either remembered an old stateless implementation or i remember another
implementation altogether.

Sep 23 2022, 11:18 AM · Gogs lister, Origin-Gitea/Gogs, System administration
ardumont closed T4478: Deploy Gogs lister to staging, a subtask of T1721: Implementation of Gogs Lister, as Resolved.
Sep 23 2022, 11:04 AM · Gogs lister, Origin-Gitea/Gogs, Archive coverage, Lister
ardumont closed T4478: Deploy Gogs lister to staging as Resolved.
Sep 23 2022, 11:04 AM · Gogs lister, Origin-Gitea/Gogs, System administration
ardumont added a comment to T4478: Deploy Gogs lister to staging.

I'll unstuck the situation by resetting that state.

Sep 23 2022, 11:02 AM · Gogs lister, Origin-Gitea/Gogs, System administration
ardumont added a comment to T4478: Deploy Gogs lister to staging.

But on my side, the "stateless" lister starts from the problematic page (17), and then
plainly stops... [2].

Note that there is a misbehavior in terms here for that "stateless" typed lister. If it
is indeed a stateless lister, it should start from scratch each time it runs. If it has
an incremental behavior (it seems so from that described behavior) then it should be
probably typed differently.

Sep 23 2022, 10:59 AM · Gogs lister, Origin-Gitea/Gogs, System administration
ardumont added a comment to T4478: Deploy Gogs lister to staging.

ah i think i got it...

Sep 23 2022, 10:51 AM · Gogs lister, Origin-Gitea/Gogs, System administration
ardumont added a comment to T4478: Deploy Gogs lister to staging.

As a shot in the dark, i've changed the list task arguments [1] but i'd be surprised if that'd change anything...

Sep 23 2022, 10:42 AM · Gogs lister, Origin-Gitea/Gogs, System administration
ardumont committed R260:f50c9c00a38d: staging: Activate debug logs for gogs listing (authored by ardumont).
staging: Activate debug logs for gogs listing
Sep 23 2022, 10:35 AM
ardumont added a comment to T4478: Deploy Gogs lister to staging.

I noticed your remarks but that's not what's happening in staging [3] for some unknown reason [1]

Sep 23 2022, 10:30 AM · Gogs lister, Origin-Gitea/Gogs, System administration
ardumont closed D8524: backend: Prevent query exception when lister ids is empty.
Sep 23 2022, 10:21 AM
ardumont committed rDSCH6d0b1d1e1ab5: backend: Prevent query exception when lister ids is empty (authored by ardumont).
backend: Prevent query exception when lister ids is empty
Sep 23 2022, 10:21 AM
ardumont moved T4545: staging: Deploy golang stack from in-progress to deployed/landed/monitoring on the System administration board.
Sep 23 2022, 10:19 AM · System administration, Golang loader, Golang lister, Archive coverage
ardumont changed the status of T4545: staging: Deploy golang stack from Open to Work in Progress.
Sep 23 2022, 10:19 AM · System administration, Golang loader, Golang lister, Archive coverage
ardumont changed the status of T4545: staging: Deploy golang stack, a subtask of T4124: Golang support, from Open to Work in Progress.
Sep 23 2022, 10:19 AM · Golang loader, Golang lister, Archive coverage
ardumont updated the task description for T4545: staging: Deploy golang stack.
Sep 23 2022, 10:18 AM · System administration, Golang loader, Golang lister, Archive coverage
ardumont added a project to T4545: staging: Deploy golang stack: System administration.
Sep 23 2022, 10:17 AM · System administration, Golang loader, Golang lister, Archive coverage
ardumont committed R260:4c967b9ed085: staging: Deploy golang stack (authored by ardumont).
staging: Deploy golang stack
Sep 23 2022, 10:11 AM
ardumont committed rSPRE83dbdd20b841: staging: Match current staging workers migrated to pompidou (authored by ardumont).
staging: Match current staging workers migrated to pompidou
Sep 23 2022, 10:02 AM
ardumont committed rSPREf00227d8965e: staging: Instantiate new elastic rancher-node-worker4 node (authored by ardumont).
staging: Instantiate new elastic rancher-node-worker4 node
Sep 23 2022, 10:02 AM
ardumont requested review of D8524: backend: Prevent query exception when lister ids is empty.
Sep 23 2022, 9:54 AM
ardumont added a revision to T4545: staging: Deploy golang stack: D8524: backend: Prevent query exception when lister ids is empty.
Sep 23 2022, 9:50 AM · System administration, Golang loader, Golang lister, Archive coverage
ardumont accepted D8523: Update value of User-Agent HTTP request header used by listers.
Sep 23 2022, 9:40 AM
ardumont accepted D8521: Use a Merkle discovery algorithm with archives.

ok, thanks, it's clearer.

Sep 23 2022, 9:37 AM
ardumont added 1 blocking reviewer(s) for D8521: Use a Merkle discovery algorithm with archives: Reviewers.
Sep 23 2022, 9:36 AM

Sep 22 2022

ardumont updated the task description for T4545: staging: Deploy golang stack.
Sep 22 2022, 6:30 PM · System administration, Golang loader, Golang lister, Archive coverage
ardumont updated the task description for T4545: staging: Deploy golang stack.
Sep 22 2022, 6:29 PM · System administration, Golang loader, Golang lister, Archive coverage
ardumont updated the task description for T4545: staging: Deploy golang stack.
Sep 22 2022, 6:20 PM · System administration, Golang loader, Golang lister, Archive coverage
ardumont committed R260:64cd15e93464: staging: Deploy golang stack (authored by ardumont).
staging: Deploy golang stack
Sep 22 2022, 6:15 PM
ardumont committed R260:005d3c0c37e3: listers: Ensure celery only receives one SIGTERM signal (authored by ardumont).
listers: Ensure celery only receives one SIGTERM signal
Sep 22 2022, 5:39 PM
ardumont committed R260:7a288f5d9975: staging: Decrease replica (authored by ardumont).
staging: Decrease replica
Sep 22 2022, 5:39 PM
ardumont committed R260:f5b96baa9290: loaders: Ensure celery only receives one SIGTERM signal (authored by ardumont).
loaders: Ensure celery only receives one SIGTERM signal
Sep 22 2022, 5:39 PM
ardumont updated the task description for T4540: staging: Migrate remaining static services to elastic workers.
Sep 22 2022, 5:32 PM · meta-task, System administration, Roadmap 2022
ardumont committed R260:c9540946d51d: staging: Decrease replica (authored by ardumont).
staging: Decrease replica
Sep 22 2022, 5:30 PM
ardumont committed R260:27c05c8b290a: loaders: Ensure celery only receives one SIGTERM signal (authored by ardumont).
loaders: Ensure celery only receives one SIGTERM signal
Sep 22 2022, 5:28 PM
ardumont committed R260:54830ad5bf88: keda: Switch to http connection for lister as well (authored by ardumont).
keda: Switch to http connection for lister as well
Sep 22 2022, 5:28 PM
ardumont committed R260:76e31254176e: keda: Switch to http connection (authored by ardumont).
keda: Switch to http connection
Sep 22 2022, 5:28 PM
ardumont committed R260:eb1c68907da4: Activate ack late messages for add forge and save code now workers (authored by ardumont).
Activate ack late messages for add forge and save code now workers
Sep 22 2022, 5:28 PM
ardumont added a comment to D8521: Use a Merkle discovery algorithm with archives.

I got some questions and suggestions inline.

Sep 22 2022, 5:18 PM
ardumont committed R260:ba5d8c6ae212: loaders: Ensure celery only receives one SIGTERM signal (authored by ardumont).
loaders: Ensure celery only receives one SIGTERM signal
Sep 22 2022, 3:44 PM
ardumont committed R260:69588b4a3f3e: loaders: Ensure celery only receives one SIGTERM signal (authored by ardumont).
loaders: Ensure celery only receives one SIGTERM signal
Sep 22 2022, 3:32 PM
ardumont committed R260:39a9fa5c10ab: loaders: Ensure celery only receives one SIGTERM call (authored by ardumont).
loaders: Ensure celery only receives one SIGTERM call
Sep 22 2022, 3:27 PM
ardumont committed R260:17ef999bb77c: loaders: Ensure celery only receives one SIGTERM call (authored by ardumont).
loaders: Ensure celery only receives one SIGTERM call
Sep 22 2022, 3:12 PM
ardumont accepted D8520: Refactor and deduplicate HTTP requests code in listers.
Sep 22 2022, 2:28 PM
ardumont committed rSPRE82a911b6558b: staging: Drop static worker3 node (authored by ardumont).
staging: Drop static worker3 node
Sep 22 2022, 11:38 AM
ardumont committed rSPREba08d27443ca: Drop experiments optional feature which is now mainstream (authored by ardumont).
Drop experiments optional feature which is now mainstream
Sep 22 2022, 11:32 AM
ardumont committed R260:5371357ee50f: keda: Switch to http connection for lister as well (authored by ardumont).
keda: Switch to http connection for lister as well
Sep 22 2022, 10:42 AM
ardumont committed R260:87d45843abcf: keda: Switch to http connection (authored by ardumont).
keda: Switch to http connection
Sep 22 2022, 10:24 AM
ardumont committed R260:a7c5b3412f9c: Activate ack late messages for add forge and save code now workers (authored by ardumont).
Activate ack late messages for add forge and save code now workers
Sep 22 2022, 10:07 AM
ardumont renamed T4540: staging: Migrate remaining static services to elastic workers from staging: Migrate remaining static loader services to elastic workers to staging: Migrate remaining static loader/lister services to elastic workers.
Sep 22 2022, 9:45 AM · meta-task, System administration, Roadmap 2022
ardumont accepted D8519: Use generic HTTP retry policy by default and rename dedicated decorator.

I'm not so sure in real life, 500 -> 500 -> 200 will happen that much often but let's see.

Sep 22 2022, 9:25 AM
ardumont committed rSPSITEf5ec1b2f53f3: staging: Deploy remaining lister in elastic staging workers (authored by ardumont).
staging: Deploy remaining lister in elastic staging workers
Sep 22 2022, 9:10 AM
ardumont committed R260:bce9935e6261: staging: Deploy remaining fast listers as a bundle (authored by ardumont).
staging: Deploy remaining fast listers as a bundle
Sep 22 2022, 9:03 AM
ardumont committed R260:5bfe01a563c6: staging: Deploy launchpad lister (authored by ardumont).
staging: Deploy launchpad lister
Sep 22 2022, 9:03 AM
ardumont committed R260:fbf64292c417: staging: Deploy opam lister (authored by ardumont).
staging: Deploy opam lister
Sep 22 2022, 9:03 AM
ardumont committed R260:fd755eaee2c1: staging: Deploy sourceforge lister (authored by ardumont).
staging: Deploy sourceforge lister
Sep 22 2022, 9:03 AM
ardumont committed R260:8dab39aece43: staging: Deploy bitbucket lister (authored by ardumont).
staging: Deploy bitbucket lister
Sep 22 2022, 9:03 AM
ardumont committed rSPSITEb99fc17be99b: staging: Migrate maven lister to elastic workers (authored by ardumont).
staging: Migrate maven lister to elastic workers
Sep 22 2022, 8:53 AM
ardumont committed R260:3105a062d05d: staging: Deploy maven lister (authored by ardumont).
staging: Deploy maven lister
Sep 22 2022, 8:49 AM
ardumont committed R260:326916a8a8a5: keda: Fix setup unifying using authenticationRef pattern (authored by ardumont).
keda: Fix setup unifying using authenticationRef pattern
Sep 22 2022, 8:49 AM

Sep 21 2022

ardumont changed the status of T4540: staging: Migrate remaining static services to elastic workers, a subtask of T4144: Elastic worker infrastructure, from Open to Work in Progress.
Sep 21 2022, 11:23 PM · meta-task, System administration, Roadmap 2022
ardumont changed the status of T4540: staging: Migrate remaining static services to elastic workers from Open to Work in Progress.
Sep 21 2022, 11:23 PM · meta-task, System administration, Roadmap 2022
ardumont updated the task description for T4540: staging: Migrate remaining static services to elastic workers.
Sep 21 2022, 11:23 PM · meta-task, System administration, Roadmap 2022
ardumont committed R260:1a1665dfe868: staging: Decrease concurrency as new workers are deployed (authored by ardumont).
staging: Decrease concurrency as new workers are deployed
Sep 21 2022, 11:15 PM
ardumont committed R260:d8e858e61193: staging: Bump swh-loader-package image (authored by ardumont).
staging: Bump swh-loader-package image
Sep 21 2022, 11:09 PM
ardumont committed R259:4dcd27abdb49: swh-graphql: Generate new frozen requirements for loader.core v4.2.0 (authored by ardumont).
swh-graphql: Generate new frozen requirements for loader.core v4.2.0
Sep 21 2022, 11:08 PM
ardumont committed R259:792260b8bc15: swh-loader-package: Update dependencies (authored by ardumont).
swh-loader-package: Update dependencies
Sep 21 2022, 11:08 PM
ardumont committed R260:1ec67bad368e: keda: Switch to amqp connection for autoscaling (authored by ardumont).
keda: Switch to amqp connection for autoscaling
Sep 21 2022, 9:24 PM
ardumont committed R260:42373b4536e1: keda: Switch to amqp connection for autoscaling (authored by ardumont).
keda: Switch to amqp connection for autoscaling
Sep 21 2022, 9:23 PM
ardumont committed R260:44151acb0bd3: keda: Drop unneeded setup already set through authenticationRef (authored by ardumont).
keda: Drop unneeded setup already set through authenticationRef
Sep 21 2022, 9:21 PM
ardumont committed R260:e8d3769dfa5f: keda: Reuse amqp connection for autoscaling (authored by ardumont).
keda: Reuse amqp connection for autoscaling
Sep 21 2022, 4:40 PM
ardumont added a reverting change for R260:90dab95f5788: keda: Drop unneeded setup already set through authenticationRef: R260:883dcb571489: Revert "keda: Drop unneeded setup already set through authenticationRef".
Sep 21 2022, 4:40 PM
ardumont committed R260:883dcb571489: Revert "keda: Drop unneeded setup already set through authenticationRef" (authored by ardumont).
Revert "keda: Drop unneeded setup already set through authenticationRef"
Sep 21 2022, 4:40 PM
ardumont committed R260:90dab95f5788: keda: Drop unneeded setup already set through authenticationRef (authored by ardumont).
keda: Drop unneeded setup already set through authenticationRef
Sep 21 2022, 4:33 PM
ardumont committed R260:c010c902d1c3: keda: Another tryout to fix scaling error (authored by ardumont).
keda: Another tryout to fix scaling error
Sep 21 2022, 4:32 PM
ardumont added a reverting change for R260:24208a72f8d9: keda: Drop extra vhost parameter: R260:afe523ae10ae: Revert "keda: Drop extra vhost parameter".
Sep 21 2022, 4:26 PM
ardumont committed R260:afe523ae10ae: Revert "keda: Drop extra vhost parameter" (authored by ardumont).
Revert "keda: Drop extra vhost parameter"
Sep 21 2022, 4:26 PM
ardumont committed R260:24208a72f8d9: keda: Drop extra vhost parameter (authored by ardumont).
keda: Drop extra vhost parameter
Sep 21 2022, 4:21 PM
ardumont committed R260:04c769c69a6d: keda: Tentatively fix an error regarding scaling (authored by ardumont).
keda: Tentatively fix an error regarding scaling
Sep 21 2022, 4:12 PM
ardumont committed R260:cc5ebd775d6d: archive-staging: Reuse common-secrets reference to access sentry dsn (authored by ardumont).
archive-staging: Reuse common-secrets reference to access sentry dsn
Sep 21 2022, 3:08 PM
ardumont committed R260:f8369d8b84f9: Ensure return of init container is ok with or without optional creds (authored by ardumont).
Ensure return of init container is ok with or without optional creds
Sep 21 2022, 3:08 PM
ardumont committed R260:b16ca531b560: loaders: Set default queue threshold autoscale to 10 (authored by ardumont).
loaders: Set default queue threshold autoscale to 10
Sep 21 2022, 2:52 PM
ardumont committed R260:2dc2453f21c0: lister: Align config and deployment according to loaders (authored by ardumont).
lister: Align config and deployment according to loaders
Sep 21 2022, 2:52 PM
ardumont committed R260:a5ca752e7153: lister/keda: Align autoscaling config (authored by ardumont).
lister/keda: Align autoscaling config
Sep 21 2022, 2:34 PM
ardumont committed R260:1489407eb150: pubdev: Activate debug loglevel (authored by ardumont).
pubdev: Activate debug loglevel
Sep 21 2022, 2:34 PM