Page MenuHomeSoftware Heritage
Feed Advanced Search

Sep 6 2022

ardumont updated the task description for T4144: Elastic worker infrastructure.
Sep 6 2022, 2:45 PM · meta-task, System administration, Roadmap 2022
ardumont committed R260:1277bbe493ca: Release new swh.loader.git on staging workers (authored by ardumont).
Release new swh.loader.git on staging workers
Sep 6 2022, 2:41 PM
ardumont committed R259:a7912986627e: swh.loader.git: Bump new release (authored by ardumont).
swh.loader.git: Bump new release
Sep 6 2022, 2:40 PM
ardumont committed R259:1acdd69c4575: swh-loader-git: Fix entrypoint.sh (authored by ardumont).
swh-loader-git: Fix entrypoint.sh
Sep 6 2022, 2:40 PM
ardumont requested review of D8397: Deploy maven-exporter production node.
Sep 6 2022, 11:50 AM
ardumont added a revision to T4330: Deploy maven stack in production: D8397: Deploy maven-exporter production node.
Sep 6 2022, 11:50 AM · System administration, Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont added inline comments to D8341: Add nixguix lister.
Sep 6 2022, 11:47 AM
ardumont updated the task description for T3691: Implement CVS loader.
Sep 6 2022, 11:27 AM · CVS loader, Archive coverage
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Sep 6 2022, 10:39 AM · Data Model, Nixguix loader
ardumont updated the task description for T3781: Replace the Nixguix loader with a lister.
Sep 6 2022, 10:32 AM · Data Model, Nixguix loader
ardumont updated the diff for D8341: Add nixguix lister.

Add coverage

Sep 6 2022, 10:29 AM
ardumont updated the summary of D8341: Add nixguix lister.
Sep 6 2022, 10:27 AM
ardumont updated the summary of D8341: Add nixguix lister.
Sep 6 2022, 10:21 AM
ardumont added inline comments to D8341: Add nixguix lister.
Sep 6 2022, 10:12 AM
ardumont added inline comments to D8341: Add nixguix lister.
Sep 6 2022, 10:11 AM
ardumont updated the diff for D8341: Add nixguix lister.

Add simple test scenario

Sep 6 2022, 10:11 AM
ardumont added a comment to T3781: Replace the Nixguix loader with a lister.

Some more information regarding extensions supported in nixpkgs and guix manifests:

In [33]: sources = "https://nix-community.github.io/nixpkgs-swh/sources-unstable.json"
Sep 6 2022, 9:42 AM · Data Model, Nixguix loader
ardumont added inline comments to D8341: Add nixguix lister.
Sep 6 2022, 9:13 AM

Sep 5 2022

ardumont updated the diff for D8341: Add nixguix lister.

Fix naive is_tarball implem so tests are fine

Sep 5 2022, 8:24 PM
ardumont updated the diff for D8341: Add nixguix lister.

Complete implementation as is and rebase on master

Sep 5 2022, 6:24 PM
ardumont closed T4495: thanos querier does not see metrics pushed by the archive staging thanos gateway in the azure bucket, a subtask of T4144: Elastic worker infrastructure, as Invalid.
Sep 5 2022, 5:29 PM · meta-task, System administration, Roadmap 2022
ardumont closed T4495: thanos querier does not see metrics pushed by the archive staging thanos gateway in the azure bucket as Invalid.
Sep 5 2022, 5:29 PM · meta-task, System administration, Roadmap 2022
ardumont added a comment to T4495: thanos querier does not see metrics pushed by the archive staging thanos gateway in the azure bucket.

In the end, this feels like an invalid task as modifying the date range, i got values in the end [1].
Which is a good news.

Sep 5 2022, 5:28 PM · meta-task, System administration, Roadmap 2022
ardumont added a reverting change for D8363: staging/graphql/ingress: Fix double if instruction: R260:aa29e31117e5: Revert "staging/graphql/ingress: Fix double if instruction".
Sep 5 2022, 5:06 PM
ardumont added a reverting change for R260:a3ebadbd9a17: staging/graphql/ingress: Fix double if instruction: R260:aa29e31117e5: Revert "staging/graphql/ingress: Fix double if instruction".
Sep 5 2022, 5:06 PM
ardumont committed R260:aa29e31117e5: Revert "staging/graphql/ingress: Fix double if instruction" (authored by ardumont).
Revert "staging/graphql/ingress: Fix double if instruction"
Sep 5 2022, 5:06 PM
ardumont triaged T4495: thanos querier does not see metrics pushed by the archive staging thanos gateway in the azure bucket as High priority.
Sep 5 2022, 4:59 PM · meta-task, System administration, Roadmap 2022
ardumont accepted D8390: package/utils: Add retry policy to download in case of throttling.

lgtm (and this unifies with how we deal the somewhat similar cases in the listers')

Sep 5 2022, 3:59 PM
ardumont added inline comments to D8390: package/utils: Add retry policy to download in case of throttling.
Sep 5 2022, 3:56 PM
ardumont accepted D8388: Fix crash when RawExtrinsicMetadata objects have the same target.
Sep 5 2022, 3:52 PM
ardumont accepted D8391: Declare lunar credentials.

lgtm

Sep 5 2022, 3:51 PM
ardumont updated the task description for T4144: Elastic worker infrastructure.
Sep 5 2022, 3:33 PM · meta-task, System administration, Roadmap 2022
ardumont triaged T4494: ingest hackage.haskell.org, the Haskell package repository as Normal priority.
Sep 5 2022, 9:57 AM · Hackage loader, Hackage lister, Archive coverage

Sep 2 2022

ardumont added a comment to T4144: Elastic worker infrastructure.

To check the live content of the thanos-sidecar in the cluster (then port-forward, k9s tool helps a lot):

Sep 2 2022, 6:35 PM · meta-task, System administration, Roadmap 2022
ardumont committed R260:2dd974fd26bb: staging: Deploy pypi loader (authored by ardumont).
staging: Deploy pypi loader
Sep 2 2022, 5:29 PM
ardumont committed R260:382ab28e186e: staging: Increase git loader (authored by ardumont).
staging: Increase git loader
Sep 2 2022, 5:28 PM
ardumont committed rSPSITE3805ce387276: thanos-gateway: Fix missing port-grpc (authored by ardumont).
thanos-gateway: Fix missing port-grpc
Sep 2 2022, 4:20 PM
ardumont closed D8385: Expose thanos gateway service to read archive staging metrics.
Sep 2 2022, 4:10 PM
ardumont committed rSPSITE90b0b1d6bc75: Expose thanos gateway service to read archive staging metrics (authored by ardumont).
Expose thanos gateway service to read archive staging metrics
Sep 2 2022, 4:10 PM
ardumont updated the diff for D8385: Expose thanos gateway service to read archive staging metrics.

Rebase

Sep 2 2022, 4:09 PM
ardumont updated the task description for T4144: Elastic worker infrastructure.
Sep 2 2022, 4:02 PM · meta-task, System administration, Roadmap 2022
ardumont updated the test plan for D8385: Expose thanos gateway service to read archive staging metrics.
Sep 2 2022, 4:02 PM
ardumont requested review of D8385: Expose thanos gateway service to read archive staging metrics.
Sep 2 2022, 4:00 PM
ardumont added a revision to T4144: Elastic worker infrastructure: D8385: Expose thanos gateway service to read archive staging metrics.
Sep 2 2022, 4:00 PM · meta-task, System administration, Roadmap 2022
ardumont updated the task description for T4144: Elastic worker infrastructure.
Sep 2 2022, 3:49 PM · meta-task, System administration, Roadmap 2022
ardumont updated the task description for T4144: Elastic worker infrastructure.
Sep 2 2022, 3:48 PM · meta-task, System administration, Roadmap 2022
ardumont closed D8381: cluster-archive-staging: Activate thanos sidecar service.
Sep 2 2022, 3:39 PM
ardumont committed rSPRE7620b1f22dde: cluster-archive-staging: Activate thanos sidecar service (authored by ardumont).
cluster-archive-staging: Activate thanos sidecar service
Sep 2 2022, 3:39 PM
ardumont added a comment to P1445 thanos tools bucket inspect ....

w00t, now i've got the proper labels:

Sep 2 2022, 3:36 PM
ardumont created P1445 thanos tools bucket inspect ....
Sep 2 2022, 1:50 PM
ardumont updated the diff for D8381: cluster-archive-staging: Activate thanos sidecar service.

Map properly external labels for thanos to take into account

Sep 2 2022, 11:44 AM
ardumont updated the diff for D8381: cluster-archive-staging: Activate thanos sidecar service.

Add external labels to metrics

Sep 2 2022, 10:32 AM
ardumont accepted D8380: Declare the missing read-only objstorage dns entry.
Sep 2 2022, 9:49 AM
ardumont added inline comments to D8381: cluster-archive-staging: Activate thanos sidecar service.
Sep 2 2022, 9:40 AM
ardumont requested review of D8381: cluster-archive-staging: Activate thanos sidecar service.
Sep 2 2022, 9:33 AM
ardumont added a revision to T4144: Elastic worker infrastructure: D8381: cluster-archive-staging: Activate thanos sidecar service.
Sep 2 2022, 9:33 AM · meta-task, System administration, Roadmap 2022
ardumont created P1444 archive-staging: push historical data to azure bucket.
Sep 2 2022, 9:32 AM
ardumont committed rSPREc911fcdd7c68: staging: Upgrade prometheus version to latest supported in cluster (authored by ardumont).
staging: Upgrade prometheus version to latest supported in cluster
Sep 2 2022, 9:27 AM

Sep 1 2022

ardumont committed rSKCONFeabc6124caa1: staging: Rename properly the monitoring namespace (authored by ardumont).
staging: Rename properly the monitoring namespace
Sep 1 2022, 5:29 PM
ardumont created P1443 (An Untitled Masterwork).
Sep 1 2022, 4:03 PM
ardumont accepted D8376: cli: Pass all journal_client config keys to the JournalClient.
Sep 1 2022, 2:01 PM
ardumont added a comment to D8376: cli: Pass all journal_client config keys to the JournalClient.

lgtm

Sep 1 2022, 2:01 PM
ardumont published D8376: cli: Pass all journal_client config keys to the JournalClient for review.
Sep 1 2022, 2:01 PM
ardumont committed rSKCONFd51b44d929d6: staging: Install monitoring namespace (authored by ardumont).
staging: Install monitoring namespace
Sep 1 2022, 1:58 PM
ardumont updated the task description for T4459: Deploy swh-indexer > v2.6 on staging then production.
Sep 1 2022, 11:50 AM · Indexer, System administration
ardumont created P1442 (An Untitled Masterwork).
Sep 1 2022, 10:11 AM
ardumont accepted D8373: Filter out more invalid URIs that make PyLD crash.
Sep 1 2022, 9:53 AM
ardumont accepted D8372: base: Filter out empty URIs so PyLD does not crash.
Sep 1 2022, 9:49 AM

Aug 31 2022

ardumont closed T4477: staging origin intrinsic metadata indexer are stuck as Resolved.
Aug 31 2022, 6:49 PM · Indexer, System administration
ardumont closed T4477: staging origin intrinsic metadata indexer are stuck, a subtask of T4459: Deploy swh-indexer > v2.6 on staging then production, as Resolved.
Aug 31 2022, 6:49 PM · Indexer, System administration
ardumont moved T4477: staging origin intrinsic metadata indexer are stuck from in-progress to deployed/landed/monitoring on the System administration board.
Aug 31 2022, 6:49 PM · Indexer, System administration
ardumont changed the status of T4477: staging origin intrinsic metadata indexer are stuck, a subtask of T4459: Deploy swh-indexer > v2.6 on staging then production, from Open to Work in Progress.
Aug 31 2022, 6:49 PM · Indexer, System administration
ardumont changed the status of T4477: staging origin intrinsic metadata indexer are stuck from Open to Work in Progress.
Aug 31 2022, 6:49 PM · Indexer, System administration
ardumont accepted D8360: docs: Document metadata formats for Gogs and Gitea.
Aug 31 2022, 6:49 PM
ardumont accepted D8353: Move swh-graph's test fixture in a pytest_plugin module.

lgtm

Aug 31 2022, 6:48 PM
ardumont accepted D8371: staging: Increase the number of workers for storage and indexer storage.
Aug 31 2022, 6:36 PM
ardumont closed D8370: staging intrinsic metadata indexer: Declare batch size to 100.
Aug 31 2022, 6:32 PM
ardumont committed rSPSITE0b1c7ad80484: staging intrinsic metadata indexer: Declare batch size to 100 (authored by ardumont).
staging intrinsic metadata indexer: Declare batch size to 100
Aug 31 2022, 6:32 PM
ardumont accepted D8371: staging: Increase the number of workers for storage and indexer storage.

thanks

Aug 31 2022, 6:22 PM
ardumont requested review of D8370: staging intrinsic metadata indexer: Declare batch size to 100.
Aug 31 2022, 6:17 PM
ardumont added a revision to T4477: staging origin intrinsic metadata indexer are stuck: D8370: staging intrinsic metadata indexer: Declare batch size to 100.
Aug 31 2022, 6:17 PM · Indexer, System administration
ardumont closed D8369: indexer.cli: Allow batch_size configuration on journal client.
Aug 31 2022, 6:11 PM
ardumont committed rDCIDX752e5d3f733e: indexer.cli: Allow batch_size configuration on journal client (authored by ardumont).
indexer.cli: Allow batch_size configuration on journal client
Aug 31 2022, 6:11 PM
ardumont added a comment to T4477: staging origin intrinsic metadata indexer are stuck.

The lag is subsiding now, slowly because only 1 journal client:

Aug 31 2022, 6:09 PM · Indexer, System administration
ardumont updated subscribers of T4477: staging origin intrinsic metadata indexer are stuck.

After further investigation w/ @vsellier, it's also related to our storage and indexer storage having too few gunicorn workers serving the journal clients (among other things).
So decreasing the batch size to something like 100 and fixing that should fairly help ^.

Aug 31 2022, 6:06 PM · Indexer, System administration
ardumont requested review of D8369: indexer.cli: Allow batch_size configuration on journal client.
Aug 31 2022, 5:51 PM
ardumont added a revision to T4477: staging origin intrinsic metadata indexer are stuck: D8369: indexer.cli: Allow batch_size configuration on journal client.
Aug 31 2022, 5:20 PM · Indexer, System administration
ardumont added a comment to T4477: staging origin intrinsic metadata indexer are stuck.

Activating debug log [1]

Aug 31 2022, 5:18 PM · Indexer, System administration
ardumont edited P1441 staging indexer origin intrinsic metadata journal client with debug.
Aug 31 2022, 4:59 PM
ardumont closed D8363: staging/graphql/ingress: Fix double if instruction.
Aug 31 2022, 4:56 PM
ardumont committed R260:a3ebadbd9a17: staging/graphql/ingress: Fix double if instruction (authored by ardumont).
staging/graphql/ingress: Fix double if instruction
Aug 31 2022, 4:56 PM
ardumont closed D8352: staging/graphql: Push issues to sentry.
Aug 31 2022, 4:56 PM
ardumont committed R260:5aed44119bb7: staging/graphql: Push issues to sentry (authored by ardumont).
staging/graphql: Push issues to sentry
Aug 31 2022, 4:56 PM
ardumont edited P1441 staging indexer origin intrinsic metadata journal client with debug.
Aug 31 2022, 4:55 PM
ardumont updated the title for P1441 staging indexer origin intrinsic metadata journal client with debug from go figure to staging indexer origin intrinsic metadata journal client with debug.
Aug 31 2022, 4:55 PM
ardumont created P1441 staging indexer origin intrinsic metadata journal client with debug.
Aug 31 2022, 4:50 PM
ardumont updated the task description for T4477: staging origin intrinsic metadata indexer are stuck.
Aug 31 2022, 4:43 PM · Indexer, System administration
ardumont triaged T4477: staging origin intrinsic metadata indexer are stuck as Normal priority.
Aug 31 2022, 4:41 PM · Indexer, System administration
ardumont renamed T4459: Deploy swh-indexer > v2.6 on staging then production from Deploy swh-indexer v2.4.2 on production and staging to Deploy swh-indexer v2.4 on production and staging.
Aug 31 2022, 4:39 PM · Indexer, System administration
ardumont accepted D8365: Remove unused middlewares package.
Aug 31 2022, 4:36 PM