Page MenuHomeSoftware Heritage
Feed Advanced Search

May 10 2022

ardumont updated the task description for T4214: Production : Deploy bzr/bazaar Loader.
May 10 2022, 10:59 AM · System administration
ardumont updated the task description for T4214: Production : Deploy bzr/bazaar Loader.
May 10 2022, 10:58 AM · System administration

May 9 2022

ardumont added inline comments to D7795: Test to detect uppercase NPM metadata file.
May 9 2022, 6:07 PM
ardumont added inline comments to D7795: Test to detect uppercase NPM metadata file.
May 9 2022, 6:05 PM
ardumont accepted D6926: First iteration of prometheus export of the e2e metrics.

A couple of suggestions inline.
And maybe even a typo in the status code to fix.

May 9 2022, 5:58 PM
ardumont renamed T4214: Production : Deploy bzr/bazaar Loader from Production : Deploy Bazaar Loader to Production : Deploy bzr/bazaar Loader.
May 9 2022, 5:47 PM · System administration
ardumont removed a parent task for T4214: Production : Deploy bzr/bazaar Loader: Unknown Object (Maniphest Task).
May 9 2022, 5:46 PM · System administration
ardumont added a subtask for T3610: Bazaar/Breezy loader: T4214: Production : Deploy bzr/bazaar Loader.
May 9 2022, 5:46 PM · BZR loader
ardumont added a parent task for T4214: Production : Deploy bzr/bazaar Loader: T3610: Bazaar/Breezy loader.
May 9 2022, 5:46 PM · System administration
ardumont added inline comments to D6926: First iteration of prometheus export of the e2e metrics.
May 9 2022, 5:31 PM
ardumont added inline comments to D6926: First iteration of prometheus export of the e2e metrics.
May 9 2022, 5:19 PM
ardumont accepted D7794: api/raw: Restrict raw object endpoint use to privileged users.

Can you please add the associated task id in the diff description?
(so one can browse to the related task more easily ;) ?

May 9 2022, 4:59 PM
ardumont accepted D7731: requirements.txt: Remove sphinx version pin.
May 9 2022, 4:53 PM
ardumont committed rSPRE179087d6fee2: staging/rancher: Bump memory to 8Gib for rancher nodes (authored by ardumont).
staging/rancher: Bump memory to 8Gib for rancher nodes
May 9 2022, 10:49 AM
ardumont accepted D7757: templates/content-display: Check content rendering for mime type first.
May 9 2022, 9:55 AM
ardumont accepted D7759: docs: Open external link in new browser tab.
May 9 2022, 9:53 AM
ardumont committed rSPRE9a4d7636d795: README: Reference the terraform part from the top-level (authored by ardumont).
README: Reference the terraform part from the top-level
May 9 2022, 9:52 AM
ardumont committed rSPRE51f802c63f79: README: Fix title typo (authored by ardumont).
README: Fix title typo
May 9 2022, 9:52 AM

Apr 29 2022

ardumont added inline comments to D7713: crates: rework to take advantage of data returned by the crates lister.
Apr 29 2022, 7:50 PM
ardumont added inline comments to D7713: crates: rework to take advantage of data returned by the crates lister.
Apr 29 2022, 7:48 PM
ardumont added 1 blocking reviewer(s) for D7713: crates: rework to take advantage of data returned by the crates lister: Reviewers.
Apr 29 2022, 7:42 PM
ardumont removed reviewers for D7713: crates: rework to take advantage of data returned by the crates lister: Reviewers, ardumont.
Apr 29 2022, 7:42 PM
ardumont resigned from D7713: crates: rework to take advantage of data returned by the crates lister.

I'll let other have the last word (i'm going afk for a bit ;)

Apr 29 2022, 7:42 PM
ardumont closed T4215: staging: Deploy latest maven stack, a subtask of T3874: staging: Analyze result of the maven listing and ingestion, as Resolved.
Apr 29 2022, 4:17 PM · Maven loader, Maven lister, Archive coverage
ardumont closed T4215: staging: Deploy latest maven stack as Resolved.
Apr 29 2022, 4:17 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a comment to T4215: staging: Deploy latest maven stack.

We can now see new maven origins [1] with multiple lists of artifacts [2]

Apr 29 2022, 4:17 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont committed rSPRE34d810097d8f: Register new elastic-worker3 to rancher cluster (authored by ardumont).
Register new elastic-worker3 to rancher cluster
Apr 29 2022, 4:14 PM
ardumont committed rDSNIP5222d5d000c1: elastic: Ignore internal files (authored by ardumont).
elastic: Ignore internal files
Apr 29 2022, 4:14 PM
ardumont committed rDSNIPe41dc37cbdc4: elastic: Upgrade image to latest softwareheritage/loaders-2022-04-29 (authored by ardumont).
elastic: Upgrade image to latest softwareheritage/loaders-2022-04-29
Apr 29 2022, 4:14 PM
ardumont added a comment to T4144: Elastic worker infrastructure.

@vsellier here is a rapid summary on:

Apr 29 2022, 3:47 PM · meta-task, System administration, Roadmap 2022
ardumont moved T4215: staging: Deploy latest maven stack from in-progress to deployed/landed/monitoring on the System administration board.
Apr 29 2022, 3:42 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a comment to T4215: staging: Deploy latest maven stack.

And ingested:

Apr 29 13:38:51 worker3 python3[215979]: [2022-04-29 13:38:51,458: INFO/ForkPoolWorker-1] Task swh.loader.package.maven.tasks.LoadMaven[4ac12ab2-7749-4152-baa1-6bf06a587cad] succeeded in 74.83338255400304s: {'status': 'eventful', 'snapshot_id': '571e40dbc59f5f5238d981986211bcabde2d73d8'}
Apr 29 2022, 3:39 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a comment to T4215: staging: Deploy latest maven stack.

New schemed maven origins are getting listed now:

Apr 29 2022, 3:37 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated the task description for T4215: staging: Deploy latest maven stack.
Apr 29 2022, 3:29 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont updated the task description for T4215: staging: Deploy latest maven stack.
Apr 29 2022, 3:12 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont changed the status of T4215: staging: Deploy latest maven stack from Open to Work in Progress.
Apr 29 2022, 2:53 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont changed the status of T4215: staging: Deploy latest maven stack, a subtask of T3874: staging: Analyze result of the maven listing and ingestion, from Open to Work in Progress.
Apr 29 2022, 2:53 PM · Maven loader, Maven lister, Archive coverage
ardumont updated the task description for T4215: staging: Deploy latest maven stack.
Apr 29 2022, 2:53 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont triaged T4215: staging: Deploy latest maven stack as Normal priority.
Apr 29 2022, 2:53 PM · Maven loader, Maven lister, System administration, Archive coverage
ardumont added a parent task for T1724: Maven Central repository support: Unknown Object (Maniphest Task).
Apr 29 2022, 2:26 PM · Maven loader, Maven lister, GSoC 2019, Archive coverage
ardumont accepted D7716: maven: Handle null mtime value in index for jar archive.
Apr 29 2022, 2:00 PM
ardumont accepted D7716: maven: Handle null mtime value in index for jar archive.
Apr 29 2022, 1:59 PM
ardumont added a revision to T3874: staging: Analyze result of the maven listing and ingestion: D7716: maven: Handle null mtime value in index for jar archive.
Apr 29 2022, 1:57 PM · Maven loader, Maven lister, Archive coverage
ardumont updated the summary of D7716: maven: Handle null mtime value in index for jar archive.
Apr 29 2022, 1:57 PM
ardumont committed rDSNIP9fe02ffb8c71: elastic: Update instructions (authored by ardumont).
elastic: Update instructions
Apr 29 2022, 1:05 PM
ardumont committed rDSNIP3528d5c40e6e: elastic: Ignore internal files (authored by ardumont).
elastic: Ignore internal files
Apr 29 2022, 1:05 PM
ardumont committed rDSNIPbcd4bbc99930: elastic: Decrease concurrency to 5 max (authored by ardumont).
elastic: Decrease concurrency to 5 max
Apr 29 2022, 1:05 PM
ardumont committed rDSNIP105640cafb1c: elastic: Allow multiple loader deployments and update documentation (authored by ardumont).
elastic: Allow multiple loader deployments and update documentation
Apr 29 2022, 1:05 PM
ardumont closed T4206: prod: Deploy metadata loader v0.0.2, a subtask of T1740: fetch extrinsic origin metadata from GitHub, as Resolved.
Apr 29 2022, 11:27 AM · Metadata workflow, Origin-GitHub
ardumont closed T4206: prod: Deploy metadata loader v0.0.2, a subtask of T3273: Use "fork" relationships to speed-up initial load of large repositories, as Resolved.
Apr 29 2022, 11:27 AM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
ardumont closed T4206: prod: Deploy metadata loader v0.0.2 as Resolved.
Apr 29 2022, 11:27 AM · System administration
ardumont closed T4204: prod: Deploy swh-scheduler 1.1.1, a subtask of T4186: Allow loaders to fetch extrinsic metadata, as Resolved.
Apr 29 2022, 11:27 AM · Core Loader, Metadata workflow
ardumont closed T4204: prod: Deploy swh-scheduler 1.1.1, a subtask of T4206: prod: Deploy metadata loader v0.0.2, as Resolved.
Apr 29 2022, 11:27 AM · System administration
ardumont closed T4204: prod: Deploy swh-scheduler 1.1.1 as Resolved.
Apr 29 2022, 11:27 AM · System administration, Core Loader
ardumont closed T4205: prod: Deploy swh-loader-core 3.2.1, a subtask of T4204: prod: Deploy swh-scheduler 1.1.1, as Resolved.
Apr 29 2022, 11:27 AM · System administration, Core Loader
ardumont closed T4205: prod: Deploy swh-loader-core 3.2.1 as Resolved.
Apr 29 2022, 11:27 AM · System administration, Core Loader
ardumont closed D7714: developer-setup: Upgrade new mypy pinned version.
Apr 29 2022, 11:25 AM
ardumont committed rDDOC551fd266362b: developer-setup: Upgrade new mypy pinned version (authored by ardumont).
developer-setup: Upgrade new mypy pinned version
Apr 29 2022, 11:25 AM
ardumont requested changes to D7713: crates: rework to take advantage of data returned by the crates lister.

It's going the right way, thanks.

Apr 29 2022, 11:25 AM
ardumont accepted D7715: maven: Remove extraction of groupId and artifactId from pom files.
Apr 29 2022, 11:23 AM
ardumont published D7715: maven: Remove extraction of groupId and artifactId from pom files for review.
Apr 29 2022, 11:23 AM
ardumont requested review of D7714: developer-setup: Upgrade new mypy pinned version.
Apr 29 2022, 11:22 AM
ardumont added a parent task for T1423: Add .crate (Rust) loader: Unknown Object (Maniphest Task).
Apr 29 2022, 11:17 AM · Crates loader, Archive coverage, Sprint 2018 12, Restricted Project
ardumont added a parent task for T4167: Document crates lister: Unknown Object (Maniphest Task).
Apr 29 2022, 11:17 AM · Documentation, Community Building
ardumont added inline comments to D7713: crates: rework to take advantage of data returned by the crates lister.
Apr 29 2022, 11:13 AM
ardumont added inline comments to D7713: crates: rework to take advantage of data returned by the crates lister.
Apr 29 2022, 11:09 AM
ardumont accepted D7712: package/maven: Fix jar archive download after changes in lister.

*thumbs up*, even better ;)

Apr 29 2022, 11:00 AM
ardumont added inline comments to D7713: crates: rework to take advantage of data returned by the crates lister.
Apr 29 2022, 10:58 AM
ardumont added a comment to D7713: crates: rework to take advantage of data returned by the crates lister.

Arf, i reloaded the page (i was reviewing) but now lost the initial content of the diff.
You need to update back the diff with the range of commits you want to use.

Apr 29 2022, 10:40 AM
ardumont accepted D7712: package/maven: Fix jar archive download after changes in lister.

lgtm, some suggestions inline.

Apr 29 2022, 10:37 AM
ardumont committed rDSNIP8326c36e8dab: README: Update instructions (authored by ardumont).
README: Update instructions
Apr 29 2022, 10:27 AM
ardumont committed rDSNIP7df0359bc105: elastic: Allow loader git to use credentials for metadata loading (authored by ardumont).
elastic: Allow loader git to use credentials for metadata loading
Apr 29 2022, 10:27 AM
ardumont committed rDSNIP1190af7e60f1: elastic: Use specific image loader version (authored by ardumont).
elastic: Use specific image loader version
Apr 29 2022, 10:27 AM

Apr 28 2022

ardumont added a comment to T4206: prod: Deploy metadata loader v0.0.2.

We now have metadata in production as well [1].
So it's working.

Apr 28 2022, 8:45 PM · System administration
ardumont committed rDSNIP1bfec6e22990: worker: Use softwareheritage/loaders image (authored by ardumont).
worker: Use softwareheritage/loaders image
Apr 28 2022, 5:07 PM
ardumont committed rDSNIPc14759091c19: worker: Align pipeline configuration with staging (authored by ardumont).
worker: Align pipeline configuration with staging
Apr 28 2022, 5:07 PM
ardumont moved T4204: prod: Deploy swh-scheduler 1.1.1 from in-progress to deployed/landed/monitoring on the System administration board.
Apr 28 2022, 4:47 PM · System administration, Core Loader
ardumont moved T4205: prod: Deploy swh-loader-core 3.2.1 from in-progress to deployed/landed/monitoring on the System administration board.
Apr 28 2022, 4:47 PM · System administration, Core Loader
ardumont moved T4206: prod: Deploy metadata loader v0.0.2 from in-progress to deployed/landed/monitoring on the System administration board.
Apr 28 2022, 4:47 PM · System administration
ardumont accepted D7710: maven: Create one origin per package instead of one per package version.

one question inline.

Apr 28 2022, 4:46 PM
ardumont added inline comments to D7710: maven: Create one origin per package instead of one per package version.
Apr 28 2022, 4:43 PM
ardumont added 1 blocking reviewer(s) for D7491: Added Moderator Name and Last Modified Time columns to the Admin Request Dashboard: Reviewers.
Apr 28 2022, 4:30 PM
ardumont committed rSPSITEd2f3a8f19b30: Deploy metadata loader with loader git (authored by ardumont).
Deploy metadata loader with loader git
Apr 28 2022, 4:13 PM
ardumont changed the status of T4206: prod: Deploy metadata loader v0.0.2, a subtask of T1740: fetch extrinsic origin metadata from GitHub, from Open to Work in Progress.
Apr 28 2022, 3:43 PM · Metadata workflow, Origin-GitHub
ardumont changed the status of T4206: prod: Deploy metadata loader v0.0.2 from Open to Work in Progress.
Apr 28 2022, 3:43 PM · System administration
ardumont changed the status of T4206: prod: Deploy metadata loader v0.0.2, a subtask of T3273: Use "fork" relationships to speed-up initial load of large repositories, from Open to Work in Progress.
Apr 28 2022, 3:43 PM · Origin-GitHub, Origin-GitLab, Git loader, Extrinsic metadata, Core Loader
ardumont changed the status of T4205: prod: Deploy swh-loader-core 3.2.1 from Open to Work in Progress.
Apr 28 2022, 3:43 PM · System administration, Core Loader
ardumont changed the status of T4205: prod: Deploy swh-loader-core 3.2.1, a subtask of T4204: prod: Deploy swh-scheduler 1.1.1, from Open to Work in Progress.
Apr 28 2022, 3:43 PM · System administration, Core Loader
ardumont changed the status of T4204: prod: Deploy swh-scheduler 1.1.1, a subtask of T4186: Allow loaders to fetch extrinsic metadata, from Open to Work in Progress.
Apr 28 2022, 3:43 PM · Core Loader, Metadata workflow
ardumont changed the status of T4204: prod: Deploy swh-scheduler 1.1.1 from Open to Work in Progress.
Apr 28 2022, 3:43 PM · System administration, Core Loader
ardumont changed the status of T4204: prod: Deploy swh-scheduler 1.1.1, a subtask of T4206: prod: Deploy metadata loader v0.0.2, from Open to Work in Progress.
Apr 28 2022, 3:43 PM · System administration
ardumont edited P1353 fork speedup!!!.
Apr 28 2022, 1:35 PM
ardumont created P1353 fork speedup!!!.
Apr 28 2022, 1:34 PM
ardumont closed T4202: staging: Deploy swh.loader.metadata and friends as Resolved.
Apr 28 2022, 12:37 PM · Metadata Loaders, System administration
ardumont closed D7709: client: Migrate to _post method call to stop deprecation warnings.
Apr 28 2022, 12:36 PM
ardumont committed rDSTOfb5514112d33: client: Migrate to _post method call to stop deprecation warnings (authored by ardumont).
client: Migrate to _post method call to stop deprecation warnings
Apr 28 2022, 12:36 PM
ardumont updated the task description for T4202: staging: Deploy swh.loader.metadata and friends.
Apr 28 2022, 12:31 PM · Metadata Loaders, System administration
ardumont added a comment to T4202: staging: Deploy swh.loader.metadata and friends.

After some tampering with the actual queue in scheduler0.staging (cleaning that up, it's staging so fine).
And limiting the number to less than 10k (100 is enough, way less workers).
We now have some github origins properly scheduled with their lister_name=github and lister_instance_name=github.
This now results in extrinsic metadata being fetched for those origins [1] (without me having to trigger the swh loader run cli).

Apr 28 2022, 12:30 PM · Metadata Loaders, System administration
ardumont accepted D7708: Update unresolved issues.
Apr 28 2022, 11:38 AM
ardumont accepted D7707: Add suggestions from current bzr maintainer.
Apr 28 2022, 11:37 AM