Page MenuHomeSoftware Heritage
Feed Advanced Search

Jul 20 2022

vlorentz added a comment to T4392: Metadata Indexer for NuGet (.nuspec).

You can translate http://schema.org/language too. It's uncommon enough to be missing from Codemeta, but it doesn't mean we can't use it.

Jul 20 2022, 9:40 AM · Indexer
vlorentz renamed T4392: Metadata Indexer for NuGet (.nuspec) from Metadata Indexer for NuGet (nuget.config) to Metadata Indexer for NuGet (.nuspec).
Jul 20 2022, 9:35 AM · Indexer
VickyMerzOwn updated the task description for T4392: Metadata Indexer for NuGet (.nuspec).
Jul 20 2022, 7:56 AM · Indexer

Jul 19 2022

vlorentz triaged T4401: Index metadata from the deposit as Normal priority.
Jul 19 2022, 1:04 PM · SWORD deposit, Indexer, Metadata workflow
ardumont added a comment to T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.

Follow journal consumption [12]

Jul 19 2022, 12:33 PM · System administration, Indexer, Metadata workflow
ardumont closed T4395: Migrate azure worker vms to cheaper and more efficient vms, a subtask of T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1, as Resolved.
Jul 19 2022, 12:29 PM · System administration, Indexer, Metadata workflow
ardumont moved T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1 from in-progress to deployed/landed/monitoring on the System administration board.
Jul 19 2022, 11:26 AM · System administration, Indexer, Metadata workflow
ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jul 19 2022, 11:26 AM · System administration, Indexer, Metadata workflow
ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jul 19 2022, 11:25 AM · System administration, Indexer, Metadata workflow
ardumont added a comment to T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
  • Stop journal client [1] (export of offset needs the topics to be inactive [1'])
  • Keep current offset dump just in case [2]
  • Reset topics to earliest
  • Restart journal client
Jul 19 2022, 11:23 AM · System administration, Indexer, Metadata workflow
vlorentz updated the task description for T4273: Rewrite indexers as journal clients when relevant.
Jul 19 2022, 11:04 AM · Indexer, Metadata workflow
ardumont updated the task description for T4273: Rewrite indexers as journal clients when relevant.
Jul 19 2022, 10:57 AM · Indexer, Metadata workflow

Jul 18 2022

ardumont renamed T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1 from staging: Deploy new origin intrinsic metadata journal client indexer > v1.1 to Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jul 18 2022, 6:35 PM · System administration, Indexer, Metadata workflow
ardumont changed the status of T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1, a subtask of T4273: Rewrite indexers as journal clients when relevant, from Open to Work in Progress.
Jul 18 2022, 6:23 PM · Indexer, Metadata workflow
ardumont changed the status of T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1 from Open to Work in Progress.
Jul 18 2022, 6:23 PM · System administration, Indexer, Metadata workflow
ardumont changed the status of T4395: Migrate azure worker vms to cheaper and more efficient vms, a subtask of T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1, from Open to Work in Progress.
Jul 18 2022, 6:23 PM · System administration, Indexer, Metadata workflow
ardumont added a subtask for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1: T4395: Migrate azure worker vms to cheaper and more efficient vms.
Jul 18 2022, 11:21 AM · System administration, Indexer, Metadata workflow

Jul 13 2022

VickyMerzOwn triaged T4392: Metadata Indexer for NuGet (.nuspec) as Normal priority.
Jul 13 2022, 11:42 AM · Indexer

Jul 12 2022

VickyMerzOwn closed T4376: Metadata Indexer for Pub (pubspec.yaml) as Resolved.
Jul 12 2022, 11:16 PM · Indexer

Jul 11 2022

ardumont moved T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1 from Backlog to Weekly backlog on the System administration board.
Jul 11 2022, 2:21 PM · System administration, Indexer, Metadata workflow

Jul 6 2022

VickyMerzOwn updated the task description for T4376: Metadata Indexer for Pub (pubspec.yaml).
Jul 6 2022, 1:21 PM · Indexer

Jul 5 2022

ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jul 5 2022, 4:55 PM · System administration, Indexer, Metadata workflow
vlorentz lowered the priority of T4277: Deal with null characters in the output of the metadata indexer from Normal to Low.

lowering priority, as it is still possible for the crash to happen, but D7992 solves the only known source

Jul 5 2022, 4:51 PM · Indexer, Metadata workflow
vlorentz closed T4274: Resolve all known crashes in the metadata indexer as Resolved.
Jul 5 2022, 4:50 PM · Indexer, Metadata workflow
vlorentz closed T4274: Resolve all known crashes in the metadata indexer, a subtask of T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1, as Resolved.
Jul 5 2022, 4:50 PM · System administration, Indexer, Metadata workflow
vlorentz closed T4274: Resolve all known crashes in the metadata indexer, a subtask of T4273: Rewrite indexers as journal clients when relevant, as Resolved.
Jul 5 2022, 4:50 PM · Indexer, Metadata workflow
vlorentz triaged T4376: Metadata Indexer for Pub (pubspec.yaml) as Normal priority.
Jul 5 2022, 12:22 PM · Indexer

Jul 4 2022

VickyMerzOwn closed T4357: Metadata Indexer for Composer as Resolved.
Jul 4 2022, 2:49 PM · Indexer

Jun 29 2022

VickyMerzOwn added a revision to T4357: Metadata Indexer for Composer: D8047: Indexer for Packagist(composer.json).
Jun 29 2022, 12:58 PM · Indexer
swh-sentry-integration added a comment to T4277: Deal with null characters in the output of the metadata indexer.

Sentry issue: SWH-INDEXER-FB

Jun 29 2022, 11:27 AM · Indexer, Metadata workflow
VickyMerzOwn closed T4275: CffMapping: Add checks for value types, a subtask of T4274: Resolve all known crashes in the metadata indexer, as Resolved.
Jun 29 2022, 11:15 AM · Indexer, Metadata workflow
VickyMerzOwn closed T4275: CffMapping: Add checks for value types as Resolved.
Jun 29 2022, 11:15 AM · Easy hack, Indexer, Metadata workflow

Jun 28 2022

VickyMerzOwn updated the task description for T4357: Metadata Indexer for Composer.
Jun 28 2022, 5:13 PM · Indexer
VickyMerzOwn triaged T4357: Metadata Indexer for Composer as Normal priority.

Sorry about the change. It was by mistake.

Jun 28 2022, 5:11 PM · Indexer
VickyMerzOwn raised the priority of T4357: Metadata Indexer for Composer from Normal to Needs Triage.
Jun 28 2022, 5:11 PM · Indexer
vlorentz triaged T4357: Metadata Indexer for Composer as Normal priority.
Jun 28 2022, 5:03 PM · Indexer
ardumont closed T4348: Deploy swh-indexer v2.0.2 on production and staging as Resolved.
Jun 28 2022, 9:58 AM · Indexer, System administration

Jun 27 2022

ardumont moved T4348: Deploy swh-indexer v2.0.2 on production and staging from in-progress to deployed/landed/monitoring on the System administration board.
Jun 27 2022, 5:48 PM · Indexer, System administration
ardumont changed the status of T4348: Deploy swh-indexer v2.0.2 on production and staging from Open to Work in Progress.
Jun 27 2022, 5:37 PM · Indexer, System administration

Jun 24 2022

VickyMerzOwn added a revision to T4275: CffMapping: Add checks for value types: D8036: Check CFF Value Types.
Jun 24 2022, 8:01 PM · Easy hack, Indexer, Metadata workflow
VickyMerzOwn added a comment to T4275: CffMapping: Add checks for value types.

I'm not so sure what needs to be done here.
If some fields have no values, does it have to skip it or put some default value?

Jun 24 2022, 3:58 PM · Easy hack, Indexer, Metadata workflow

Jun 22 2022

vlorentz updated the task description for T4348: Deploy swh-indexer v2.0.2 on production and staging.
Jun 22 2022, 3:19 PM · Indexer, System administration
vlorentz updated the task description for T4348: Deploy swh-indexer v2.0.2 on production and staging.
Jun 22 2022, 3:19 PM · Indexer, System administration
vlorentz triaged T4348: Deploy swh-indexer v2.0.2 on production and staging as Normal priority.
Jun 22 2022, 3:18 PM · Indexer, System administration

Jun 21 2022

vlorentz assigned T4275: CffMapping: Add checks for value types to VickyMerzOwn.
Jun 21 2022, 1:32 PM · Easy hack, Indexer, Metadata workflow
vlorentz added a subtask for T4274: Resolve all known crashes in the metadata indexer: T4333: test_npm_adversarial fails.
Jun 21 2022, 1:32 PM · Indexer, Metadata workflow
vlorentz added a parent task for T4333: test_npm_adversarial fails: T4274: Resolve all known crashes in the metadata indexer.
Jun 21 2022, 1:32 PM · Indexer
vlorentz closed T4333: test_npm_adversarial fails as Resolved.
Jun 21 2022, 1:32 PM · Indexer
VickyMerzOwn added a revision to T4333: test_npm_adversarial fails: D8007: Fix crash when npm description is not a string.
Jun 21 2022, 11:33 AM · Indexer
VickyMerzOwn closed T4276: CffMapping: ignore invalid yaml files, a subtask of T4274: Resolve all known crashes in the metadata indexer, as Resolved.
Jun 21 2022, 11:27 AM · Indexer, Metadata workflow
VickyMerzOwn closed T4276: CffMapping: ignore invalid yaml files as Resolved.
Jun 21 2022, 11:27 AM · Indexer, Metadata workflow
VickyMerzOwn added a revision to T4276: CffMapping: ignore invalid yaml files: D8002: CffMapping: Ignores invalid yaml files.
Jun 21 2022, 8:54 AM · Indexer, Metadata workflow

Jun 17 2022

VickyMerzOwn updated the language for P1386 normalize_description method in NpmMapping class from autodetect to python.
Jun 17 2022, 6:10 PM · Indexer
VickyMerzOwn edited P1386 normalize_description method in NpmMapping class.
Jun 17 2022, 6:08 PM · Indexer
VickyMerzOwn updated the language for P1386 normalize_description method in NpmMapping class from python to autodetect.
Jun 17 2022, 6:07 PM · Indexer
VickyMerzOwn created P1386 normalize_description method in NpmMapping class.
Jun 17 2022, 6:05 PM · Indexer
vlorentz triaged T4333: test_npm_adversarial fails as Low priority.
Jun 17 2022, 12:19 PM · Indexer
VickyMerzOwn created T4333: test_npm_adversarial fails.
Jun 17 2022, 12:15 PM · Indexer

Jun 15 2022

vlorentz added a revision to T4277: Deal with null characters in the output of the metadata indexer: D7992: npm: Add workaround for mangled package descriptions.
Jun 15 2022, 6:30 PM · Indexer, Metadata workflow

Jun 13 2022

ardumont closed T4319: Deploy indexer v2.0 as Resolved.
Jun 13 2022, 1:11 PM · System administration, Indexer

Jun 10 2022

ardumont updated the task description for T4319: Deploy indexer v2.0.
Jun 10 2022, 5:33 PM · System administration, Indexer
ardumont added a comment to T4319: Deploy indexer v2.0.

workers restarting.
At least one is done and stuff are being written accordingly.

Jun 10 2022, 5:33 PM · System administration, Indexer
ardumont moved T4319: Deploy indexer v2.0 from in-progress to deployed/landed/monitoring on the System administration board.
Jun 10 2022, 5:32 PM · System administration, Indexer
ardumont updated the task description for T4319: Deploy indexer v2.0.
Jun 10 2022, 5:27 PM · System administration, Indexer
ardumont added a comment to T4319: Deploy indexer v2.0.

Migrate schema:

swhstorage@saam:~$ swh db --config-file indexer.yml upgrade indexer --to-version=134 --module-config-key=indexer_storage
INFO:swh.core.db.db_utils:Executing migration script '/usr/lib/python3/dist-packages/swh/indexer/sql/upgrades/134.sql'
Migration to version 134 done
Jun 10 2022, 5:23 PM · System administration, Indexer
ardumont updated the task description for T4319: Deploy indexer v2.0.
Jun 10 2022, 5:21 PM · System administration, Indexer
ardumont added a comment to T4319: Deploy indexer v2.0.
softwareheritage-indexer=# create table origin_intrinsic_metadata_backup as table origin_intrinsic_metadata;
SELECT 22359694
softwareheritage-indexer=# create table revision_intrinsic_metadata_backup as table revision_intrinsic_metadata;
SELECT 16955557
softwareheritage-indexer=# alter table origin_intrinsic_metadata_backup owner to swhstorage;
ALTER TABLE
softwareheritage-indexer=# alter table revision_intrinsic_metadata_backup owner to swhstorage;
ALTER TABLE
Jun 10 2022, 5:20 PM · System administration, Indexer
vlorentz assigned T4276: CffMapping: ignore invalid yaml files to VickyMerzOwn.
Jun 10 2022, 12:26 PM · Indexer, Metadata workflow
ardumont added a comment to T4319: Deploy indexer v2.0.

With those applied ^, workers are happier now.

Jun 10 2022, 10:24 AM · System administration, Indexer
ardumont added a revision to T4319: Deploy indexer v2.0: D7978: upgrades/134: Add missing index creation.
Jun 10 2022, 10:23 AM · System administration, Indexer
ardumont added a comment to T4319: Deploy indexer v2.0.
Jun 10 2022, 10:17 AM · System administration, Indexer
ardumont added a comment to T4319: Deploy indexer v2.0.

staging
...

  • Checks
Jun 10 2022, 10:14 AM · System administration, Indexer
ardumont changed the status of T4319: Deploy indexer v2.0 from Open to Work in Progress.
Jun 10 2022, 10:06 AM · System administration, Indexer
ardumont updated the task description for T4319: Deploy indexer v2.0.
Jun 10 2022, 10:06 AM · System administration, Indexer
ardumont added a comment to T4319: Deploy indexer v2.0.
  • Backup tables that will get dropped [1]
  • current deployed db version: 133 [2]
  • current version to deploy: 134
  • Upgrade db version [3]
Jun 10 2022, 10:04 AM · System administration, Indexer
ardumont updated the task description for T4319: Deploy indexer v2.0.
Jun 10 2022, 10:04 AM · System administration, Indexer
ardumont updated the task description for T4319: Deploy indexer v2.0.
Jun 10 2022, 9:28 AM · System administration, Indexer

Jun 9 2022

ardumont triaged T4319: Deploy indexer v2.0 as Normal priority.
Jun 9 2022, 2:52 PM · System administration, Indexer

Jun 2 2022

ardumont changed the status of T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1 from Work in Progress to Open.
Jun 2 2022, 6:00 PM · System administration, Indexer, Metadata workflow
ardumont changed the status of T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1, a subtask of T4273: Rewrite indexers as journal clients when relevant, from Work in Progress to Open.
Jun 2 2022, 6:00 PM · Indexer, Metadata workflow
ardumont moved T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1 from deployed/landed/monitoring to Backlog on the System administration board.
Jun 2 2022, 6:00 PM · System administration, Indexer, Metadata workflow
ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jun 2 2022, 5:57 PM · System administration, Indexer, Metadata workflow
ardumont added a revision to T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1: D7951: Deploy new origin intrinsic metadata journal client indexer.
Jun 2 2022, 5:57 PM · System administration, Indexer, Metadata workflow
ardumont added a comment to T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.

Reverting:

  • Stopping and disabling journal client services [1]
  • D7950: Revert puppet manifest changes
  • scheduler0.staging: deploy manifest changes [2]
  • workers.staging: Deploy manifest changes [3]
  • check everything is back to normal [4]
Jun 2 2022, 5:53 PM · System administration, Indexer, Metadata workflow
ardumont added a revision to T4274: Resolve all known crashes in the metadata indexer: D7950: staging: Revert indexer journal client deployment.
Jun 2 2022, 5:46 PM · Indexer, Metadata workflow
ardumont added a parent task for T4274: Resolve all known crashes in the metadata indexer: T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jun 2 2022, 5:37 PM · Indexer, Metadata workflow
ardumont added a subtask for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1: T4274: Resolve all known crashes in the metadata indexer.
Jun 2 2022, 5:37 PM · System administration, Indexer, Metadata workflow
ardumont added a comment to T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.

This needs to be reverted in waiting for [1] to be resolved.
I'll attend to it tomorrow.

Jun 2 2022, 4:38 PM · System administration, Indexer, Metadata workflow

Jun 1 2022

ardumont moved T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1 from in-progress to deployed/landed/monitoring on the System administration board.
Jun 1 2022, 5:37 PM · System administration, Indexer, Metadata workflow
ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jun 1 2022, 5:37 PM · System administration, Indexer, Metadata workflow
ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jun 1 2022, 5:22 PM · System administration, Indexer, Metadata workflow
ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jun 1 2022, 5:15 PM · System administration, Indexer, Metadata workflow
ardumont renamed T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1 from staging: Deploy new origin intrinsic metadata journal client indexer v1.1 to staging: Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jun 1 2022, 5:09 PM · System administration, Indexer, Metadata workflow
vlorentz added a revision to T4273: Rewrite indexers as journal clients when relevant: D7899: Add support for indexing directly from the journal client.
Jun 1 2022, 4:46 PM · Indexer, Metadata workflow
vlorentz added a revision to T4273: Rewrite indexers as journal clients when relevant: D7940: Switch origin-intrinsic-metadata from celery- to journal-based workers.
Jun 1 2022, 4:46 PM · Indexer, Metadata workflow
ardumont updated the task description for T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.
Jun 1 2022, 4:33 PM · System administration, Indexer, Metadata workflow
ardumont changed the status of T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1 from Open to Work in Progress.
Jun 1 2022, 11:56 AM · System administration, Indexer, Metadata workflow
ardumont changed the status of T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1, a subtask of T4273: Rewrite indexers as journal clients when relevant, from Open to Work in Progress.
Jun 1 2022, 11:56 AM · Indexer, Metadata workflow
ardumont added a project to T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1: System administration.
Jun 1 2022, 11:56 AM · System administration, Indexer, Metadata workflow
ardumont added a comment to T4282: Deploy new origin intrinsic metadata journal client indexer > v1.1.

Should be ready to be deployed now.

Jun 1 2022, 11:56 AM · System administration, Indexer, Metadata workflow