Page MenuHomeSoftware Heritage
Feed All Stories

Oct 15 2021

vlorentz added a comment to D6470: Make it explicit that the "main" docs page is actually devel doc.

I don't know if it's really relevant. The docs/ folders of other packages are the devel docs too; and I feel that adding a symlink here might just make things more confusing when debugging.

Oct 15 2021, 11:34 AM
anlambert added a reviewer for D6484: jobs/swh-environment: Simplify build script and workaround pip issue: Reviewers.
Oct 15 2021, 11:29 AM
anlambert added a comment to T3663: Make the swh-environment jenkins job green and activate notifications.

I pushed D6484 to fix the build issue. Instead of pinning pip version, I used the --no-use-pep517 pip option as suggested in the pip github issue.

Oct 15 2021, 11:28 AM · System administration
anlambert added a revision to T3663: Make the swh-environment jenkins job green and activate notifications: D6484: jobs/swh-environment: Simplify build script and workaround pip issue.
Oct 15 2021, 11:26 AM · System administration
anlambert requested review of D6484: jobs/swh-environment: Simplify build script and workaround pip issue.
Oct 15 2021, 11:26 AM
ardumont updated the task description for T3664: Activate sentry for journal clients.
Oct 15 2021, 11:25 AM · System administration, Monitoring
vlorentz updated the diff for D6472: Add a script for a 'monthly roadmap report' bot email.

install on pergamon

Oct 15 2021, 11:23 AM
vlorentz added inline comments to D6472: Add a script for a 'monthly roadmap report' bot email.
Oct 15 2021, 11:22 AM
ardumont added a revision to T3664: Activate sentry for journal clients: D6483: Activate sentry for scheduler journal client.
Oct 15 2021, 11:22 AM · System administration, Monitoring
ardumont requested review of D6483: Activate sentry for scheduler journal client.
Oct 15 2021, 11:22 AM
vlorentz added a comment to T75: Check integrity of directories, revisions, and releases.

analysis on directories (some are also part of the fixable_trivial above, but I don't have the exact number, I lost it in my analysis):

Oct 15 2021, 11:21 AM · Archive content, Restricted Project
ardumont updated the task description for T3664: Activate sentry for journal clients.
Oct 15 2021, 11:19 AM · System administration, Monitoring
ardumont closed D6482: Activate sentry for indexer journal client.
Oct 15 2021, 11:18 AM
ardumont committed rSPSITE8805a9e64d1e: Activate sentry for indexer journal client (authored by ardumont).
Activate sentry for indexer journal client
Oct 15 2021, 11:18 AM
vsellier accepted D6482: Activate sentry for indexer journal client.

LGTM

Oct 15 2021, 11:12 AM
vsellier added a comment to T3630: staging - journal0 needs more space.

This migrations are still in progress:

root@storage1:/opt/kafka/bin# ./kafka-reassign-partitions.sh --bootstrap-server $SERVER --list | cut -f1 -d"-" | uniq -c
      1 __consumer_offsets
     64 swh.journal.objects.content
     64 swh.journal.objects.directory
      2 swh.journal.objects.metadata_authority
      1 swh.journal.objects.metadata_fetcher
     64 swh.journal.objects.raw_extrinsic_metadata
     64 swh.journal.objects.revision
     64 swh.journal.objects_privileged.revision
Oct 15 2021, 11:10 AM · System administration
ardumont changed the status of T3664: Activate sentry for journal clients from Open to Work in Progress.
Oct 15 2021, 11:10 AM · System administration, Monitoring
ardumont requested review of D6482: Activate sentry for indexer journal client.
Oct 15 2021, 11:10 AM
ardumont added a revision to T3664: Activate sentry for journal clients: D6482: Activate sentry for indexer journal client.
Oct 15 2021, 11:10 AM · System administration, Monitoring
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDCNTbaca25faf484: Updated backport on buster-swh from debian/0.9.0-1_swh1 (unstable-swh) (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated backport on buster-swh from debian/0.9.0-1_swh1 (unstable-swh)
Oct 15 2021, 11:04 AM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDCNT03cebdb8b99f: Merge tag 'debian/0.9.0-1_swh1' into debian/buster-swh (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Merge tag 'debian/0.9.0-1_swh1' into debian/buster-swh
Oct 15 2021, 11:04 AM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDCNT94ff8b4c857d: pristine-tar data for swh.counters_0.9.0.orig.tar.gz (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
pristine-tar data for swh.counters_0.9.0.orig.tar.gz
Oct 15 2021, 11:02 AM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDCNTb866530bc421: Updated debian changelog for version 0.9.0 (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated debian changelog for version 0.9.0
Oct 15 2021, 11:02 AM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDCNT5bd476b2be75: Update upstream source from tag 'debian/upstream/0.9.0' (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Update upstream source from tag 'debian/upstream/0.9.0'
Oct 15 2021, 11:02 AM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDCNTf66442f2f058: New upstream version 0.9.0 (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
New upstream version 0.9.0
Oct 15 2021, 11:02 AM
anlambert added a comment to T3663: Make the swh-environment jenkins job green and activate notifications.

I quickly hacked on Jenkins job to remove the pip upgrade but the default pip version on the docker image is too outdated
and some swh dependencies will fail to install, see console output.

Oct 15 2021, 11:02 AM · System administration
ardumont closed T3619: Enable Sentry for swh-graph as Resolved.
Oct 15 2021, 11:01 AM · System administration, Sentry
ardumont moved T3664: Activate sentry for journal clients from Backlog to Weekly backlog on the System administration board.
Oct 15 2021, 10:59 AM · System administration, Monitoring
ardumont added projects to T3664: Activate sentry for journal clients: Monitoring, System administration.
Oct 15 2021, 10:58 AM · System administration, Monitoring
ardumont triaged T3664: Activate sentry for journal clients as Normal priority.
Oct 15 2021, 10:58 AM · System administration, Monitoring
vsellier closed D6481: Revert "journal_client: Add origins processing".
Oct 15 2021, 10:58 AM
vsellier added a reverting change for rDCNTcd595e71aef4: journal_client: Add origins processing: rDCNTce8e5e88c840: Revert "journal_client: Add origins processing".
Oct 15 2021, 10:58 AM
vsellier added a reverting change for D5910: journal_client: Add origins processing: rDCNTce8e5e88c840: Revert "journal_client: Add origins processing".
Oct 15 2021, 10:58 AM
vsellier committed rDCNTce8e5e88c840: Revert "journal_client: Add origins processing" (authored by vsellier).
Revert "journal_client: Add origins processing"
Oct 15 2021, 10:58 AM
ardumont moved T3619: Enable Sentry for swh-graph from code-review/monitoring to deployed/landed on the System administration board.
Oct 15 2021, 10:55 AM · System administration, Sentry
anlambert added a comment to D6479: bib/install: disable pip self ugprade.

There is a typo in commit message: s/bib/bin/.

Oct 15 2021, 10:52 AM
ardumont closed D6480: Activate sentry for swh.graph.
Oct 15 2021, 10:52 AM
ardumont committed rSPSITEd69b1ea5a682: Activate sentry for swh.graph (authored by ardumont).
Activate sentry for swh.graph
Oct 15 2021, 10:52 AM
anlambert accepted D6481: Revert "journal_client: Add origins processing".
Oct 15 2021, 10:50 AM
vsellier requested review of D6481: Revert "journal_client: Add origins processing".
Oct 15 2021, 10:50 AM
vsellier added a revision to T3659: staging - counters journal client failed to process origins: D6481: Revert "journal_client: Add origins processing".
Oct 15 2021, 10:48 AM · Counters
vsellier added a reverting change for D5910: journal_client: Add origins processing: D6481: Revert "journal_client: Add origins processing".
Oct 15 2021, 10:48 AM
vsellier added a reverting change for rDCNTcd595e71aef4: journal_client: Add origins processing: D6481: Revert "journal_client: Add origins processing".
Oct 15 2021, 10:48 AM
douardda triaged T3663: Make the swh-environment jenkins job green and activate notifications as High priority.
Oct 15 2021, 10:45 AM · System administration
vsellier accepted D6480: Activate sentry for swh.graph.

LGTM thanks

Oct 15 2021, 10:43 AM
ardumont closed T3662: Activate sentry for counter journal client as Resolved.
Oct 15 2021, 10:42 AM · System administration, Counters
ardumont moved T3619: Enable Sentry for swh-graph from in-progress to code-review/monitoring on the System administration board.
Oct 15 2021, 10:42 AM · System administration, Sentry
ardumont changed the status of T3619: Enable Sentry for swh-graph from Open to Work in Progress.
Oct 15 2021, 10:42 AM · System administration, Sentry
ardumont committed rSPPRIVC06952c7f3e67: Update censored data (authored by ardumont).
Update censored data
Oct 15 2021, 10:41 AM
ardumont requested review of D6480: Activate sentry for swh.graph.
Oct 15 2021, 10:33 AM
ardumont added a revision to T3619: Enable Sentry for swh-graph: D6480: Activate sentry for swh.graph.
Oct 15 2021, 10:33 AM · System administration, Sentry
ardumont moved T3619: Enable Sentry for swh-graph from Backlog to Weekly backlog on the System administration board.
Oct 15 2021, 10:29 AM · System administration, Sentry
ardumont moved T3662: Activate sentry for counter journal client from code-review/monitoring to deployed/landed on the System administration board.
Oct 15 2021, 10:29 AM · System administration, Counters
ardumont moved T3662: Activate sentry for counter journal client from in-progress to code-review/monitoring on the System administration board.
Oct 15 2021, 10:28 AM · System administration, Counters
ardumont changed the status of T3662: Activate sentry for counter journal client from Open to Work in Progress.
Oct 15 2021, 10:28 AM · System administration, Counters
ardumont edited projects for T3662: Activate sentry for counter journal client, added: System administration; removed System administrators.
Oct 15 2021, 10:28 AM · System administration, Counters
ardumont edited projects for T3639: prepare quote for "granet2", next gen swh-graph compression server, added: System administration; removed System administrators.
Oct 15 2021, 10:28 AM · System administration
ardumont edited projects for T3579: Meta-task: upgrade infrastructure to Debian Bullseye, added: System administration; removed System administrators.
Oct 15 2021, 10:28 AM · System administration (Component upgrades)
ardumont closed D6478: Activate sentry for counter journal client.
Oct 15 2021, 10:25 AM
ardumont committed rSPSITE9acf27203f76: Activate sentry for counter journal client (authored by ardumont).
Activate sentry for counter journal client
Oct 15 2021, 10:25 AM
vsellier accepted D6478: Activate sentry for counter journal client.

thanks \o/

Oct 15 2021, 10:24 AM
ardumont updated the diff for D6478: Activate sentry for counter journal client.

rebase

Oct 15 2021, 10:24 AM
olasd accepted D6478: Activate sentry for counter journal client.
Oct 15 2021, 10:23 AM
olasd accepted D6479: bib/install: disable pip self ugprade.
Oct 15 2021, 10:22 AM
zack requested review of D6479: bib/install: disable pip self ugprade.
Oct 15 2021, 10:20 AM
ardumont added a reviewer for D6478: Activate sentry for counter journal client: System administrators.
Oct 15 2021, 10:19 AM
ardumont requested review of D6478: Activate sentry for counter journal client.
Oct 15 2021, 10:18 AM
ardumont added a revision to T3662: Activate sentry for counter journal client: D6478: Activate sentry for counter journal client.
Oct 15 2021, 10:18 AM · System administration, Counters
vsellier added a project to T3662: Activate sentry for counter journal client: System administrators.
Oct 15 2021, 10:12 AM · System administration, Counters
ardumont triaged T3662: Activate sentry for counter journal client as Normal priority.
Oct 15 2021, 10:08 AM · System administration, Counters
ardumont closed T3648: Fix swh-docs's broken dev build as Resolved.

It's finally green! Closing this.

Oct 15 2021, 10:03 AM · Documentation
vsellier updated subscribers of T3659: staging - counters journal client failed to process origins.

@anlambert according to T3402#67318 the counters are not used to count the origins per forge. Is this portion of code still needed ?

Oct 15 2021, 9:56 AM · Counters
ardumont changed the status of T3658: Reference bitbucket mercurial origins, a subtask of T3338: Load the archived bitbucket mercurial repositories, from Open to Work in Progress.
Oct 15 2021, 9:49 AM · System administration, Mercurial loader
ardumont changed the status of T3658: Reference bitbucket mercurial origins from Open to Work in Progress.
Oct 15 2021, 9:49 AM · System administration, Mercurial loader
ardumont added a comment to T3658: Reference bitbucket mercurial origins.

A first simple solution has been implemented in the webapp for now [1].
It's not deployed yet.

Oct 15 2021, 9:48 AM · System administration, Mercurial loader
vsellier added a comment to T3659: staging - counters journal client failed to process origins.

This is the content of the netloc causing the exception:

Oct 15 07:45:21 counters0 swh[1254115]: INFO:swh.counters.journal_client:origin_netloc:bitbucket.org : {'https://bitbucket.org/trackerlab/boxes-public.git', 'https://bitbucket.org/fet_lab/neurotables.git', 'https://bitbucket.org/DataMinerUK/infinite-interns-2.git'}
Oct 15 2021, 9:47 AM · Counters
vsellier changed the status of T3659: staging - counters journal client failed to process origins from Open to Work in Progress.
Oct 15 2021, 9:46 AM · Counters
ardumont renamed T3658: Reference bitbucket mercurial origins from Reference bitbucket mercurial origins in scheduler metrics to Reference bitbucket mercurial origins.
Oct 15 2021, 9:45 AM · System administration, Mercurial loader
ardumont closed T3315: archive SourceForge as Resolved.
Oct 15 2021, 9:44 AM · Archive coverage
ardumont closed T735: SourceForge lister, a subtask of T3315: archive SourceForge, as Resolved.
Oct 15 2021, 9:44 AM · Archive coverage
ardumont closed T735: SourceForge lister as Resolved.
Oct 15 2021, 9:44 AM · Origin-SourceForge
ardumont closed T3470: lister-sourceforge: Activate sourceforge origins when listed as Resolved.
Oct 15 2021, 9:44 AM · System administration, Archive coverage, Origin-SourceForge
ardumont closed T3470: lister-sourceforge: Activate sourceforge origins when listed, a subtask of T3374: Ingest sourceforge repositories (origins of type git, svn, hg), as Resolved.
Oct 15 2021, 9:44 AM · System administration, Archive coverage, Origin-SourceForge
ardumont moved T3470: lister-sourceforge: Activate sourceforge origins when listed from code-review/monitoring to deployed/landed on the System administration board.
Oct 15 2021, 9:43 AM · System administration, Archive coverage, Origin-SourceForge
ardumont added a comment to T3470: lister-sourceforge: Activate sourceforge origins when listed.

Sourceforge origins are progressively activated back and getting ingested along the way [1].

Oct 15 2021, 9:43 AM · System administration, Archive coverage, Origin-SourceForge
ardumont committed rDDOC015a2b07ad8f: Drop swh.loader.cvs.rcsparse from autodoc_mock_imports settings (authored by ardumont).
Drop swh.loader.cvs.rcsparse from autodoc_mock_imports settings
Oct 15 2021, 9:37 AM
zack updated subscribers of T3656: Survey revisions/releases with partially loaded history.
Oct 15 2021, 9:34 AM · Archive content
zack added a comment to T3656: Survey revisions/releases with partially loaded history.
In T3656#72364, @grouss wrote:

according to the list of nodes provided by seirl there were ~21,000,000 revisions without ancestors according to swh-graph snapshot (2020-12-15)

Oct 15 2021, 9:33 AM · Archive content
grouss added a comment to T3656: Survey revisions/releases with partially loaded history.

according to the list of nodes provided by seirl there were ~21,000,000 revisions without ancestors according to swh-graph snapshot (2020-12-15)
checking in the current live swh DAG 2 days ago 98% have one in release or snapshot_branch.
indeed I was surprised because I did'nt have to loop over the revision history.

Oct 15 2021, 9:25 AM · Archive content
olasd added a project to T3660: Nodes with missing ancestors in SWH DAG / SWH-graph: Archive content.
Oct 15 2021, 9:17 AM · Archive content
ardumont added a comment to T3656: Survey revisions/releases with partially loaded history.

You might be interested by what @grouss just opened in T3660
(ah scratched that, zack already mentioned it)

Oct 15 2021, 9:07 AM · Archive content
ardumont updated the task description for T3661: docs: Activate build on docs diff.
Oct 15 2021, 9:05 AM · Documentation
ardumont updated the task description for T3661: docs: Activate build on docs diff.
Oct 15 2021, 9:04 AM · Documentation
ardumont triaged T3661: docs: Activate build on docs diff as Normal priority.
Oct 15 2021, 9:04 AM · Documentation
ardumont added a comment to T3648: Fix swh-docs's broken dev build.

Note that activating that build on diff will take some time...
I mean the build sphinx-dev takes a long time to finish.

Oct 15 2021, 9:02 AM · Documentation
ardumont added a comment to T3648: Fix swh-docs's broken dev build.

Looks like the only remaining warning is the following: WARNING: A mocked object is
detected: 'swh.loader.cvs.rcsparse'.

Oct 15 2021, 9:01 AM · Documentation
zack triaged T3660: Nodes with missing ancestors in SWH DAG / SWH-graph as Low priority.
Oct 15 2021, 8:56 AM · Archive content
zack added a parent task for T3660: Nodes with missing ancestors in SWH DAG / SWH-graph: T3656: Survey revisions/releases with partially loaded history.
Oct 15 2021, 8:56 AM · Archive content
zack added a subtask for T3656: Survey revisions/releases with partially loaded history: T3660: Nodes with missing ancestors in SWH DAG / SWH-graph.
Oct 15 2021, 8:56 AM · Archive content
zack updated subscribers of T3656: Survey revisions/releases with partially loaded history.

In T3660, @grouss has found many more.
Might be for a different reason (the dataset he analyzed is not the live one), but it's worth a comparison.

Oct 15 2021, 8:55 AM · Archive content