I don't know if it's really relevant. The docs/ folders of other packages are the devel docs too; and I feel that adding a symlink here might just make things more confusing when debugging.

I pushed D6484 to fix the build issue. Instead of pinning pip version, I used the --no-use-pep517 pip option as suggested in the pip github issue.

install on pergamon

analysis on directories (some are also part of the fixable_trivial above, but I don't have the exact number, I lost it in my analysis):

LGTM

This migrations are still in progress:

root@storage1:/opt/kafka/bin# ./kafka-reassign-partitions.sh --bootstrap-server $SERVER --list | cut -f1 -d"-" | uniq -c
      1 __consumer_offsets
     64 swh.journal.objects.content
     64 swh.journal.objects.directory
      2 swh.journal.objects.metadata_authority
      1 swh.journal.objects.metadata_fetcher
     64 swh.journal.objects.raw_extrinsic_metadata
     64 swh.journal.objects.revision
     64 swh.journal.objects_privileged.revision

I quickly hacked on Jenkins job to remove the pip upgrade but the default pip version on the docker image is too outdated
and some swh dependencies will fail to install, see console output.

There is a typo in commit message: s/bib/bin/.

LGTM thanks

thanks \o/

rebase

It's finally green! Closing this.

@anlambert according to T3402#67318 the counters are not used to count the origins per forge. Is this portion of code still needed ?

A first simple solution has been implemented in the webapp for now [1].
It's not deployed yet.

This is the content of the netloc causing the exception:

Oct 15 07:45:21 counters0 swh[1254115]: INFO:swh.counters.journal_client:origin_netloc:bitbucket.org : {'https://bitbucket.org/trackerlab/boxes-public.git', 'https://bitbucket.org/fet_lab/neurotables.git', 'https://bitbucket.org/DataMinerUK/infinite-interns-2.git'}

Sourceforge origins are progressively activated back and getting ingested along the way [1].

In T3656#72364, @grouss wrote:

according to the list of nodes provided by seirl there were ~21,000,000 revisions without ancestors according to swh-graph snapshot (2020-12-15)

according to the list of nodes provided by seirl there were ~21,000,000 revisions without ancestors according to swh-graph snapshot (2020-12-15)
checking in the current live swh DAG 2 days ago 98% have one in release or snapshot_branch.
indeed I was surprised because I did'nt have to loop over the revision history.

You might be interested by what @grouss just opened in T3660
(ah scratched that, zack already mentioned it)

Note that activating that build on diff will take some time...
I mean the build sphinx-dev takes a long time to finish.

Looks like the only remaining warning is the following: WARNING: A mocked object is
detected: 'swh.loader.cvs.rcsparse'.

In T3660, @grouss has found many more.
Might be for a different reason (the dataset he analyzed is not the live one), but it's worth a comparison.