yes, agreed.

Analyzing further the suggestions using the deprecated swh-lister cache db table as data
point (production data) [1], 3 instances so far will generate sometimes wrong origin
urls with the suggested approach.

Thanks @ardumont for experimenting with this. The 500 seems normal: we need to tell Eclipse about us first, I'll put you in touch. So maybe it's still a no-brainer, and we just need to document the "contant the owner to get whitelisted" human step :-)

Current status of all listings:

npm listing done, so status ok as well:

Build is green

Add mapping for definitions and harvests
Add functions map_row, map_definition, map_harvest to check whether swh archive is able to map clearlydefined object or not

Build has FAILED

Add mapping for definitions and harvests
Add functions map_row, map_definition, map_harvest to check whether swh archive is able to map clearlydefined object or not

Build is green

Add mapping for definitions and harvests
Add functions map_row, map_definition, map_harvest to check whether swh archive is able to map clearlydefined object or not

Bloom filters are still on the table for other use cases, like testing super quickly for contents that we do not have, but if nobody has strong objections, this seems the way to go for the counters (very small footprint, small under/over counting errors, thanks Philippe Flajolet's magic :-))

Nice catch

npm run scheduled, run in progress:

pypi run triggered itself and went well all alone (cool):

Status lister: ok (with local patch):

Build is green

Add mapping for definitions and harvests
Add functions map_row, map_definition, map_harvest to check whether swh archive is able to map clearlydefined object or not

Ticket opened via the dell support.
The disk should be delivered the Monday 1st February 2021, the DSI is informed

I thought having fixed that bug (also encountered when developing the lister).

Build is green

lister-cran status: run ko [1]

swhworker@worker0:~$ SWH_CONFIG_FILENAME=/etc/softwareheritage/lister.yml swh lister run --lister cran
Traceback (most recent call last):
  File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.11.0', 'console_scripts', 'swh')()
...
  File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 344, in raise_for_status
    raise exception from None
swh.core.api.RemoteException: <RemoteException 500 TypeError: ["can not serialize 'Attribute' object"]>

[1] https://sentry.softwareheritage.org/share/issue/2cd53c7575834b1aaf65760b80bcbcef/

Rebase

:)

It's ok now.

Yes, I completely forgot about your diff that fixed it.

Node patch D4965, this gets better, the launchpad listed origins:

Nevertheless, errors like T3003#57551 can still appear if there is duplicate origins in the sent list.

In the context of deploying the next gen lister in staging (T2998), i also tried the eclipse cgit instance

Looks good to me !

I thought having fixed that bug (also encountered when developing the lister).

Note that this lister seems to need some writing improvments though.
It seemed to have flushed the writing only at the end of the listing.
If that's the real behavior (i'll need to check), that won't bode well for relatively high dimensioned instance like the cgit eclispe instance for example.

cgit lister should flush origins after each page, which instance has been listed here ?

Some listers like debian might flush a large amount of origins per page, will be curious to see how it goes.

lister launchpad run ko, see details [1]

Make them flush their current state at regular interval sounds like a better behavior.

My guess is that swh-scheduler 0.9.2 is not installed on staging, issue has been fixed in rDSCH2906b4e8a08517f5e3d86272232ad8ba926a43d7.