In D8553#222288, @vlorentz wrote:

Although, as using extra_loader_arguments as an escape hatch to pass lists of artifacts is starting to spread across lister/loader pairs (GNU, Maven, Arch, AUR, CRAN, Crates, and maybe OPAM), I would be fine with defining a new way to pass these objects (as well as arch_metadata/aur_metadata/crates_metadata?), but it should be done for all these loaders and using the same format.

(This will also make it easier to change the storage of these from large JSON objects to rows in the scheduler DB in the future, if their size becomes an issue for postgresql or celery; but not now)

Sep 27 2022, 6:10 PM

anlambert added a comment to D8553: arch/aur: Use dicts instead of lists as extra loader arguments.

My point is that we should not create dozens of undocumented ad-hoc formats in the scheduler database just to remove two lines of code from each loader

Sep 27 2022, 6:03 PM

anlambert added a comment to D8553: arch/aur: Use dicts instead of lists as extra loader arguments.

In D8553#222284, @vlorentz wrote:

It avoids the proliferation of ad-hoc formats

Sep 27 2022, 5:51 PM

anlambert added a comment to D8553: arch/aur: Use dicts instead of lists as extra loader arguments.

In D8553#222275, @anlambert wrote:

In D8553#222234, @vlorentz wrote:

They intentionally use the same format as https://docs.softwareheritage.org/devel/swh-storage/extrinsic-metadata-specification.html#original-artifacts-json

And so ? This makes the loader simpler and has no effect on what we archive, so better using dicts here.

Sep 27 2022, 5:50 PM

anlambert added a comment to D8553: arch/aur: Use dicts instead of lists as extra loader arguments.

In D8553#222234, @vlorentz wrote:

They intentionally use the same format as https://docs.softwareheritage.org/devel/swh-storage/extrinsic-metadata-specification.html#original-artifacts-json

Sep 27 2022, 5:43 PM

anlambert updated the diff for D8535: docker/conf: Declare task queues for new listers and loaders.

Add cpan lister queue

Sep 27 2022, 5:26 PM

anlambert requested review of D8553: arch/aur: Use dicts instead of lists as extra loader arguments.

Sep 27 2022, 5:21 PM

anlambert requested review of D8554: arch/aur: Align loader parameters with lister output.

Sep 27 2022, 5:20 PM

anlambert added a revision to T4233: Ingest Arch Linux: D8554: arch/aur: Align loader parameters with lister output.

Sep 27 2022, 5:16 PM · Arch loader, Arch Lister, Archive coverage

anlambert added a revision to T4466: Ingest Aur repository (Arch User Repository): D8554: arch/aur: Align loader parameters with lister output.

Sep 27 2022, 5:16 PM · AUR loader, AUR lister, Archive coverage

anlambert added a revision to T4466: Ingest Aur repository (Arch User Repository): D8553: arch/aur: Use dicts instead of lists as extra loader arguments.

Sep 27 2022, 5:16 PM · AUR loader, AUR lister, Archive coverage

anlambert added a revision to T4233: Ingest Arch Linux: D8553: arch/aur: Use dicts instead of lists as extra loader arguments.

Sep 27 2022, 5:16 PM · Arch loader, Arch Lister, Archive coverage

anlambert accepted D8528: Nuget: Lister for NuGet the package manager for .NET.

Sep 27 2022, 4:22 PM

anlambert closed D8550: developer-setup: Update packages list and repo URL.

Sep 27 2022, 3:15 PM

anlambert closed D8548: developer-setup: Add instructions to install latest yarn version.

Sep 27 2022, 3:15 PM

anlambert committed rDDOC2824e64cb192: developer-setup: Use https protocol for apt.postgresql.org (authored by anlambert).

developer-setup: Use https protocol for apt.postgresql.org

Sep 27 2022, 3:15 PM

anlambert committed rDDOC4aaa6749a898: developer-setup: Add instructions to install latest yarn version (authored by anlambert).

developer-setup: Add instructions to install latest yarn version

Sep 27 2022, 3:15 PM

anlambert committed rDDOCff12b8c8e9df: developer-setup: Remove postgresql-autodoc from debian packages list (authored by anlambert).

developer-setup: Remove postgresql-autodoc from debian packages list

Sep 27 2022, 3:15 PM

anlambert added a comment to D8542: Cpan: List Perl module origins from cpan.org.

In D8542#222044, @franckbret wrote:

In D8542#221864, @anlambert wrote:

@franckbret , have you considered exploiting the https://fastapi.metacpan.org/v1/release/_search endpoint of the CPAN elasticsearch ?

It seems to list all CPAN releases with dates, links to tarballs and checksums. You could build a list of artifacts for each package as in the crates loader
and pass them as loader arguments.

Thanks for the review.

Yes I have checked this one too. When exploring CPAN I found a lot of inconsistencies when parsing data, so i've gone a simplier way.
The main cons is that /release/ returns a lot of useless data and a big amount of records, the idea is that the loader will load related versions via https://fastapi.metacpan.org/v1/release/versions/{pkgname}

Let's talk about this next week.

Sep 27 2022, 3:10 PM

anlambert requested review of D8550: developer-setup: Update packages list and repo URL.

Sep 27 2022, 2:55 PM

anlambert requested review of D8548: developer-setup: Add instructions to install latest yarn version.

Sep 27 2022, 2:15 PM

anlambert requested changes to D8171: crates: Loader implements incremental mode.

The loader requires some adaptation to align with lister output.

Sep 27 2022, 2:12 PM

anlambert added a comment to D8535: docker/conf: Declare task queues for new listers and loaders.

In D8535#221927, @ardumont wrote:

Are those new listers landed yet?

Not yet but they will be soon and it allows to test them in docker environment.

ok. If we land this now and someone starts from scratch or rebuild its local dev, that will break though.

Sep 27 2022, 2:08 PM

anlambert requested changes to D8454: Crates.io: Add last_update for each version of a crate.

@franckbret, I added the improvements for the crates lister we discussed last week as inline comments.

Sep 27 2022, 12:18 PM

anlambert closed D8537: browse: Add 'Save again' button in origin visits view.

Sep 27 2022, 11:50 AM

anlambert committed rDWAPPS4a03dc7bf6af: browse: Add 'Save again' button in origin visits view (authored by anlambert).

browse: Add 'Save again' button in origin visits view

Sep 27 2022, 11:50 AM

anlambert requested review of D8537: browse: Add 'Save again' button in origin visits view.

Sep 27 2022, 11:43 AM

anlambert accepted D8531: Puppet: Lister for Puppet modules.

Sep 27 2022, 11:23 AM

anlambert added a comment to D8542: Cpan: List Perl module origins from cpan.org.

@franckbret , have you considered exploiting the https://fastapi.metacpan.org/v1/release/_search endpoint of the CPAN elasticsearch ?

Sep 27 2022, 11:15 AM

anlambert added a comment to D8535: docker/conf: Declare task queues for new listers and loaders.

In D8535#221690, @ardumont wrote:

Are those new listers landed yet?

Sep 27 2022, 11:03 AM

anlambert accepted D8338: Hackage: List origins from hackage.haskell.org, The Haskell Package Repository.

Looks good to me !

Sep 27 2022, 10:47 AM

anlambert accepted D8517: Conda: List origins from anaconda.com, the Package, dependency and environment management for any language.

Looks good to me, thanks !

Sep 27 2022, 10:45 AM

Sep 26 2022

anlambert added a revision to T4553: Add “Save code now” to the Visits page: D8537: browse: Add 'Save again' button in origin visits view.

Sep 26 2022, 5:17 PM · Web app

anlambert requested changes to D8528: Nuget: Lister for NuGet the package manager for .NET.

Overall looks good to me, just a couple of inline comments to handle.

Sep 26 2022, 4:30 PM

anlambert requested changes to D8517: Conda: List origins from anaconda.com, the Package, dependency and environment management for any language.

Still some changes to bring to that diff now HTTP requests and user-agent setting codes have been deduplicated.

Sep 26 2022, 4:03 PM

anlambert requested review of D8536: Update instructions for running a lister in docker.

Sep 26 2022, 3:59 PM

anlambert requested review of D8535: docker/conf: Declare task queues for new listers and loaders.

Sep 26 2022, 3:39 PM

anlambert requested changes to D8379: Hackage: Loads Hackage Listed origins.

That diff requires some changes as the api_info function got renamed (see inline comments).

Sep 26 2022, 3:24 PM

anlambert requested changes to D8338: Hackage: List origins from hackage.haskell.org, The Haskell Package Repository.

Sep 26 2022, 2:50 PM

anlambert committed rDLDBASE26fe954bd9dc: golang: Fix imports ordering reported by isort (authored by anlambert).

golang: Fix imports ordering reported by isort

Sep 26 2022, 1:45 PM

anlambert added a comment to T4553: Add “Save code now” to the Visits page.

I think it is better to reuse the "Save again" button from code tab.

Sep 26 2022, 12:41 PM · Web app

anlambert triaged T4553: Add “Save code now” to the Visits page as Normal priority.

Sep 26 2022, 12:38 PM · Web app

anlambert triaged T4552: Add link to search for origins loaded from an add forge now request as Normal priority.

Sep 26 2022, 11:44 AM · Web app, Add Forge Now

anlambert closed D8523: Update value of User-Agent HTTP request header used by listers.

Sep 26 2022, 11:36 AM

anlambert closed D8520: Refactor and deduplicate HTTP requests code in listers.

Sep 26 2022, 11:36 AM

anlambert committed rDLSdb6ce12e9e3b: Refactor and deduplicate HTTP requests code in listers (authored by anlambert).

Refactor and deduplicate HTTP requests code in listers

Sep 26 2022, 11:36 AM

anlambert committed rDLSd5c30a3ce371: Update value of User-Agent HTTP request header used by listers (authored by anlambert).

Update value of User-Agent HTTP request header used by listers

Sep 26 2022, 11:36 AM

anlambert closed D8519: Use generic HTTP retry policy by default and rename dedicated decorator.

Sep 26 2022, 11:36 AM

anlambert committed rDLS9c55acd28609: Use generic HTTP retry policy by default and rename dedicated decorator (authored by anlambert).

Use generic HTTP retry policy by default and rename dedicated decorator

Sep 26 2022, 11:36 AM

anlambert updated the diff for D8523: Update value of User-Agent HTTP request header used by listers.

Rebase

Sep 26 2022, 10:51 AM

anlambert updated the diff for D8520: Refactor and deduplicate HTTP requests code in listers.

Rebase

Sep 26 2022, 10:50 AM

anlambert updated the diff for D8519: Use generic HTTP retry policy by default and rename dedicated decorator.

Rebase

Sep 26 2022, 10:50 AM

anlambert closed D8516: cypress: Use click command instead of submit one for form submission.

Sep 26 2022, 10:47 AM

anlambert committed rDWAPPS772928988c91: cypress: Use click command instead of submit one for form submission (authored by anlambert).

cypress: Use click command instead of submit one for form submission

Sep 26 2022, 10:47 AM

Sep 22 2022

anlambert requested review of D8523: Update value of User-Agent HTTP request header used by listers.

Sep 22 2022, 3:54 PM

Sep 21 2022

anlambert requested review of D8520: Refactor and deduplicate HTTP requests code in listers.

Sep 21 2022, 8:14 PM

anlambert requested review of D8519: Use generic HTTP retry policy by default and rename dedicated decorator.

Sep 21 2022, 5:07 PM

anlambert requested changes to D8517: Conda: List origins from anaconda.com, the Package, dependency and environment management for any language.

Overall, looks god to me but there is still some rooms for a couple of improvements (see inline comments).

Sep 21 2022, 4:17 PM

anlambert committed rDLDBASEe1f2135f241c: package/utils: rename api_info function to get_url_body (authored by anlambert).

package/utils: rename api_info function to get_url_body

Sep 21 2022, 3:49 PM

anlambert closed D8495: package/utils: Add debug log and throttling retry to api_info.

Sep 21 2022, 3:49 PM

anlambert committed rDLDBASEbcbb554e625f: package/utils: Add debug log and throttling retry to api_info (authored by anlambert).

package/utils: Add debug log and throttling retry to api_info

Sep 21 2022, 3:49 PM

anlambert updated the diff for D8495: package/utils: Add debug log and throttling retry to api_info.

Update doctring

Sep 21 2022, 3:45 PM

anlambert accepted D8515: Add filters for latest visit status.

Sep 21 2022, 3:27 PM

anlambert added a comment to T4478: Deploy Gogs lister to staging.

Nevertheless, they are less origins in the scheduler db, maybe some get listed more than once, need to check.

Sep 21 2022, 3:23 PM · Gogs lister, Origin-Gitea/Gogs, System administration

anlambert added inline comments to D8515: Add filters for latest visit status.

Sep 21 2022, 3:15 PM

anlambert accepted D8511: Change targettype in directory entry; add directory entry filter by name.

LGTM

Sep 21 2022, 2:57 PM

anlambert added a comment to T4478: Deploy Gogs lister to staging.

Just tested the lister in docker and I got a lot more origins listed:

docker-swh-lister-1  | [2022-09-21 12:24:57,583: INFO/MainProcess] Task swh.lister.gogs.tasks.FullGogsRelister[2b0ba017-4917-44dc-9511-e860bb431322] received
docker-swh-lister-1  | [2022-09-21 12:24:57,584: DEBUG/ForkPoolWorker-1] Loading config file /lister.yml
docker-swh-lister-1  | [2022-09-21 12:24:57,594: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search with params {'limit': 50, 'page': 1}
docker-swh-lister-1  | [2022-09-21 12:24:58,556: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=2 with params {}
docker-swh-lister-1  | [2022-09-21 12:24:58,791: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=3 with params {}
docker-swh-lister-1  | [2022-09-21 12:24:59,018: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=4 with params {}
docker-swh-lister-1  | [2022-09-21 12:24:59,263: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=5 with params {}
docker-swh-lister-1  | [2022-09-21 12:24:59,580: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=6 with params {}
docker-swh-lister-1  | [2022-09-21 12:24:59,845: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=7 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:00,094: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=8 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:00,323: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=9 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:00,560: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=10 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:00,787: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=11 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:01,020: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=12 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:01,324: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=13 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:01,794: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=14 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:02,191: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=15 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:02,549: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=16 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:02,967: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=17 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:03,250: WARNING/ForkPoolWorker-1] Unexpected HTTP status code 500 on https://try.gogs.io/api/v1/repos/search?page=17: b''
docker-swh-lister-1  | [2022-09-21 12:25:03,250: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=18 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:03,515: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=19 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:03,765: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=20 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:04,078: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=21 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:04,444: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=22 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:04,695: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=23 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:04,948: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=24 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:05,262: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=25 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:05,814: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=26 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:06,184: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=27 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:06,454: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=28 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:06,688: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=29 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:06,935: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=30 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:07,247: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=31 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:07,500: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=32 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:07,734: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=33 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:08,064: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=34 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:08,337: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=35 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:08,581: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=36 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:08,818: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=37 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:09,057: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=38 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:09,289: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=39 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:09,531: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=40 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:09,756: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=41 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:09,999: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=42 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:10,213: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=43 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:10,444: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=44 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:10,689: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=45 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:10,914: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=46 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:11,163: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=47 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:11,390: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=48 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:11,710: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=49 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:12,068: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=50 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:12,342: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=51 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:12,619: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=52 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:12,862: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=53 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:13,169: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=54 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:13,549: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=55 with params {}
docker-swh-lister-1  | [2022-09-21 12:25:13,827: DEBUG/ForkPoolWorker-1] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@0f4428ad1388', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL.  See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 19.2.1', 'product': 'RabbitMQ', 'version': '3.6.16'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
docker-swh-lister-1  | [2022-09-21 12:25:13,828: DEBUG/ForkPoolWorker-1] using channel_id: 1
docker-swh-lister-1  | [2022-09-21 12:25:13,831: DEBUG/ForkPoolWorker-1] Channel open
docker-swh-lister-1  | [2022-09-21 12:25:13,833: INFO/ForkPoolWorker-1] Task swh.lister.gogs.tasks.FullGogsRelister[2b0ba017-4917-44dc-9511-e860bb431322] succeeded in 16.23296177299926s: {'pages': 55, 'origins': 575}

Sep 21 2022, 2:33 PM · Gogs lister, Origin-Gitea/Gogs, System administration

anlambert added a comment to T4478: Deploy Gogs lister to staging.

I was wondering why the number of listed repos was so low, turns out gogs has an option to not list a repository through the API when creating it:

Sep 21 2022, 2:23 PM · Gogs lister, Origin-Gitea/Gogs, System administration

anlambert requested changes to D8511: Change targettype in directory entry; add directory entry filter by name.

In D8511#220866, @jayeshv wrote:

In D8511#220865, @vlorentz wrote:

Using .lower() on byte strings of unknown encoding is going to have surprising results.

What is the intended use of making this case-insensitive?

The intention is to have some kind of a search for contents. Do you think a case sensitive filter is good enough, or should we explicitly handle the error here?