I just ran the lister using swh lister run -l gogs url=https://try.gogs.io/api/v1/ api_token=xxx

"denied" gives the feeling that the forge admin has refused the arcival

which is true according to the help page: Denied: The forge administrator(s) denied the request to list their forge.

one suggestion inline.

I forgot to mention it to you explicitely (you may have seen this very task's
description update though). Can you please make sure you run the lister within "docker".
You will find what you need regarding lister in swh-environment:/docker/conf/lister.yml
(add the cog lister entry there).

Maybe it'd be worth opening an issue upstream regarding this behavior.

Done https://github.com/gogs/docs-api/issues/34

Hello, was there a discussion leading to this implementation? If so, it'd be nice to mention it in the description (please).

(I've dropped val from reviewers since they are in vacation and you attended most if not all of their request changes.)

couple of suggestions inline.

Hey @ardumont, I noticed the following problem:

If we follow the Gogs v1 API, The last page for try.gogs.io repo search is 28 (with page size = 20)

But if open the explore section, they go up to 685 pages instead! (with page size = 20)

To clearly understand the problem, I skimmed through Gogs source code and found ExploreRepos and Search functions which are responsible for these results. Both of them seem to use db.SearchRepositoryByName and the differences are in terms of args like OwnerID, UserID, OrderBy and Private.

Tomorrow, I'll properly go through their code and post updates here. Feel free to add your thoughts on this :)

I feel like "Blocked" which is shorter convey the same meaning.

I've landed and deployed the diffs to make /tmp a tmpfs partition and activated the swap (as swapfile) on their local disk /mnt.

Since the normal ingestion is mostly done (1 last normal ingestion ongoing), i've now make worker17-18 consumes 1 more task for the large repositories queue as well (vs. letting them twiddle their thumbs ;).

Adapt according to review to:

• also allow swap deactivation if needed
• escape regexp pattern matching

one remark on the main class docstring that needs a tad rework to match the actual pagination implementation.

Thanks for your feedback @ardumont

i'm assuming then that the last line will have the expected values (when there is redundant configuration line).

Adapt according to suggestion

Let's use file_line resources instead of using a template which will be out of date 28 seconds after the change is pushed.

Update

Restrict declaration to /tmp

• Decrease parallelism on indexer workers running journal clients

In D8167#212470, @olasd wrote:

The origin extrinsic metadata indexer only runs on a single file at a time, so I don't think it needs to run alone on dedicated hosts.

At this point in time:

• 1 "normal" origin
• 22 "large" origins

Use proper fqdn