Rebase and add docstring

🤷🤷🤷 let's try it and see...

I've started working on this, but this needs a backport of a newer prometheus client library, so it's a bit involved.

LGTM, thanks

Awesome, thanks! (With one last stroke of paint on the bikeshed :D)

Thanks!

Maybe the localized statsd methods could use a docstring and type annotations.

Should we add redirects for the previous page names?

In D7726#201823, @vlorentz wrote:

Good idea, I'll send another diff (based on D7727) to do this

Very nice, thanks!

Nice, thanks

LGTM

Make the comment make sense

In T4185#84547, @vlorentz wrote:

Loaders do not have this information themselves, it is handled by the "filtering storage proxy" (swh.storage.proxies.filter); which runs in the same process as loaders

In the future this may warrant a config knob (or enablement only above a given threshold in the target offsets) to avoid thrashing on 16 indexes all the time even for small(er) topics.

Nice, thanks!

I'm closing this. I've submitted T4216 to track the actual packfile limit issue.

Deployed the puppet changes and ran puppet on rp1.internal.admin. The websocket requests from hedgedoc get properly handled. Websocket requests for grafana also still work.

Very nice, thanks!

LGTM (save from a couple of nitpicks), thanks

Thanks. Your commit message needs an update :-)

I think the separate mechanic for save code now is much more than a detail; it's the main reason we've kept the old task scheduler.

Should we be using the parent, or the source (which, afaik, is the root of all forks) repo? Or both?

In D7647#200065, @vlorentz wrote:

Thanks. On the diagram, it looks like 9' is causing 12 (via 10 and 11), which is not true (yet! D7332 isn't agreed upon) and could be very confusing.

This is because 9' writes to the origin_visit table, but 10/11/12 are based on the origin table/topic.

Could you split swh-storage in two, to avoid this misunderstanding?

I have a few nits, but overall the text looks good to me, thank you!

Cool, thanks.

Thanks!

To be able to reuse this code somewhere else, you will want to make it accept a user-agent value explicitly too. That way the loader.metadata can just set its own value.

In terms of architecture, I think it would make sense to move this to a new swh.lister.github.utils (or swh.lister.utils.github) module, so that third-party users don't import the whole listing machinery but just the relevant bits for platform access.

Looks good, except for a small issue in the logic with repeated policies (which probably warrants adding a test for this usecase too).

Sounds good! I would suggest making the policies for a given visit type a list:

Add test to ensure that one can override the gunicorn (flipped) default with an envvar

Make the cli tests clearer by running the asserts in the subcommand

Switch approach for preventing changes to the global logging setup