rebase

rebase

rebase

rebase

move (forgotten) import stmts top level (thx vlorentz)

rebase

small cleanup in cli_new/50-data.sql

The problem with step 2. is that we do not have an API to declare the latest version, something that should come with swh.core.db is suppose.

yeah!

See T3893 instead.

From IRC:

we can always improve it, but now we have a decent dashboard, so let's consider this done.

Seems moslty ok, even if these errors still pop now and then in the logs.

In D7039#183023, @olasd wrote:

Thanks.

--build-dep-resolver=aptitude should only be used when building with an extra-repository which has a non-default priority, that is only when using a -backports suite (so only for the bullseye and buster instructions). It should probably be documented in the list of "useful options" rather than as the default.

It might need a dedicated bot user to be created on hedgedoc also.

Things to fix in the script:

forgot one...

Use roles as suggested by vlorentz

yat (yet-another-typo)

more typos

and fix the spurious () (thx ardumont)

fix rst syntax (thx D6995 review)

thx

Is there a reason not to close this task?

update using new statsd.status_gauge() context manager (in swh.core 1.1)

In D6944#181139, @olasd wrote:

The code for the gauges feels like something that would be usefully handled with a context manager.

Something like (untested)

class StatsdStatusGauges:
    def __init__(self, metric_name: str, statuses: Collection[str], common_tags: Optional[Dict[str, str]] = None):
        self.metric_name = metric_name
        self.statuses = set(statuses)
        self.common_metrics = common_tags or {}
        self.current_status: Optional[str] = None

    def reset_gauges(self):
        self.current_status = None
        for status in self.statuses:
            statsd.gauge(self.metric_name, 0, {**self.common_tags, "status": status})

    def send_current_gauge(self, value: int):
        if self.current_status is not None:
            statsd.gauge(self.metric_name, value, {**self.common_tags, "status": self.current_status})

    def set(self, new_status: str):
        if new_status not in self.statuses:
            raise ValueError(f'{new_status} not in {self.statuses}')

        # May not be needed; May even be counter-productive if we want to send the gauges to keep them around in the statsd exporter
        if new_status == self.current_status:
            return

        self.send_current_gauge(0)
        self.current_status = new_status
        self.send_current_gauge(1)

    def __enter__(self):
        self.reset_gauges()
        return self

    def __exit__(self, *exc):
        self.reset_gauges()
        return False

Which would be used like:

with StatsdStatusGauges(JOURNAL_STATUS_METRIC, {"processing", "waiting"}) as status_gauge:
    [...]
    status_gauge.set("waiting")
    [...]
    status_gauge.set("processing")

In D6884#181115, @olasd wrote:

I'm kinda wondering if this import stuff should move to a common module - I think we do kind of the same thing with entrypoints?

Rebase and update according suggestions

improve comment as suggested by ardumont

Add the cli option to configure this concurrency value

fine for me (but plz give a bit more insight)

I guess this is then related to T3653 somehow