• an increment of 250Go was added via the proxmox ui (for a total of 500Go now)
• the disk was resized on the os side :

root@pergamon:~# parted /dev/vdc
GNU Parted 3.2
Using /dev/vdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print                                                            
Model: Virtio Block Device (virtblk)
Disk /dev/vdc: 537GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

puppet was disabled on production nodes to avoid this diff to be applied. We will perform a rolling restart of the production cluster after the next scheduled kernel upgrade

The disk will be resized to avoid the service disruption in a short term before looking at T1362

esnode3 was restarted and updated.

~/src/swh/puppet-environment master* ❯ curl -XPUT -H "Content-Type: application/json" http://$ES_NODE:9200/_cluster/settings -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'
{"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"primaries"}}}},"transient":{}}%

The puppet configurarion is applied on esnode1 and esnode2 but we should have taken the opportunity to perform a system update.

A naive upgrade was started but the cluster has collapsed with a node falling in OutOfMemory during the shard rebalancing.

fake LGTM to be able to change the status

dedicated ES node for staging deployed (search-esnode0.internal.staging.swh.network) with D4658 and D4651

LGTM

According the description and to the latest discussion we haved decided to not continue in this direction.

@olasd We have tested to use the official elasticsearch puppet plugin (D4654).
There is several issues to use it. WDYT?

This is not viable as the opnsense api doesn't allow to retrieve the network rules

After T2828, It's more clear of what must be deployed to have the counters working on staging:

• the counters can be intialized via the /stat/refresh endpoint of the storage api (Note: It will create more counters than production as directory_entry_* and revision_history are not counted in production)
• Add a script/service to execute the `swh_update_counter_bucketed` in an infinite loop
• Create the buckets in the object_counts_bucketed
  • per object type : identifier|bucket_start|bucket_end. value and last_update will be updated be the stored procedures.
• configure prometheus sql exporter for db1.staging [1]
• configure profile_exporter on pergamon
  • Update the script to ensure the data are filtered by environments (to avoid staging data to be included in production counts [2])
  • Configure a new cron
    • loading an empty file for historical data
    • creating a new export_file
• update webapp to be able to configure the counter origin

I took the time to create a schema of the pipeline to help me summarize the subject.
It should help to deploy the counters in staging.
SVG :

I was right until the last point :). My mistake was to look at an old version of stored procedure to base my reflection.
Thanks again for the explanation, it's crystal clear now.

Thanks for the clarification.
I missed those counters, I was only focused on the sql_swh_archive_object_count metrics. Could you give some pointers or information on how it's called ? I can only found the stored procedure declaration on storage [1].

All stopped workers are restarted :

vsellier@pergamon ~ % sudo clush -b -w @swh-workers16 'puppet agent --enable; systemctl default'

Erratum: the counters are not yet visible on the "Object added by time period" dashboard due to the aggregation per day.

In T2828#53740, @rdicosmo wrote:

Thanks for looking into this. It would be great to make sure that statistics are collected only once at a time (every X hours), and cached, so to avoid rerunning expensive queries regularly.

The postgresql statistics come back online [1].
The "Object added by time period" dashboard[2] has also data to display

D4635 is landed.

The installation doesn't look too complicated [1], we need to install a new firewall dedicated to admin/internal tools.
The unknown part is on the sso part, but as a quick win, we can try to plug it on the current softwareheritage keycloak's scheme with a dedicated group

In D4635#115756, @olasd wrote:

I was also somewhat concerned that this filtering would, in fact, show a partial picture of what's happening on the database, as we do a lot of processing / I/Os in temp tables. But in practice the aggregate value of i/os on temp tables is orders of magnitude lower than that of any actual table, so I guess this is fine...

In D4635#115756, @olasd wrote:

I was also somewhat concerned that this filtering would, in fact, show a partial picture of what's happening on the database, as we do a lot of processing / I/Os in temp tables. But in practice the aggregate value of i/os on temp tables is orders of magnitude lower than that of any actual table, so I guess this is fine...

Exclude only pg_temp schemas

This is the complete result of the statistics with the temporary tables : P886

As the slowness of the monitoring requests doesn't seem to be related to the direct load on the database, the indexers were restarted :

vsellier@pergamon ~ % sudo clush -b -w @azure-workers 'cd /etc/systemd/system/multi-user.target.wants; for unit in "swh-worker@indexer_origin_intrinsic_metadata.service swh-worker@indexer_fossology_license.service swh-worker@indexer_content_mimetype.service"; do systemctl enable $unit; done; systemctl start swh-worker@indexer_origin_intrinsic_metadata.service swh-worker@indexer_fossology_license.service swh-worker@indexer_content_mimetype.service; puppet agent --enable'

Update description

D4635 is a proposal to solve the performance issues on the statistic queries

@olasd has stopped the backfilling with :

pkill -2 -u swhstorage -f revision

(allow to flush the logs before exiting)

Half of the workers were stopped :

root@pergamon:~# sudo clush -b -w @swh-workers16 'puppet agent --disable "Reduce load of belvedere"; cd /etc/systemd/system/multi-user.target.wants; for unit in swh-worker@*; do systemctl disable $unit; done; systemctl stop swh-worker@*'
worker12: Removed /etc/systemd/system/multi-user.target.wants/swh-worker@checker_deposit.service.
worker11: Removed /etc/systemd/system/multi-user.target.wants/swh-worker@checker_deposit.service.
worker10: Removed /etc/systemd/system/multi-user.target.wants/swh-worker@checker_deposit.service.
worker09: Removed /etc/systemd/system/multi-user.target.wants/swh-worker@checker_deposit.service.
worker12: Removed /etc/systemd/system/multi-user.target.wants/swh-worker@lister.service.
...

It seems there is no other solution then reducing the load on belvedere.
There is an aggressive backfill in progress from getty(192.168.100.102) :

postgres=# select client_addr, count(datid) from pg_stat_activity where state != 'idle' group by client_addr;
select client_addr, count(datid) from pg_stat_activity where state != 'idle' group by client_addr;
   client_addr   | count 
-----------------+-------
                 |     3
 192.168.100.18  |     0
 ::1             |     1
 192.168.100.210 |    60
 192.168.100.102 |    64
(5 rows)

I don't want to kill the job running since several day (2020-11-27) to avoid losing any work, The temporary solution is to reduce the number of workers to relieve the load on belvedere

In T2828#53672, @rdicosmo wrote:

Hmmm... there is definitely no need to update the counters more than once a day

Let's try a temporary workaround :

root@belvedere:/etc/prometheus-sql-exporter# puppet agent --disable "Diagnose prometheus-exporter timeout" 
root@belvedere:/etc/prometheus-sql-exporter# mv swh-scheduler.yml ~
root@belvedere:/etc/prometheus-sql-exporter# systemctl restart prometheus-sql-exporter

It seems some queries are executed on the database each time the metrics are requested.
This one is too long (on the swh-scheduler instance):

After retracing the counter computation pipeline, it seems they are computed from the values stored on prometheus.

The swh-indexer stack is deployed on staging and the initial loading is done.
The volumes are quite low :

this a description of the pipeline to clarify the interaction between the components (source: P883) :

T2814 needs to be released before

Improve test coverage and change mandatory configuration validation

Fix tests

the backfilling is complete (except for the metadatas). We will focus now on some clients to ensure all the local configuration is correct (T2814 for example), and then we will focus on exposing kafka to the outside.

The diff fixes the configuration issue but it seems there is another problem with the visits :

LGTM tested on staging