Query: Advanced Search

Remove deprecated documentation

	Include stories about projects I am a member of.

Add the related task

It looks it's the normal behavior of celery with the current configuration:

After analyzing further the logs, it seems the celery process is accepting the task eb038b42-713e-4c23-a655-17d04f23870a whereas a previous loading is still running
A more contextual logs:

Previous task is received and started:

Jan 17 16:19:31 worker14 python3[2180500]: [2022-01-17 16:19:31,868: INFO/MainProcess] Received task: swh.loader.git.tasks.UpdateGitRepository[70c13ea8-19f4-4ac7-ae4f-ea71f80fc25a]  
Jan 17 16:19:32 worker14 python3[2721731]: [2022-01-17 16:19:32,467: INFO/ForkPoolWorker-133] Load origin 'https://git.libreoffice.org/translations' with type 'git'

The new task is accepted whereas the previous one is not yet finished:

Jan 17 16:26:10 worker14 python3[2180500]: [2022-01-17 16:26:10,683: INFO/MainProcess] Received task: swh.loader.git.tasks.UpdateGitRepository[eb038b42-713e-4c23-a655-17d04f23870a]

the first task ends:

Jan 17 19:48:20 worker14 python3[2721731]: [51B blob data]
Jan 17 19:48:20 worker14 python3[2721731]: [3.6K blob data]
Jan 17 19:49:57 worker14 python3[2721731]: [2022-01-17 19:49:57,085: ERROR/ForkPoolWorker-133] Loading failure, updating to `failed` status
                                           Traceback (most recent call last):
                                             File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 338, in load
                                               more_data_to_fetch = self.fetch_data()
                                             File "/usr/lib/python3/dist-packages/swh/loader/git/loader.py", line 277, in fetch_data
                                               self.origin.url, base_repo, do_progress
                                             File "/usr/lib/python3/dist-packages/swh/loader/git/loader.py", line 213, in fetch_pack_from_origin
                                               progress=do_activity,
                                             File "/usr/lib/python3/dist-packages/dulwich/client.py", line 2001, in fetch_pack
                                               progress,
                                             File "/usr/lib/python3/dist-packages/dulwich/client.py", line 845, in _handle_upload_pack_tail
                                               SIDE_BAND_CHANNEL_PROGRESS: progress,
                                             File "/usr/lib/python3/dist-packages/dulwich/client.py", line 604, in _read_side_band64k_data
                                               cb(pkt)
                                             File "/usr/lib/python3/dist-packages/swh/loader/git/loader.py", line 201, in do_pack
                                               f"Pack file too big for repository {origin_url}, "
                                           OSError: Pack file too big for repository https://git.libreoffice.org/translations, limit is 4294967296 bytes, current size is 4294966867, would write 65515
Jan 17 19:49:59 worker14 python3[2721731]: [2022-01-17 19:49:59,468: INFO/ForkPoolWorker-133] Task swh.loader.git.tasks.UpdateGitRepository[70c13ea8-19f4-4ac7-ae4f-ea71f80fc25a] succeeded in 12627.452697345056s: {'status': 'failed'}

the second task loading starts after 3 hours:

Jan 17 19:50:02 worker14 python3[2736039]: [2022-01-17 19:50:02,768: INFO/ForkPoolWorker-134] Load origin 'https://github.com/webtorrent/webtorrent-desktop' with type 'git'

create the admin vm grafana0

Update after the apply

pergamon cleanup done with D6952.

grafana: clean pergamon resources

remove unecessary spaces

restore useful removed comment

grafana: disable grafana0 backups

grafana0 server created
configuration applied on the reverse proxy
network configuration checks:
- grafana0 -> pergamon:9090 (prometheus): Allowed for all in a floating rule
- grafana0 -> esnode[1-3]:9200 (elasticsearch queries):
public DNS updated to use rp1 (swh-rproxy3.inria.fr) as the public entry point
pergamon apache configuration temporary updated to point to the new grafana0 backend server ( and add a temporary rule on the firewall to allow it)

grafana: fix the missing database name properties

webapp: Temporary use the main database

Move grafana on a dedicated server behind the admin RP

Escape file name pattern to not depend on the /root content

Rebase
Use the centralized database instead of a local postgresql
Move the prometheus sql exporter on dali

rebase
reduce the memory as the database will not be running on this host but in the centralized admin database
add the output section

Upgrade bullseye template to 11.2

netbox: use the centralized admin db

journalbeat: create a missing directory not created in the last packages

LGTM Thanks for fixing this.

We will need a larger disk than 32Go as the sentry db is almost 100GB

We will wait for the availability of the dedicated admin database server (T3833) before moving grafana. It will avoid to move the data 2 times and will upgrade the database to the last postgresql version.

The first github milestone was reached today.
All the github loading are failing since 00:00 UTC.

The build of swh-search is now ok in master and swh-envrionment.
The swh-environment build is still failing but it's due to the swh-graph tests flakiness (it will be solved in T3831)

Unpin tree-sitter dependency

Remove useless remaining comment

I don't know, it was pinned so I kept it pinned but yes as the versions specified in the diffs are the last ones, I guess we can remove it

Why the duplication of profile::grafana into profile::grafana::backend?

The goal was to let the initial profile untouched during the migration and to clearly identify the public reverse_proxy is not installed by this part

Add more documentation on the backend profile

Use the postgresql module to configure the database Indeed, it's simpler
fix a wrong hostname declaration for vagrant
fix grafana plugin installation and configuration calls

vagrant: declare the grafana0 server

workaround for puppet certificates revocation in vagrant

vagrant: fix misconfigured puppet node types

Advanced Search
Use Results
Edit Query
Hide Query

Jan 19 2022

Jan 18 2022

Jan 14 2022

Jan 13 2022

Jan 12 2022

Jan 11 2022

Jan 10 2022

Jan 7 2022

Advanced SearchUse ResultsEdit QueryHide Query