Please note a recent version of docker-compose is needed to allow the environment to start with the healthcheck keywork,
It was not working with the version 1.26.2 but is ok with 1.28.2

❯ docker-compose version
docker-compose version 1.26.2, build eefe0d31
docker-py version: 4.2.2
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

Feb 17 2021, 4:02 PM

vsellier added a comment to T2774: Fix vault end-to-end check.

With tcpdump, it seems swh-web don't add the headers to don't cache the response in case of a 404:

GET /api/1/vault/directory/a317baff051f68e83557d51e59539dac2ff55b34/ HTTP/1.1
Host: archive.softwareheritage.org
User-Agent: python-requests/2.21.0
Accept: */*
X-Forwarded-For: 128.93.166.14
X-Forwarded-Proto: https
Accept-Encoding: gzip
X-Varnish: 230399

Feb 17 2021, 12:46 PM · Vault, System administration, Monitoring

vsellier added a comment to T2774: Fix vault end-to-end check.

after digging, it seems the request with a 404 return code are cached by varnish.
When the test is launched, a first request is done which returns a 404 and the post is issued. When the check try to get the status of the cooking, the initial 404 is returned by varnish

Feb 17 2021, 12:40 PM · Vault, System administration, Monitoring

vsellier added a comment to T2774: Fix vault end-to-end check.

It seems the scheduler has missed some updates. After an upgrade of the python3-swh-.* packages, the error is again the initial one.

Feb 17 2021, 12:02 PM · Vault, System administration, Monitoring

vsellier changed the status of T2774: Fix vault end-to-end check from Open to Work in Progress.

Feb 17 2021, 11:51 AM · Vault, System administration, Monitoring

vsellier added a comment to T2774: Fix vault end-to-end check.

After an upgrade of the packages on pergamon and vangogh, the error is now :

Feb 17 10:49:38 vangogh python3[1990225]: 2021-02-17 10:49:38 [1990225] root:ERROR <RemoteException 500 InvalidDatetimeFormat: ['invalid input syntax for type timestamp with time zone: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\nCONTEXT:  COPY tmp_task, line 1, column next_run: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\n']>
                                          Traceback (most recent call last):
                                            File "/usr/lib/python3/dist-packages/swh/core/api/asynchronous.py", line 71, in middleware_handler
                                              return await handler(request)
                                            File "/usr/lib/python3/dist-packages/swh/core/api/asynchronous.py", line 178, in decorated_meth
                                              result = obj_meth(**kw)
                                            File "/usr/lib/python3/dist-packages/swh/core/db/common.py", line 62, in _meth
                                              return meth(self, *args, db=db, cur=cur, **kwargs)
                                            File "/usr/lib/python3/dist-packages/swh/vault/backend.py", line 220, in cook
                                              self.create_task(obj_type, obj_id, sticky)
                                            File "/usr/lib/python3/dist-packages/swh/core/db/common.py", line 62, in _meth
                                              return meth(self, *args, db=db, cur=cur, **kwargs)
                                            File "/usr/lib/python3/dist-packages/swh/vault/backend.py", line 163, in create_task
                                              task_id = self._send_task(obj_type, hex_id)
                                            File "/usr/lib/python3/dist-packages/swh/vault/backend.py", line 139, in _send_task
                                              added_tasks = self.scheduler.create_tasks([task])
                                            File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 181, in meth_
                                              return self.post(meth._endpoint_path, post_data)
                                            File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 278, in post
                                              return self._decode_response(response)
                                            File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 354, in _decode_response
                                              self.raise_for_status(response)
                                            File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 344, in raise_for_status
                                              raise exception from None
                                          swh.core.api.RemoteException: <RemoteException 500 InvalidDatetimeFormat: ['invalid input syntax for type timestamp with time zone: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\nCONTEXT:  COPY tmp_task, line 1, column next_run: "Timestamp(seconds=1613558977, nanoseconds=999614000)"\n']>

Feb 17 2021, 11:51 AM · Vault, System administration, Monitoring

vsellier added projects to T2774: Fix vault end-to-end check: System administration, Vault.

Feb 17 2021, 10:46 AM · Vault, System administration, Monitoring

Feb 16 2021

vsellier changed the status of T3042: swh-search: add statsd/prometheus metrics from Open to Work in Progress.

Feb 16 2021, 6:22 PM · System administration, Archive search

vsellier committed rDSNIP194a686fc404: Add counters schemas sources (authored by vsellier).

Add counters schemas sources

Feb 16 2021, 5:32 PM

vsellier added a comment to T2912: Next generation archive counters.

I wrote a proposal for the next steps [1] so we could start the work on these counters. All comments/contributions are welcome.

Feb 16 2021, 5:25 PM · Roadmap 2021, System administration, Monitoring, Web app

vsellier updated the title for P951 Cassandra dependencies's hell from Cassandra dependencies hell to Cassandra dependencies's hell.

Feb 16 2021, 1:03 PM

vsellier updated the title for P951 Cassandra dependencies's hell from untitled to Cassandra dependencies hell.

Feb 16 2021, 1:03 PM

vsellier added a comment to P951 Cassandra dependencies's hell.

root@0a15636c2914:/# aptitude install cassandra openjdk-11-jre 
The following NEW packages will be installed:
  alsa-topology-conf{a} alsa-ucm-conf{a} at-spi2-core{a} ca-certificates-java{a} cassandra dbus{a} file{a} fontconfig-config{a} fonts-dejavu-core{a} fonts-dejavu-extra{a} java-common{a} libapparmor1{a} 
  libasound2{a} libasound2-data{a} libatk-bridge2.0-0{a} libatk-wrapper-java{a} libatk-wrapper-java-jni{a} libatk1.0-0{a} libatk1.0-data{a} libatspi2.0-0{a} libavahi-client3{a} libavahi-common-data{a} 
  libavahi-common3{a} libbsd0{a} libcap2{a} libcups2{a} libdbus-1-3{a} libdrm-amdgpu1{a} libdrm-common{a} libdrm-intel1{a} libdrm-nouveau2{a} libdrm-radeon1{a} libdrm2{a} libedit2{a} libelf1{a} 
  libevent-core-2.1-7{a} libevent-pthreads-2.1-7{a} libexpat1{a} libfontconfig1{a} libfontenc1{a} libfreetype6{a} libgif7{a} libgl1{a} libgl1-mesa-dri{a} libglapi-mesa{a} libglib2.0-0{a} libglib2.0-data{a} 
  libglvnd0{a} libglx-mesa0{a} libglx0{a} libgraphite2-3{a} libharfbuzz0b{a} libice6{a} libicu67{a} libjpeg62-turbo{a} liblcms2-2{a} libllvm11{a} libmagic-mgc{a} libmagic1{a} libmd0{a} libnspr4{a} 
  libnss3{a} libopts25{a} libpciaccess0{a} libpcsclite1{a} libpng16-16{a} libpython2-stdlib{a} libpython2.7-minimal{a} libpython2.7-stdlib{a} libsensors-config{a} libsensors5{a} libsm6{a} libvulkan1{a} 
  libwayland-client0{a} libx11-6{a} libx11-data{a} libx11-xcb1{a} libxau6{a} libxaw7{a} libxcb-dri2-0{a} libxcb-dri3-0{a} libxcb-glx0{a} libxcb-present0{a} libxcb-randr0{a} libxcb-shape0{a} libxcb-shm0{a} 
  libxcb-sync1{a} libxcb-xfixes0{a} libxcb1{a} libxcomposite1{a} libxdamage1{a} libxdmcp6{a} libxext6{a} libxfixes3{a} libxft2{a} libxi6{a} libxinerama1{a} libxkbfile1{a} libxml2{a} libxmu6{a} libxmuu1{a} 
  libxpm4{a} libxrandr2{a} libxrender1{a} libxshmfence1{a} libxt6{a} libxtst6{a} libxv1{a} libxxf86dga1{a} libxxf86vm1{a} libz3-4{a} mailcap{a} media-types{a} mesa-vulkan-drivers{a} mime-support{a} ntp{a} 
  openjdk-11-jre openjdk-11-jre-headless{a} python-is-python2{a} python2{a} python2-minimal{a} python2.7{a} python2.7-minimal{a} shared-mime-info{a} sntp{a} ucf{a} x11-common{a} x11-utils{a} 
  xdg-user-dirs{a}

root@0a15636c2914:/# aptitude install cassandra 
The following NEW packages will be installed:
  adwaita-icon-theme{a} alsa-topology-conf{a} alsa-ucm-conf{a} at-spi2-core{a} ca-certificates-java{a} cassandra dbus{a} file{a} fontconfig{a} fontconfig-config{a} fonts-dejavu-core{a} fonts-dejavu-extra{a} 
  gtk-update-icon-cache{a} hicolor-icon-theme{a} java-common{a} libapparmor1{a} libasound2{a} libasound2-data{a} libatk-bridge2.0-0{a} libatk-wrapper-java{a} libatk-wrapper-java-jni{a} libatk1.0-0{a} 
  libatk1.0-data{a} libatspi2.0-0{a} libavahi-client3{a} libavahi-common-data{a} libavahi-common3{a} libbsd0{a} libcairo-gobject2{a} libcairo2{a} libcap2{a} libcups2{a} libdatrie1{a} libdbus-1-3{a} 
  libdeflate0{a} libdrm-amdgpu1{a} libdrm-common{a} libdrm-intel1{a} libdrm-nouveau2{a} libdrm-radeon1{a} libdrm2{a} libedit2{a} libelf1{a} libevent-core-2.1-7{a} libevent-pthreads-2.1-7{a} libexpat1{a} 
  libfontconfig1{a} libfontenc1{a} libfreetype6{a} libfribidi0{a} libgail-common{a} libgail18{a} libgdk-pixbuf-2.0-0{a} libgdk-pixbuf2.0-bin{a} libgdk-pixbuf2.0-common{a} libgif7{a} libgl1{a} 
  libgl1-mesa-dri{a} libglapi-mesa{a} libglib2.0-0{a} libglib2.0-data{a} libglvnd0{a} libglx-mesa0{a} libglx0{a} libgraphite2-3{a} libgtk2.0-0{a} libgtk2.0-bin{a} libgtk2.0-common{a} libharfbuzz0b{a} 
  libice6{a} libicu67{a} libjbig0{a} libjpeg62-turbo{a} liblcms2-2{a} libllvm11{a} libmagic-mgc{a} libmagic1{a} libmd0{a} libnspr4{a} libnss3{a} libopts25{a} libpango-1.0-0{a} libpangocairo-1.0-0{a} 
  libpangoft2-1.0-0{a} libpciaccess0{a} libpcsclite1{a} libpixman-1-0{a} libpng16-16{a} libpython2-stdlib{a} libpython2.7-minimal{a} libpython2.7-stdlib{a} librsvg2-2{a} librsvg2-common{a} 
  libsensors-config{a} libsensors5{a} libsm6{a} libthai-data{a} libthai0{a} libtiff5{a} libvulkan1{a} libwayland-client0{a} libwebp6{a} libx11-6{a} libx11-data{a} libx11-xcb1{a} libxau6{a} libxaw7{a} 
  libxcb-dri2-0{a} libxcb-dri3-0{a} libxcb-glx0{a} libxcb-present0{a} libxcb-randr0{a} libxcb-render0{a} libxcb-shape0{a} libxcb-shm0{a} libxcb-sync1{a} libxcb-xfixes0{a} libxcb1{a} libxcomposite1{a} 
  libxcursor1{a} libxdamage1{a} libxdmcp6{a} libxext6{a} libxfixes3{a} libxft2{a} libxi6{a} libxinerama1{a} libxkbfile1{a} libxml2{a} libxmu6{a} libxmuu1{a} libxpm4{a} libxrandr2{a} libxrender1{a} 
  libxshmfence1{a} libxt6{a} libxtst6{a} libxv1{a} libxxf86dga1{a} libxxf86vm1{a} libz3-4{a} mailcap{a} media-types{a} mesa-vulkan-drivers{a} mime-support{a} ntp{a} openjdk-17-jre{a} 
  openjdk-17-jre-headless{a} python-is-python2{a} python2{a} python2-minimal{a} python2.7{a} python2.7-minimal{a} shared-mime-info{a} sntp{a} ucf{a} x11-common{a} x11-utils{a} xdg-user-dirs{a} 
0 packages upgraded, 159 newly installed, 0 to remove and 12 not upgraded.

Feb 16 2021, 1:03 PM

vsellier created P951 Cassandra dependencies's hell.

Feb 16 2021, 12:41 PM

Feb 15 2021

vsellier accepted D5074: mercurial.loader: Make error comparison case-insensitive.

Feb 15 2021, 5:22 PM

vsellier closed D5052: loader: mark visit as 'not_found' when relevant.

Feb 15 2021, 4:22 PM

vsellier committed rDLDHG4c84a86b371e: loader: mark visit as 'not_found' when relevant (authored by vsellier).

loader: mark visit as 'not_found' when relevant

Feb 15 2021, 4:22 PM

vsellier added inline comments to D5052: loader: mark visit as 'not_found' when relevant.

Feb 15 2021, 4:10 PM

vsellier updated the diff for D5052: loader: mark visit as 'not_found' when relevant.

Adapt according the review

Feb 15 2021, 4:09 PM

vsellier renamed T3041: [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata from Provision enough space for the search ES cluster to ingest all intrinsic metadata to [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata.

Feb 15 2021, 10:02 AM · System administration, Archive search

vsellier renamed T3040: [production] Enable swh-search's journal-client for indexed objects from Enable swh-search's journal-client for indexed objects to [production] Enable swh-search's journal-client for indexed objects.

Feb 15 2021, 10:02 AM · System administration, Journal, Archive search

Feb 12 2021

vsellier added a comment to T3042: swh-search: add statsd/prometheus metrics.

A basic dashboard [1] is created on garfana based on the number of log line.
It's too limited as it's not possible to isolate the logs per environment as the information is not available.
It will be added in T3043

Feb 12 2021, 5:53 PM · System administration, Archive search

vsellier moved T3042: swh-search: add statsd/prometheus metrics from Backlog to Weekly backlog on the System administration board.

Feb 12 2021, 5:44 PM · System administration, Archive search

vsellier moved T3043: journalbeat:/filebeat Add an environment field on the logs from Backlog to Weekly backlog on the System administration board.

Feb 12 2021, 5:44 PM · System administration

vsellier triaged T3043: journalbeat:/filebeat Add an environment field on the logs as Normal priority.

Feb 12 2021, 5:44 PM · System administration

vsellier updated the diff for D5052: loader: mark visit as 'not_found' when relevant.

Fix tests to increase code coverage

Feb 12 2021, 4:35 PM

vsellier updated the diff for D5052: loader: mark visit as 'not_found' when relevant.

The not_found status is now only set when the repository is really not found.

Feb 12 2021, 4:17 PM

vsellier triaged T3042: swh-search: add statsd/prometheus metrics as Normal priority.

Feb 12 2021, 12:19 PM · System administration, Archive search

vsellier added a comment to D5064: Enable to filter searched origins by visit types.

In D5064#127855, @vlorentz wrote:

It also misses a test where an origin is initially inserted with visit_types, then visit_types is added

Feb 12 2021, 11:43 AM

vsellier added a comment to T3033: Replace first disk on storage1.staging.

The disk has been sent to the manufacturer this morning, we now have to wait to hear from him.

Feb 12 2021, 10:20 AM · System administration

vsellier accepted D5066: Simplify max_date().

👍nice! Thanks

Feb 12 2021, 10:18 AM

Feb 11 2021

vsellier closed D5049: loader: Mark visit status as failed when relevant.

Feb 11 2021, 6:21 PM

vsellier committed rDLDHG1f96493ef318: loader: Mark visit status as failed when relevant (authored by vsellier).

loader: Mark visit status as failed when relevant

Feb 11 2021, 6:21 PM

vsellier closed T2566: Add an icinga check on whether the puppet agent is enabled as Resolved.

After puppet has added the group to the user nagios[1], the icinga services needed to be restarted.

# clush -b -w @staging 'systemctl restart icinga2'
# clush -b -w @all 'systemctl restart icinga2'

Feb 11 2021, 4:55 PM · System administration

vsellier committed rSPSITE08c1f568a165: icinga: convert puppet checks threshold to seconds (authored by vsellier).

icinga: convert puppet checks threshold to seconds

Feb 11 2021, 3:55 PM

vsellier closed D5065: icinga: grant access to private puppet directories.

Feb 11 2021, 3:35 PM

vsellier committed rSPSITE8bc7ef05cca4: icinga: grant access to private puppet directories (authored by vsellier).

icinga: grant access to private puppet directories

Feb 11 2021, 3:35 PM

vsellier claimed T3015: Sentry should have two different projects for swh-indexer and swh-indexer-storage.

Feb 11 2021, 3:34 PM · System administration, Sentry

vsellier placed T3041: [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata up for grabs.

Feb 11 2021, 3:33 PM · System administration, Archive search

vsellier placed T3040: [production] Enable swh-search's journal-client for indexed objects up for grabs.

Feb 11 2021, 3:32 PM · System administration, Journal, Archive search

vsellier added a comment to T3015: Sentry should have two different projects for swh-indexer and swh-indexer-storage.

I'm not sure to understand the real problem here.
As the indexer and indexer-storage are in same source repository, the versions should match or increase in //. Sentry should be able to deal with it as any other version upgrade.

Feb 11 2021, 3:23 PM · System administration, Sentry

vsellier changed the status of T3015: Sentry should have two different projects for swh-indexer and swh-indexer-storage from Open to Work in Progress.

Feb 11 2021, 3:05 PM · System administration, Sentry

vsellier updated the test plan for D5065: icinga: grant access to private puppet directories.

Feb 11 2021, 3:04 PM

vsellier requested review of D5065: icinga: grant access to private puppet directories.

Feb 11 2021, 3:03 PM

vsellier added a revision to T2566: Add an icinga check on whether the puppet agent is enabled: D5065: icinga: grant access to private puppet directories.

Feb 11 2021, 3:03 PM · System administration

vsellier added a comment to T3040: [production] Enable swh-search's journal-client for indexed objects.

T3041 needs to be done before this one (for the production environment)

Feb 11 2021, 2:21 PM · System administration, Journal, Archive search

vsellier added a project to T2182: Switch production swh-web to use swh-search instead of postgresql search.: System administration.

Feb 11 2021, 12:14 PM · System administration, Archive search, Storage manager

vsellier requested review of D5052: loader: mark visit as 'not_found' when relevant.

Feb 11 2021, 12:13 PM

vsellier closed T2182: Switch production swh-web to use swh-search instead of postgresql search., a subtask of T1910: Redesign origin search using a dedicated component (swh-search), as Resolved.

Feb 11 2021, 12:10 PM · Archive search, Storage manager

vsellier closed T2182: Switch production swh-web to use swh-search instead of postgresql search. as Resolved.

D5063 is applied, the main webapp is now using swh-search by default.

Feb 11 2021, 12:10 PM · System administration, Archive search, Storage manager

vsellier closed D5063: webapp: use swh-search as main search engine in production.

Feb 11 2021, 12:00 PM

vsellier committed rSPSITE6f18a7797f1c: webapp: use swh-search as main search engine (authored by vsellier).

webapp: use swh-search as main search engine

Feb 11 2021, 12:00 PM

vsellier requested review of D5049: loader: Mark visit status as failed when relevant.

Feb 11 2021, 11:55 AM

vsellier updated the test plan for D5063: webapp: use swh-search as main search engine in production.

Feb 11 2021, 11:50 AM

vsellier created P946 production deployment of swh-search.

Feb 11 2021, 11:29 AM

vsellier added a comment to D5063: webapp: use swh-search as main search engine in production.

This is (or should be ;) ) the state after the diff is applied :

Feb 11 2021, 11:28 AM

vsellier requested review of D5063: webapp: use swh-search as main search engine in production.

Feb 11 2021, 11:27 AM

vsellier added a revision to T2182: Switch production swh-web to use swh-search instead of postgresql search.: D5063: webapp: use swh-search as main search engine in production.

Feb 11 2021, 11:27 AM · System administration, Archive search, Storage manager

vsellier changed the status of T2182: Switch production swh-web to use swh-search instead of postgresql search., a subtask of T1910: Redesign origin search using a dedicated component (swh-search), from Open to Work in Progress.

Feb 11 2021, 9:24 AM · Archive search, Storage manager

vsellier changed the status of T2182: Switch production swh-web to use swh-search instead of postgresql search. from Open to Work in Progress.

The main webapp search can be switch from the sql search to the swh-search as all the tests performed on staging and https://webapp1.internal.softwareheritage.org are ok

Feb 11 2021, 9:24 AM · System administration, Archive search, Storage manager

Feb 10 2021

vsellier accepted D5060: Install properly backfill configuration to getty and storage1.staging.

Feb 10 2021, 6:07 PM

vsellier accepted D5058: backfill: Dedicate storage configuration for the backfiller.

Feb 10 2021, 6:03 PM

vsellier accepted D5059: Move indexer_journal_client to scheduler instances.

Feb 10 2021, 5:30 PM

vsellier added a comment to D5059: Move indexer_journal_client to scheduler instances.

LGTM but it's biased as we worked onit together ;)

Feb 10 2021, 5:30 PM

vsellier accepted D5056: staging: Dedicate an indexer worker.

Feb 10 2021, 12:36 PM

vsellier accepted D5054: Enable the journal-writer for the swh-idx-storage in production.

Feb 10 2021, 12:06 PM

vsellier closed D5047: journal_client: Fix date computations for (un)eventful visits.

Feb 10 2021, 11:17 AM

vsellier committed rDSCHcf32e3764920: journal_client: Fix date computations for (un)eventful visits (authored by vsellier).

journal_client: Fix date computations for (un)eventful visits

Feb 10 2021, 11:17 AM

vsellier edited P945 origin_visit_stats scenario.

Feb 10 2021, 10:11 AM

vsellier created P945 origin_visit_stats scenario.

Feb 10 2021, 10:00 AM

Feb 9 2021

vsellier accepted D5053: staging: Activate swh-search-journal-client@indexed.

LGTM

Feb 9 2021, 6:44 PM

vsellier updated the title for P944 search_journal_client @origin offsets matching the index backup (csv) from search_journal_client @origin offsets matching the index backup to search_journal_client @origin offsets matching the index backup (csv).

Feb 9 2021, 6:34 PM

vsellier created P944 search_journal_client @origin offsets matching the index backup (csv).

Feb 9 2021, 6:33 PM

vsellier requested review of D5047: journal_client: Fix date computations for (un)eventful visits.

Feb 9 2021, 6:19 PM

vsellier added a revision to T3030: Improve loaders to deal with new visit status events: D5052: loader: mark visit as 'not_found' when relevant.

Feb 9 2021, 5:09 PM · Core Loader

vsellier committed rSPSITEfb01f43ad2cd: icinga: use the right parameter name for ignoremissing (authored by vsellier).

icinga: use the right parameter name for ignoremissing

Feb 9 2021, 4:43 PM

vsellier closed D5043: icinga: monitor puppet agent activation.

Feb 9 2021, 4:26 PM