Page MenuHomeSoftware Heritage
Feed Advanced Search

Mon, Jan 10

vsellier closed T3838: swh-search build is ko as Resolved.

The build of swh-search is now ok in master and swh-envrionment.
The swh-environment build is still failing but it's due to the swh-graph tests flakiness (it will be solved in T3831)

Mon, Jan 10, 7:06 PM · Archive search
vsellier added a comment to T3838: swh-search build is ko.

I don't know, it was pinned so I kept it pinned but yes as it's the versions specified in the diffs are the last ones, I guess we can remove it

Mon, Jan 10, 5:14 PM · Archive search
olasd added a comment to T3838: swh-search build is ko.

Why are we pinning tree-sitter at all?

Mon, Jan 10, 4:44 PM · Archive search
vsellier added a revision to T3838: swh-search build is ko: D6897: Unpin the tree-sitter dependency.
Mon, Jan 10, 3:14 PM · Archive search
vsellier added a revision to T3838: swh-search build is ko: D6896: Unpin tree-sitter dependency.
Mon, Jan 10, 2:54 PM · Archive search
vsellier changed the status of T3838: swh-search build is ko from Open to Work in Progress.
Mon, Jan 10, 12:13 PM · Archive search

Dec 13 2021

anlambert closed T3803: swh-search tests are hanging since elasticsearch 7.16 release as Resolved by committing rDSEA933c78830448: conftest: Fix tests hang since elasticsearch 7.16 release.
Dec 13 2021, 4:13 PM · Archive search
anlambert added a comment to T3803: swh-search tests are hanging since elasticsearch 7.16 release.

Explicitly setting the LIBFFI_TMPDIR environment variable indeed fixes the hang: D6828.

Dec 13 2021, 4:06 PM · Archive search
anlambert added a revision to T3803: swh-search tests are hanging since elasticsearch 7.16 release: D6828: conftest: Fix tests hang since elasticsearch 7.16 release.
Dec 13 2021, 4:03 PM · Archive search
vsellier added a comment to T3803: swh-search tests are hanging since elasticsearch 7.16 release.

If not defined, this variable is set by the elasticsearch launch script https://github.com/elastic/elasticsearch/pull/80699/files#diff-ddfc3a6ea1404997e56f2e771adede06b173f0fea37b4779d827c85d6cc52897R35
I guess as the fixture is not starting elasticsearch[1] throught the startup script, the variable is not defined

Dec 13 2021, 3:27 PM · Archive search
vsellier added a comment to T3803: swh-search tests are hanging since elasticsearch 7.16 release.

This link is interesting: https://www.elastic.co/guide/en/elasticsearch/reference/current/executable-jna-tmpdir.html
(from https://github.com/elastic/elasticsearch/issues/73309)

Dec 13 2021, 3:05 PM · Archive search
anlambert triaged T3803: swh-search tests are hanging since elasticsearch 7.16 release as Normal priority.
Dec 13 2021, 2:42 PM · Archive search

Dec 9 2021

vsellier triaged T3784: swh-search / staging: transient timeouts on elasticsearch queries as Normal priority.
Dec 9 2021, 12:38 PM · Archive search, System administration

Dec 6 2021

ardumont changed the status of T3708: Upgrade swh-search elasticsearch version from Duplicate to Resolved.
Dec 6 2021, 5:35 PM · Archive search, System administration (Component upgrades)
ardumont moved T3741: swh-search - upgrade elasticsearch backend from Backlog to Done on the System administration (Component upgrades) board.
Dec 6 2021, 5:35 PM · System administration (Component upgrades), Archive search
ardumont edited projects for T3741: swh-search - upgrade elasticsearch backend, added: System administration (Component upgrades); removed System administration.
Dec 6 2021, 5:35 PM · System administration (Component upgrades), Archive search
ardumont merged T3708: Upgrade swh-search elasticsearch version into T3741: swh-search - upgrade elasticsearch backend.
Dec 6 2021, 5:35 PM · System administration (Component upgrades), Archive search
ardumont merged task T3708: Upgrade swh-search elasticsearch version into T3741: swh-search - upgrade elasticsearch backend.
Dec 6 2021, 5:35 PM · Archive search, System administration (Component upgrades)

Nov 24 2021

vsellier closed T3741: swh-search - upgrade elasticsearch backend as Resolved.
Nov 24 2021, 6:11 PM · System administration (Component upgrades), Archive search
vsellier added a comment to T3741: swh-search - upgrade elasticsearch backend.

production nodes are upgraded :

  • stop the journal clients:
root@search1:~# systemctl stop swh-search-journal-client@indexed
root@search1:~# systemctl stop swh-search-journal-client@objects
  • flush the index to speedup the recovery
curl -XPOST http://search-esnode4:9200/_flush

For each node :

  • disable shard allocation:
cat > /tmp/shard_allocation.json <<EOF
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}
EOF
Nov 24 2021, 5:26 PM · System administration (Component upgrades), Archive search
vsellier added a revision to T3741: swh-search - upgrade elasticsearch backend: D6685: swh-search: upgrade elasticsearch to 7.15.2.
Nov 24 2021, 4:24 PM · System administration (Component upgrades), Archive search
vsellier added a revision to T3741: swh-search - upgrade elasticsearch backend: D6682: swh-search: Upgrade elasticsearch to 7.15.2.
Nov 24 2021, 11:53 AM · System administration (Component upgrades), Archive search
vsellier added a comment to T3741: swh-search - upgrade elasticsearch backend.

The staging elasticsearch is migrated to 7.15.2, everything looks good.

Nov 24 2021, 10:39 AM · System administration (Component upgrades), Archive search

Nov 23 2021

vsellier added a revision to T3741: swh-search - upgrade elasticsearch backend: D6677: staging: upgrade swh-search elasticsearch to 7.15.2.
Nov 23 2021, 7:53 PM · System administration (Component upgrades), Archive search

Nov 19 2021

seirl closed T3742: yarn called in swh-search setup.py but not present in developer setup docs as Resolved.

Fixed in rDDOC55cdfd9ee957f57cf91b0f6932cc941d2887d933

Nov 19 2021, 5:29 PM · Archive search
seirl triaged T3742: yarn called in swh-search setup.py but not present in developer setup docs as Normal priority.
Nov 19 2021, 5:17 PM · Archive search
vsellier triaged T3741: swh-search - upgrade elasticsearch backend as Normal priority.
Nov 19 2021, 5:01 PM · System administration (Component upgrades), Archive search

Nov 5 2021

vsellier triaged T3708: Upgrade swh-search elasticsearch version as Normal priority.
Nov 5 2021, 2:31 PM · Archive search, System administration (Component upgrades)

Oct 19 2021

vsellier renamed T3671: staging - swh-search (metadata indexer) is unable to update a document due to an unparseable date from staging - swh-search unable to update a document due to an unparseable date to staging - swh-search (metadata indexer) is unable to update a document due to an unparseable date.
Oct 19 2021, 11:03 AM · Intrinsic metadata, Archive search
vsellier updated the task description for T3671: staging - swh-search (metadata indexer) is unable to update a document due to an unparseable date.
Oct 19 2021, 10:54 AM · Intrinsic metadata, Archive search
vsellier triaged T3671: staging - swh-search (metadata indexer) is unable to update a document due to an unparseable date as Normal priority.
Oct 19 2021, 10:48 AM · Intrinsic metadata, Archive search

Oct 1 2021

anlambert added a revision to T2254: textual search language for the Web UI: D6390: search: Add query language support for staff users.
Oct 1 2021, 3:03 PM · Archive search, Web app

Sep 29 2021

ardumont closed T3620: Deploy swh.search v0.11.6 as Resolved.
Sep 29 2021, 6:20 PM · System administration, Archive search
ardumont moved T3620: Deploy swh.search v0.11.6 from code-review to deployed/landed/monitoring on the System administration board.
Sep 29 2021, 6:20 PM · System administration, Archive search
ardumont moved T3620: Deploy swh.search v0.11.6 from in-progress to code-review on the System administration board.
Sep 29 2021, 6:20 PM · System administration, Archive search
ardumont updated the task description for T3620: Deploy swh.search v0.11.6.
Sep 29 2021, 6:12 PM · System administration, Archive search
ardumont changed the status of T3620: Deploy swh.search v0.11.6 from Open to Work in Progress.
Sep 29 2021, 6:00 PM · System administration, Archive search
ardumont added projects to T3620: Deploy swh.search v0.11.6: Archive search, System administration.
Sep 29 2021, 5:06 PM · System administration, Archive search

Sep 23 2021

vlorentz triaged T3606: Document the swh-search design as Normal priority.
Sep 23 2021, 2:55 PM · Archive search, Documentation

Sep 20 2021

vsellier added a revision to T3433: Deploy swh.search v0.10/v0.11: D6303: swh-web: fix the metadata backend configuration in the swh-search override.
Sep 20 2021, 1:42 PM · System administration, Archive search

Sep 13 2021

vlorentz added a comment to T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch.

On the other hand, journal clients are sort of a resolution to T2063.

Sep 13 2021, 2:42 PM · Archive search, Metadata workflow
olasd added a comment to T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch.

I'm tempted to postpone this issue until we resolve T2063...

Sep 13 2021, 11:36 AM · Archive search, Metadata workflow

Sep 10 2021

vlorentz added a comment to T2073: Index extrinsic metadata from the journal in swh-search/Elasticsearch.

I'm tempted to postpone this issue until we resolve T2063...

Sep 10 2021, 4:05 PM · Archive search, Metadata workflow
anlambert closed T3441: Implement query to get origin visit types dynamically as Resolved.

This has been implemented and is now used by swh-web in production, closing this.

Sep 10 2021, 10:49 AM · Archive search

Sep 8 2021

anlambert added a revision to T3441: Implement query to get origin visit types dynamically: D6219: common/utils: Add function to get origin visit types dynamically.
Sep 8 2021, 5:42 PM · Archive search
vlorentz closed T2590: Finish the indexer -> swh-search pipeline, a subtask of T2182: Switch production swh-web to use swh-search instead of postgresql search., as Resolved.
Sep 8 2021, 3:35 PM · System administration, Archive search, Storage manager
vlorentz closed T2590: Finish the indexer -> swh-search pipeline as Resolved.
Sep 8 2021, 3:35 PM · Journal, Archive search
vsellier closed T3040: [production] Enable swh-search's journal-client for indexed objects, a subtask of T2590: Finish the indexer -> swh-search pipeline, as Resolved.
Sep 8 2021, 3:24 PM · Journal, Archive search
vsellier closed T3040: [production] Enable swh-search's journal-client for indexed objects as Resolved.

metadata searches are now done in Elasticsearch since the deployment of T3433

Sep 8 2021, 3:24 PM · System administration, Journal, Archive search
vsellier renamed T3433: Deploy swh.search v0.10/v0.11 from Deploy swh.search v0.10/v0.11 on staging to Deploy swh.search v0.10/v0.11.
Sep 8 2021, 3:21 PM · System administration, Archive search
vsellier closed T3433: Deploy swh.search v0.10/v0.11 as Resolved.

Everything is deployed and look functional.

Sep 8 2021, 3:21 PM · System administration, Archive search

Sep 7 2021

vsellier added a revision to T3433: Deploy swh.search v0.10/v0.11: D6206: webapp: support new metadata search backend configuation.
Sep 7 2021, 4:08 PM · System administration, Archive search
vlorentz closed T3562: [swh-search] Document version conflict during parallel indexation as Resolved by committing rDSEA7479282c70db: Retry on concurrent conflicting updates.
Sep 7 2021, 3:31 PM · Archive search
vlorentz added a revision to T3562: [swh-search] Document version conflict during parallel indexation: D6203: Retry on concurrent conflicting updates.
Sep 7 2021, 2:53 PM · Archive search
vsellier added a revision to T3433: Deploy swh.search v0.10/v0.11: D6197: swh-search: use the consumer group used during the reindexation.
Sep 7 2021, 11:22 AM · System administration, Archive search

Sep 6 2021

vsellier triaged T3562: [swh-search] Document version conflict during parallel indexation as Normal priority.
Sep 6 2021, 2:52 PM · Archive search
vlorentz updated the task description for T3560: Polish the swh-search QL.
Sep 6 2021, 10:38 AM · Archive search, System administration
vlorentz removed a project from T3559: Enable the swh-search QL in staging: meta-task.
Sep 6 2021, 10:37 AM · Archive search, System administration, Intrinsic metadata, Extrinsic metadata
vlorentz removed a project from T3558: Enable the swh-search QL in production: meta-task.
Sep 6 2021, 10:37 AM · Archive search, System administration, Intrinsic metadata, Extrinsic metadata
vlorentz triaged T3560: Polish the swh-search QL as Normal priority.
Sep 6 2021, 10:37 AM · Archive search, System administration
vlorentz added a project to T3558: Enable the swh-search QL in production: Archive search.
Sep 6 2021, 10:36 AM · Archive search, System administration, Intrinsic metadata, Extrinsic metadata
vlorentz triaged T3559: Enable the swh-search QL in staging as Normal priority.
Sep 6 2021, 10:36 AM · Archive search, System administration, Intrinsic metadata, Extrinsic metadata

Sep 3 2021

vsellier added a comment to T3433: Deploy swh.search v0.10/v0.11.

production deployment:

  • disable puppet
  • stop and disable the journal clients and the search backend
  • update the swh-search configuration to change the index name to origin-v0.11
root@search1:/etc/softwareheritage/search# diff -U3 /tmp/server.yml server.yml
--- /tmp/server.yml	2021-09-03 14:06:07.896137122 +0000
+++ server.yml	2021-09-03 14:05:47.072081879 +0000
@@ -10,7 +10,7 @@
     port: 9200
   indexes:
     origin:
-      index: origin-production
+      index: origin-v0.11
       read_alias: origin-read
       write_alias: origin-write
  • update the journal-clients to use a group id swh.search.journal_client.[indexed|object]-v0.11
root@search1:/etc/softwareheritage/search# diff -U3 /tmp/journal_client_objects.yml journal_client_objects.yml 
--- /tmp/journal_client_objects.yml	2021-09-03 14:06:52.660255797 +0000
+++ journal_client_objects.yml	2021-09-03 14:07:10.684303568 +0000
@@ -8,7 +8,7 @@
   - kafka2.internal.softwareheritage.org
   - kafka3.internal.softwareheritage.org
   - kafka4.internal.softwareheritage.org
-  group_id: swh.search.journal_client
+  group_id: swh.search.journal_client-v0.11
   prefix: swh.journal.objects
   object_types:
   - origin
root@search1:/etc/softwareheritage/search# diff -U3 /tmp/journal_client_indexed.yml journal_client_indexed.yml 
--- /tmp/journal_client_indexed.yml	2021-09-03 14:06:52.660255797 +0000
+++ journal_client_indexed.yml	2021-09-03 14:07:25.760343512 +0000
@@ -8,7 +8,7 @@
   - kafka2.internal.softwareheritage.org
   - kafka3.internal.softwareheritage.org
   - kafka4.internal.softwareheritage.org
-  group_id: swh.search.journal_client.indexed
+  group_id: swh.search.journal_client.indexed-v0.11
   prefix: swh.journal.indexed
   object_types:
   - origin_intrinsic_metadata
  • perform a system upgrade
root@search1:/etc/softwareheritage/search# apt dist-upgrade -V
...
The following NEW packages will be installed:
   python3-tree-sitter (0.19.0-1+swh1~bpo10+1)
The following packages will be upgraded:
   libnss-systemd (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   libpam-systemd (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   libsystemd0 (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   libudev1 (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   python3-swh.core (0.14.3-1~swh1~bpo10+1 => 0.14.5-1~swh1~bpo10+1)
   python3-swh.model (2.6.1-1~swh1~bpo10+1 => 2.8.0-1~swh1~bpo10+1)
   python3-swh.scheduler (0.15.0-1~swh1~bpo10+1 => 0.18.0-1~swh1~bpo10+1)
   python3-swh.search (0.9.0-1~swh1~bpo10+1 => 0.11.4-2~swh1~bpo10+1)
   python3-swh.storage (0.30.1-1~swh1~bpo10+1 => 0.36.0-1~swh1~bpo10+1)
   systemd (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   systemd-sysv (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   systemd-timesyncd (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   udev (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
13 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
...

There is no need to reboot

  • enable and restart the swh-search backend
  • check the new index creation
root@search1:/etc/softwareheritage/search# curl ${ES_SERVER}/_cat/indices\?v
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin-v0.11      XOUR_jKcTtWKjlPk_8EAlA  90   1          0            0     34.3kb         18.2kb
green  open   origin-v0.9.0     TH9xlECuS4CcJTDw0Fqieg  90   1  175001478     36494554      293gb        146.9gb
green  open   origin-production hZfuv0lVRImjOjO_rYgDzg  90   1  176722078     56232582      311gb        155.1gb
  • update the write index alias
root@search1:~/T3433# ./update-write-alias.sh 
{"acknowledged":true}{"acknowledged":true}root@search1:~/T3433# 
root@search1:~/T3433# curl ${ES_SERVER}/_cat/aliases\?v
alias               index             filter routing.index routing.search is_write_index
origin-write        origin-v0.11      -      -             -              -
origin-read-v0.9.0  origin-v0.9.0     -      -             -              -
origin-v0.9.0-read  origin-v0.9.0     -      -             -              -
origin-v0.9.0-write origin-v0.9.0     -      -             -              -
origin-write-v0.9.0 origin-v0.9.0     -      -             -              -
origin-read         origin-production -      -             -              -

All the v0.9.0 stuff will be cleared once the migration to the v0.11 done

  • restart the journal clients
root@search1:~# systemctl enable swh-search-journal-client@objects
Created symlink /etc/systemd/system/multi-user.target.wants/swh-search-journal-client@objects.service → /etc/systemd/system/swh-search-journal-client@.service.
root@search1:~# systemctl enable swh-search-journal-client@indexed
Created symlink /etc/systemd/system/multi-user.target.wants/swh-search-journal-client@indexed.service → /etc/systemd/system/swh-search-journal-client@.service.
root@search1:~# systemctl start swh-search-journal-client@objects
root@search1:~# systemctl start swh-search-journal-client@indexed
  • wait for the lag to recover, create additional journal clients if necessary
  • update the read index alias
  • land D6182, D6183, D6197
  • Update swh-web configuration to support the new way to configure the metadata search backend (D6202)
  • deploy them on webapp1 and moma
Sep 3 2021, 4:03 PM · System administration, Archive search
vsellier added a revision to T3040: [production] Enable swh-search's journal-client for indexed objects: D6183: swh-search: activate metadata search all ES on the main webapp.
Sep 3 2021, 3:45 PM · System administration, Journal, Archive search
vsellier added a revision to T3433: Deploy swh.search v0.10/v0.11: D6182: swh-search: update the configuration for the deployment of v0.11.4.
Sep 3 2021, 3:44 PM · System administration, Archive search
vsellier added a comment to T3433: Deploy swh.search v0.10/v0.11.
  • puppet configuration deployed in staging
  • read index updated with this script:
#!/bin/bash
Sep 3 2021, 9:57 AM · System administration, Archive search
vsellier added a revision to T3433: Deploy swh.search v0.10/v0.11: D6176: swh-search: deploy v0.11.4 in staging.
Sep 3 2021, 8:42 AM · System administration, Archive search
vsellier added a comment to T3433: Deploy swh.search v0.10/v0.11.

The lag has recovered in ~ 12hours.
The content of the index looks goods (just cherry picked a couple of origin).

Sep 3 2021, 8:34 AM · System administration, Archive search

Sep 1 2021

vsellier added a comment to T3433: Deploy swh.search v0.10/v0.11.
  • package python3-swh.search upgraded to version 0.11.4-2, the problem is fixed
  • the new index is well created:
root@search0:/# curl -s http://search-esnode0:9200/_cat/indices\?v
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin-v0.11                HljzsdD9SmKI7-8ekB_q3Q  80   0          0            0      4.2kb          4.2kb
green  close  origin                      HthJj42xT5uO7w3Aoxzppw  80   0                                                  
green  close  origin-v0.9.0               o7FiYJWnTkOViKiAdCXCuA  80   0                                                  
green  open   origin-v0.10.0              -fvf4hK9QDeN8qYTJBBlxQ  80   0    1981623       559384      2.3gb          2.3gb
green  close  origin-backup-20210209-1736 P1CKjXW0QiWM5zlzX46-fg  80   0                                                  
green  close  origin-v0.5.0               SGplSaqPR_O9cPYU4ZsmdQ  80   0
  • journal clients enabled and restarted
  • the journal clients lags should recover in less than 12h
  • waiting some time to estimate the duration with only one journal client per type
Sep 1 2021, 5:46 PM · System administration, Archive search
vsellier added a comment to T3433: Deploy swh.search v0.10/v0.11.

The problem was fixed by rDSEA68347a5604c74150197f691593cbb05bdd34396f
thanks @olasd

Sep 1 2021, 5:22 PM · System administration, Archive search
vsellier added a comment to T3433: Deploy swh.search v0.10/v0.11.

Deployment of version v0.11.4 in staging:
On search0:

  • puppet stopped
  • stop and disable the journal clients and search backend
  • update the swh-search configuration to use a origin-v0.11 index
root@search0:/etc/softwareheritage/search# diff -U2 /tmp/server.yml server.yml 
--- /tmp/server.yml	2021-09-01 13:42:29.347951302 +0000
+++ server.yml	2021-09-01 13:42:35.739953523 +0000
@@ -7,5 +7,5 @@
   indexes:
     origin:
-      index: origin-v0.10.0
+      index: origin-v0.11
       read_alias: origin-read
       write_alias: origin-write
  • update the journal-clients to use a group id swh.search.journal_client.[indexed|object]-v0.11
root@search0:/etc/softwareheritage/search# diff -U3 /tmp/journal_client_objects.yml journal_client_objects.yml 
--- /tmp/journal_client_objects.yml	2021-09-01 13:44:49.843999978 +0000
+++ journal_client_objects.yml	2021-09-01 13:45:03.972004852 +0000
@@ -5,7 +5,7 @@
 journal:
   brokers:
   - journal0.internal.staging.swh.network
-  group_id: swh.search.journal_client-v0.10.0
+  group_id: swh.search.journal_client-v0.11
   prefix: swh.journal.objects
   object_types:
   - origin
root@search0:/etc/softwareheritage/search# diff -U3 /tmp/journal_client_indexed.yml journal_client_indexed.yml 
--- /tmp/journal_client_indexed.yml	2021-09-01 13:44:44.847998252 +0000
+++ journal_client_indexed.yml	2021-09-01 13:44:57.020002454 +0000
@@ -5,7 +5,7 @@
 journal:
   brokers:
   - journal0.internal.staging.swh.network
-  group_id: swh.search.journal_client.indexed-v0.10.0
+  group_id: swh.search.journal_client.indexed-v0.11
   prefix: swh.journal.indexed
   object_types:
   - origin_intrinsic_metadata
  • perform a system upgrade, a reboot was not required
  • enable and start swh-search backend
  • An error occurs after the restart:
Sep 01 14:19:12 search0 python3[4066688]: 2021-09-01 14:19:12 [4066688] root:ERROR command 'cc' failed with exit status 1
                                          Traceback (most recent call last):
                                            File "/usr/lib/python3.7/distutils/unixccompiler.py", line 118, in _compile
                                              extra_postargs)
                                            File "/usr/lib/python3.7/distutils/ccompiler.py", line 909, in spawn
                                              spawn(cmd, dry_run=self.dry_run)
                                            File "/usr/lib/python3.7/distutils/spawn.py", line 36, in spawn
                                              _spawn_posix(cmd, search_path, dry_run=dry_run)
                                            File "/usr/lib/python3.7/distutils/spawn.py", line 159, in _spawn_posix
                                              % (cmd, exit_status))
                                          distutils.errors.DistutilsExecError: command 'cc' failed with exit status 1
Sep 1 2021, 5:15 PM · System administration, Archive search
vsellier closed T3484: Fix the release builds for swh-search, a subtask of T3433: Deploy swh.search v0.10/v0.11, as Resolved.
Sep 1 2021, 3:21 PM · System administration, Archive search
vsellier closed T3484: Fix the release builds for swh-search as Resolved.

The build is now fixed and the v0.11.4 version is ready to be deployed on the environments

Sep 1 2021, 3:21 PM · System administration, Archive search

Aug 31 2021

ardumont closed T2102: Clean up Debian packaging branch bootstrapping scripts, a subtask of T2101: Bootstrap Debian packaging for swh.search, as Resolved.
Aug 31 2021, 10:48 AM · Archive search

Aug 30 2021

vlorentz assigned T3040: [production] Enable swh-search's journal-client for indexed objects to vsellier.
Aug 30 2021, 10:41 AM · System administration, Journal, Archive search
vlorentz closed T3373: Metadata search is failing due to a boolean field in the mapping of the metadata fields, a subtask of T3041: [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata, as Resolved.
Aug 30 2021, 9:37 AM · System administration, Archive search
vlorentz closed T3373: Metadata search is failing due to a boolean field in the mapping of the metadata fields as Resolved.
Aug 30 2021, 9:37 AM · System administration, Archive search

Aug 25 2021

anlambert added a revision to T3441: Implement query to get origin visit types dynamically: D6137: search: Add count_visit_types to interface.
Aug 25 2021, 6:29 PM · Archive search

Aug 17 2021

vsellier added a comment to T3484: Fix the release builds for swh-search.

One very important thing to get right is the Build-Depends line in the source package stanza. setuptools/distribute-based packages have the nasty habit of downloading dependencies from PyPI if they are needed at python setup.py build time. If the package is available from the system (as would be the case when Build-Depends > is up-to-date), then distribute will not try to download the package, otherwise it will try to download it. This is a huge no-no, and pybuild internally sets the http_proxy and https_proxy environment variables (to 127.0.0.1:9) to prevent this from happening.

Aug 17 2021, 6:13 PM · System administration, Archive search
vsellier added a comment to T3484: Fix the release builds for swh-search.

The pypi build is still working well with the 2 last diff.
Now there is a new error during the debian ones:

dh: warning: Compatibility levels before 10 are deprecated (level 9 in use)
   dh_auto_clean -O--buildsystem=pybuild
dh_auto_clean: warning: Compatibility levels before 10 are deprecated (level 9 in use)
I: pybuild base:232: python3.9 setup.py clean 
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101bcd0>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101beb0>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101b850>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101b730>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fef2101b610>: Failed to establish a new connection: [Errno -2] Name or service not known'))': /simple/tree-sitter/
ERROR: Could not find a version that satisfies the requirement tree-sitter==0.19.0
ERROR: No matching distribution found for tree-sitter==0.19.0
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/setuptools/installer.py", line 75, in fetch_build_egg
    subprocess.check_call(cmd)
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3.9', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpdrbws3hq', '--quiet', 'tree-sitter==0.19.0']' returned non-zero exit status 1.
Aug 17 2021, 5:57 PM · System administration, Archive search

Aug 16 2021

vsellier renamed T3484: Fix the release builds for swh-search from Fix the pypi-upload build for swh-search to Fix the release builds for swh-search.
Aug 16 2021, 2:54 PM · System administration, Archive search

Aug 13 2021

vlorentz added a comment to T3484: Fix the release builds for swh-search.

you also need to add yarn as a build-dep for debian packages

Aug 13 2021, 5:25 PM · System administration, Archive search
vlorentz added a comment to T3484: Fix the release builds for swh-search.

D6088 should help

Aug 13 2021, 5:24 PM · System administration, Archive search
ardumont added a comment to T3484: Fix the release builds for swh-search.

So that unstuck the pypi part ;)
Now on with unstucking the debian build [1]

Aug 13 2021, 4:45 PM · System administration, Archive search
vsellier added a revision to T3484: Fix the release builds for swh-search: D6086: Document the dependency on the tree-sitter python module.
Aug 13 2021, 4:18 PM · System administration, Archive search
vsellier added a revision to T3484: Fix the release builds for swh-search: D6085: Install a missing python module for the swh-search build.
Aug 13 2021, 4:14 PM · System administration, Archive search
vsellier moved T3484: Fix the release builds for swh-search from Backlog to in-progress on the System administration board.
Aug 13 2021, 10:35 AM · System administration, Archive search
vsellier changed the status of T3484: Fix the release builds for swh-search from Open to Work in Progress.
Aug 13 2021, 10:35 AM · System administration, Archive search
vsellier added a comment to T3373: Metadata search is failing due to a boolean field in the mapping of the metadata fields.

there are no more errors. The fix will deployed in production with the deployment of swh-search:v0.11.0 (T3433)

Aug 13 2021, 10:28 AM · System administration, Archive search

Aug 11 2021

ardumont renamed T3433: Deploy swh.search v0.10/v0.11 from Deploy swh.search ~~v0.10~~ v0.11 on staging to Deploy swh.search v0.10/v0.11 on staging.
Aug 11 2021, 11:25 AM · System administration, Archive search
ardumont renamed T3433: Deploy swh.search v0.10/v0.11 from Deploy swh.search v0.10 on staging to Deploy swh.search ~~v0.10~~ v0.11 on staging.
Aug 11 2021, 11:24 AM · System administration, Archive search
ardumont added a comment to T3433: Deploy swh.search v0.10/v0.11.

A new swh.search v0.11 got tagged (this includes the current blocking point
deactivation). That's a workaround though. I've opened a task to avoid forgetting about
the conclusion on the discussion started.

Aug 11 2021, 11:24 AM · System administration, Archive search
ardumont triaged T3479: Define a proper way to upgrade swh.search as Normal priority.
Aug 11 2021, 11:23 AM · System administration, Archive search
ardumont added a project to T3479: Define a proper way to upgrade swh.search: System administration.
Aug 11 2021, 11:23 AM · System administration, Archive search
ardumont created T3479: Define a proper way to upgrade swh.search.
Aug 11 2021, 11:23 AM · System administration, Archive search

Jul 29 2021

ardumont moved T3060: Deploy swh-search v0.6.0 in **staging** from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration, Archive search
ardumont moved T3041: [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata from in-progress to done on the System administration board.
Jul 29 2021, 1:23 PM · System administration, Archive search

Jul 28 2021

ardumont moved T3433: Deploy swh.search v0.10/v0.11 from Weekly backlog to in-progress on the System administration board.
Jul 28 2021, 12:05 PM · System administration, Archive search

Jul 27 2021

ardumont added a comment to T3433: Deploy swh.search v0.10/v0.11.

By the way, a small list of caveats encountered when deploying search (not to fix
immediately, just to mention them).

Jul 27 2021, 8:59 AM · System administration, Archive search

Jul 26 2021

vlorentz added a comment to T3433: Deploy swh.search v0.10/v0.11.

Another idea: move this fetching to a new indexer, and make it write to a new topic, which the swh-search journal client can read from.

Jul 26 2021, 7:31 PM · System administration, Archive search