Query: All Stories

Reorganised motivations

	Include stories about projects I am a member of.

@ardumont @olasd can we do this:

Add a new property fallback_schema = ['https', 'http'] in PackageLoader
Change download_package to try different schemas from fallback_schema list (if url/schema is valid)
Change this (fallback_schema) property in the inherited classes (from PackageLoader) as per requirements

Handle missing schema only for niguix (instead of doing it for all types of downloads)

So, what would be useful as metadata? What we have is groupId, artifactId, version.

maven jar-loader: fix tests.

Old versions of Dulwich (eg. 0.16.3, the version in stretch), dropped newlines at the end of the gpgsig header.

vault: Fix content-types

With the new concurrent replay of the directory, the disk usage grow up rapidly:

Some feedback, I had to delay the benchmarks because the servers was almost full and the cluster needed to be expanded to 7 nodes. The cluster is in stabilization phase (rebuild/repair of the new node and cleanup of the old one)
When it will be done, I will be able to finalize the tests Hopefully at the beginning of the next week

Closing this as resolved, archive coverage widget has been updated accordingly and deployed.

@stsp , thanks for this !

Improve connection error handling on both client and server side

No need for the _metrics suffix

Keep the _metrics in the metric name

Leave as is for a first deployment.
We will test on the partners platform and iterate.

production deployment:

disable puppet
stop and disable the journal clients and the search backend
update the swh-search configuration to change the index name to origin-v0.11

root@search1:/etc/softwareheritage/search# diff -U3 /tmp/server.yml server.yml
--- /tmp/server.yml	2021-09-03 14:06:07.896137122 +0000
+++ server.yml	2021-09-03 14:05:47.072081879 +0000
@@ -10,7 +10,7 @@
     port: 9200
   indexes:
     origin:
-      index: origin-production
+      index: origin-v0.11
       read_alias: origin-read
       write_alias: origin-write

update the journal-clients to use a group id swh.search.journal_client.[indexed|object]-v0.11

root@search1:/etc/softwareheritage/search# diff -U3 /tmp/journal_client_objects.yml journal_client_objects.yml 
--- /tmp/journal_client_objects.yml	2021-09-03 14:06:52.660255797 +0000
+++ journal_client_objects.yml	2021-09-03 14:07:10.684303568 +0000
@@ -8,7 +8,7 @@
   - kafka2.internal.softwareheritage.org
   - kafka3.internal.softwareheritage.org
   - kafka4.internal.softwareheritage.org
-  group_id: swh.search.journal_client
+  group_id: swh.search.journal_client-v0.11
   prefix: swh.journal.objects
   object_types:
   - origin
root@search1:/etc/softwareheritage/search# diff -U3 /tmp/journal_client_indexed.yml journal_client_indexed.yml 
--- /tmp/journal_client_indexed.yml	2021-09-03 14:06:52.660255797 +0000
+++ journal_client_indexed.yml	2021-09-03 14:07:25.760343512 +0000
@@ -8,7 +8,7 @@
   - kafka2.internal.softwareheritage.org
   - kafka3.internal.softwareheritage.org
   - kafka4.internal.softwareheritage.org
-  group_id: swh.search.journal_client.indexed
+  group_id: swh.search.journal_client.indexed-v0.11
   prefix: swh.journal.indexed
   object_types:
   - origin_intrinsic_metadata

perform a system upgrade

root@search1:/etc/softwareheritage/search# apt dist-upgrade -V
...
The following NEW packages will be installed:
   python3-tree-sitter (0.19.0-1+swh1~bpo10+1)
The following packages will be upgraded:
   libnss-systemd (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   libpam-systemd (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   libsystemd0 (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   libudev1 (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   python3-swh.core (0.14.3-1~swh1~bpo10+1 => 0.14.5-1~swh1~bpo10+1)
   python3-swh.model (2.6.1-1~swh1~bpo10+1 => 2.8.0-1~swh1~bpo10+1)
   python3-swh.scheduler (0.15.0-1~swh1~bpo10+1 => 0.18.0-1~swh1~bpo10+1)
   python3-swh.search (0.9.0-1~swh1~bpo10+1 => 0.11.4-2~swh1~bpo10+1)
   python3-swh.storage (0.30.1-1~swh1~bpo10+1 => 0.36.0-1~swh1~bpo10+1)
   systemd (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   systemd-sysv (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   systemd-timesyncd (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
   udev (247.3-3~bpo10+1 => 247.3-6~bpo10+1)
13 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
...

There is no need to reboot

enable and restart the swh-search backend
check the new index creation

root@search1:/etc/softwareheritage/search# curl ${ES_SERVER}/_cat/indices\?v
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   origin-v0.11      XOUR_jKcTtWKjlPk_8EAlA  90   1          0            0     34.3kb         18.2kb
green  open   origin-v0.9.0     TH9xlECuS4CcJTDw0Fqieg  90   1  175001478     36494554      293gb        146.9gb
green  open   origin-production hZfuv0lVRImjOjO_rYgDzg  90   1  176722078     56232582      311gb        155.1gb

update the write index alias

root@search1:~/T3433# ./update-write-alias.sh 
{"acknowledged":true}{"acknowledged":true}root@search1:~/T3433# 
root@search1:~/T3433# curl ${ES_SERVER}/_cat/aliases\?v
alias               index             filter routing.index routing.search is_write_index
origin-write        origin-v0.11      -      -             -              -
origin-read-v0.9.0  origin-v0.9.0     -      -             -              -
origin-v0.9.0-read  origin-v0.9.0     -      -             -              -
origin-v0.9.0-write origin-v0.9.0     -      -             -              -
origin-write-v0.9.0 origin-v0.9.0     -      -             -              -
origin-read         origin-production -      -             -              -

All the v0.9.0 stuff will be cleared once the migration to the v0.11 done

restart the journal clients

root@search1:~# systemctl enable swh-search-journal-client@objects
Created symlink /etc/systemd/system/multi-user.target.wants/swh-search-journal-client@objects.service → /etc/systemd/system/swh-search-journal-client@.service.
root@search1:~# systemctl enable swh-search-journal-client@indexed
Created symlink /etc/systemd/system/multi-user.target.wants/swh-search-journal-client@indexed.service → /etc/systemd/system/swh-search-journal-client@.service.
root@search1:~# systemctl start swh-search-journal-client@objects
root@search1:~# systemctl start swh-search-journal-client@indexed

wait for the lag to recover, create additional journal clients if necessary
update the read index alias
land D6182, D6183, D6197
Update swh-web configuration to support the new way to configure the metadata search backend (D6202)
deploy them on webapp1 and moma

Oh right, I missed those too during my yesterday reviews.

maven jar-loader: fix tests.

maven jar-loader: fix tests.

test_directory_get_entries_pagination: don't depend on result order

maven jar-loader: fix tests.

For git-bare cooking to work, upgrade swh.storage to the latest version (> v0.35) and
restart the storage service (done).

Would you have time during September to help on this task? (it is a roadmap task, btw)

maven jar-loader: fix tests.

I forgot to update the diff after local rebase, closed by rSPSITEe009324d13e8a6f3821bdc755a1c3ae9c044bf28.

maven jar-loader: fix minor typos + tests.

maven jar-loader: fix minor typos + tests.

Drop the other secondary cluster instruction

webapp/config: Bump content_display_max_size to 5 MB

setup.py: Ensure tree-sitter-cli gets installed in develop command

vault-ui: Remove remaining references to object_type

All Stories
Use Results
Edit Query
Hide Query

Sep 6 2021

Sep 5 2021

Sep 4 2021

Sep 3 2021

All StoriesUse ResultsEdit QueryHide Query

Sep 6 2021

Sep 5 2021

Sep 4 2021

Sep 3 2021

All Stories
Use Results
Edit Query
Hide Query