Query: Advanced Search

Add sentry token for swh-counters

vagrant: add staging-counters0

Remove the swh/__init__.py file from the package

docker: Install swh-counters in the docker image

	Include stories about projects I am a member of.

To identify the disks to replace, the front led can be activated via the idrac interface.
The disks are :
scsi-35000c500ae759873 | /dev/sdd: Serial ZA1G3R1S -> Physical Disk 0:1:3
scsi-35000c500ae750b2f | /dev/sdf : Serial ZA1G3H81 -> Physical Disk 0:1:5

Removing the mirror mirror-2 with the disks scsi-35000c500ae750b2f and scsi-35000c500ae759873

root@granet:~# zpool remove hdd mirror-2
root@granet:~# zpool status hdd
  pool: hdd
 state: ONLINE
  scan: scrub repaired 0B in 0 days 17:39:22 with 0 errors on Sun Mar 14 18:03:23 2021
remove: Evacuation of mirror in progress since Fri Mar 19 10:45:22 2021
    1.03G copied out of 6.19T at 118M/s, 0.02% done, 15h18m to go
config:

And added to the hdd zfs pool:

before

root@granet:~# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
hdd   29.1T  22.8T  6.29T        -         -    20%    78%  1.00x    ONLINE  -
ssd   10.3T  7.91T  2.44T        -         -    24%    76%  1.00x    ONLINE  -

configuration

root@granet:~# ls -l /dev/disk/by-id | grep -e "wwn.*sdk" -e "wwn.*sdl"
lrwxrwxrwx 1 root root  9 Mar 19 10:10 wwn-0x5000c500cb46da4b -> ../../sdl
lrwxrwxrwx 1 root root  9 Mar 19 10:09 wwn-0x5000c500cb46e41b -> ../../sdk

root@granet:~# zpool add hdd mirror wwn-0x5000c500cb46da4b wwn-0x5000c500cb46e41b

root@granet:~# zpool status hdd
  pool: hdd
 state: ONLINE
  scan: scrub repaired 0B in 0 days 17:39:22 with 0 errors on Sun Mar 14 18:03:23 2021
config:

Disks configured in JBOD mode:

root@granet:~# megacli -PDMakeJBOD  -physdrv[32:10] -a0

ZFS status was reset and a scrub restart after the upgrade of the zfs packages.
No more errors are detected.

All the servers were updated. We took the opportunity to upgrade and restart them to apply the last updates.

webapp: use replica as main database on production

plan for hypervisors / nodes upgrades:

*esnode*
- delaying node down detection and limit shard allocation to primaries

esnode2 ~ % export ES_NODE=192.168.100.62:9200                                              
esnode2 ~ % curl -XPUT -H "Content-Type: application/json" http://$ES_NODE/_cluster/settings -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

Plan:

first the upgrade will be done on the elasticsearch server
in parallel somerset can be updated
after the webapp can be configured to use somerset as the principal database
Upgrade of belveder
upgrade of saam (with the help of @olasd)
upgrade of belvedere
and finally the kafka servers

branches unstable and buster configured
The builds passed and the package is available :

root@search0:~# apt search python3-swh.counters
Sorting... Done
Full Text Search... Done
python3-swh.counters/unknown 0.1.0-1+swh2~bpo10+1 all
  Software Heritage counters utilities

Rebuild to unstuck the debian packaging.

Rebuild for buster-swh

pristine-tar data for swh.counters_0.1.0.orig.tar.gz

Initial packaging for swh.counters

New upstream version 0.1.0

Build is working for unstable and buster.

Complete rebuild from unstable

Rebuild for buster-swh

Initial packaging for pytest-redis

jobs/dependency-packages: Add pytest-redis deps packages

Initial packaging for pytest-redis

New upstream version 2.0.0

pristine-tar data for pytest-redis_2.0.0.orig.tar.gz

New upstream version 2.0.0

New repository created and configured: https://forge.softwareheritage.org/source/pytest-redis/

Implement remote service

fix tear down method

2 disks were installed on the 2 remaining free slots.
They are detected by the raid card but need to be configured in JBOD mode.
It's postponed to Thursday morning as granet is sensible until a demonstration on wednesday afternoon.

Change the flask app implementation method

changing status to pending changes as I will reworke the app initialization to use the same method as swh-objstorage

Implement counters pipeline

review feedbacks:

rename the strangly named test
use a getter for the redis client property
cleanup code

Add redis to the base build image

fix the download url of kafka

The build was failing with a cryptic error :

14:20:43 error committing wijin0be0ztxj3pd6l64aeahh: invalid mutable ref 0xc001a73760: invalid: error committing 8bvh4xv6le4xw14m4fr8zvwho: invalid mutable ref 0xc001a72040: invalid: executor failed running [/bin/sh -c export DEBIAN_FRONTEND=noninteractive &&   apt-get update &&   apt-get install -y apt-transport-https curl ca-certificates gpg  &&   echo deb [signed-by=/usr/share/keyrings/postgres-archive-keyring.gpg] http://apt.postgresql.org/pub/repos/apt/ buster-pgdg main > /etc/apt/sources.list.d/postgres.list &&   curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor > /usr/share/keyrings/postgres-archive-keyring.gpg &&   echo deb [signed-by=/usr/share/keyrings/yarnpkg-archive-keyring.gpg] https://dl.yarnpkg.com/debian/ stable main > /etc/apt/sources.list.d/yarnpkg.list &&   curl -fsSL https://dl.yarnpkg.com/debian/pubkey.gpg | gpg --dearmor > /usr/share/keyrings/yarnpkg-archive-keyring.gpg &&   echo deb [signed-by=/usr/share/keyrings/elasticsearch-archive-keyring.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main > /etc/apt/sources.list.d/elastic-7.x.list &&   curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor > /usr/share/keyrings/elasticsearch-archive-keyring.gpg &&   echo deb [signed-by=/usr/share/keyrings/cassandra.gpg] http://www.apache.org/dist/cassandra/debian 40x main > /etc/apt/sources.list.d/cassandra.list &&   curl -fsSL https://downloads.apache.org/cassandra/KEYS | gpg --dearmor > /usr/share/keyrings/cassandra.gpg &&   apt-get update &&   apt-get upgrade -y &&   apt-get install -y   arcanist   build-essential   cassandra   elasticsearch   fuse3   git-lfs   jq   libfuse3-dev   libsvn-dev   libsystemd-dev   lzip   maven   mercurial   pkg-config   postgresql-11   postgresql-client-11   postgresql-server-dev-11   python3-dev   python3-pip   python3-venv   subversion   tini   yarn   zstd]: stat /var/lib/docker/overlay2/8bvh4xv6le4xw14m4fr8zvwho: no such file or directory
14:20:43 make: *** [Makefile:45: swh-jenkins/base-buster] Error 1
14:20:43 Build step 'Execute shell' marked build as failure

docker: Revert installation of swh-counters

Use the recommanded way to declare the repository signing keys

Used signed-by for all the 3d party repositories

docker-compose: declare swh counters containers

adapt the kafka's healthcheck
remove the typo on the initial log of the swh-counters container

Add the swh-counters in the docker image

~~The debian package~~ a first release is needed to be able to launch the counters without overrides

Advanced Search
Use Results
Edit Query
Hide Query

Mar 22 2021

Mar 21 2021

Mar 19 2021

Mar 18 2021

Mar 17 2021

Mar 16 2021

Mar 15 2021

Mar 12 2021

Advanced SearchUse ResultsEdit QueryHide Query