Page MenuHomeSoftware Heritage
Feed Advanced Search

Dec 7 2021

vsellier committed rDENV6f30000b9f19: Ensure swh-web is started before trying to refresh the save code now statuses (authored by vsellier).
Ensure swh-web is started before trying to refresh the save code now statuses
Dec 7 2021, 3:40 PM
vsellier added a comment to D6775: Ensure swh-web is started before trying to refresh the save code now statuses.

the dependency is already declared on the docker-compose, it just ensure the container is started before launching swh-web-cron, not the service is well initialized and responding.

Dec 7 2021, 3:38 PM
vsellier added a comment to T3778: The docker-dev build is often failing.

thanks @anlambert for the tips, it works as expected

Dec 7 2021, 3:35 PM · System administration
vsellier added a revision to T3778: The docker-dev build is often failing: D6775: Ensure swh-web is started before trying to refresh the save code now statuses.
Dec 7 2021, 3:33 PM · System administration
vsellier requested review of D6775: Ensure swh-web is started before trying to refresh the save code now statuses.
Dec 7 2021, 3:33 PM
vsellier closed D6774: Add durations to nginx and swh-storage access log.
Dec 7 2021, 3:32 PM
vsellier committed rDENVf9cd5ca7780a: Add durations to nginx and swh-storage access log (authored by vsellier).
Add durations to nginx and swh-storage access log
Dec 7 2021, 3:32 PM
vsellier added a comment to D6774: Add durations to nginx and swh-storage access log.

it's related to this change because if it's let to DEBUG, the access log is logged 2 times.

Dec 7 2021, 3:30 PM
vsellier updated the summary of D6774: Add durations to nginx and swh-storage access log.
Dec 7 2021, 3:24 PM
vsellier requested review of D6774: Add durations to nginx and swh-storage access log.
Dec 7 2021, 3:18 PM
vsellier added a revision to T3778: The docker-dev build is often failing: D6774: Add durations to nginx and swh-storage access log.
Dec 7 2021, 3:18 PM · System administration
vsellier accepted D6754: public_keys: Update jenkins keys according to latest changes.

LGTM thanks

Dec 7 2021, 2:52 PM
vsellier accepted D6757: jenkins: Install required jre in jenkins base profile.

LGTM

Dec 7 2021, 2:50 PM
vsellier changed the status of T3778: The docker-dev build is often failing from Open to Work in Progress.
Dec 7 2021, 2:30 PM · System administration
vsellier closed T3777: Cleanup old ghtorrent scheduler services as Resolved.
Dec 7 2021, 2:05 PM · System administration
vsellier added a comment to T3777: Cleanup old ghtorrent scheduler services.

on saatchi:

root@saatchi:/etc/systemd/system# rm -v swh-scheduler-updater* ssh-ghtorrent.service 
removed 'swh-scheduler-updater-consumer-ghtorrent.service'
removed 'swh-scheduler-updater-writer.service'
removed 'swh-scheduler-updater-writer.timer'
removed 'ssh-ghtorrent.service'
Dec 7 2021, 2:05 PM · System administration
vsellier added a comment to T3777: Cleanup old ghtorrent scheduler services.

on scheduler0:

root@scheduler0:/etc/systemd/system# systemctl stop ssh-ghtorrent
root@scheduler0:/etc/systemd/system# systemctl disable ssh-ghtorrent
root@scheduler0:/etc/systemd/system# systemctl stop swh-scheduler-updater*
root@scheduler0:/etc/systemd/system# systemctl disable swh-scheduler-updater*
root@scheduler0:/etc/systemd/system# rm -v swh-scheduler-updater*
removed 'swh-scheduler-updater-consumer-ghtorrent.service'
removed 'swh-scheduler-updater-writer.service'
removed 'swh-scheduler-updater-writer.timer'
root@scheduler0:/etc/systemd/system# rm ssh-ghtorrent.service 
removed 'ssh-ghtorrent.service'
root@scheduler0:/etc/systemd/system# systemctl reset-failed
Dec 7 2021, 2:00 PM · System administration
vsellier moved T3777: Cleanup old ghtorrent scheduler services from Backlog to in-progress on the System administration board.
Dec 7 2021, 1:54 PM · System administration
vsellier added a project to T3777: Cleanup old ghtorrent scheduler services: System administration.
Dec 7 2021, 1:54 PM · System administration
vsellier changed the status of T3777: Cleanup old ghtorrent scheduler services from Open to Work in Progress.
Dec 7 2021, 1:54 PM · System administration
vsellier triaged T3776: cassandra tests are failing in the swh-environment build as High priority.
Dec 7 2021, 1:16 PM · Storage manager
vsellier closed D6756: stop to use the deprecated docker-compose --no-ansi command line option.
Dec 7 2021, 11:34 AM
vsellier committed rDENV11a2ad19a7b0: stop to use the deprecated docker-compose --no-ansi command line option (authored by vsellier).
stop to use the deprecated docker-compose --no-ansi command line option
Dec 7 2021, 11:34 AM
vsellier added a comment to T3773: Deploy swh-scheduler v0.22.0.

Version v0.21.0 deployed in staging:

root@scheduler0:~# apt list --upgradable 2>/dev/null | grep swh | cut -f1 -d'/' | xargs -t apt install
apt install python3-swh.core python3-swh.counters python3-swh.journal python3-swh.lister python3-swh.loader.core python3-swh.model python3-swh.objstorage python3-swh.scheduler python3-swh.storage 
...
root@scheduler0:~# systemctl reload gunicorn-swh-scheduler.service
Dec 7 2021, 9:48 AM · System administration, Npm Lister
vsellier added a comment to T3773: Deploy swh-scheduler v0.22.0.

The problem is reproduced in staging before the deployment

swhworker@worker1:~$ swh lister -C /etc/softwareheritage/lister.yml run -l npm 
Traceback (most recent call last):
  File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.15.0', 'console_scripts', 'swh')()
  File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
    return swh(auto_envvar_prefix="SWH")
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/swh/lister/cli.py", line 65, in run
    get_lister(lister, **config).run()
  File "/usr/lib/python3/dist-packages/swh/lister/pattern.py", line 130, in run
    full_stats.origins += self.send_origins(origins)
  File "/usr/lib/python3/dist-packages/swh/lister/pattern.py", line 234, in send_origins
    ret = self.scheduler.record_listed_origins(batch_origins)
  File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 181, in meth_
    return self.post(meth._endpoint_path, post_data)
  File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 278, in post
    return self._decode_response(response)
  File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 354, in _decode_response
    self.raise_for_status(response)
  File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 344, in raise_for_status
    raise exception from None
swh.core.api.RemoteException: <RemoteException 500 CardinalityViolation: ['ON CONFLICT DO UPDATE command cannot affect row a second time\nHINT:  Ensure that no rows proposed for insertion within the same command have duplicate constrained values.\n']>
Dec 7 2021, 9:02 AM · System administration, Npm Lister
vsellier changed the status of T3773: Deploy swh-scheduler v0.22.0 from Open to Work in Progress.
Dec 7 2021, 8:59 AM · System administration, Npm Lister
vsellier requested review of D6756: stop to use the deprecated docker-compose --no-ansi command line option.
Dec 7 2021, 8:47 AM
vsellier closed D6753: Ensure there is no duplicated origins in the insertion batches.
Dec 7 2021, 8:40 AM
vsellier committed rDSCH0a6aac583adf: Ensure there is no duplicated origins in the insertion batches (authored by vsellier).
Ensure there is no duplicated origins in the insertion batches
Dec 7 2021, 8:40 AM

Dec 6 2021

vsellier updated the diff for D6753: Ensure there is no duplicated origins in the insertion batches.

keep the dict comprehension version

Dec 6 2021, 9:12 PM
vsellier requested review of D6753: Ensure there is no duplicated origins in the insertion batches.
Dec 6 2021, 4:37 PM
vsellier added a revision to T3769: NPM lister is failing with a database update conflict: D6753: Ensure there is no duplicated origins in the insertion batches.
Dec 6 2021, 4:35 PM · System administration, Npm Lister
vsellier renamed T3769: NPM lister is failing with a database update conflict from Unstuck npm lister in production to NPM lister is failing with a database update conflict.
Dec 6 2021, 12:34 PM · System administration, Npm Lister
vsellier added a comment to T3769: NPM lister is failing with a database update conflict.

The last run failed with this error:

Dec 6 2021, 12:32 PM · System administration, Npm Lister
vsellier added a comment to T3769: NPM lister is failing with a database update conflict.

Unstuck the task scheduling:

softwareheritage-scheduler=> begin; update task set next_run=now(), status='next_run_not_scheduled' where id=153874548;
BEGIN
UPDATE 1
softwareheritage-scheduler=*> commit;
COMMIT
Dec 6 2021, 12:30 PM · System administration, Npm Lister
vsellier changed the status of T3769: NPM lister is failing with a database update conflict from Open to Work in Progress.
Dec 6 2021, 12:29 PM · System administration, Npm Lister

Dec 2 2021

vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Dec 2 2021, 6:23 PM · System administration (Component upgrades)
vsellier closed D6736: kafka: remove the physical interface declaration.
Dec 2 2021, 5:40 PM
vsellier committed rSPSITEca930fb95203: kafka: remove the physical interface declaration (authored by vsellier).
kafka: remove the physical interface declaration
Dec 2 2021, 5:40 PM
vsellier requested review of D6736: kafka: remove the physical interface declaration.
Dec 2 2021, 5:38 PM
vsellier added a revision to T3762: Upgrade kafka nodes to bullseye: D6736: kafka: remove the physical interface declaration.
Dec 2 2021, 5:38 PM · System administration (Component upgrades)
vsellier closed D6733: kafka: remove the physical interface declaration.
Dec 2 2021, 5:22 PM
vsellier committed rSPSITE40bbfa808515: kafka: remove the physical interface declaration (authored by vsellier).
kafka: remove the physical interface declaration
Dec 2 2021, 5:22 PM
vsellier requested review of D6733: kafka: remove the physical interface declaration.
Dec 2 2021, 5:16 PM
vsellier added a revision to T3762: Upgrade kafka nodes to bullseye: D6733: kafka: remove the physical interface declaration.
Dec 2 2021, 5:16 PM · System administration (Component upgrades)
vsellier closed D6732: kafka: remove the physical interface declaration.
Dec 2 2021, 4:43 PM
vsellier committed rSPSITEd8bd43fcacc6: kafka: remove the physical interface declaration (authored by vsellier).
kafka: remove the physical interface declaration
Dec 2 2021, 4:43 PM
vsellier requested review of D6732: kafka: remove the physical interface declaration.
Dec 2 2021, 4:39 PM
vsellier added a revision to T3762: Upgrade kafka nodes to bullseye: D6732: kafka: remove the physical interface declaration.
Dec 2 2021, 4:39 PM · System administration (Component upgrades)
vsellier updated the task description for T3762: Upgrade kafka nodes to bullseye.
Dec 2 2021, 3:49 PM · System administration (Component upgrades)
vsellier updated the task description for T3762: Upgrade kafka nodes to bullseye.
Dec 2 2021, 3:38 PM · System administration (Component upgrades)
vsellier closed D6728: debian: prepare the kafka servers upgrade.
Dec 2 2021, 3:04 PM
vsellier committed rSPSITEf752bfd2658d: debian: prepare the kafka servers upgrade (authored by vsellier).
debian: prepare the kafka servers upgrade
Dec 2 2021, 3:04 PM
vsellier requested review of D6728: debian: prepare the kafka servers upgrade.
Dec 2 2021, 2:49 PM
vsellier added a revision to T3762: Upgrade kafka nodes to bullseye: D6728: debian: prepare the kafka servers upgrade.
Dec 2 2021, 2:49 PM · System administration (Component upgrades)
vsellier committed rSENVdec312429e19: vagrant: declare production kafka nodes (authored by vsellier).
vagrant: declare production kafka nodes
Dec 2 2021, 12:49 PM
vsellier committed rSENV0eb1f25cf338: Upgrade migrated boxes to bullseye (authored by vsellier).
Upgrade migrated boxes to bullseye
Dec 2 2021, 12:49 PM
vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Dec 2 2021, 10:49 AM · System administration (Component upgrades)
vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Dec 2 2021, 10:48 AM · System administration (Component upgrades)
vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Dec 2 2021, 10:37 AM · System administration (Component upgrades)
vsellier updated the task description for T3762: Upgrade kafka nodes to bullseye.
Dec 2 2021, 10:37 AM · System administration (Component upgrades)
vsellier changed the status of T3762: Upgrade kafka nodes to bullseye from Open to Work in Progress.
Dec 2 2021, 10:13 AM · System administration (Component upgrades)
vsellier closed T3357: Perform some tests of the cassandra storage on Grid5000, a subtask of T1892: Cassandra as a storage backend, as Resolved.
Dec 2 2021, 10:10 AM · meta-task, Storage manager
vsellier closed T3357: Perform some tests of the cassandra storage on Grid5000 as Resolved.

The slide of the restrospective of the experiment are available at : https://hedgedoc.softwareheritage.org/VOP9qh1MTqm4DjPQfFgNbQ

Dec 2 2021, 10:10 AM · System administration, Storage manager
vsellier closed T3573: [cassandra] directory and content read benchmarks, a subtask of T3357: Perform some tests of the cassandra storage on Grid5000, as Resolved.
Dec 2 2021, 10:08 AM · System administration, Storage manager
vsellier closed T3573: [cassandra] directory and content read benchmarks as Resolved.

It was not easy to know if it's a lot of call or long running calls because it's regular sample and we don't have this granularity.

Dec 2 2021, 10:08 AM · System administration, Storage manager

Dec 1 2021

vsellier updated the task description for T3706: (Minor) upgrade Opnsense to 21.1.9.
Dec 1 2021, 1:14 PM · System administration (Component upgrades)

Nov 29 2021

vsellier accepted D6703: Decomission old vault nodes (orangerie and orangeriedev).

LGTM

Nov 29 2021, 12:09 PM

Nov 26 2021

vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Nov 26 2021, 6:52 PM · System administration (Component upgrades)
vsellier closed T3751: Upgrade swh-search elasticsearch servers to bullseye , a subtask of T3579: Meta-task: upgrade infrastructure to Debian Bullseye, as Resolved.
Nov 26 2021, 5:52 PM · System administration (Component upgrades)
vsellier closed T3751: Upgrade swh-search elasticsearch servers to bullseye as Resolved.

All the production nodes (`search-esnode[4-6]') are upgraded to bullseye

Nov 26 2021, 5:52 PM · System administration (Component upgrades)
vsellier closed T3750: Upgrade ELK stack to bullseye, a subtask of T3579: Meta-task: upgrade infrastructure to Debian Bullseye, as Resolved.
Nov 26 2021, 1:58 PM · System administration (Component upgrades)
vsellier closed T3750: Upgrade ELK stack to bullseye as Resolved.

All the nodes are updated

Nov 26 2021, 1:58 PM · System administration (Component upgrades)
vsellier moved T3750: Upgrade ELK stack to bullseye from Backlog to In progress on the System administration (Component upgrades) board.
Nov 26 2021, 9:39 AM · System administration (Component upgrades)
vsellier edited projects for T3750: Upgrade ELK stack to bullseye, added: System administration (Component upgrades); removed System administration.
Nov 26 2021, 9:39 AM · System administration (Component upgrades)
vsellier moved T3751: Upgrade swh-search elasticsearch servers to bullseye from Backlog to In progress on the System administration (Component upgrades) board.
Nov 26 2021, 9:38 AM · System administration (Component upgrades)
vsellier added a comment to T3751: Upgrade swh-search elasticsearch servers to bullseye .
  • search-esnode0 updated without errors
Nov 26 2021, 9:38 AM · System administration (Component upgrades)
vsellier updated the task description for T3751: Upgrade swh-search elasticsearch servers to bullseye .
Nov 26 2021, 9:36 AM · System administration (Component upgrades)
vsellier changed the status of T3751: Upgrade swh-search elasticsearch servers to bullseye from Open to Work in Progress.
Nov 26 2021, 8:55 AM · System administration (Component upgrades)

Nov 25 2021

vsellier added a comment to T3750: Upgrade ELK stack to bullseye.

The upgrade by itself was made with the same command as explained previously.
The shard allocation was disabled during the process to avoid unnecessary movements of shard in the cluster

Nov 25 2021, 7:19 PM · System administration (Component upgrades)
vsellier added a comment to T3750: Upgrade ELK stack to bullseye.
  • The server was updated using the same procedure used for logstash0
  • there is no error detected when puppet is running
  • all the services are correctly started
Nov 25 2021, 3:52 PM · System administration (Component upgrades)
vsellier updated subscribers of T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Nov 25 2021, 2:59 PM · System administration (Component upgrades)
vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Nov 25 2021, 2:58 PM · System administration (Component upgrades)
vsellier added a comment to T3750: Upgrade ELK stack to bullseye.
  1. logstash0 upgrade
Nov 25 2021, 2:50 PM · System administration (Component upgrades)
vsellier closed D6689: logstash: remove unecessary dependency to the openjdk package.
Nov 25 2021, 2:23 PM
vsellier committed rSPSITE668a23c476c8: logstash: remove unecessary dependency to the openjdk package (authored by vsellier).
logstash: remove unecessary dependency to the openjdk package
Nov 25 2021, 2:23 PM
vsellier requested review of D6689: logstash: remove unecessary dependency to the openjdk package.
Nov 25 2021, 12:37 PM
vsellier added a revision to T3750: Upgrade ELK stack to bullseye: D6689: logstash: remove unecessary dependency to the openjdk package.
Nov 25 2021, 12:37 PM · System administration (Component upgrades)
vsellier updated subscribers of T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Nov 25 2021, 9:53 AM · System administration (Component upgrades)
vsellier changed the status of T3750: Upgrade ELK stack to bullseye from Open to Work in Progress.
Nov 25 2021, 9:53 AM · System administration (Component upgrades)

Nov 24 2021

vsellier closed T3741: swh-search - upgrade elasticsearch backend as Resolved.
Nov 24 2021, 6:11 PM · System administration (Component upgrades), Archive search
vsellier added a comment to T3741: swh-search - upgrade elasticsearch backend.

production nodes are upgraded :

  • stop the journal clients:
root@search1:~# systemctl stop swh-search-journal-client@indexed
root@search1:~# systemctl stop swh-search-journal-client@objects
  • flush the index to speedup the recovery
curl -XPOST http://search-esnode4:9200/_flush

For each node :

  • disable shard allocation:
cat > /tmp/shard_allocation.json <<EOF
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}
EOF
Nov 24 2021, 5:26 PM · System administration (Component upgrades), Archive search
vsellier closed D6682: swh-search: Upgrade elasticsearch to 7.15.2.
Nov 24 2021, 5:22 PM
vsellier committed rSPSITE3c6fdbb59608: swh-search: Upgrade elasticsearch to 7.15.2 (authored by vsellier).
swh-search: Upgrade elasticsearch to 7.15.2
Nov 24 2021, 5:22 PM
vsellier updated the diff for D6682: swh-search: Upgrade elasticsearch to 7.15.2.

rebase

Nov 24 2021, 5:20 PM
vsellier closed D6685: swh-search: upgrade elasticsearch to 7.15.2.
Nov 24 2021, 4:56 PM
vsellier committed rDENV839946f79dc1: swh-search: upgrade elasticsearch to 7.15.2 (authored by vsellier).
swh-search: upgrade elasticsearch to 7.15.2
Nov 24 2021, 4:56 PM
vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Nov 24 2021, 4:35 PM · System administration (Component upgrades)
vsellier updated the diff for D6685: swh-search: upgrade elasticsearch to 7.15.2.

fix a typo in the commit message

Nov 24 2021, 4:31 PM
vsellier requested review of D6685: swh-search: upgrade elasticsearch to 7.15.2.
Nov 24 2021, 4:24 PM