Page MenuHomeSoftware Heritage
Feed Advanced Search

Sep 21 2021

vsellier committed rDENV65c9df141733: allow kafka to start without being killed by an oom (authored by vsellier).
allow kafka to start without being killed by an oom
Sep 21 2021, 9:09 AM
vsellier committed rDENV74fdf9dd861c: webapp: configure rolling upgrade (authored by vsellier).
webapp: configure rolling upgrade
Sep 21 2021, 9:09 AM
vsellier committed rDENV84e11cfbb781: persist zookeeper data to fix kafka restart (authored by vsellier).
persist zookeeper data to fix kafka restart
Sep 21 2021, 9:09 AM
vsellier committed rDENV8de8aac8a0c3: add directory initialisation task (authored by vsellier).
add directory initialisation task
Sep 21 2021, 9:09 AM
vsellier committed rDENV3964067f868a: webapp: correctly install source via pip (authored by vsellier).
webapp: correctly install source via pip
Sep 21 2021, 9:09 AM
vsellier committed rDENV5f858c17a96b: Make webapp use an entrypoint and update to use postgres backend (authored by ardumont).
Make webapp use an entrypoint and update to use postgres backend
Sep 21 2021, 9:09 AM
vsellier committed rDENVc179807029bd: objstorage: correctly install source via pip (authored by vsellier).
objstorage: correctly install source via pip
Sep 21 2021, 9:09 AM
vsellier committed rDENV0b4f636f95b9: loaders: Decrease loaders verbosity (authored by ardumont).
loaders: Decrease loaders verbosity
Sep 21 2021, 9:09 AM
vsellier committed rDENVfd862386cd7a: Add loaders (#2) (authored by ardumont).
Add loaders (#2)
Sep 21 2021, 9:09 AM
vsellier committed rDENV352bdba91166: Dockerfile: Clean up whitespaces (authored by ardumont).
Dockerfile: Clean up whitespaces
Sep 21 2021, 9:09 AM
vsellier committed rDENV583216cd3a81: add service declaration on the /etc/hosts (authored by vsellier).
add service declaration on the /etc/hosts
Sep 21 2021, 9:09 AM
vsellier committed rDENVf4a9466adf23: declare the webapp (authored by vsellier).
declare the webapp
Sep 21 2021, 9:09 AM
vsellier committed rDENVcad7404c392c: cleanup base image (authored by vsellier).
cleanup base image
Sep 21 2021, 9:09 AM
vsellier committed rDENV11eda1b7d58b: deploy kafka (authored by vsellier).
deploy kafka
Sep 21 2021, 9:09 AM
vsellier committed rDENVd5ec7c0da9ca: add util scripts on the base image (authored by vsellier).
add util scripts on the base image
Sep 21 2021, 9:09 AM
vsellier committed rDENV59e4bb336101: README: Fix and unify host mount points creation (authored by ardumont).
README: Fix and unify host mount points creation
Sep 21 2021, 9:09 AM
vsellier committed rDENV91faced8d3e3: Add rabbitmq service (authored by ardumont).
Add rabbitmq service
Sep 21 2021, 9:09 AM
vsellier committed rDENVd133a93d07ff: Add scheduler service (authored by ardumont).
Add scheduler service
Sep 21 2021, 9:09 AM
vsellier committed rDENVae03f9b30ff4: Readme: Update reference names (authored by ardumont).
Readme: Update reference names
Sep 21 2021, 9:09 AM
vsellier committed rDENVf1211ab704b6: Reference how to build the objstorage (authored by ardumont).
Reference how to build the objstorage
Sep 21 2021, 9:09 AM
vsellier committed rDENV34e281df7b0e: Cleanup whitespace noises (authored by ardumont).
Cleanup whitespace noises
Sep 21 2021, 9:09 AM
vsellier committed rDENV2bd2c3b5e02f: restore missing python dependencies (authored by vsellier).
restore missing python dependencies
Sep 21 2021, 9:09 AM
vsellier committed rDENV4e4b70df59d1: docker: stop ignoring .git directory to allow pip to perform local install (authored by vsellier).
docker: stop ignoring .git directory to allow pip to perform local install
Sep 21 2021, 9:09 AM
vsellier committed rDENV3d03bfe2ca68: Specify the righ patch creation on the doc (authored by vsellier).
Specify the righ patch creation on the doc
Sep 21 2021, 9:09 AM
vsellier committed rDENVf8fb72262d8f: fix zookeeper configuration (authored by vsellier).
fix zookeeper configuration
Sep 21 2021, 9:09 AM
vsellier committed rDENV86af89854652: add limit on containers (authored by vsellier).
add limit on containers
Sep 21 2021, 9:09 AM
vsellier committed rDENV08507040f8b4: storage: Initialize database schema on startup (authored by vsellier).
storage: Initialize database schema on startup
Sep 21 2021, 9:09 AM
vsellier committed rDENV92d27602b706: fix database startup (authored by vsellier).
fix database startup
Sep 21 2021, 9:09 AM
vsellier committed rDENVb36a68c8910d: make modules discoverable for swh command line (authored by vsellier).
make modules discoverable for swh command line
Sep 21 2021, 9:09 AM
vsellier committed rDENVe2907c2429bb: add skaffold section on the readme (authored by vsellier).
add skaffold section on the readme
Sep 21 2021, 9:09 AM
vsellier committed rDENVe58734e5e7e6: wip try journal (authored by vsellier).
wip try journal
Sep 21 2021, 9:09 AM
vsellier committed rDENV28adc161dc7b: split image to efficiently manage dependencies (authored by vsellier).
split image to efficiently manage dependencies
Sep 21 2021, 9:09 AM
vsellier committed rDENVfabe52ceccbb: wip - kubernetes and skaffold initialization (authored by vsellier).
wip - kubernetes and skaffold initialization
Sep 21 2021, 9:09 AM
vsellier triaged T3592: POC elastic worker infrastructure as Normal priority.
Sep 21 2021, 8:29 AM · System administration
vsellier updated the summary of D6308: Add a documentation page to list the services urls.
Sep 21 2021, 7:53 AM · System administration
vsellier retitled D6308: Add a documentation page to list the services urls from WIP - add a documentation page to list the services urls to Add a documentation page to list the services urls.
Sep 21 2021, 7:52 AM · System administration

Sep 20 2021

vsellier updated the diff for D6308: Add a documentation page to list the services urls.

complete other tools section

Sep 20 2021, 9:45 PM · System administration
vsellier updated the diff for D6308: Add a documentation page to list the services urls.
  • fix a couple of staging urls
  • add the production urls
Sep 20 2021, 7:42 PM · System administration
vsellier added a comment to D6308: Add a documentation page to list the services urls.

Do you reference only the staging part?

Sep 20 2021, 6:23 PM · System administration
vsellier updated the diff for D6308: Add a documentation page to list the services urls.

fix typos

Sep 20 2021, 6:20 PM · System administration
vsellier planned changes to D6308: Add a documentation page to list the services urls.
Sep 20 2021, 3:57 PM · System administration
vsellier requested review of D6308: Add a documentation page to list the services urls.
Sep 20 2021, 3:57 PM · System administration
vsellier added a revision to T3408: Provide read-only access to production servers: D6308: Add a documentation page to list the services urls.
Sep 20 2021, 3:57 PM · System administration
vsellier closed D6304: swh-counters: use the right docker override file name.
Sep 20 2021, 1:52 PM
vsellier committed rDENVc232b0c6e10e: swh-counters: use the right docker override file name (authored by vsellier).
swh-counters: use the right docker override file name
Sep 20 2021, 1:52 PM
vsellier closed D6303: swh-web: fix the metadata backend configuration in the swh-search override.
Sep 20 2021, 1:52 PM
vsellier committed rDENVa0939bff7344: swh-web: fix the metadata backend configuration in the swh-search override (authored by vsellier).
swh-web: fix the metadata backend configuration in the swh-search override
Sep 20 2021, 1:52 PM
vsellier requested review of D6304: swh-counters: use the right docker override file name.
Sep 20 2021, 1:49 PM
vsellier requested review of D6303: swh-web: fix the metadata backend configuration in the swh-search override.
Sep 20 2021, 1:42 PM
vsellier added a revision to T3433: Deploy swh.search v0.10/v0.11: D6303: swh-web: fix the metadata backend configuration in the swh-search override.
Sep 20 2021, 1:42 PM · System administration, Archive search

Sep 17 2021

vsellier closed D6294: mirror: Add the extid topic in the list of topics to synchronize.
Sep 17 2021, 4:51 PM
vsellier committed rDDOCe893f6eda11b: mirror: Add the extid topic in the list of topics to synchronize (authored by vsellier).
mirror: Add the extid topic in the list of topics to synchronize
Sep 17 2021, 4:51 PM
vsellier requested review of D6294: mirror: Add the extid topic in the list of topics to synchronize.
Sep 17 2021, 4:31 PM
vsellier committed rDSNIP41c771d59b0c: grid5000/cassandra: add statsd metrics dashboards (authored by vsellier).
grid5000/cassandra: add statsd metrics dashboards
Sep 17 2021, 1:58 PM
vsellier committed rDSNIP7ec3d29c5c19: grid5000/cassandra: allow to test scheduler db persistance locally (authored by vsellier).
grid5000/cassandra: allow to test scheduler db persistance locally
Sep 17 2021, 1:58 PM
vsellier committed rDSNIPd4b4d44f1e61: grid5000/cassandra: allow to instantiate a best_effort node without an… (authored by vsellier).
grid5000/cassandra: allow to instantiate a best_effort node without an…
Sep 17 2021, 1:58 PM

Sep 16 2021

vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Sep 16 2021, 6:09 PM · System administration (Component upgrades)
vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Sep 16 2021, 6:07 PM · System administration (Component upgrades)
vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Sep 16 2021, 6:06 PM · System administration (Component upgrades)
vsellier updated subscribers of T3579: Meta-task: upgrade infrastructure to Debian Bullseye.

the content of this comment was moved to the task description

Sep 16 2021, 6:00 PM · System administration (Component upgrades)
vsellier closed T3408: Provide read-only access to production servers, a subtask of T1526: Install a new VPN endpoint at Rocquencourt, as Resolved.
Sep 16 2021, 2:40 PM · System administration
vsellier closed T3408: Provide read-only access to production servers as Resolved.
Sep 16 2021, 2:40 PM · System administration
vsellier closed T3493: [cassandra] Git loader performance are very bad as Resolved.

changing the status to resolved as the main issues are solved.
Other tests with more parallel workers will be launched, if other problems will be detected, they will be tracked in new dedicated tickets.

Sep 16 2021, 11:31 AM · System administration, Storage manager
vsellier closed T3493: [cassandra] Git loader performance are very bad, a subtask of T3357: Perform some tests of the cassandra storage on Grid5000, as Resolved.
Sep 16 2021, 11:31 AM · System administration, Storage manager
vsellier added a comment to T3579: Meta-task: upgrade infrastructure to Debian Bullseye.

WIP document: https://hedgedoc.softwareheritage.org/GYxIMBMXSRGVNxcCBrfA3w#

Sep 16 2021, 9:53 AM · System administration (Component upgrades)
vsellier updated the task description for T3579: Meta-task: upgrade infrastructure to Debian Bullseye.
Sep 16 2021, 9:44 AM · System administration (Component upgrades)
vsellier triaged T3579: Meta-task: upgrade infrastructure to Debian Bullseye as Normal priority.
Sep 16 2021, 9:43 AM · System administration (Component upgrades)
vsellier accepted D6276: keyrings/cassandra: Update.

LGTM

Sep 16 2021, 9:01 AM

Sep 15 2021

vsellier committed rDSNIP118ba1a0f735: grid5000/cassandra: declare a server type with only a big zfs dataset (authored by vsellier).
grid5000/cassandra: declare a server type with only a big zfs dataset
Sep 15 2021, 6:19 PM
vsellier updated the task description for T3577: Parallel loaders performances .
Sep 15 2021, 5:49 PM · System administration, Storage manager
vsellier changed the status of T3577: Parallel loaders performances from Open to Work in Progress.
Sep 15 2021, 5:49 PM · System administration, Storage manager
vsellier updated the task description for T3357: Perform some tests of the cassandra storage on Grid5000.
Sep 15 2021, 5:42 PM · System administration, Storage manager
vsellier added a comment to T3573: [cassandra] directory and content read benchmarks.

Test of a the new D6269 patch:

Sep 15 2021, 5:12 PM · System administration, Storage manager
vsellier added a comment to T3573: [cassandra] directory and content read benchmarks.

2 flame graphs of the previous directory_ls:

  • one-by-one

first run (cache cold):

Sep 15 2021, 2:55 PM · System administration, Storage manager
vsellier added a comment to T3573: [cassandra] directory and content read benchmarks.

This is the results of the different runs:

Sep 15 2021, 2:40 PM · System administration, Storage manager
vsellier created P1164 flush cassandra buffers.
Sep 15 2021, 12:04 PM
vsellier added a comment to T3573: [cassandra] directory and content read benchmarks.

The directory_ls and indirectly the get_content performace was tested with this small script: P1163
A cold restart (all buffer cleared, cassandra restarted) is done between each tests (P1164)

Sep 15 2021, 11:20 AM · System administration, Storage manager
vsellier renamed T3573: [cassandra] directory and content read benchmarks from [cassandra] directory and content read benchmarkss to [cassandra] directory and content read benchmarks.
Sep 15 2021, 11:19 AM · System administration, Storage manager
vsellier created P1163 directory_ls.py.
Sep 15 2021, 11:18 AM
vsellier changed the status of T3573: [cassandra] directory and content read benchmarks from Open to Work in Progress.
Sep 15 2021, 11:11 AM · System administration, Storage manager
vsellier closed T3476: One of the system disks of beaubourg is out of order, a subtask of T3444: 26/07/2021: Unstuck infrastructure outage then post-mortem, as Resolved.
Sep 15 2021, 8:32 AM · System administration
vsellier closed T3476: One of the system disks of beaubourg is out of order as Resolved.

The disk was received Monday and replaced Thuesday by Christophe from the DSI.
The raid card automatically launch the raid rebuild. Everything is ok now.

root@beaubourg:~#  megacli -PDList -aALL
...
Sep 15 2021, 8:32 AM · System administration

Sep 13 2021

vsellier closed T3465: Test multidatacenter replication, a subtask of T3357: Perform some tests of the cassandra storage on Grid5000, as Resolved.
Sep 13 2021, 1:48 PM · System administration, Storage manager
vsellier closed T3465: Test multidatacenter replication as Resolved.
Sep 13 2021, 1:48 PM · System administration, Storage manager
vsellier added a comment to T3465: Test multidatacenter replication.

The new datacenter is active since a couple of week.
It allowed to test:

  • how to declare a new dc and bootstrap it
  • how the data is replicated between the DC
  • how to perform inter/intra DC repairs
  • how to add nodes on a DC on bootstrap it
  • how to remove a datacenter
Sep 13 2021, 1:48 PM · System administration, Storage manager
vsellier closed T3464: Prepare a quote for the cassandra servers, a subtask of T3357: Perform some tests of the cassandra storage on Grid5000, as Resolved.
Sep 13 2021, 1:24 PM · System administration, Storage manager
vsellier closed T3464: Prepare a quote for the cassandra servers as Resolved.

The quote is done and validated, we will launch the command when the new matinfo deal will be available

Sep 13 2021, 1:24 PM · System administration, Storage manager

Sep 10 2021

vsellier added a comment to T3476: One of the system disks of beaubourg is out of order.

A replacement disk will be sent by DELL. It should be delivered the 2021-09-13 if everything is ok.
The DSI is notified of the delivery

Sep 10 2021, 3:23 PM · System administration
vsellier committed rSPRE45e67ae40690: Add a vm template for debian 11.0 (authored by vsellier).
Add a vm template for debian 11.0
Sep 10 2021, 10:20 AM
vsellier closed D6237: Add a template for debian 10.10.
Sep 10 2021, 10:20 AM
vsellier committed rSPRE41e5dd01a609: Add a template for debian 10.10 (authored by vsellier).
Add a template for debian 10.10
Sep 10 2021, 10:20 AM
vsellier closed D6227: Adapt the debian security repository release for bullseye distribution.
Sep 10 2021, 10:16 AM
vsellier committed rSPSITE0a3411f454e6: Adapt the debian security repository release for bullseye distribution (authored by vsellier).
Adapt the debian security repository release for bullseye distribution
Sep 10 2021, 10:16 AM
vsellier updated the diff for D6227: Adapt the debian security repository release for bullseye distribution.

rebase

Sep 10 2021, 10:15 AM
vsellier closed D6226: Prepare the debian 11 vagrant template.
Sep 10 2021, 10:14 AM
vsellier committed rSENV874a41909a12: Prepare the debian 11 vagrant template (authored by vsellier).
Prepare the debian 11 vagrant template
Sep 10 2021, 10:14 AM
vsellier updated the diff for D6226: Prepare the debian 11 vagrant template.

rebase

Sep 10 2021, 10:14 AM
vsellier updated the diff for D6226: Prepare the debian 11 vagrant template.
  • fix the wrong box url
  • Explain the move to nfs v4
  • Explain the puppet directories hack
Sep 10 2021, 10:09 AM
vsellier added inline comments to D6226: Prepare the debian 11 vagrant template.
Sep 10 2021, 9:52 AM
vsellier requested review of D6237: Add a template for debian 10.10.
Sep 10 2021, 8:43 AM