Page MenuHomeSoftware Heritage
Feed Advanced Search

Sep 28 2021

vsellier requested review of D6364: provenance: declare rabbitmq users.
Sep 28 2021, 2:36 PM
vsellier added a revision to T3487: Installation of the new provenance server: D6364: provenance: declare rabbitmq users.
Sep 28 2021, 2:36 PM · System administration
vsellier added a comment to T3487: Installation of the new provenance server.

The zfs pool and dataset are configured:

  • pool configuration
## nvme drives pool
#zpool create data mirror nvme-eui.36315030525005540025384500000003 nvme-eui.36315030525005800025384500000003 mirror nvme-eui.36315030525005620025384500000003 nvme-eui.36315030525005890025384500000003
Sep 28 2021, 11:45 AM · System administration
vsellier added a comment to T3487: Installation of the new provenance server.

I forgot to mention there is a gift from dell on the server: an additional 600Go 10rpm disk

Sep 28 2021, 9:45 AM · System administration
vsellier added a comment to T3487: Installation of the new provenance server.

The server is installed. It remains few task to perform manually:

  • configure the zfs datasets (will configure 2 mirror pool for ~12To available, tell me if it's not what it's expected)
  • build few missing packages for bullseye (relative to the monitoring: prometheus-rabbitmq-exporter, prometheus-statsd-exporter, journalbeat)
  • configure a rabbitmq admin user
Sep 28 2021, 9:38 AM · System administration

Sep 27 2021

vsellier committed rSPPRIVC44570ad137d7: Generate censored data from uncensored repository (authored by vsellier).
Generate censored data from uncensored repository
Sep 27 2021, 7:45 PM
vsellier committed rSPSITE5a7dc21e8403: Fix database reference name (authored by vsellier).
Fix database reference name
Sep 27 2021, 7:38 PM
vsellier closed D6359: Prepare the configuration of the provenance server.
Sep 27 2021, 7:35 PM
vsellier committed rSPSITEa35c3550f9a6: Prepare the configuration of the provenance server (authored by vsellier).
Prepare the configuration of the provenance server
Sep 27 2021, 7:35 PM
vsellier closed D6356: Upgrade the debian sid release name.
Sep 27 2021, 7:35 PM
vsellier committed rSPSITE27a4f043fb04: Upgrade the debian sid release name (authored by vsellier).
Upgrade the debian sid release name
Sep 27 2021, 7:35 PM
vsellier closed D6354: Improve vagrant initialization time.
Sep 27 2021, 7:32 PM
vsellier committed rSENVfce61dca6c18: add provenance server (authored by vsellier).
add provenance server
Sep 27 2021, 7:32 PM
vsellier committed rSENV5105d6c4ada7: Improve vagrant initialization time (authored by vsellier).
Improve vagrant initialization time
Sep 27 2021, 7:32 PM
vsellier requested review of D6359: Prepare the configuration of the provenance server.
Sep 27 2021, 5:02 PM
vsellier added a revision to T3487: Installation of the new provenance server: D6359: Prepare the configuration of the provenance server.
Sep 27 2021, 5:02 PM · System administration
vsellier added a comment to T3487: Installation of the new provenance server.

yes pgbouncer will be used and it's configured by default to 2000 // connections
I don't know the kind of load the provenance client will generate but the default 100 connections allowed by postgres will be probably too short and needed to be increased too

Sep 27 2021, 4:55 PM · System administration
vsellier added a comment to T3487: Installation of the new provenance server.

As see with @aeviso , we will install the following components on the server (the os will be debian11)

  • rabbitmq
  • postgresql:13
    • a default swh-storage database will be managed by puppet
    • 1000 parallel connections allowed
    • shared_buffers 50go
  • docker
Sep 27 2021, 4:08 PM · System administration
vsellier requested review of D6356: Upgrade the debian sid release name.
Sep 27 2021, 3:38 PM
vsellier added a revision to T3579: Meta-task: upgrade infrastructure to Debian Bullseye: D6356: Upgrade the debian sid release name.
Sep 27 2021, 3:38 PM · System administration (Component upgrades)
vsellier requested review of D6354: Improve vagrant initialization time.
Sep 27 2021, 3:01 PM
vsellier closed D6350: service urls: Fix the public url of the staging brocker.
Sep 27 2021, 11:25 AM
vsellier committed rDDOCef4409979126: service urls: Fix the public url of the staging brocker (authored by vsellier).
service urls: Fix the public url of the staging brocker
Sep 27 2021, 11:25 AM
vsellier requested review of D6350: service urls: Fix the public url of the staging brocker.
Sep 27 2021, 10:57 AM
vsellier added a revision to T3408: Provide read-only access to production servers: D6350: service urls: Fix the public url of the staging brocker.
Sep 27 2021, 10:57 AM · System administration

Sep 24 2021

vsellier accepted D6305: opam: Install and maintain up-to-date shared opam root directories.

It looks ok for the puppet code

Sep 24 2021, 4:15 PM

Sep 23 2021

vsellier added a comment to T3592: POC elastic worker infrastructure.

Interesting documentations on how to manage jobs:

Sep 23 2021, 10:32 AM · System administration

Sep 22 2021

vsellier closed D6308: Add a documentation page to list the services urls.
Sep 22 2021, 12:35 PM · System administration
vsellier committed rDDOC3a935af020e7: add a documentation page to list the services urls (authored by vsellier).
add a documentation page to list the services urls
Sep 22 2021, 12:35 PM
vsellier committed rDENV2507d723a1e6: POC a default smaller profile equivalent to the default docker-compose (authored by vsellier).
POC a default smaller profile equivalent to the default docker-compose
Sep 22 2021, 12:28 PM
vsellier committed rDENV46090bb0b458: Upgrade the registry-ui and fix a CORS issue blocking the access to the registry (authored by vsellier).
Upgrade the registry-ui and fix a CORS issue blocking the access to the registry
Sep 22 2021, 12:28 PM
vsellier updated the task description for T3592: POC elastic worker infrastructure.
Sep 22 2021, 12:03 PM · System administration
vsellier updated the diff for D6308: Add a documentation page to list the services urls.
  • Remove useless ending '/'
  • define VPN / private meanings
Sep 22 2021, 11:25 AM · System administration
vsellier added inline comments to D6308: Add a documentation page to list the services urls.
Sep 22 2021, 10:56 AM · System administration
vsellier added inline comments to D6308: Add a documentation page to list the services urls.
Sep 22 2021, 10:41 AM · System administration

Sep 21 2021

vsellier updated the diff for D6308: Add a documentation page to list the services urls.

RabbitMq GUI -> RabbitMQ

Sep 21 2021, 11:49 AM · System administration
vsellier committed rDENV6444f0e6c3ee: webapp: Add cron to periodically refresh counters (authored by ardumont).
webapp: Add cron to periodically refresh counters
Sep 21 2021, 9:09 AM
vsellier committed rDENV7f69420a18fb: dockerignore: Ignore swh-web/cypress/ generated folder (authored by ardumont).
dockerignore: Ignore swh-web/cypress/ generated folder
Sep 21 2021, 9:09 AM
vsellier committed rDENV6dfdcdc52bed: Revert "Add the counters-journal-client-messages deployment" (authored by vsellier).
Revert "Add the counters-journal-client-messages deployment"
Sep 21 2021, 9:09 AM
vsellier committed rDENVc4a1af25c893: Add the counters-journal-client-messages deployment (authored by vsellier).
Add the counters-journal-client-messages deployment
Sep 21 2021, 9:09 AM
vsellier committed rDENVa213976dec44: kubernetes/readme: Explicit how to connect to db in a new section (authored by ardumont).
kubernetes/readme: Explicit how to connect to db in a new section
Sep 21 2021, 9:09 AM
vsellier committed rDENV3cf1960b2212: webapp: Add cron to periodically refresh counters (authored by ardumont).
webapp: Add cron to periodically refresh counters
Sep 21 2021, 9:09 AM
vsellier committed rDENVad3c127b26d5: Remove unused dockerfile (authored by vsellier).
Remove unused dockerfile
Sep 21 2021, 9:09 AM
vsellier committed rDENV420b1da20106: kube/web-db: Merge clusterIP and nodePort into 1 to expose 32457 (authored by ardumont).
kube/web-db: Merge clusterIP and nodePort into 1 to expose 32457
Sep 21 2021, 9:09 AM
vsellier committed rDENVa90a05a2ecd9: kubernetes/web-db: Increase slightly cpu resource (authored by ardumont).
kubernetes/web-db: Increase slightly cpu resource
Sep 21 2021, 9:09 AM
vsellier committed rDENV77150cf5109c: improve documentation (authored by vsellier).
improve documentation
Sep 21 2021, 9:09 AM
vsellier committed rDENV42f5e36f0cc2: Add dedicated deposit workers (checker and loader) (authored by ardumont).
Add dedicated deposit workers (checker and loader)
Sep 21 2021, 9:09 AM
vsellier committed rDENVe12c760ef6f2: use swh-counters in the webapp (authored by vsellier).
use swh-counters in the webapp
Sep 21 2021, 9:09 AM
vsellier committed rDENV26c06d7ef54b: increase loader limits (authored by vsellier).
increase loader limits
Sep 21 2021, 9:09 AM
vsellier committed rDENV0cd7bccf6d06: Increase directory slicing level (authored by vsellier).
Increase directory slicing level
Sep 21 2021, 9:09 AM
vsellier committed rDENVce7a5f41d5c5: kube/loaders: Declare save code now queues to consume from (authored by ardumont).
kube/loaders: Declare save code now queues to consume from
Sep 21 2021, 9:09 AM
vsellier committed rDENV44cb0ae400d4: kube/Readme: Update registry-ui.default (authored by ardumont).
kube/Readme: Update registry-ui.default
Sep 21 2021, 9:09 AM
vsellier committed rDENV03d17f049c47: Activate metadata search via swh-search (authored by vsellier).
Activate metadata search via swh-search
Sep 21 2021, 9:09 AM
vsellier committed rDENV62832bc4b06d: Add deposit db service (authored by ardumont).
Add deposit db service
Sep 21 2021, 9:09 AM
vsellier committed rDENVf44538decc87: Add deposit service (authored by ardumont).
Add deposit service
Sep 21 2021, 9:09 AM
vsellier committed rDENVf28e53974dcc: fix elasticsearch mount point (authored by vsellier).
fix elasticsearch mount point
Sep 21 2021, 9:09 AM
vsellier committed rDENV3f940bdcd76e: webapp: Activate vault interaction (authored by ardumont).
webapp: Activate vault interaction
Sep 21 2021, 9:09 AM
vsellier committed rDENV6e309b671a9c: kafka: avoid oom on startup when data is growing (authored by vsellier).
kafka: avoid oom on startup when data is growing
Sep 21 2021, 9:09 AM
vsellier committed rDENV141be01f8ef2: Add vault cooker workers (authored by ardumont).
Add vault cooker workers
Sep 21 2021, 9:09 AM
vsellier committed rDENV40461360ca54: Add indexer journal client (authored by ardumont).
Add indexer journal client
Sep 21 2021, 9:09 AM
vsellier committed rDENV951e0682d6ed: Add vault service (authored by ardumont).
Add vault service
Sep 21 2021, 9:09 AM
vsellier committed rDENV44ce11185320: Add vault db service (authored by ardumont).
Add vault db service
Sep 21 2021, 9:09 AM
vsellier committed rDENV8fdaf8bf70da: Add indexer workers (authored by ardumont).
Add indexer workers
Sep 21 2021, 9:09 AM
vsellier committed rDENVfcd3ad88d87e: Add indexer storage rpc service (authored by ardumont).
Add indexer storage rpc service
Sep 21 2021, 9:09 AM
vsellier committed rDENVe490f03b489d: kubernetes/Readme: Simplify readme instructions (authored by ardumont).
kubernetes/Readme: Simplify readme instructions
Sep 21 2021, 9:09 AM
vsellier committed rDENV1c0611f993cc: Add indexer storage db service (authored by ardumont).
Add indexer storage db service
Sep 21 2021, 9:09 AM
vsellier committed rDENVf612ee1def6c: counters: use labels in prometheus queries (authored by vsellier).
counters: use labels in prometheus queries
Sep 21 2021, 9:09 AM
vsellier committed rDENV44a26dea1d34: speedup rebuild (authored by vsellier).
speedup rebuild
Sep 21 2021, 9:09 AM
vsellier committed rDENV6ce318b29e89: Parallelize builds (authored by vsellier).
Parallelize builds
Sep 21 2021, 9:09 AM
vsellier committed rDENVe1a052d784c6: webapp use counters to display the history count graph (authored by vsellier).
webapp use counters to display the history count graph
Sep 21 2021, 9:09 AM
vsellier committed rDENV12624ad7f657: use a local storage for the registry (authored by vsellier).
use a local storage for the registry
Sep 21 2021, 9:09 AM
vsellier committed rDENV15c067b51de7: reduce kafka startup time (authored by vsellier).
reduce kafka startup time
Sep 21 2021, 9:09 AM
vsellier committed rDENV3e3e66dcd1d4: Add a UI on top of the registry (authored by vsellier).
Add a UI on top of the registry
Sep 21 2021, 9:09 AM
vsellier committed rDENVe97a1302cf49: monitoring: use consistent names for exporter job (authored by vsellier).
monitoring: use consistent names for exporter job
Sep 21 2021, 9:09 AM
vsellier committed rDENV9eafb9338c0f: Decrease verbosity for scheduler and counter stack (authored by ardumont).
Decrease verbosity for scheduler and counter stack
Sep 21 2021, 9:09 AM
vsellier committed rDENVe10cb9006c39: loaders: use a dynamic hostname (authored by vsellier).
loaders: use a dynamic hostname
Sep 21 2021, 9:09 AM
vsellier committed rDENV38ccdc227604: try to perform a warm shutdown of the lister and loaders (authored by vsellier).
try to perform a warm shutdown of the lister and loaders
Sep 21 2021, 9:09 AM
vsellier committed rDENV541f8aa6ceb8: kubernetes/02-monitoring: Add scraping webapp metrics to prometheus (authored by ardumont).
kubernetes/02-monitoring: Add scraping webapp metrics to prometheus
Sep 21 2021, 9:09 AM
vsellier committed rDENVbbfeb6b78bf2: declare the counters stack (authored by vsellier).
declare the counters stack
Sep 21 2021, 9:09 AM
vsellier committed rDENVc7d32ae93656: add missing parts for counters (authored by vsellier).
add missing parts for counters
Sep 21 2021, 9:09 AM
vsellier committed rDENVc7c22087ab0d: Add missing dockerfile for the lister's image (authored by vsellier).
Add missing dockerfile for the lister's image
Sep 21 2021, 9:09 AM
vsellier committed rDENV9289a32b9d28: adapt limits (authored by vsellier).
adapt limits
Sep 21 2021, 9:09 AM
vsellier committed rDENV0497a7754261: add scheduler journal client (authored by vsellier).
add scheduler journal client
Sep 21 2021, 9:09 AM
vsellier committed rDENVd15417f3cf3b: Increase storage limit to avoid OOM (authored by vsellier).
Increase storage limit to avoid OOM
Sep 21 2021, 9:09 AM
vsellier committed rDENV252bc86b6256: reorder skaffold configuration (authored by vsellier).
reorder skaffold configuration
Sep 21 2021, 9:09 AM
vsellier committed rDENV8865be7b2a77: Add search journal client (authored by vsellier).
Add search journal client
Sep 21 2021, 9:09 AM
vsellier committed rDENVc10123504517: add missing point on the documentation (authored by vsellier).
add missing point on the documentation
Sep 21 2021, 9:09 AM
vsellier committed rDENV320915468418: Adjust limits (authored by vsellier).
Adjust limits
Sep 21 2021, 9:09 AM
vsellier committed rDENV4ec0d59d4125: configure the webapp to use postgresql (authored by vsellier).
configure the webapp to use postgresql
Sep 21 2021, 9:09 AM
vsellier committed rDENV40703c8303df: add listers (authored by vsellier).
add listers
Sep 21 2021, 9:09 AM
vsellier committed rDENV62e6e66ca3da: Improve elasticsearch startup (authored by vsellier).
Improve elasticsearch startup
Sep 21 2021, 9:09 AM
vsellier committed rDENV6c35cec47f6e: rabbitmq: Expose the rabbitmq manager port (authored by ardumont).
rabbitmq: Expose the rabbitmq manager port
Sep 21 2021, 9:09 AM
vsellier committed rDENV68657a3bddc6: Drop scheduler-runner verbosity to INFO (authored by ardumont).
Drop scheduler-runner verbosity to INFO
Sep 21 2021, 9:09 AM
vsellier committed rDENVdb0c7db1fe7b: Add swh-scheduler-listener service (authored by ardumont).
Add swh-scheduler-listener service
Sep 21 2021, 9:09 AM
vsellier committed rDENV6cfc56ef75df: Add swh search service (authored by ardumont).
Add swh search service
Sep 21 2021, 9:09 AM
vsellier committed rDENV12fd07f79e6e: Add scheduler-runner service (authored by ardumont).
Add scheduler-runner service
Sep 21 2021, 9:09 AM
vsellier committed rDENV6ad6b5aa769c: Add elasticsearch service (authored by ardumont).
Add elasticsearch service
Sep 21 2021, 9:09 AM
vsellier committed rDENV084cab747d3c: Add prometheus and grafana (authored by vsellier).
Add prometheus and grafana
Sep 21 2021, 9:09 AM
vsellier committed rDENVea436d1facad: fix postgresql persistence (authored by vsellier).
fix postgresql persistence
Sep 21 2021, 9:09 AM
vsellier committed rDENVb8a3b1cd88c4: Move rabbitmq definition prior to the scheduler (authored by ardumont).
Move rabbitmq definition prior to the scheduler
Sep 21 2021, 9:09 AM