Page MenuHomeSoftware Heritage
Feed Advanced Search

Apr 20 2021

vsellier committed rDENVe788685ffd03: allow kafka to start without being killed by an oom (authored by vsellier).
allow kafka to start without being killed by an oom
Apr 20 2021, 4:20 PM
vsellier committed rDENV8b26181dd29e: Move rabbitmq definition prior to the scheduler (authored by ardumont).
Move rabbitmq definition prior to the scheduler
Apr 20 2021, 4:20 PM
vsellier committed rDENVd2078d9e1f58: Add prometheus and grafana (authored by vsellier).
Add prometheus and grafana
Apr 20 2021, 4:20 PM
vsellier committed rDENVe7423de91e1f: add directory initialisation task (authored by vsellier).
add directory initialisation task
Apr 20 2021, 4:20 PM
vsellier committed rDENVfad6ffd1f12a: persist zookeeper data to fix kafka restart (authored by vsellier).
persist zookeeper data to fix kafka restart
Apr 20 2021, 4:20 PM
vsellier committed rDENV02c68c6193f3: webapp: configure rolling upgrade (authored by vsellier).
webapp: configure rolling upgrade
Apr 20 2021, 4:20 PM
vsellier committed rDENV1ab6aeedca06: webapp: correctly install source via pip (authored by vsellier).
webapp: correctly install source via pip
Apr 20 2021, 4:20 PM
vsellier committed rDENV3496dfe7d6f0: Make webapp use an entrypoint and update to use postgres backend (authored by ardumont).
Make webapp use an entrypoint and update to use postgres backend
Apr 20 2021, 4:20 PM
vsellier committed rDENVdbe2a7d8a20d: objstorage: correctly install source via pip (authored by vsellier).
objstorage: correctly install source via pip
Apr 20 2021, 4:20 PM
vsellier committed rDENV56591249a5eb: Dockerfile: Clean up whitespaces (authored by ardumont).
Dockerfile: Clean up whitespaces
Apr 20 2021, 4:20 PM
vsellier committed rDENV5f3dfa662f54: loaders: Decrease loaders verbosity (authored by ardumont).
loaders: Decrease loaders verbosity
Apr 20 2021, 4:20 PM
vsellier committed rDENV8564f0d2c001: Add loaders (#2) (authored by ardumont).
Add loaders (#2)
Apr 20 2021, 4:20 PM
vsellier committed rDENVb09640e4fa5a: add service declaration on the /etc/hosts (authored by vsellier).
add service declaration on the /etc/hosts
Apr 20 2021, 4:20 PM
vsellier committed rDENVb1d314ab971c: cleanup base image (authored by vsellier).
cleanup base image
Apr 20 2021, 4:20 PM
vsellier committed rDENV265e3e412909: declare the webapp (authored by vsellier).
declare the webapp
Apr 20 2021, 4:20 PM
vsellier committed rDENV885dabee1d78: deploy kafka (authored by vsellier).
deploy kafka
Apr 20 2021, 4:20 PM
vsellier committed rDENV844c4620cb50: add util scripts on the base image (authored by vsellier).
add util scripts on the base image
Apr 20 2021, 4:20 PM
vsellier committed rDENV35d3074b2fb1: Add rabbitmq service (authored by ardumont).
Add rabbitmq service
Apr 20 2021, 4:20 PM
vsellier committed rDENVa7b622631244: README: Fix and unify host mount points creation (authored by ardumont).
README: Fix and unify host mount points creation
Apr 20 2021, 4:20 PM
vsellier committed rDENV6c1f8354a594: Reference how to build the objstorage (authored by ardumont).
Reference how to build the objstorage
Apr 20 2021, 4:20 PM
vsellier committed rDENV899449c9c99e: Add scheduler service (authored by ardumont).
Add scheduler service
Apr 20 2021, 4:20 PM
vsellier committed rDENV37bf87db47b2: Readme: Update reference names (authored by ardumont).
Readme: Update reference names
Apr 20 2021, 4:20 PM
vsellier committed rDENVa2974219dea5: Cleanup whitespace noises (authored by ardumont).
Cleanup whitespace noises
Apr 20 2021, 4:20 PM
vsellier committed rDENVb6c95efb90be: restore missing python dependencies (authored by vsellier).
restore missing python dependencies
Apr 20 2021, 4:20 PM
vsellier committed rDENV957b5399b9f1: Specify the righ patch creation on the doc (authored by vsellier).
Specify the righ patch creation on the doc
Apr 20 2021, 4:20 PM
vsellier committed rDENV3de0baef88d1: docker: stop ignoring .git directory to allow pip to perform local install (authored by vsellier).
docker: stop ignoring .git directory to allow pip to perform local install
Apr 20 2021, 4:20 PM
vsellier committed rDENVd917810e87b1: fix zookeeper configuration (authored by vsellier).
fix zookeeper configuration
Apr 20 2021, 4:20 PM
vsellier committed rDENV454aa1ef5eb0: add limit on containers (authored by vsellier).
add limit on containers
Apr 20 2021, 4:20 PM
vsellier committed rDENV6cf5f524ed23: storage: Initialize database schema on startup (authored by vsellier).
storage: Initialize database schema on startup
Apr 20 2021, 4:20 PM
vsellier committed rDENV51377bcdf873: fix database startup (authored by vsellier).
fix database startup
Apr 20 2021, 4:20 PM
vsellier committed rDENVf99fb0d09a45: make modules discoverable for swh command line (authored by vsellier).
make modules discoverable for swh command line
Apr 20 2021, 4:20 PM
vsellier committed rDENV5075e104e233: add skaffold section on the readme (authored by vsellier).
add skaffold section on the readme
Apr 20 2021, 4:19 PM
vsellier committed rDENV0cfe5be92631: split image to efficiently manage dependencies (authored by vsellier).
split image to efficiently manage dependencies
Apr 20 2021, 4:19 PM
vsellier committed rDENV5c8b476ed513: wip try journal (authored by vsellier).
wip try journal
Apr 20 2021, 4:19 PM
vsellier committed rDENV71bab54065af: wip - kubernetes and skaffold initialization (authored by vsellier).
wip - kubernetes and skaffold initialization
Apr 20 2021, 4:19 PM
vsellier added a comment to T3041: [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata.

Apparently the order was lost somewhere after it was sent to dell the 6th april 馃
It was reissued yesterday...

Apr 20 2021, 10:37 AMSystem administration, Archive search

Apr 19 2021

vsellier added a revision to T3242: Decommission ClearlyDefined resources: D5557: Remove clearly-defined resources.
Apr 19 2021, 7:02 PMSystem administration
vsellier requested review of D5557: Remove clearly-defined resources.
Apr 19 2021, 7:02 PM
vsellier committed rSPSITEe749fd9a244c: Remove tenma's access (authored by vsellier).
Remove tenma's access
Apr 19 2021, 6:38 PM
vsellier added a comment to T3242: Decommission ClearlyDefined resources.

The clearly-defined dump is backuped on /srv/storage/space/mirrors/clearly-defined/clearcode-dump-2021-01-05/clearcode_backup-20210105-2111.dump

Apr 19 2021, 6:35 PMSystem administration
vsellier changed the status of T3251: Count authors from revisions and releases, a subtask of T2912: Next generation archive counters, from Open to Work in Progress.
Apr 19 2021, 3:52 PMRoadmap 2021, System administration, Monitoring, Web app
vsellier changed the status of T3251: Count authors from revisions and releases from Open to Work in Progress.
Apr 19 2021, 3:52 PMMonitoring, Web app
vsellier closed T2501: Proxmox reliability improvements (Summer 2020) as Resolved.

changing the status to resolved as all the subtasks are done. feel free to reopen if it shouldn't.

Apr 19 2021, 3:44 PMSystem administration

Apr 16 2021

vsellier accepted D5549: Deploy check save code now end to end check.

just one question inline regarding the icinga test name.
Otherwise, looks good

Apr 16 2021, 6:00 PM
vsellier accepted D5541: Add end to end save code now check.

lgtm

Apr 16 2021, 3:52 PM
vsellier closed T3254: Outboarding of tenma as Resolved.
Apr 16 2021, 2:23 PMSystem administration
vsellier added a comment to T3254: Outboarding of tenma.
  • he was not in the list of members of the swh-team (thanks @ardumont for adding me to the owners of the list)
Apr 16 2021, 2:23 PMSystem administration
vsellier updated subscribers of T3254: Outboarding of tenma.
  • unix credentials disabled
  • user tenma removed from the groups Staff and Reviewers in phabricator
  • VPN credential revoked:
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ls
dh1024.pem  dh2048.pem	easyrsa  old  pki  vars  x509-types
root@louvre:/etc/openvpn/keys# ./easyrsa revoke tenma
Apr 16 2021, 11:51 AMSystem administration
vsellier placed T2743: Investigate why logging is inconsistent up for grabs.
Apr 16 2021, 11:48 AMVault, Object storage, Core & foundations
vsellier placed T1410: Kill implicit configuration: new configuration scheme up for grabs.
Apr 16 2021, 11:48 AMCore & foundations
vsellier placed T2885: investigate the persistent seemingly benign errors on cooking progress up for grabs.
Apr 16 2021, 11:48 AMVault
vsellier removed a member for Staff: Tanya.
Apr 16 2021, 11:44 AM
vsellier removed a member for Reviewers: tenma.
Apr 16 2021, 11:43 AM
vsellier removed a member for Staff: tenma.
Apr 16 2021, 11:41 AM
vsellier committed rSPPRIVC8abf1d0534b8: refresh compared to uncensored data (authored by vsellier).
refresh compared to uncensored data
Apr 16 2021, 11:39 AM
vsellier added a revision to T3254: Outboarding of tenma: D5542: Remove tenma's access.
Apr 16 2021, 11:31 AMSystem administration
vsellier requested review of D5542: Remove tenma's access.
Apr 16 2021, 11:31 AM
vsellier changed the status of T3254: Outboarding of tenma from Open to Work in Progress.
Apr 16 2021, 10:42 AMSystem administration
vsellier moved T3203: docs: Document the firewall installation and procedures from Backlog to Weekly backlog on the System administration board.
Apr 16 2021, 10:13 AMDocumentation, System administration
vsellier moved T3242: Decommission ClearlyDefined resources from Backlog to in-progress on the System administration board.
Apr 16 2021, 10:13 AMSystem administration
vsellier moved T3243: Replace /dev/sdb and /dev/sdc on storage1.staging from Backlog to in-progress on the System administration board.
Apr 16 2021, 10:12 AMSystem administration, Staging environment
vsellier accepted D5541: Add end to end save code now check.

I'm not a big fan of the random pickup of repo because it adds entropy on the result, but otherwise it's great to be able to monitor this 馃憤

Apr 16 2021, 9:35 AM

Apr 15 2021

vsellier closed T2939: Replace out of order disks on db1.staging and storage1.staging as Resolved.

closing this issue as the quota of disks replacement will be over after T3243.
The disk on db1 looks stable for the moment. It will be removed from the zfs pool in case of problems.

Apr 15 2021, 6:14 PMSystem administration
vsellier closed D5539: Outboarding of tg1999.
Apr 15 2021, 5:57 PM
vsellier committed rSPSITE73f28d682007: Outboarding of tg1999 (authored by vsellier).
Outboarding of tg1999
Apr 15 2021, 5:57 PM
vsellier added a comment to T3242: Decommission ClearlyDefined resources.

The compression and the transfer of the dump from clearly-defined server to saam is in progress.

ssh clearly-defined.internal.staging.swh.network "pbzip2 -c -9 /srv/softwareheritage/clearlydefined/clearcode-dump-2021-01-05/clearcode_backup-20210105-2111.dump" | pv | cat > clearcode_backup-20210105-2111.dump.bz2

When it will be done, and if the compression ratio is good, the compressed archive will be added to the public annex in a dataset/clearly-defined directory

Apr 15 2021, 5:49 PMSystem administration
vsellier updated the task description for T3242: Decommission ClearlyDefined resources.
Apr 15 2021, 5:43 PMSystem administration
vsellier added a comment to T3242: Decommission ClearlyDefined resources.

According to https://intranet.softwareheritage.org/wiki/Outboarding:

  • tg19999 unix account disabled

*TG1999 removed from interns members in the forge, no tasks were assigned to the user

  • He was not on the #swh-team channel
  • He was not a member of the swh-team ml
  • VPN certificated revoked
root@louvre:/etc/openvpn/keys# ./easyrsa revoke tg1999
Apr 15 2021, 5:42 PMSystem administration
vsellier removed a member for Interns: TG1999.
Apr 15 2021, 5:32 PM
vsellier added a revision to T3242: Decommission ClearlyDefined resources: D5539: Outboarding of tg1999.
Apr 15 2021, 5:26 PMSystem administration
vsellier requested review of D5539: Outboarding of tg1999.
Apr 15 2021, 5:26 PM
vsellier changed the status of T3242: Decommission ClearlyDefined resources from Open to Work in Progress.

thanks, I will move the dump in a secure place where it will be backuped.

Apr 15 2021, 3:05 PMSystem administration
vsellier added a comment to T3243: Replace /dev/sdb and /dev/sdc on storage1.staging.

Email sent to the dsi to launch the replacement.

Apr 15 2021, 3:03 PMSystem administration, Staging environment
vsellier added a comment to T3243: Replace /dev/sdb and /dev/sdc on storage1.staging.

In preparation of the disk replacement, their leds must be activated to make the emplacement identifiable:

  • Ensure all the led are off
root@storage1:~# ls /dev/sd* | grep -e "[a-z]$" | xargs -n1 -t -i{} ledctl normal={} 
ledctl normal=/dev/sda 
ledctl normal=/dev/sdb 
ledctl normal=/dev/sdc 
ledctl normal=/dev/sdd 
ledctl normal=/dev/sde 
ledctl normal=/dev/sdf 
ledctl normal=/dev/sdg 
ledctl normal=/dev/sdh 
ledctl normal=/dev/sdi 
ledctl normal=/dev/sdj 
ledctl normal=/dev/sdk 
ledctl normal=/dev/sdl 
ledctl normal=/dev/sdm 
ledctl normal=/dev/sdn
  • light on
root@storage1:~# ledctl locate=/dev/sdb
root@storage1:~# ledctl locate=/dev/sdc
Apr 15 2021, 2:31 PMSystem administration, Staging environment
vsellier triaged T3251: Count authors from revisions and releases as Normal priority.
Apr 15 2021, 1:13 PMMonitoring, Web app
vsellier added a comment to T2912: Next generation archive counters.

I would really like to keep the author counter: how complex is it to add it?

Apr 15 2021, 1:03 PMRoadmap 2021, System administration, Monitoring, Web app
vsellier added a comment to T2912: Next generation archive counters.

Staging webapp[1] and webapp1 on production [2] are now configured to use swh-counters to display the historical values and the live object counts.

Apr 15 2021, 11:40 AMRoadmap 2021, System administration, Monitoring, Web app
vsellier closed T3250: Deploy the counters in production (step1 webapp1 only), a subtask of T2912: Next generation archive counters, as Resolved.
Apr 15 2021, 11:32 AMRoadmap 2021, System administration, Monitoring, Web app
vsellier closed T3250: Deploy the counters in production (step1 webapp1 only) as Resolved.

Deployment done on staging and production. The new counters are currently only activated on webapp1

Apr 15 2021, 11:32 AMSystem administration, Monitoring, Web app
vsellier closed D5531: counters: Configure staging and webapp1 to use swh-counters.
Apr 15 2021, 11:11 AM
vsellier committed rSPSITE6fbdb75919c1: counters: Configure staging and webapp1 to use swh-counters (authored by vsellier).
counters: Configure staging and webapp1 to use swh-counters
Apr 15 2021, 11:11 AM
vsellier added inline comments to D5531: counters: Configure staging and webapp1 to use swh-counters.
Apr 15 2021, 11:06 AM
vsellier updated the test plan for D5531: counters: Configure staging and webapp1 to use swh-counters.
Apr 15 2021, 10:33 AM
vsellier updated the summary of D5531: counters: Configure staging and webapp1 to use swh-counters.
Apr 15 2021, 10:30 AM
vsellier updated the diff for D5531: counters: Configure staging and webapp1 to use swh-counters.

fix a typo on the commit description

Apr 15 2021, 10:29 AM
vsellier requested review of D5531: counters: Configure staging and webapp1 to use swh-counters.
Apr 15 2021, 10:23 AM
vsellier added a revision to T3250: Deploy the counters in production (step1 webapp1 only): D5531: counters: Configure staging and webapp1 to use swh-counters.
Apr 15 2021, 10:23 AMSystem administration, Monitoring, Web app

Apr 14 2021

vsellier changed the status of T3250: Deploy the counters in production (step1 webapp1 only) from Open to Work in Progress.
Apr 14 2021, 7:12 PMSystem administration, Monitoring, Web app
vsellier closed T3231: Make the source of the object counts configurable, a subtask of T2912: Next generation archive counters, as Resolved.
Apr 14 2021, 6:40 PMRoadmap 2021, System administration, Monitoring, Web app
vsellier closed T3231: Make the source of the object counts configurable as Resolved.
  • version 0.1.296(swh2) released and deployed on all the webapp
Apr 14 2021, 6:40 PMSystem administration, Monitoring, Web app
vsellier accepted D5526: conf/loader: Declare save code now queues to consume from.
Apr 14 2021, 5:38 PM
vsellier accepted D5486: Declare new service worker to consume save code now queues.

lgtm

Apr 14 2021, 5:36 PM
vsellier closed D5524: Add the configuration for the source of object counters.
Apr 14 2021, 4:23 PM
vsellier committed rDWAPPS05d55040f31c: Add the configuration for the source of object counters (authored by vsellier).
Add the configuration for the source of object counters
Apr 14 2021, 4:23 PM
vsellier updated the diff for D5524: Add the configuration for the source of object counters.

rename the object variable

Apr 14 2021, 4:13 PM
vsellier requested review of D5524: Add the configuration for the source of object counters.
Apr 14 2021, 3:12 PM
vsellier updated the task description for T3231: Make the source of the object counts configurable.
Apr 14 2021, 3:11 PMSystem administration, Monitoring, Web app
vsellier added a revision to T3231: Make the source of the object counts configurable: D5524: Add the configuration for the source of object counters.
Apr 14 2021, 3:05 PMSystem administration, Monitoring, Web app
vsellier closed D5518: Make the project usable in the webapp tests.
Apr 14 2021, 11:18 AM
vsellier committed rDCNT0e5d551ff59b: Make the project usable in the webapp tests (authored by vsellier).
Make the project usable in the webapp tests
Apr 14 2021, 11:18 AM