Add a script to avoid to commit each change before being able to test it (watch-vagrant-conf)
Update the documentation accordingly
Declare the admin-inventory server

Mon, Sep 28, 2:52 PM

vsellier added inline comments to D4062: declare the swh-docker-dev job.

Mon, Sep 28, 2:20 PM

vsellier updated the diff for D4062: declare the swh-docker-dev job.

rebase

Mon, Sep 28, 2:19 PM

vsellier updated the diff for D4061: Add documentation to test locally.

Change the commit message according to the diff's feedback

Mon, Sep 28, 2:19 PM

vsellier updated the diff for D4062: declare the swh-docker-dev job.

rebase
Remove the now useless CASS_DRIVER_NO_CYTHON option

Mon, Sep 28, 2:15 PM

vsellier updated the diff for D4061: Add documentation to test locally.

fix typo on the commit message

Mon, Sep 28, 2:13 PM

vsellier retitled D4061: Add documentation to test locally from Add documention test locally to Add documentation to test locally.

Mon, Sep 28, 2:11 PM

vsellier created D4062: declare the swh-docker-dev job.

Mon, Sep 28, 2:08 PM

vsellier retitled D4061: Add documentation to test locally from Declare the swh-docker-dev job to Add documention test locally.

Mon, Sep 28, 2:08 PM

vsellier updated the diff for D4061: Add documentation to test locally.

Use only one commit per diff

Mon, Sep 28, 2:08 PM

vsellier retitled D4061: Add documentation to test locally from - Add documention test locally - Declare the swh-docker-dev job to Declare the swh-docker-dev job.

Mon, Sep 28, 2:02 PM

vsellier created D4061: Add documentation to test locally.

Mon, Sep 28, 2:00 PM

vsellier closed T2644: swh-docker-dev CI build - broken since build #569 - 404 not found for psql apt repo as Resolved.

I have added the `--pull` option in the job configuration so the python image will always be pulled (if needed) during the build :

Mon, Sep 28, 10:59 AM · Docker environment, System administration

Yesterday

vsellier claimed T2644: swh-docker-dev CI build - broken since build #569 - 404 not found for psql apt repo.

Sun, Sep 27, 11:36 AM · Docker environment, System administration

vsellier added a comment to T2644: swh-docker-dev CI build - broken since build #569 - 404 not found for psql apt repo.

It seems the python 3.7 images on the jenkins master is quite old :

root@thyssen:~# docker images | grep python
python                     3.7                  a4cc999cf2aa        16 months ago       929MB

Sun, Sep 27, 11:36 AM · Docker environment, System administration

Fri, Sep 25

vsellier committed rDSNIPf14410a67888: netbox - import interfaces and ip addresses (authored by vsellier).

netbox - import interfaces and ip addresses

Fri, Sep 25, 6:09 PM

vsellier closed D4044: Upgrade to postgresql 12.

Fri, Sep 25, 5:06 PM

vsellier committed rDENV4cb4a9133914: Upgrade to postgresql 12 (authored by vsellier).

Upgrade to postgresql 12

Fri, Sep 25, 5:06 PM

vsellier updated subscribers of D4025: Add netbox inventory tool support.

@olasd All the points we check yesterday should be addressed.

Fri, Sep 25, 3:31 PM

vsellier updated the diff for D4025: Add netbox inventory tool support.

isolate the reverse_proxy configuration in a dedicated profile

Fri, Sep 25, 3:24 PM

vsellier updated the diff for D4025: Add netbox inventory tool support.

Use the archive module to install the tgz
Declare missing dependencies to fix the initial puppet run
Install dir is now read-only for the netbox user
Restore the whitespaces on the cassandra configuration

Fri, Sep 25, 12:56 PM

vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

Rebase

Fri, Sep 25, 10:58 AM

vsellier created D4044: Upgrade to postgresql 12.

Fri, Sep 25, 9:56 AM

Thu, Sep 24

vsellier committed rDSNIP8cc97eaa5f1b: initial commit of the netbox importer (authored by vsellier).

initial commit of the netbox importer

Thu, Sep 24, 7:24 PM

vsellier updated the diff for D4025: Add netbox inventory tool support.

add apache and redis configurations
Use the swh forge's redis repository
use role::swh_server as base roleto activate the backup

Thu, Sep 24, 5:29 PM

vsellier committed rSENV5fe9303121a6: Declare puppet/redis repository on swh forge (authored by vsellier).

Declare puppet/redis repository on swh forge

Thu, Sep 24, 2:54 PM

vsellier changed the visibility for D4027: Add netbox password.

Thu, Sep 24, 10:06 AM

vsellier changed the visibility for D4027: Add netbox password.

Thu, Sep 24, 10:04 AM

vsellier planned changes to D4027: Add netbox password.

Thu, Sep 24, 10:04 AM

vsellier retitled D4027: Add netbox password from Add netbox password (the addition of my password is not related to the issue itself but it was not push before) to Add netbox password.

Thu, Sep 24, 10:04 AM

vsellier added a revision to T2620: Inventory of the swh infrastructure: D4027: Add netbox password.

Thu, Sep 24, 10:03 AM · System administration

vsellier created D4027: Add netbox password.

Thu, Sep 24, 10:03 AM

vsellier added a revision to T2620: Inventory of the swh infrastructure: Unknown Object (Differential Revision).

Thu, Sep 24, 9:59 AM · System administration

vsellier planned changes to D4025: Add netbox inventory tool support.

Thu, Sep 24, 9:41 AM

vsellier created D4025: Add netbox inventory tool support.

Thu, Sep 24, 9:41 AM

vsellier added a revision to T2620: Inventory of the swh infrastructure: D4025: Add netbox inventory tool support.

Thu, Sep 24, 9:41 AM · System administration

vsellier updated the diff for D3989: adapt configuration for the vagrant environment.

Add the properties for netbox on vagrant

Thu, Sep 24, 9:32 AM

Wed, Sep 23

vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

declare site.pp to override the default puppet entrypoint
add the inventory vm

Wed, Sep 23, 6:33 PM

Tue, Sep 22

vsellier closed T2591: Filebeat fails to start on moma as Resolved.

The configuration is now managed by puppet.

Tue, Sep 22, 3:17 PM · System administration

vsellier closed D3961: Manage filebeat configuration via puppet.

Tue, Sep 22, 3:11 PM

vsellier committed rSPSITEc56d9612ca75: Manage filebeat configuration via puppet (authored by vsellier).

Manage filebeat configuration via puppet

Tue, Sep 22, 3:11 PM

vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

Update the base image to use ethX interfaces

Tue, Sep 22, 3:05 PM

vsellier updated the diff for D3989: adapt configuration for the vagrant environment.

fix hiera configuration in the context of vagrant
fix vms network configuration

Tue, Sep 22, 2:38 PM

Mon, Sep 21

vsellier claimed T2620: Inventory of the swh infrastructure.

Mon, Sep 21, 5:54 PM · System administration

vsellier changed the status of T2620: Inventory of the swh infrastructure from Open to Work in Progress.

Mon, Sep 21, 5:54 PM · System administration

vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

Remove useless comment

Mon, Sep 21, 4:26 PM

vsellier added inline comments to D3989: adapt configuration for the vagrant environment.

Mon, Sep 21, 4:01 PM

vsellier updated the diff for D3989: adapt configuration for the vagrant environment.

remove useless properties

Mon, Sep 21, 4:01 PM

vsellier planned changes to D3989: adapt configuration for the vagrant environment.

Mon, Sep 21, 3:42 PM

vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

Remove unecessary comments

Mon, Sep 21, 2:59 PM

vsellier planned changes to D3967: Vagrant environment to test puppet locally.

I don't know what's the current state of the diff is (I see quite a lot of commented code, don't know if you want to merge it like this or not, e.g. Vagrantfile).

I just discover there is still an issue with the network interfaces not configured after a restart of a vm. IMO we can wait for this problem to be solved before landing the diff.

Mon, Sep 21, 2:52 PM

vsellier updated the test plan for D3961: Manage filebeat configuration via puppet.

Mon, Sep 21, 2:18 PM

vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

add vagrant configuration and documentation

Mon, Sep 21, 1:44 PM

vsellier closed T2594: production: Running nixguix on guix sources, a subtask of T1352: ingest Guix (SD) packages, as Resolved.

Mon, Sep 21, 12:09 PM · Archive coverage

vsellier closed T2594: production: Running nixguix on guix sources, a subtask of T2485: staging: Running nixguix on guix sources , as Resolved.

Mon, Sep 21, 12:09 PM · Archive coverage

vsellier closed T2594: production: Running nixguix on guix sources as Resolved.

The recurring visits looks good.

Mon, Sep 21, 12:09 PM · System administration

vsellier updated the task description for T2606: Test puppet configuration in a local vagrant environment.

Mon, Sep 21, 11:49 AM · System administration

vsellier updated the diff for D3961: Manage filebeat configuration via puppet.

update the diff according the previous feedbacks :

The logstash hosts are declared on a uniq properties
No more yaml templates ;)
The profile::filbeat don't use parameters anymore
Ensure the permissions are correctly set
Add the purge option on the inputs.d directory

Mon, Sep 21, 9:28 AM

Fri, Sep 18

vsellier created D3989: adapt configuration for the vagrant environment.

Fri, Sep 18, 3:09 PM

vsellier added a revision to T2606: Test puppet configuration in a local vagrant environment: D3989: adapt configuration for the vagrant environment.

Fri, Sep 18, 3:09 PM · System administration

vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

Add vagrant configuration and documentation

Fri, Sep 18, 2:55 PM

vsellier retitled D3967: Vagrant environment to test puppet locally from Create the virtualbox image used to instantiate the vagrant environment to Vagrant environment to test puppet locally.

Fri, Sep 18, 2:54 PM

Thu, Sep 17

vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

add vagrant usage documentation

Thu, Sep 17, 7:02 PM

vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

prefix the preseed file by the debian version name

Thu, Sep 17, 5:47 PM

vsellier updated the diff for D3961: Manage filebeat configuration via puppet.

rebase

Thu, Sep 17, 3:50 PM

vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.

lister's cache truncated :

swh-lister=> truncate gitea_repo;
TRUNCATE TABLE
swh-lister=> truncate launchpad_repo;
TRUNCATE TABLE

recurring task for full listing created :
- gitea

swhscheduler@saatchi:~$ swh  scheduler --config-file /etc/softwareheritage/scheduler.yml task add list-gitea-full url=https://codeberg.org/api/v1/ limit=100
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
Created 1 tasks

Thu, Sep 17, 2:52 PM · System administration

vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.

Task types registered on the scheduler :

swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.launchpad
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad
swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.gitea    
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea

Thu, Sep 17, 2:37 PM · System administration

vsellier added a comment to D3967: Vagrant environment to test puppet locally.

the iso is available on the public annex at this url : https://annex.softwareheritage.org/public/isos/virtualbox/debian/

Thu, Sep 17, 1:27 PM

vsellier added a comment to D3967: Vagrant environment to test puppet locally.

In D3967#98011, @olasd wrote:

In D3967#97981, @ardumont wrote:

I think this should be moved to the sysadm-provisioning repository [1] instead.

[1] https://forge.softwareheritage.org/source/swh-sysadmin-provisioning/

No, I think a puppet-specific thing like this should definitely be in the puppet repository.

Thu, Sep 17, 1:26 PM

vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.

initial gitea lister launched :

swhworker@worker02:~$ SWH_CONFIG_FILENAME=/etc/softwareheritage/lister.yml swh lister run --lister gitea --priority high url=https://codeberg.org/api/v1/ limit=100
...
INFO:root:listing repos starting at 1198
INFO:root:listing repos starting at 1199
INFO:root:listing repos starting at 1200
INFO:root:stopping after page 1200, no next link found

Thu, Sep 17, 11:53 AM · System administration

vsellier closed T2577: Test gitea lister on staging environment, a subtask of T2313: Archive git.fsfe.org (Gitea), as Resolved.

Thu, Sep 17, 11:41 AM · Archive coverage, Lister

vsellier closed T2577: Test gitea lister on staging environment as Resolved.

everything seems to work well, the production deployment will be done in T2608

Thu, Sep 17, 11:41 AM · Lister

vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.

initial manual launchpad listing launched :

swhworker@worker02:~$ SWH_CONFIG_FILENAME=/etc/softwareheritage/lister.yml swh lister run --lister launchpad --priority high
INFO:swh.core.config:Loading config file /etc/softwareheritage/lister.yml
INFO:swh.core.config:Loading config file /etc/softwareheritage/global.ini
INFO:swh.core.config:Loading config file /etc/softwareheritage/lister.yml

Thu, Sep 17, 11:39 AM · System administration

vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.

schedult tak-types created:

swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.gitea
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea
INFO:swh.scheduler.cli.task_type:Create task type list-gitea-full in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-gitea-incremental in scheduler
swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.launchpad
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad
INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-full in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-incremental in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-new in scheduler

Thu, Sep 17, 11:34 AM · System administration

vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.

user guest granted to access the new tables :

swh-lister=>    grant select
swh-lister->    on all tables in schema public
swh-lister->    to guest;
GRANT

Thu, Sep 17, 11:27 AM · System administration

vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.

lister model updated from worker01:

swhworker@worker01:/etc/softwareheritage$ swh lister --db-url postgresql://swh-lister:anime-pelvis-showy@db.internal.softwareheritage.org:5432/swh-lister db-init
INFO:swh.lister.cli:Loading lister bitbucket
INFO:swh.lister.cli:Loading lister cgit
INFO:swh.lister.cli:Loading lister cran
INFO:swh.lister.cli:Loading lister debian
INFO:swh.lister.cli:Loading lister gitea
INFO:swh.lister.cli:Loading lister github
INFO:swh.lister.cli:Loading lister gitlab
INFO:swh.lister.cli:Loading lister gnu
INFO:swh.lister.cli:Loading lister launchpad
INFO:swh.lister.cli:Loading lister npm
INFO:swh.lister.cli:Loading lister packagist
INFO:swh.lister.cli:Loading lister phabricator
INFO:swh.lister.cli:Loading lister pypi
INFO:swh.lister.cli:Initializing database
INFO:swh.lister.core.models:Creating tables
INFO:swh.lister.cli:Calling init hook for debian

Thu, Sep 17, 11:21 AM · System administration

vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.

New version of the lister package deployed :

on workers

root@pergamon:~# clush -b -w @swh-workers 'apt-get update; apt install -y python3-swh.lister' 
...
root@pergamon:~# clush -b -w @swh-workers "dpkg -l python3-swh.lister"
---------------
worker[01-16] (16)
---------------
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name               Version              Architecture Description
+++-==================-====================-============-=================================================================
ii  python3-swh.lister 0.1.4-1~swh1~bpo10+1 all          Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)

on the scheduler :

root@saatchi:~# apt update && apt install python3-swh.lister
...
Restarting services...
 systemctl restart gunicorn-swh-scheduler.service icinga2.service journalbeat.service postfix@-.service rabbitmq-server.service rpcbind.service ssh.service swh-scheduler-runner.service unbound.service

Thu, Sep 17, 11:13 AM · System administration

vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.

Actions :

deploy the new version of the list on each worker
create the new task-type on the scheduler
manually launch a listing to create high loading task for launchpad and gitea repository
schedule the recurring listing tasks for both repositories

Thu, Sep 17, 10:44 AM · System administration

vsellier added a subtask for T1734: Create a Lister for launchpad.net: T2608: Deploy launchpad and gitea listers on production.

Thu, Sep 17, 10:40 AM · Lister, Archive coverage

vsellier added parent tasks for T2608: Deploy launchpad and gitea listers on production: T1734: Create a Lister for launchpad.net, T2313: Archive git.fsfe.org (Gitea).

Thu, Sep 17, 10:40 AM · System administration

vsellier added a subtask for T2313: Archive git.fsfe.org (Gitea): T2608: Deploy launchpad and gitea listers on production.

Thu, Sep 17, 10:40 AM · Archive coverage, Lister

vsellier changed the status of T2608: Deploy launchpad and gitea listers on production from Open to Work in Progress.

Thu, Sep 17, 10:39 AM · System administration

Wed, Sep 16

vsellier created D3967: Vagrant environment to test puppet locally.

Wed, Sep 16, 6:28 PM

vsellier added a revision to T2606: Test puppet configuration in a local vagrant environment: D3967: Vagrant environment to test puppet locally.

Wed, Sep 16, 6:28 PM · System administration

vsellier changed the status of T2606: Test puppet configuration in a local vagrant environment from Open to Work in Progress.

Wed, Sep 16, 6:16 PM · System administration

vsellier updated the test plan for D3961: Manage filebeat configuration via puppet.

Wed, Sep 16, 5:14 PM

vsellier updated the diff for D3961: Manage filebeat configuration via puppet.

Use the capability of filebeat to use a file per input.

Wed, Sep 16, 5:12 PM

vsellier edited reviewers for D3961: Manage filebeat configuration via puppet, added: System administrators; removed: Reviewers.

Wed, Sep 16, 11:11 AM

vsellier added a revision to T2591: Filebeat fails to start on moma: D3961: Manage filebeat configuration via puppet.

Wed, Sep 16, 11:11 AM · System administration

vsellier created D3961: Manage filebeat configuration via puppet.

Wed, Sep 16, 11:11 AM

Tue, Sep 15

vsellier added a comment to T2594: production: Running nixguix on guix sources.

The loading is in progress and seems ok

Tue, Sep 15, 11:56 AM · System administration

vsellier added a comment to T2594: production: Running nixguix on guix sources.

The loading task for guix is scheduled in production :

swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task add load-nixguix url=https://guix.gnu.org/sources.json
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
Created 1 tasks

Tue, Sep 15, 11:41 AM · System administration

vsellier (Vincent Sellier)User

Projects

User Details

Recent ActivityView All

Today

Yesterday

Fri, Sep 25

Thu, Sep 24

Wed, Sep 23

Tue, Sep 22

Mon, Sep 21

Fri, Sep 18

Thu, Sep 17

Wed, Sep 16

Tue, Sep 15

vsellier (Vincent Sellier)
User

Recent Activity
View All