Page MenuHomeSoftware Heritage

vsellier (Vincent Sellier)
User

Projects

User Details

User Since
Mar 21 2019, 4:14 PM (79 w, 4 d)

Recent Activity

Today

vsellier updated the diff for D4025: Add netbox inventory tool support.
  • rebase
  • add the vagrant configuration
Mon, Sep 28, 3:35 PM
vsellier committed rSPSITE0c8e46a88da2: adapt the configuration for the vagrant environment (authored by vsellier).
adapt the configuration for the vagrant environment
Mon, Sep 28, 3:26 PM
vsellier closed D3989: adapt configuration for the vagrant environment.
Mon, Sep 28, 3:26 PM
vsellier updated the diff for D3989: adapt configuration for the vagrant environment.

rebase

Mon, Sep 28, 3:25 PM
vsellier committed rSENVc6ff01ebc08b: add vagrant configuration and documentation (authored by vsellier).
add vagrant configuration and documentation
Mon, Sep 28, 2:55 PM
vsellier committed rSENV7585ad03eb24: Create the virtualbox image used to instantiate the vagrant environment (authored by vsellier).
Create the virtualbox image used to instantiate the vagrant environment
Mon, Sep 28, 2:55 PM
vsellier closed D3967: Vagrant environment to test puppet locally.
Mon, Sep 28, 2:55 PM
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

Switch the default test branch to staging

Mon, Sep 28, 2:54 PM
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
  • Add a script to avoid to commit each change before being able to test it (watch-vagrant-conf)
  • Update the documentation accordingly
  • Declare the admin-inventory server
Mon, Sep 28, 2:52 PM
vsellier added inline comments to D4062: declare the swh-docker-dev job.
Mon, Sep 28, 2:20 PM
vsellier updated the diff for D4062: declare the swh-docker-dev job.

rebase

Mon, Sep 28, 2:19 PM
vsellier updated the diff for D4061: Add documentation to test locally.

Change the commit message according to the diff's feedback

Mon, Sep 28, 2:19 PM
vsellier updated the diff for D4062: declare the swh-docker-dev job.
  • rebase
  • Remove the now useless CASS_DRIVER_NO_CYTHON option
Mon, Sep 28, 2:15 PM
vsellier updated the diff for D4061: Add documentation to test locally.

fix typo on the commit message

Mon, Sep 28, 2:13 PM
vsellier retitled D4061: Add documentation to test locally from Add documention test locally to Add documentation to test locally.
Mon, Sep 28, 2:11 PM
vsellier created D4062: declare the swh-docker-dev job.
Mon, Sep 28, 2:08 PM
vsellier retitled D4061: Add documentation to test locally from Declare the swh-docker-dev job to Add documention test locally.
Mon, Sep 28, 2:08 PM
vsellier updated the diff for D4061: Add documentation to test locally.

Use only one commit per diff

Mon, Sep 28, 2:08 PM
vsellier retitled D4061: Add documentation to test locally from - Add documention test locally - Declare the swh-docker-dev job to Declare the swh-docker-dev job.
Mon, Sep 28, 2:02 PM
vsellier created D4061: Add documentation to test locally.
Mon, Sep 28, 2:00 PM
vsellier closed T2644: swh-docker-dev CI build - broken since build #569 - 404 not found for psql apt repo as Resolved.

I have added the `--pull` option in the job configuration so the python image will always be pulled (if needed) during the build :

Mon, Sep 28, 10:59 AM · Docker environment, System administration

Yesterday

vsellier claimed T2644: swh-docker-dev CI build - broken since build #569 - 404 not found for psql apt repo.
Sun, Sep 27, 11:36 AM · Docker environment, System administration
vsellier added a comment to T2644: swh-docker-dev CI build - broken since build #569 - 404 not found for psql apt repo.

It seems the python 3.7 images on the jenkins master is quite old :

root@thyssen:~# docker images | grep python
python                     3.7                  a4cc999cf2aa        16 months ago       929MB
Sun, Sep 27, 11:36 AM · Docker environment, System administration

Fri, Sep 25

vsellier committed rDSNIPf14410a67888: netbox - import interfaces and ip addresses (authored by vsellier).
netbox - import interfaces and ip addresses
Fri, Sep 25, 6:09 PM
vsellier closed D4044: Upgrade to postgresql 12.
Fri, Sep 25, 5:06 PM
vsellier committed rDENV4cb4a9133914: Upgrade to postgresql 12 (authored by vsellier).
Upgrade to postgresql 12
Fri, Sep 25, 5:06 PM
vsellier updated subscribers of D4025: Add netbox inventory tool support.

@olasd All the points we check yesterday should be addressed.

Fri, Sep 25, 3:31 PM
vsellier updated the diff for D4025: Add netbox inventory tool support.

isolate the reverse_proxy configuration in a dedicated profile

Fri, Sep 25, 3:24 PM
vsellier updated the diff for D4025: Add netbox inventory tool support.
  • Use the archive module to install the tgz
  • Declare missing dependencies to fix the initial puppet run
  • Install dir is now read-only for the netbox user
  • Restore the whitespaces on the cassandra configuration
Fri, Sep 25, 12:56 PM
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

Rebase

Fri, Sep 25, 10:58 AM
vsellier created D4044: Upgrade to postgresql 12.
Fri, Sep 25, 9:56 AM

Thu, Sep 24

vsellier committed rDSNIP8cc97eaa5f1b: initial commit of the netbox importer (authored by vsellier).
initial commit of the netbox importer
Thu, Sep 24, 7:24 PM
vsellier updated the diff for D4025: Add netbox inventory tool support.
  • add apache and redis configurations
  • Use the swh forge's redis repository
  • use role::swh_server as base roleto activate the backup
Thu, Sep 24, 5:29 PM
vsellier committed rSENV5fe9303121a6: Declare puppet/redis repository on swh forge (authored by vsellier).
Declare puppet/redis repository on swh forge
Thu, Sep 24, 2:54 PM
vsellier changed the visibility for D4027: Add netbox password.
Thu, Sep 24, 10:06 AM
vsellier changed the visibility for D4027: Add netbox password.
Thu, Sep 24, 10:04 AM
vsellier planned changes to D4027: Add netbox password.
Thu, Sep 24, 10:04 AM
vsellier retitled D4027: Add netbox password from Add netbox password (the addition of my password is not related to the issue itself but it was not push before) to Add netbox password.
Thu, Sep 24, 10:04 AM
vsellier added a revision to T2620: Inventory of the swh infrastructure: D4027: Add netbox password.
Thu, Sep 24, 10:03 AM · System administration
vsellier created D4027: Add netbox password.
Thu, Sep 24, 10:03 AM
vsellier added a revision to T2620: Inventory of the swh infrastructure: Unknown Object (Differential Revision).
Thu, Sep 24, 9:59 AM · System administration
vsellier planned changes to D4025: Add netbox inventory tool support.
Thu, Sep 24, 9:41 AM
vsellier created D4025: Add netbox inventory tool support.
Thu, Sep 24, 9:41 AM
vsellier added a revision to T2620: Inventory of the swh infrastructure: D4025: Add netbox inventory tool support.
Thu, Sep 24, 9:41 AM · System administration
vsellier updated the diff for D3989: adapt configuration for the vagrant environment.

Add the properties for netbox on vagrant

Thu, Sep 24, 9:32 AM

Wed, Sep 23

vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
  • declare site.pp to override the default puppet entrypoint
  • add the inventory vm
Wed, Sep 23, 6:33 PM

Tue, Sep 22

vsellier closed T2591: Filebeat fails to start on moma as Resolved.

The configuration is now managed by puppet.

Tue, Sep 22, 3:17 PM · System administration
vsellier closed D3961: Manage filebeat configuration via puppet.
Tue, Sep 22, 3:11 PM
vsellier committed rSPSITEc56d9612ca75: Manage filebeat configuration via puppet (authored by vsellier).
Manage filebeat configuration via puppet
Tue, Sep 22, 3:11 PM
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

Update the base image to use ethX interfaces

Tue, Sep 22, 3:05 PM
vsellier updated the diff for D3989: adapt configuration for the vagrant environment.
  • fix hiera configuration in the context of vagrant
  • fix vms network configuration
Tue, Sep 22, 2:38 PM

Mon, Sep 21

vsellier claimed T2620: Inventory of the swh infrastructure.
Mon, Sep 21, 5:54 PM · System administration
vsellier changed the status of T2620: Inventory of the swh infrastructure from Open to Work in Progress.
Mon, Sep 21, 5:54 PM · System administration
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

Remove useless comment

Mon, Sep 21, 4:26 PM
vsellier added inline comments to D3989: adapt configuration for the vagrant environment.
Mon, Sep 21, 4:01 PM
vsellier updated the diff for D3989: adapt configuration for the vagrant environment.

remove useless properties

Mon, Sep 21, 4:01 PM
vsellier planned changes to D3989: adapt configuration for the vagrant environment.
Mon, Sep 21, 3:42 PM
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

Remove unecessary comments

Mon, Sep 21, 2:59 PM
vsellier planned changes to D3967: Vagrant environment to test puppet locally.

I don't know what's the current state of the diff is (I see quite a lot of commented code, don't know if you want to merge it like this or not, e.g. Vagrantfile).

I just discover there is still an issue with the network interfaces not configured after a restart of a vm. IMO we can wait for this problem to be solved before landing the diff.

Mon, Sep 21, 2:52 PM
vsellier updated the test plan for D3961: Manage filebeat configuration via puppet.
Mon, Sep 21, 2:18 PM
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
  • add vagrant configuration and documentation
Mon, Sep 21, 1:44 PM
vsellier closed T2594: production: Running nixguix on guix sources, a subtask of T1352: ingest Guix (SD) packages, as Resolved.
Mon, Sep 21, 12:09 PM · Archive coverage
vsellier closed T2594: production: Running nixguix on guix sources, a subtask of T2485: staging: Running nixguix on guix sources , as Resolved.
Mon, Sep 21, 12:09 PM · Archive coverage
vsellier closed T2594: production: Running nixguix on guix sources as Resolved.

The recurring visits looks good.

Mon, Sep 21, 12:09 PM · System administration
vsellier updated the task description for T2606: Test puppet configuration in a local vagrant environment.
Mon, Sep 21, 11:49 AM · System administration
vsellier updated the diff for D3961: Manage filebeat configuration via puppet.

update the diff according the previous feedbacks :

  • The logstash hosts are declared on a uniq properties
  • No more yaml templates ;)
  • The profile::filbeat don't use parameters anymore
  • Ensure the permissions are correctly set
  • Add the purge option on the inputs.d directory
Mon, Sep 21, 9:28 AM

Fri, Sep 18

vsellier created D3989: adapt configuration for the vagrant environment.
Fri, Sep 18, 3:09 PM
vsellier added a revision to T2606: Test puppet configuration in a local vagrant environment: D3989: adapt configuration for the vagrant environment.
Fri, Sep 18, 3:09 PM · System administration
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
Add vagrant configuration and documentation
Fri, Sep 18, 2:55 PM
vsellier retitled D3967: Vagrant environment to test puppet locally from Create the virtualbox image used to instantiate the vagrant environment to Vagrant environment to test puppet locally.
Fri, Sep 18, 2:54 PM

Thu, Sep 17

vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

add vagrant usage documentation

Thu, Sep 17, 7:02 PM
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.

prefix the preseed file by the debian version name

Thu, Sep 17, 5:47 PM
vsellier updated the diff for D3961: Manage filebeat configuration via puppet.

rebase

Thu, Sep 17, 3:50 PM
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
  • lister's cache truncated :
swh-lister=> truncate gitea_repo;
TRUNCATE TABLE
swh-lister=> truncate launchpad_repo;
TRUNCATE TABLE
  • recurring task for full listing created :
    • gitea
swhscheduler@saatchi:~$ swh  scheduler --config-file /etc/softwareheritage/scheduler.yml task add list-gitea-full url=https://codeberg.org/api/v1/ limit=100
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
Created 1 tasks
Thu, Sep 17, 2:52 PM · System administration
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
  • Task types registered on the scheduler :
swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.launchpad
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad
swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.gitea    
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea
Thu, Sep 17, 2:37 PM · System administration
vsellier added a comment to D3967: Vagrant environment to test puppet locally.

the iso is available on the public annex at this url : https://annex.softwareheritage.org/public/isos/virtualbox/debian/

Thu, Sep 17, 1:27 PM
vsellier added a comment to D3967: Vagrant environment to test puppet locally.
In D3967#98011, @olasd wrote:

I think this should be moved to the sysadm-provisioning repository [1] instead.

[1] https://forge.softwareheritage.org/source/swh-sysadmin-provisioning/

No, I think a puppet-specific thing like this should definitely be in the puppet repository.

Thu, Sep 17, 1:26 PM
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
  • initial gitea lister launched :
swhworker@worker02:~$ SWH_CONFIG_FILENAME=/etc/softwareheritage/lister.yml swh lister run --lister gitea --priority high url=https://codeberg.org/api/v1/ limit=100
...
INFO:root:listing repos starting at 1198
INFO:root:listing repos starting at 1199
INFO:root:listing repos starting at 1200
INFO:root:stopping after page 1200, no next link found
Thu, Sep 17, 11:53 AM · System administration
vsellier closed T2577: Test gitea lister on staging environment, a subtask of T2313: Archive git.fsfe.org (Gitea), as Resolved.
Thu, Sep 17, 11:41 AM · Archive coverage, Lister
vsellier closed T2577: Test gitea lister on staging environment as Resolved.

everything seems to work well, the production deployment will be done in T2608

Thu, Sep 17, 11:41 AM · Lister
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
  • initial manual launchpad listing launched :
swhworker@worker02:~$ SWH_CONFIG_FILENAME=/etc/softwareheritage/lister.yml swh lister run --lister launchpad --priority high
INFO:swh.core.config:Loading config file /etc/softwareheritage/lister.yml
INFO:swh.core.config:Loading config file /etc/softwareheritage/global.ini
INFO:swh.core.config:Loading config file /etc/softwareheritage/lister.yml
Thu, Sep 17, 11:39 AM · System administration
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
  • schedult tak-types created:
swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.gitea
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea
INFO:swh.scheduler.cli.task_type:Create task type list-gitea-full in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-gitea-incremental in scheduler
swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.launchpad
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad
INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-full in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-incremental in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-new in scheduler
Thu, Sep 17, 11:34 AM · System administration
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
  • user guest granted to access the new tables :
swh-lister=>    grant select
swh-lister->    on all tables in schema public
swh-lister->    to guest;
GRANT
Thu, Sep 17, 11:27 AM · System administration
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
  • lister model updated from worker01:
swhworker@worker01:/etc/softwareheritage$ swh lister --db-url postgresql://swh-lister:anime-pelvis-showy@db.internal.softwareheritage.org:5432/swh-lister db-init
INFO:swh.lister.cli:Loading lister bitbucket
INFO:swh.lister.cli:Loading lister cgit
INFO:swh.lister.cli:Loading lister cran
INFO:swh.lister.cli:Loading lister debian
INFO:swh.lister.cli:Loading lister gitea
INFO:swh.lister.cli:Loading lister github
INFO:swh.lister.cli:Loading lister gitlab
INFO:swh.lister.cli:Loading lister gnu
INFO:swh.lister.cli:Loading lister launchpad
INFO:swh.lister.cli:Loading lister npm
INFO:swh.lister.cli:Loading lister packagist
INFO:swh.lister.cli:Loading lister phabricator
INFO:swh.lister.cli:Loading lister pypi
INFO:swh.lister.cli:Initializing database
INFO:swh.lister.core.models:Creating tables
INFO:swh.lister.cli:Calling init hook for debian
Thu, Sep 17, 11:21 AM · System administration
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.

New version of the lister package deployed :

  • on workers
root@pergamon:~# clush -b -w @swh-workers 'apt-get update; apt install -y python3-swh.lister' 
...
root@pergamon:~# clush -b -w @swh-workers "dpkg -l python3-swh.lister"
---------------
worker[01-16] (16)
---------------
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name               Version              Architecture Description
+++-==================-====================-============-=================================================================
ii  python3-swh.lister 0.1.4-1~swh1~bpo10+1 all          Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)
  • on the scheduler :
root@saatchi:~# apt update && apt install python3-swh.lister
...
Restarting services...
 systemctl restart gunicorn-swh-scheduler.service icinga2.service journalbeat.service postfix@-.service rabbitmq-server.service rpcbind.service ssh.service swh-scheduler-runner.service unbound.service
Thu, Sep 17, 11:13 AM · System administration
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.

Actions :

  • deploy the new version of the list on each worker
  • create the new task-type on the scheduler
  • manually launch a listing to create high loading task for launchpad and gitea repository
  • schedule the recurring listing tasks for both repositories
Thu, Sep 17, 10:44 AM · System administration
vsellier added a subtask for T1734: Create a Lister for launchpad.net: T2608: Deploy launchpad and gitea listers on production.
Thu, Sep 17, 10:40 AM · Lister, Archive coverage
vsellier added parent tasks for T2608: Deploy launchpad and gitea listers on production: T1734: Create a Lister for launchpad.net, T2313: Archive git.fsfe.org (Gitea).
Thu, Sep 17, 10:40 AM · System administration
vsellier added a subtask for T2313: Archive git.fsfe.org (Gitea): T2608: Deploy launchpad and gitea listers on production.
Thu, Sep 17, 10:40 AM · Archive coverage, Lister
vsellier changed the status of T2608: Deploy launchpad and gitea listers on production from Open to Work in Progress.
Thu, Sep 17, 10:39 AM · System administration

Wed, Sep 16

vsellier created D3967: Vagrant environment to test puppet locally.
Wed, Sep 16, 6:28 PM
vsellier added a revision to T2606: Test puppet configuration in a local vagrant environment: D3967: Vagrant environment to test puppet locally.
Wed, Sep 16, 6:28 PM · System administration
vsellier changed the status of T2606: Test puppet configuration in a local vagrant environment from Open to Work in Progress.
Wed, Sep 16, 6:16 PM · System administration
vsellier updated the test plan for D3961: Manage filebeat configuration via puppet.
Wed, Sep 16, 5:14 PM
vsellier updated the diff for D3961: Manage filebeat configuration via puppet.

Use the capability of filebeat to use a file per input.

Wed, Sep 16, 5:12 PM
vsellier edited reviewers for D3961: Manage filebeat configuration via puppet, added: System administrators; removed: Reviewers.
Wed, Sep 16, 11:11 AM
vsellier added a revision to T2591: Filebeat fails to start on moma: D3961: Manage filebeat configuration via puppet.
Wed, Sep 16, 11:11 AM · System administration
vsellier created D3961: Manage filebeat configuration via puppet.
Wed, Sep 16, 11:11 AM

Tue, Sep 15

vsellier added a comment to T2594: production: Running nixguix on guix sources.

The loading is in progress and seems ok

Tue, Sep 15, 11:56 AM · System administration
vsellier added a comment to T2594: production: Running nixguix on guix sources.

The loading task for guix is scheduled in production :

swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task add load-nixguix url=https://guix.gnu.org/sources.json
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
Created 1 tasks
Tue, Sep 15, 11:41 AM · System administration