I have added the `--pull` option in the job configuration so the python image will always be pulled (if needed) during the build :
Feed Advanced Search
Sep 28 2020
Sep 28 2020
vsellier retitled D4061: Add documentation to test locally from - Add documention test locally
- Declare the swh-docker-dev job to Declare the swh-docker-dev job.
vsellier closed T2644: swh-docker-dev CI build - broken since build #569 - 404 not found for psql apt repo as Resolved.
Sep 27 2020
Sep 27 2020
vsellier added a comment to T2644: swh-docker-dev CI build - broken since build #569 - 404 not found for psql apt repo.
It seems the python 3.7 images on the jenkins master is quite old :
root@thyssen:~# docker images | grep python python 3.7 a4cc999cf2aa 16 months ago 929MB
Sep 25 2020
Sep 25 2020
vsellier committed rDSNIPf14410a67888: netbox - import interfaces and ip addresses (authored by vsellier).
netbox - import interfaces and ip addresses
Upgrade to postgresql 12
vsellier updated subscribers of D4025: Add netbox inventory tool support.
@olasd All the points we have checked yesterday should be addressed.
vsellier updated the diff for D4025: Add netbox inventory tool support.
isolate the reverse_proxy configuration in a dedicated profile
vsellier updated the diff for D4025: Add netbox inventory tool support.
- Use the archive module to install the tgz
- Declare missing dependencies to fix the initial puppet run
- Install dir is now read-only for the netbox user
- Restore the whitespaces on the cassandra configuration
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
Rebase
Sep 24 2020
Sep 24 2020
vsellier committed rDSNIP8cc97eaa5f1b: initial commit of the netbox importer (authored by vsellier).
initial commit of the netbox importer
vsellier updated the diff for D4025: Add netbox inventory tool support.
- add apache and redis configurations
- Use the swh forge's redis repository
- use role::swh_server as base roleto activate the backup
vsellier committed rSENV5fe9303121a6: Declare puppet/redis repository on swh forge (authored by vsellier).
Declare puppet/redis repository on swh forge
vsellier changed the visibility for D4027: Add netbox password.
vsellier changed the visibility for D4027: Add netbox password.
vsellier planned changes to D4027: Add netbox password.
vsellier retitled D4027: Add netbox password from Add netbox password
(the addition of my password is not related to the issue itself but it was not push before) to Add netbox password.
vsellier added a revision to T2620: Inventory of the swh infrastructure: D4027: Add netbox password.
vsellier added a revision to T2620: Inventory of the swh infrastructure: Unknown Object (Differential Revision).
vsellier planned changes to D4025: Add netbox inventory tool support.
vsellier added a revision to T2620: Inventory of the swh infrastructure: D4025: Add netbox inventory tool support.
vsellier updated the diff for D3989: adapt configuration for the vagrant environment.
Add the properties for netbox on vagrant
Sep 23 2020
Sep 23 2020
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
- declare site.pp to override the default puppet entrypoint
- add the inventory vm
Sep 22 2020
Sep 22 2020
The configuration is now managed by puppet.
vsellier committed rSPSITEc56d9612ca75: Manage filebeat configuration via puppet (authored by vsellier).
Manage filebeat configuration via puppet
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
Update the base image to use ethX interfaces
vsellier updated the diff for D3989: adapt configuration for the vagrant environment.
- fix hiera configuration in the context of vagrant
- fix vms network configuration
Sep 21 2020
Sep 21 2020
vsellier changed the status of T2620: Inventory of the swh infrastructure from Open to Work in Progress.
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
Remove useless comment
vsellier added inline comments to D3989: adapt configuration for the vagrant environment.
vsellier updated the diff for D3989: adapt configuration for the vagrant environment.
remove useless properties
vsellier planned changes to D3989: adapt configuration for the vagrant environment.
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
Remove unecessary comments
vsellier planned changes to D3967: Vagrant environment to test puppet locally.
I don't know what's the current state of the diff is (I see quite a lot of commented code, don't know if you want to merge it like this or not, e.g. Vagrantfile).
I just discover there is still an issue with the network interfaces not configured after a restart of a vm. IMO we can wait for this problem to be solved before landing the diff.
vsellier updated the test plan for D3961: Manage filebeat configuration via puppet.
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
Fix plenty of " :" ;)
vsellier closed T2594: production: Running nixguix on guix sources, a subtask of T1352: ingest Guix (SD) packages, as Resolved.
vsellier closed T2594: production: Running nixguix on guix sources, a subtask of T2485: staging: Running nixguix on guix sources , as Resolved.
The recurring visits looks good.
vsellier updated the task description for T2606: Test puppet configuration in a local vagrant environment.
vsellier updated the diff for D3961: Manage filebeat configuration via puppet.
update the diff according the previous feedbacks :
- The logstash hosts are declared on a uniq property
- No more yaml templates ;)
- The profile::filbeat doesn't use parameters anymore
- Ensure the permissions are correctly set
- Add the purge option on the inputs.d directory
Sep 18 2020
Sep 18 2020
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
Add vagrant configuration and documentation
vsellier retitled D3967: Vagrant environment to test puppet locally from Create the virtualbox image used to instantiate the vagrant environment to Vagrant environment to test puppet locally.
Sep 17 2020
Sep 17 2020
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
add vagrant usage documentation
vsellier updated the diff for D3967: Vagrant environment to test puppet locally.
prefix the preseed file by the debian version name
vsellier updated the diff for D3961: Manage filebeat configuration via puppet.
rebase
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
- lister's cache truncated :
swh-lister=> truncate gitea_repo; TRUNCATE TABLE swh-lister=> truncate launchpad_repo; TRUNCATE TABLE
- recurring task for full listing created :
- gitea
swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task add list-gitea-full url=https://codeberg.org/api/v1/ limit=100 INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml Created 1 tasks
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
- Task types registered on the scheduler :
swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.launchpad INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.gitea INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea
vsellier added a comment to D3967: Vagrant environment to test puppet locally.
the iso is available on the public annex at this url : https://annex.softwareheritage.org/public/isos/virtualbox/debian/
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
- initial gitea lister launched :
swhworker@worker02:~$ SWH_CONFIG_FILENAME=/etc/softwareheritage/lister.yml swh lister run --lister gitea --priority high url=https://codeberg.org/api/v1/ limit=100 ... INFO:root:listing repos starting at 1198 INFO:root:listing repos starting at 1199 INFO:root:listing repos starting at 1200 INFO:root:stopping after page 1200, no next link found
vsellier closed T2577: Test gitea lister on staging environment, a subtask of T2313: Archive git.fsfe.org (Gitea), as Resolved.
everything seems to work well, the production deployment will be done in T2608
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
- initial manual launchpad listing launched :
swhworker@worker02:~$ SWH_CONFIG_FILENAME=/etc/softwareheritage/lister.yml swh lister run --lister launchpad --priority high INFO:swh.core.config:Loading config file /etc/softwareheritage/lister.yml INFO:swh.core.config:Loading config file /etc/softwareheritage/global.ini INFO:swh.core.config:Loading config file /etc/softwareheritage/lister.yml
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
- schedult tak-types created:
swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.gitea INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea INFO:swh.scheduler.cli.task_type:Create task type list-gitea-full in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-gitea-incremental in scheduler swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.launchpad INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.launchpad INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-full in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-incremental in scheduler INFO:swh.scheduler.cli.task_type:Create task type list-launchpad-new in scheduler
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
- user guest granted to access the new tables :
swh-lister=> grant select swh-lister-> on all tables in schema public swh-lister-> to guest; GRANT
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
- lister model updated from worker01:
swhworker@worker01:/etc/softwareheritage$ swh lister --db-url postgresql://*****@db.internal.softwareheritage.org:5432/swh-lister db-init INFO:swh.lister.cli:Loading lister bitbucket INFO:swh.lister.cli:Loading lister cgit INFO:swh.lister.cli:Loading lister cran INFO:swh.lister.cli:Loading lister debian INFO:swh.lister.cli:Loading lister gitea INFO:swh.lister.cli:Loading lister github INFO:swh.lister.cli:Loading lister gitlab INFO:swh.lister.cli:Loading lister gnu INFO:swh.lister.cli:Loading lister launchpad INFO:swh.lister.cli:Loading lister npm INFO:swh.lister.cli:Loading lister packagist INFO:swh.lister.cli:Loading lister phabricator INFO:swh.lister.cli:Loading lister pypi INFO:swh.lister.cli:Initializing database INFO:swh.lister.core.models:Creating tables INFO:swh.lister.cli:Calling init hook for debian
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
New version of the lister package deployed :
- on workers
root@pergamon:~# clush -b -w @swh-workers 'apt-get update; apt install -y python3-swh.lister' ... root@pergamon:~# clush -b -w @swh-workers "dpkg -l python3-swh.lister" --------------- worker[01-16] (16) --------------- Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==================-====================-============-================================================================= ii python3-swh.lister 0.1.4-1~swh1~bpo10+1 all Software Heritage Listers (bitbucket, git(lab|hub), pypi, etc...)
- on the scheduler :
root@saatchi:~# apt update && apt install python3-swh.lister ... Restarting services... systemctl restart gunicorn-swh-scheduler.service icinga2.service journalbeat.service postfix@-.service rabbitmq-server.service rpcbind.service ssh.service swh-scheduler-runner.service unbound.service
vsellier added a comment to T2608: Deploy launchpad and gitea listers on production.
Actions :
- deploy the new version of the lister on each worker
- update the lister data model
- create the new task-type on the scheduler
- manually launch a listing to create high priority loading tasks for launchpad and gitea repository to ingest soon the repositories and not at the end of the current git queue
- truncate lister cache to allow the recurring loading tasks to be created
- schedule the recurring listing tasks for both repositories
vsellier changed the status of T2608: Deploy launchpad and gitea listers on production from Open to Work in Progress.
Sep 16 2020
Sep 16 2020
vsellier changed the status of T2606: Test puppet configuration in a local vagrant environment from Open to Work in Progress.
vsellier updated the test plan for D3961: Manage filebeat configuration via puppet.
vsellier updated the diff for D3961: Manage filebeat configuration via puppet.
Use the capability of filebeat to use a file per input.
vsellier edited reviewers for D3961: Manage filebeat configuration via puppet, added: System administrators; removed: Reviewers.
vsellier added a revision to T2591: Filebeat fails to start on moma: D3961: Manage filebeat configuration via puppet.
Sep 15 2020
Sep 15 2020
vsellier added a comment to T2594: production: Running nixguix on guix sources.
The loading is in progress and seems ok
vsellier added a comment to T2594: production: Running nixguix on guix sources.
The loading task for guix is scheduled in production :
swhscheduler@saatchi:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task add load-nixguix url=https://guix.gnu.org/sources.json INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml Created 1 tasks
vsellier removed a subtask for T1991: Implement a Guix/Nix loader: T2594: production: Running nixguix on guix sources.
vsellier edited parent tasks for T2594: production: Running nixguix on guix sources, added: T2485: staging: Running nixguix on guix sources ; removed: T1991: Implement a Guix/Nix loader.
vsellier added a subtask for T1991: Implement a Guix/Nix loader: T2594: production: Running nixguix on guix sources.
vsellier added a parent task for T2594: production: Running nixguix on guix sources: T1991: Implement a Guix/Nix loader.
vsellier added a parent task for T2594: production: Running nixguix on guix sources: T1352: ingest Guix (SD) packages.
vsellier added a subtask for T1352: ingest Guix (SD) packages: T2594: production: Running nixguix on guix sources.
vsellier changed the status of T2594: production: Running nixguix on guix sources from Open to Work in Progress.
vsellier added a comment to T2591: Filebeat fails to start on moma.
The configuration was fixed on moma :
root@moma:/etc/filebeat# diff -U3 /tmp/filebeat.yml /etc/filebeat/filebeat.yml --- /tmp/filebeat.yml 2020-09-15 08:10:20.512838905 +0000 +++ /etc/filebeat/filebeat.yml 2020-09-15 08:16:13.096135043 +0000 @@ -1,4 +1,4 @@ -filebeat.prospectors: +filebeat.inputs: - type: log paths: - /var/log/apache2/archive.softwareheritage.org_non-ssl_access.log
and the logs since the last rotation correctly ingested :
vsellier updated the task description for T2591: Filebeat fails to start on moma.
vsellier changed the status of T2591: Filebeat fails to start on moma from Open to Work in Progress.
Sep 14 2020
Sep 14 2020
vsellier added a comment to T2358: Deploy launchpad lister on staging.
An email was sent on the swh-devel mailing list to ask for reviews.
The deployment in production will be performed in the middle of week 38 is no problems are raised.
vsellier added a comment to T2577: Test gitea lister on staging environment.
An email was sent on the swh-devel mailing list to ask for reviews.
The deployment in production will be performed in the middle of week 38 is no problems are raised.
Sep 11 2020
Sep 11 2020
Sep 10 2020
Sep 10 2020
vsellier reopened T2577: Test gitea lister on staging environment, a subtask of T2313: Archive git.fsfe.org (Gitea), as Work in Progress.
vsellier reopened T2577: Test gitea lister on staging environment as "Work in Progress".
reopened to validate to complete process from the listing to the loading of some repository
vsellier closed T2577: Test gitea lister on staging environment, a subtask of T2313: Archive git.fsfe.org (Gitea), as Resolved.