Feed Advanced Search

Advanced Search
Use Results
Edit Query
Hide Query

	Include stories about projects I am a member of.

Sep 10 2020

vsellier added a comment to T2577: Test gitea lister on staging environment.

The test of new version v0.1.4 including the fix on the the range split, the uid change and the incremental task fix is ok.

Sep 10 2020, 1:06 PM · Lister

vsellier accepted D3903: gitea.lister: Fix uid to be unique across instance.

LGTM

Sep 10 2020, 11:29 AM

vsellier accepted D3899: utils.split_range: Make computed ranges not overlap.

Tested in the docker-environment, the problem is not reproduced anymore with 5 concurrent listers.

Sep 10 2020, 10:39 AM · Lister

vsellier accepted D3898: test_utils: Migrate to pytest.

Sep 10 2020, 10:24 AM · Lister

Sep 9 2020

vsellier added a comment to T2577: Test gitea lister on staging environment.

The concurrency issue was reproduced locally on the docker environment with a concurrency of 5.

Sep 9 2020, 4:26 PM · Lister

vsellier committed rSPSITEcb56ae71235b: Deploy svn loader on staging (authored by vsellier).

Deploy svn loader on staging

Sep 9 2020, 1:59 PM

vsellier added a comment to T2577: Test gitea lister on staging environment.

I have tested to create a list-gitea-incremental task but it fails to but this time with another exception relative to an unexpected "sort" parameter : https://sentry.softwareheritage.org/share/issue/b0119b56f24347bcb58ac28c68685c62/

Sep 9 2020, 11:43 AM · Lister

vsellier added a comment to T2577: Test gitea lister on staging environment.

the configuration is deployed and the listers were restarted.

Sep 9 2020, 11:17 AM · Lister

vsellier committed rSPSITEb51b0ceb29b0: lister configuration : add gitea lister tasks (authored by vsellier).

lister configuration : add gitea lister tasks

Sep 9 2020, 10:43 AM

vsellier closed D3896: lister configuration : add gitea lister tasks.

Sep 9 2020, 10:43 AM

vsellier added a comment to T2577: Test gitea lister on staging environment.

For info, on my desktop with the docker environment, with a limit of 100, the lister takes 3s to list the complete codeberg forge :

swh-lister_1                    | [2020-09-08 18:33:19,259: INFO/ForkPoolWorker-1] Task swh.lister.gitea.tasks.RangeGiteaLister[363e0b30-b13a-4f62-bd31-9847dfe62450] succeeded in 3.7196799100056523s: {'status': 'eventful'}

Sep 9 2020, 10:36 AM · Lister

vsellier added a comment to T2358: Deploy launchpad lister on staging.

The task ran in 30mn (1887s):

Sep 08 13:45:34 worker1 python3[237586]: [2020-09-08 13:45:34,851: INFO/ForkPoolWorker-4] Task swh.lister.launchpad.tasks.FullLaunchpadLister[73e298be-aeda-4882-b52d-dfe5a2ec316c] succeeded in 1887.75128286588s: {'status': 'eventful'}

Sep 9 2020, 10:25 AM · System administration, Lister, Archive coverage

vsellier added a comment to T2577: Test gitea lister on staging environment.

The data model does't need to be created because it was already done in T2358
The task is created :

swhscheduler@scheduler0:~$ swh  scheduler --config-file /etc/softwareheritage/scheduler.yml task add --policy oneshot list-gitea-full url=https://codeberg.org/api/v1/ limit=100
WARNING:swh.core.cli:Could not load subcommand storage: No module named 'swh.journal'
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
Created 1 tasks

Sep 9 2020, 9:30 AM · Lister

Sep 8 2020

vsellier added a comment to T2577: Test gitea lister on staging environment.

task-type registered :

swhscheduler@scheduler0:/etc/softwareheritage/backend$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type register -p lister.gitea
WARNING:swh.core.cli:Could not load subcommand storage: No module named 'swh.journal'
INFO:swh.core.config:Loading config file /etc/softwareheritage/scheduler.yml
INFO:swh.scheduler.cli.task_type:Loading entrypoint for plugin lister.gitea
INFO:swh.scheduler.cli.task_type:Create task type list-gitea-full in scheduler
INFO:swh.scheduler.cli.task_type:Create task type list-gitea-incremental in scheduler

Sep 8 2020, 5:06 PM · Lister

vsellier updated the diff for D3896: lister configuration : add gitea lister tasks.

fix mix with launchpad tasks

Sep 8 2020, 4:56 PM

vsellier created D3896: lister configuration : add gitea lister tasks.

Sep 8 2020, 4:48 PM

vsellier added a revision to T2577: Test gitea lister on staging environment: D3896: lister configuration : add gitea lister tasks.

Sep 8 2020, 4:48 PM · Lister

vsellier changed the status of T2577: Test gitea lister on staging environment, a subtask of T2313: Archive git.fsfe.org (Gitea), from Open to Work in Progress.

Sep 8 2020, 4:35 PM · Archive coverage, Lister

vsellier changed the status of T2577: Test gitea lister on staging environment from Open to Work in Progress.

Sep 8 2020, 4:35 PM · Lister

vsellier claimed T2577: Test gitea lister on staging environment.

Sep 8 2020, 4:35 PM · Lister

vsellier added a subtask for T2313: Archive git.fsfe.org (Gitea): T2577: Test gitea lister on staging environment.

Sep 8 2020, 4:34 PM · Archive coverage, Lister

vsellier added a parent task for T2577: Test gitea lister on staging environment: T2313: Archive git.fsfe.org (Gitea).

Sep 8 2020, 4:34 PM · Lister

vsellier created T2577: Test gitea lister on staging environment.

Sep 8 2020, 4:34 PM · Lister

vsellier closed T2358: Deploy launchpad lister on staging, a subtask of T1734: Create a Lister for launchpad.net, as Resolved.

Sep 8 2020, 3:57 PM · Lister, Archive coverage

vsellier closed T2358: Deploy launchpad lister on staging as Resolved.

The launchpad lister (v0.1.2) is deployed and running on staging

Sep 8 2020, 3:57 PM · System administration, Lister, Archive coverage

vsellier closed T2574: "No attribute get_task_types" error on scheduler's task_type cli as Invalid.

A parameter was missing in the call :

swhscheduler@scheduler0:~$ swh scheduler --config-file /etc/softwareheritage/scheduler.yml task-type list

Sep 8 2020, 2:42 PM · Scheduling utilities

vsellier committed rDLS74ca3d0b8732: Launchpad: rename task name to match conventions (authored by vsellier).

Launchpad: rename task name to match conventions

Sep 8 2020, 2:39 PM

vsellier closed D3887: Launchpad: rename task name to match conventions.

Sep 8 2020, 2:39 PM

vsellier added a revision to T2358: Deploy launchpad lister on staging: D3887: Launchpad: rename task name to match conventions.

Sep 8 2020, 2:25 PM · System administration, Lister, Archive coverage

vsellier created D3887: Launchpad: rename task name to match conventions.

Sep 8 2020, 2:25 PM

vsellier updated the task description for T2574: "No attribute get_task_types" error on scheduler's task_type cli.

Sep 8 2020, 12:26 PM · Scheduling utilities

vsellier created T2574: "No attribute get_task_types" error on scheduler's task_type cli.

Sep 8 2020, 12:20 PM · Scheduling utilities

vsellier accepted D3884: lister configuration: Add launchpad lister tasks.

Sep 8 2020, 10:59 AM

vsellier added a comment to D3884: lister configuration: Add launchpad lister tasks.

Octocatalog-diff test :

➜  puppet-environment git:(master) ✗ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to arcpatch_D3884 worker0.internal.staging.swh.network
Found host worker0.internal.staging.swh.network
WARN     -> Environment "arcpatch-D3884" contained non-word characters, correcting name to arcpatch_D3884
Cloning into '/tmp/swh-ocd.e8lQMoZ0/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.e8lQMoZ0/environments/arcpatch_D3884/data/private'...
done.
*** Running octocatalog-diff on host worker0.internal.staging.swh.network
I, [2020-09-08T10:45:12.720125 #4652]  INFO -- : Catalogs compiled for worker0.internal.staging.swh.network
I, [2020-09-08T10:45:13.707209 #4652]  INFO -- : Diffs computed for worker0.internal.staging.swh.network
diff origin/production/worker0.internal.staging.swh.network current/worker0.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/lister.yml] =>
   parameters =>
     content =>
      @@ -24,4 +24,6 @@
         - swh.lister.gitlab.tasks.FullGitLabRelister
         - swh.lister.gnu.tasks.GNUListerTask
      +  - swh.lister.launchpad.tasks.FullLaunchpadLister
      +  - swh.lister.launchpad.tasks.IncrementalLaunchpadLister
         - swh.lister.npm.tasks.NpmListerTask
         - swh.lister.phabricator.tasks.FullPhabricatorLister
*******************************************
*** End octocatalog-diff on worker0.internal.staging.swh.network

Sep 8 2020, 10:58 AM

Sep 4 2020

vsellier added a comment to T1414: Set up an inventory app.

Wikimedia is using netbox as the source of trust in their infrastructure and puppet is configuring the facts from it. It's not exactly the same use case we want as we would like to have netbox automatically provisioned.

Sep 4 2020, 5:14 PM · System administration, Sprint 2018 12

vsellier added a comment to T1414: Set up an inventory app.

and their documentation : https://wikitech.wikimedia.org/wiki/Netbox

Sep 4 2020, 11:20 AM · System administration, Sprint 2018 12

vsellier added a comment to T1414: Set up an inventory app.

A docker-compose is available to easily test netbox : https://github.com/netbox-community/netbox-docker
This is the puppet configuration used at wikimedia : https://gerrit.wikimedia.org/r/c/operations/puppet/+/387880/

Sep 4 2020, 11:17 AM · System administration, Sprint 2018 12

vsellier committed rSPSITE14b15b2f4f63: Allow vsellier to use sudo on the servers (authored by vsellier).

Allow vsellier to use sudo on the servers

Sep 4 2020, 9:59 AM

vsellier committed rSPSITEa44916e4edc2: Allow vsellier to use sudo on the servers (authored by vsellier).

Allow vsellier to use sudo on the servers