Changeset View
Changeset View
Standalone View
Standalone View
README.md
Show First 20 Lines • Show All 103 Lines • ▼ Show 20 Lines | |||||
At this point, the archive is empty and needs to be filled with some content. | At this point, the archive is empty and needs to be filled with some content. | ||||
To do so, you can create tasks that will scrape a forge. For example, to inject | To do so, you can create tasks that will scrape a forge. For example, to inject | ||||
the code from the https://0xacab.org gitlab forge: | the code from the https://0xacab.org gitlab forge: | ||||
``` | ``` | ||||
~/swh-environment/swh-docker-dev$ docker-compose exec swh-scheduler-api \ | ~/swh-environment/swh-docker-dev$ docker-compose exec swh-scheduler-api \ | ||||
swh scheduler task add list-gitlab-full \ | swh scheduler task add list-gitlab-full \ | ||||
-p oneshot api_baseurl=https://0xacab.org/api/v4 | -p oneshot url=https://0xacab.org/api/v4 | ||||
Created 1 tasks | Created 1 tasks | ||||
Task 1 | Task 1 | ||||
Next run: just now (2018-12-19 14:58:49+00:00) | Next run: just now (2018-12-19 14:58:49+00:00) | ||||
Interval: 90 days, 0:00:00 | Interval: 90 days, 0:00:00 | ||||
Type: list-gitlab-full | Type: list-gitlab-full | ||||
Policy: oneshot | Policy: oneshot | ||||
Args: | Args: | ||||
Keyword args: | Keyword args: | ||||
api_baseurl=https://0xacab.org/api/v4 | url=https://0xacab.org/api/v4 | ||||
``` | ``` | ||||
This task will scrape the forge's project list and create subtasks to inject | This task will scrape the forge's project list and create subtasks to inject | ||||
each git repository found there. | each git repository found there. | ||||
This will take a bit af time to complete. | This will take a bit af time to complete. | ||||
To increase the speed at which git repositories are imported, you can spawn more | To increase the speed at which git repositories are imported, you can spawn more | ||||
▲ Show 20 Lines • Show All 145 Lines • ▼ Show 20 Lines | |||||
and keep it up to date. | and keep it up to date. | ||||
For example, to add a (one shot) task that will list git repos on the | For example, to add a (one shot) task that will list git repos on the | ||||
0xacab.org gitlab instance, one can do (from this git repository): | 0xacab.org gitlab instance, one can do (from this git repository): | ||||
``` | ``` | ||||
~/swh-environment/swh-docker-dev$ docker-compose exec swh-scheduler-api \ | ~/swh-environment/swh-docker-dev$ docker-compose exec swh-scheduler-api \ | ||||
swh scheduler task add list-gitlab-full \ | swh scheduler task add list-gitlab-full \ | ||||
-p oneshot api_baseurl=https://0xacab.org/api/v4 | -p oneshot url=https://0xacab.org/api/v4 | ||||
Created 1 tasks | Created 1 tasks | ||||
Task 12 | Task 12 | ||||
Next run: just now (2018-12-19 14:58:49+00:00) | Next run: just now (2018-12-19 14:58:49+00:00) | ||||
Interval: 90 days, 0:00:00 | Interval: 90 days, 0:00:00 | ||||
Type: list-gitlab-full | Type: list-gitlab-full | ||||
Policy: oneshot | Policy: oneshot | ||||
Args: | Args: | ||||
Keyword args: | Keyword args: | ||||
api_baseurl=https://0xacab.org/api/v4 | url=https://0xacab.org/api/v4 | ||||
``` | ``` | ||||
This will insert a new task in the scheduler. To list existing tasks for a | This will insert a new task in the scheduler. To list existing tasks for a | ||||
given task type: | given task type: | ||||
``` | ``` | ||||
~/swh-environment/swh-docker-dev$ docker-compose exec swh-scheduler-api \ | ~/swh-environment/swh-docker-dev$ docker-compose exec swh-scheduler-api \ | ||||
swh scheduler task list-pending list-gitlab-full | swh scheduler task list-pending list-gitlab-full | ||||
Found 1 list-gitlab-full tasks | Found 1 list-gitlab-full tasks | ||||
Task 12 | Task 12 | ||||
Next run: 2 minutes ago (2018-12-19 14:58:49+00:00) | Next run: 2 minutes ago (2018-12-19 14:58:49+00:00) | ||||
Interval: 90 days, 0:00:00 | Interval: 90 days, 0:00:00 | ||||
Type: list-gitlab-full | Type: list-gitlab-full | ||||
Policy: oneshot | Policy: oneshot | ||||
Args: | Args: | ||||
Keyword args: | Keyword args: | ||||
api_baseurl=https://0xacab.org/api/v4 | url=https://0xacab.org/api/v4 | ||||
``` | ``` | ||||
To list all existing task types: | To list all existing task types: | ||||
``` | ``` | ||||
~/swh-environment/swh-docker-dev$ docker-compose exec swh-scheduler-api \ | ~/swh-environment/swh-docker-dev$ docker-compose exec swh-scheduler-api \ | ||||
swh scheduler task-type list | swh scheduler task-type list | ||||
▲ Show 20 Lines • Show All 356 Lines • Show Last 20 Lines |