diff --git a/sysadm/deployment/add-forge-now.rst b/sysadm/deployment/add-forge-now.rst new file mode 100644 --- /dev/null +++ b/sysadm/deployment/add-forge-now.rst @@ -0,0 +1,144 @@ +.. _process-add-forge-now-request: + +Process Add-forge-now request +============================= + +.. admonition:: Intended audience + :class: important + + sysadm staff members + +That process is semi-automatic for the moment. Referencing those steps is a kickstarter +for automation. + + +Introduction +------------ + +A forge ticket (`see for example the git.afpy.org ticket +`_) should +have been opened. + +Meaning the `moderation process has been started +`_. Hence, the +upstream forge has been notified we will start the ingestion soon. + + +.. _add-forge-now-testing-on-staging: + +Testing on staging +------------------ + +To make sure everything we can ingest that forge, we will start first by testing out a +subset of that forge listing on staging. That will ensure we have the right amounts of +information to deal with it. + +On a staging node (scheduler0.internal.staging.swh.network). + +.. code:: + + swh lister run \ + -l \ + -p oneshot url= \ + max_pages=1 max_origins_per_page=15 enable_origins=False + + +For example, for the git.afpy.org gitea instance: + +.. code:: + + swh@lister-all-bd469fc7-9hq9h:~$ swh lister run \ + -l gitea \ + url=https://git.afpy.org/api/v1/ \ + max_pages=1 max_origins_per_page=15 enable_origins=False + + INFO:swh.lister.pattern:Max origins per page set, truncated 36 page results down to 15 + INFO:swh.lister.pattern:Disabling origins before sending them to the scheduler + INFO:swh.lister.pattern:Reached page limit of 1, terminating + + +Ensure the run got referenced in the staging scheduler db: + +.. code:: + + 2022-12-06 11:50:17 swh-scheduler@db1:5432 λ select * from listers where name='gitea' and instance_name='git.afpy.org'; + +--------------------------------------+-------+---------------+-------------------------------+--------------------------------------------------------+-------------------------------+ + | id | name | instance_name | created | current_state | updated | + +--------------------------------------+-------+---------------+-------------------------------+--------------------------------------------------------+-------------------------------+ + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | gitea | git.afpy.org | 2022-12-06 10:47:46.975571+00 | {"last_seen_repo_id": 12, "last_seen_next_link": null} | 2022-12-06 10:47:48.180968+00 | + +--------------------------------------+-------+---------------+-------------------------------+--------------------------------------------------------+-------------------------------+ + (1 row) + + Time: 4.109 ms + + 2022-12-06 11:50:24 swh-scheduler@db1:5432 λ select * from listed_origins where lister_id = (select id from listers where name='gitea' and instance_name='git.afpy.org') ; + +--------------------------------------+-----------------------------------------------------------+------------+------------------------+---------+-------------------------------+-------------------------------+------------------------+ + | lister_id | url | visit_type | extra_loader_arguments | enabled | first_seen | last_seen | last_update | + +--------------------------------------+-----------------------------------------------------------+------------+------------------------+---------+-------------------------------+-------------------------------+------------------------+ + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/AFPy/afpy.org.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-11-19 09:15:30+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/foxmask/baeuda.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-10-25 12:51:18+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/fcode/boilerplate-python.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-10-27 22:07:07+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/fcode/delarte.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-12-06 00:39:26+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/foxmask/django-th.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-11-05 23:03:10+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/foxmask/dj-diabetes.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-11-05 23:03:50+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/freezed/free_zed.gitlab.io.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-11-17 10:29:07+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/AFPy/gestion.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-12-03 20:39:47+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/mdk/git-xss-locator.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-10-12 14:57:03+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/HARFANG3D/harfang3d.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-11-25 06:56:39+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/mdk/hkis-au-campus-du-libre-2022.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-11-27 15:39:42+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/mdk/hl-vt100.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-11-01 16:19:41+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/mdk/horrlang.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-10-31 16:52:08+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/mdk/infra.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-11-18 09:23:54+00 | + | d07d1c90-5016-4ab6-91ac-3300f8eb4fc6 | https://git.afpy.org/AFPy/infra.git | git | {} | f | 2022-12-06 10:47:47.535655+00 | 2022-12-06 10:47:47.535655+00 | 2022-12-01 09:31:50+00 | + +--------------------------------------+-----------------------------------------------------------+------------+------------------------+---------+-------------------------------+-------------------------------+------------------------+ + (15 rows) + + Time: 1225.399 ms (00:01.225) + +Trigger the ingestion for those few origins: + +.. code:: + + ~/addforgenow/addforge-now-schedule-with-url-lister-and-optional-visit-type-and-queue.sh gitea git.afpy.org + Tue Dec 6 17:16:20 UTC 2022 scheduling git origins with policy never_visited_oldest_update_first to queue add_forge_now:swh.loader.git.tasks.UpdateGitRepository for lister gitea (instance git.afpy.org) (tablesample 1) + 100 slots available in celery queue + 15 visits to send to celery + +Give it some time to properly ingest some of those origins, and then check they were +ingested: + +.. code:: + + 2022-12-06 18:47:43 swh-scheduler@db1:5432 λ select * from origin_visit_stats where visit_type='git' and url like 'https://git.afpy.org%'; + +-----------------------------------------------------------+------------+--------------------------------------------+-------------------------------+---------------------------+----------------------+-------------------+-------------------------------+-------------------------------+-------------------+ + | url | visit_type | last_snapshot | last_scheduled | next_visit_queue_position | next_position_offset | successive_visits | last_successful | last_visit | last_visit_status | + +-----------------------------------------------------------+------------+--------------------------------------------+-------------------------------+---------------------------+----------------------+-------------------+-------------------------------+-------------------------------+-------------------+ + | https://git.afpy.org/fcode/boilerplate-python.git | git | \x812e8ff75a6424c51cfd22a9202503f21cccf13d | 2022-12-06 17:16:22.384523+00 | 167043832784 | 4 | 1 | 2022-12-06 17:17:28.58761+00 | 2022-12-06 17:17:28.58761+00 | successful | + | https://git.afpy.org/AFPy/infra.git | git | \x2696466a836f7411e35d3802d0de31fb5a3a1c5d | 2022-12-06 17:16:22.384523+00 | 167043839992 | 4 | 1 | 2022-12-06 17:27:31.499761+00 | 2022-12-06 17:27:31.499761+00 | successful | + | https://git.afpy.org/mdk/git-xss-locator.git | git | \xa55b9be6a67299b668084a68c2775081f3cfe255 | 2022-12-06 17:16:22.384523+00 | 167043822872 | 4 | 1 | 2022-12-06 17:17:11.161265+00 | 2022-12-06 17:17:11.161265+00 | successful | + | https://git.afpy.org/HARFANG3D/harfang3d.git | git | \x140a4581ac338e0196f71634eb5c8a388afaf419 | 2022-12-06 17:16:22.384523+00 | 167043836254 | 4 | 1 | 2022-12-06 17:25:33.642212+00 | 2022-12-06 17:25:33.642212+00 | successful | + | https://git.afpy.org/mdk/horrlang.git | git | \x88039a3ad3630e246880707ac7b5052879f78810 | 2022-12-06 17:16:22.384523+00 | 167043844551 | 4 | 1 | 2022-12-06 17:17:31.745645+00 | 2022-12-06 17:17:31.745645+00 | successful | + | https://git.afpy.org/mdk/infra.git | git | \x6d4974ee86a729ad36c512b8ca5caaa4314c2106 | 2022-12-06 17:16:22.384523+00 | 167043829585 | 4 | 1 | 2022-12-06 17:20:56.603273+00 | 2022-12-06 17:20:56.603273+00 | successful | + | https://git.afpy.org/foxmask/dj-diabetes.git | git | \x7c7a184719d6b66a93e283b6711b79677c514fb9 | 2022-12-06 17:16:22.384523+00 | 167043844794 | 4 | 1 | 2022-12-06 17:18:30.15001+00 | 2022-12-06 17:18:30.15001+00 | successful | + | https://git.afpy.org/foxmask/baeuda.git | git | \x07d1cfe8e163562f035473c420eb45acb48a359d | 2022-12-06 17:16:22.384523+00 | 167043853107 | 4 | 1 | 2022-12-06 17:17:34.682106+00 | 2022-12-06 17:17:34.682106+00 | successful | + | https://git.afpy.org/mdk/hl-vt100.git | git | \x7a649953bce78849007625bddc8dfc69b152f1a8 | 2022-12-06 17:16:22.384523+00 | 167043834780 | 4 | 1 | 2022-12-06 17:17:38.292726+00 | 2022-12-06 17:17:38.292726+00 | successful | + | https://git.afpy.org/freezed/free_zed.gitlab.io.git | git | \x8a5941dd3ec9a4350ed8fadfd3486531fe57c4ff | 2022-12-06 17:16:22.384523+00 | 167043839901 | 4 | 1 | 2022-12-06 17:20:07.322192+00 | 2022-12-06 17:20:07.322192+00 | successful | + | https://git.afpy.org/AFPy/afpy.org.git | git | \x8e7d398a34ed2a2a084397660733e08216770daa | 2022-12-06 17:16:22.384523+00 | 167043853574 | 4 | 1 | 2022-12-06 17:23:14.834908+00 | 2022-12-06 17:23:14.834908+00 | successful | + | https://git.afpy.org/AFPy/gestion.git | git | \xb8464eef6f90f6d2e836e25122c9b506b01f81f4 | 2022-12-06 17:16:22.384523+00 | 167043841956 | 4 | 1 | 2022-12-06 17:31:49.301271+00 | 2022-12-06 17:31:49.301271+00 | successful | + | https://git.afpy.org/mdk/hkis-au-campus-du-libre-2022.git | git | \x0a2116e2b561f881ca9d6b5ef73b20fddbcd5f48 | 2022-12-06 17:16:22.384523+00 | 167043833627 | 4 | 1 | 2022-12-06 17:25:38.914527+00 | 2022-12-06 17:25:38.914527+00 | successful | + | https://git.afpy.org/foxmask/django-th.git | git | \x004bc22e7b2e2a1158fceed1a86a7209936859c8 | 2022-12-06 17:16:22.384523+00 | 167043824244 | 4 | 1 | 2022-12-06 17:27:21.051499+00 | 2022-12-06 17:27:21.051499+00 | successful | + | https://git.afpy.org/fcode/delarte.git | git | \x76517eca13c0858067d391e082b86033d16ab17e | 2022-12-06 17:16:22.384523+00 | 167043856375 | 4 | 1 | 2022-12-06 17:27:37.400123+00 | 2022-12-06 17:27:37.400123+00 | successful | + +-----------------------------------------------------------+------------+--------------------------------------------+-------------------------------+---------------------------+----------------------+-------------------+-------------------------------+-------------------------------+-------------------+ + (15 rows) + + Time: 933.779 ms + + Now that everything went fine, schedule properly the forge in production. + +.. _add-forge-now-deploying-on-production: + +Deploying on production +----------------------- + +Now that we make sure we can ingest a subset, we can deploy the full listing and +ingestion on that forge. diff --git a/sysadm/deployment/index.rst b/sysadm/deployment/index.rst --- a/sysadm/deployment/index.rst +++ b/sysadm/deployment/index.rst @@ -11,3 +11,4 @@ howto-debian-packaging jenkins argocd + process-add-forge-now-request