Jan 8 2023
Jan 8 2023
gitlab-migration changed the status of T4451: Fetch extrinsic metadata from Gitea/Gogs, a subtask of T4457: Index metadata from Gitea/Gogs, from Resolved to Migrated.
gitlab-migration changed the status of T4451: Fetch extrinsic metadata from Gitea/Gogs from Resolved to Migrated.
gitlab-migration changed the status of T4457: Index metadata from Gitea/Gogs from Resolved to Migrated.
gitlab-migration changed the status of T4423: Gogs pagination API breaks because of fatal repos, a subtask of T1721: Implementation of Gogs Lister, from Resolved to Migrated.
gitlab-migration changed the status of T4423: Gogs pagination API breaks because of fatal repos from Resolved to Migrated.
Dec 1 2022
Dec 1 2022
Nov 3 2022
Nov 3 2022
bchauvet added a parent task for T4478: Deploy Gogs lister to staging: T4079: Extend archive coverage.
Oct 24 2022
Oct 24 2022
vlorentz closed T4451: Fetch extrinsic metadata from Gitea/Gogs, a subtask of T4457: Index metadata from Gitea/Gogs, as Resolved.
Oct 19 2022
Oct 19 2022
gitlab-migration closed T4606: Deploy swh-indexer v2.7.0, a subtask of T4457: Index metadata from Gitea/Gogs, as Migrated.
gitlab-migration closed T4605: Deploy swh-loader-metadata v0.0.3, a subtask of T4451: Fetch extrinsic metadata from Gitea/Gogs, as Migrated.
gitlab-migration changed the status of T4478: Deploy Gogs lister to staging, a subtask of T1721: Implementation of Gogs Lister, from Resolved to Migrated.
gitlab-migration changed the status of T4478: Deploy Gogs lister to staging from Resolved to Migrated.
Oct 5 2022
Oct 5 2022
ardumont added a comment to T4478: Deploy Gogs lister to staging.
Thanks to @anlambert, a possible gogs public instance [1] to check again.
vlorentz added a subtask for T4457: Index metadata from Gitea/Gogs: T4606: Deploy swh-indexer v2.7.0.
Sep 29 2022
Sep 29 2022
ardumont updated the task description for T1721: Implementation of Gogs Lister.
ardumont closed T4423: Gogs pagination API breaks because of fatal repos, a subtask of T1721: Implementation of Gogs Lister, as Resolved.
That's been worked around in the lister to skip those 500 which were breaking the pagination listing.
There is no reply to the upstream bug yet.
Sep 23 2022
Sep 23 2022
ardumont updated the task description for T1721: Implementation of Gogs Lister.
ardumont added a comment to T4478: Deploy Gogs lister to staging.
I either remembered an old stateless implementation or i remember another
implementation altogether.
ardumont closed T4478: Deploy Gogs lister to staging, a subtask of T1721: Implementation of Gogs Lister, as Resolved.
ardumont added a comment to T4478: Deploy Gogs lister to staging.
I'll unstuck the situation by resetting that state.
ardumont added a comment to T4478: Deploy Gogs lister to staging.
But on my side, the "stateless" lister starts from the problematic page (17), and then
plainly stops... [2].Note that there is a misbehavior in terms here for that "stateless" typed lister. If it
is indeed a stateless lister, it should start from scratch each time it runs. If it has
an incremental behavior (it seems so from that described behavior) then it should be
probably typed differently.
ardumont added a comment to T4478: Deploy Gogs lister to staging.
ah i think i got it...
ardumont added a comment to T4478: Deploy Gogs lister to staging.
As a shot in the dark, i've changed the list task arguments [1] but i'd be surprised if that'd change anything...
ardumont added a comment to T4478: Deploy Gogs lister to staging.
I noticed your remarks but that's not what's happening in staging [3] for some unknown reason [1]
Sep 21 2022
Sep 21 2022
anlambert added a comment to T4478: Deploy Gogs lister to staging.
Nevertheless, they are less origins in the scheduler db, maybe some get listed more than once, need to check.
anlambert added a comment to T4478: Deploy Gogs lister to staging.
Just tested the lister in docker and I got a lot more origins listed:
docker-swh-lister-1 | [2022-09-21 12:24:57,583: INFO/MainProcess] Task swh.lister.gogs.tasks.FullGogsRelister[2b0ba017-4917-44dc-9511-e860bb431322] received
docker-swh-lister-1 | [2022-09-21 12:24:57,584: DEBUG/ForkPoolWorker-1] Loading config file /lister.yml
docker-swh-lister-1 | [2022-09-21 12:24:57,594: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search with params {'limit': 50, 'page': 1}
docker-swh-lister-1 | [2022-09-21 12:24:58,556: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=2 with params {}
docker-swh-lister-1 | [2022-09-21 12:24:58,791: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=3 with params {}
docker-swh-lister-1 | [2022-09-21 12:24:59,018: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=4 with params {}
docker-swh-lister-1 | [2022-09-21 12:24:59,263: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=5 with params {}
docker-swh-lister-1 | [2022-09-21 12:24:59,580: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=6 with params {}
docker-swh-lister-1 | [2022-09-21 12:24:59,845: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=7 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:00,094: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=8 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:00,323: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=9 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:00,560: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=10 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:00,787: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=11 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:01,020: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=12 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:01,324: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=13 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:01,794: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=14 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:02,191: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=15 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:02,549: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=16 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:02,967: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=17 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:03,250: WARNING/ForkPoolWorker-1] Unexpected HTTP status code 500 on https://try.gogs.io/api/v1/repos/search?page=17: b''
docker-swh-lister-1 | [2022-09-21 12:25:03,250: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=18 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:03,515: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=19 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:03,765: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=20 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:04,078: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=21 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:04,444: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=22 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:04,695: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=23 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:04,948: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=24 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:05,262: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=25 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:05,814: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=26 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:06,184: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=27 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:06,454: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=28 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:06,688: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=29 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:06,935: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=30 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:07,247: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=31 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:07,500: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=32 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:07,734: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=33 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:08,064: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=34 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:08,337: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=35 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:08,581: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=36 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:08,818: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=37 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:09,057: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=38 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:09,289: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=39 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:09,531: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=40 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:09,756: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=41 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:09,999: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=42 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:10,213: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=43 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:10,444: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=44 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:10,689: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=45 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:10,914: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=46 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:11,163: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=47 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:11,390: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=48 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:11,710: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=49 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:12,068: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=50 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:12,342: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=51 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:12,619: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=52 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:12,862: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=53 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:13,169: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=54 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:13,549: DEBUG/ForkPoolWorker-1] Fetching URL https://try.gogs.io/api/v1/repos/search?page=55 with params {}
docker-swh-lister-1 | [2022-09-21 12:25:13,827: DEBUG/ForkPoolWorker-1] Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@0f4428ad1388', 'copyright': 'Copyright (C) 2007-2018 Pivotal Software, Inc.', 'information': 'Licensed under the MPL. See http://www.rabbitmq.com/', 'platform': 'Erlang/OTP 19.2.1', 'product': 'RabbitMQ', 'version': '3.6.16'}, mechanisms: [b'AMQPLAIN', b'PLAIN'], locales: ['en_US']
docker-swh-lister-1 | [2022-09-21 12:25:13,828: DEBUG/ForkPoolWorker-1] using channel_id: 1
docker-swh-lister-1 | [2022-09-21 12:25:13,831: DEBUG/ForkPoolWorker-1] Channel open
docker-swh-lister-1 | [2022-09-21 12:25:13,833: INFO/ForkPoolWorker-1] Task swh.lister.gogs.tasks.FullGogsRelister[2b0ba017-4917-44dc-9511-e860bb431322] succeeded in 16.23296177299926s: {'pages': 55, 'origins': 575}anlambert added a comment to T4478: Deploy Gogs lister to staging.
I was wondering why the number of listed repos was so low, turns out gogs has an option to not list a repository through the API when creating it:
ardumont moved T4478: Deploy Gogs lister to staging from code-review/await-feedback/pause to deployed/landed/monitoring on the System administration board.
ardumont updated the task description for T4478: Deploy Gogs lister to staging.
Sep 20 2022
Sep 20 2022
ardumont added a comment to T4478: Deploy Gogs lister to staging.
@ardumont because of T4423 we will face 500 for https://try.gogs.io/api/v1/repos/search?page=17 no matter how many times we retry (until it is fixed by Gogs maintainers).
KShivendu added a comment to T4478: Deploy Gogs lister to staging.
@ardumont because of T4423 we will face 500 for https://try.gogs.io/api/v1/repos/search?page=17 no matter how many times we retry (until it is fixed by Gogs maintainers). So should we not ignore the page and move on to the next one?
ardumont updated subscribers of T4478: Deploy Gogs lister to staging.
try.gogs.io being down must have been a coincidence last week.
ardumont updated the task description for T4478: Deploy Gogs lister to staging.
Sep 16 2022
Sep 16 2022
KShivendu added a comment to T4478: Deploy Gogs lister to staging.
thanks. will try to reproduce this locally.
Sep 13 2022
Sep 13 2022
ardumont moved T4478: Deploy Gogs lister to staging from in-progress to code-review/await-feedback/pause on the System administration board.
ardumont added a comment to T4478: Deploy Gogs lister to staging.
It appeared to not be doing much [0]. Although, T4533 got raised and [1] went down a bit earlier.
I've stopped the lister in the mean time.
vlorentz added a revision to T4457: Index metadata from Gitea/Gogs: D8460: Add Gitea metadata mapping.
ardumont updated the task description for T4478: Deploy Gogs lister to staging.
ardumont changed the status of T4478: Deploy Gogs lister to staging, a subtask of T1721: Implementation of Gogs Lister, from Open to Work in Progress.
ardumont updated the task description for T4478: Deploy Gogs lister to staging.
ardumont updated the task description for T4478: Deploy Gogs lister to staging.
ardumont updated the task description for T4478: Deploy Gogs lister to staging.