Page MenuHomeSoftware Heritage
Create Task
Maniphest T4545

staging: Deploy golang stack
Closed, MigratedEdits Locked
Actions

Description

  • Package swh.loader.core v4.2
  • Edit chart to add the golang loader and lister
  • Add golang lister task to recurring listing [1]
  • Lister triggered [2]
  • T4545#92221: Let loaders ingest [3]

For information, sentry issues should pop up along the way [4]

[1]

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-golang-incremental
Created 1 tasks

Task 33419557
  Next run: today (2022-09-22T16:19:15.520781+00:00)
  Interval: 1 day, 0:00:00
  Type: list-golang-incremental
  Policy: recurring
  Args:
  Keyword args:

[2]

17:54:45 swh-scheduler@db1:5432=> select now(), lister_id, visit_type, count(*) from listed_origins where lister_id=(select id from listers where name='golang') group by lister_id, visit_type;
+-------------------------------+--------------------------------------+------------+-------+
|              now              |              lister_id               | visit_type | count |
+-------------------------------+--------------------------------------+------------+-------+
| 2022-09-22 16:29:35.206863+00 | 46b62892-60b2-4252-b37a-5c3cc84c9a6d | golang     | 98612 |
+-------------------------------+--------------------------------------+------------+-------+
(1 row)

Time: 967.805 ms
18:29:36 swh-scheduler@db1:5432=> select now(), lister_id, visit_type, count(*) from listed_origins where lister_id=(select id from listers where name='golang') group by lister_id, visit_type;
+------------------------------+--------------------------------------+------------+-------+
|             now              |              lister_id               | visit_type | count |
+------------------------------+--------------------------------------+------------+-------+
| 2022-09-22 16:29:57.37355+00 | 46b62892-60b2-4252-b37a-5c3cc84c9a6d | golang     | 99616 |
+------------------------------+--------------------------------------+------------+-------+
(1 row)

Time: 795.562 ms

[3]

│ loaders [2022-09-23 08:14:40,355: INFO/ForkPoolWorker-1] Task swh.loader.package.golang.tasks.LoadGolang[3e4aaa49-bebf-4343-9768-05c1d2ed6862] succeeded in 0.7992246079957113s: {'status': 'failed'}                                       │
│ loaders [2022-09-23 08:14:40,367: INFO/MainProcess] Task swh.loader.package.golang.tasks.LoadGolang[d7c25e0e-42a6-4ed2-b390-c9de03ad7976] received                                                                                          │
│ loaders [2022-09-23 08:14:57,376: INFO/ForkPoolWorker-1] Task swh.loader.package.golang.tasks.LoadGolang[a4db278c-4ba6-48c8-b3f2-c1e0f71043a2] succeeded in 17.01177186399582s: {'status': 'eventful', 'snapshot_id': 'db5d1dbb46a4246d4972 │
│ loaders [2022-09-23 08:14:57,398: INFO/MainProcess] Task swh.loader.package.golang.tasks.LoadGolang[860bbdb0-f7b3-4a26-aae8-bfece0c146e4] received                                                                                          │
│ loaders [2022-09-23 08:15:05,873: INFO/ForkPoolWorker-1] Task swh.loader.package.golang.tasks.LoadGolang[d7c25e0e-42a6-4ed2-b390-c9de03ad7976] succeeded in 8.471439613000257s: {'status': 'eventful', 'snapshot_id': 'c9151adbafd2216b1f85 │
│ loaders [2022-09-23 08:15:05,900: INFO/MainProcess] Task swh.loader.package.golang.tasks.LoadGolang[2e523a68-8eca-44d6-b6fb-07789bcc77d7] received                                                                                          │
│ loaders [2022-09-23 08:15:06,327: ERROR/ForkPoolWorker-1] Failed to load any release for https://pkg.go.dev/github.com/btcboost/copernicus                                                                                                  │
│ loaders [2022-09-23 08:15:06,450: INFO/ForkPoolWorker-1] Task swh.loader.package.golang.tasks.LoadGolang[860bbdb0-f7b3-4a26-aae8-bfece0c146e4] succeeded in 0.5533570959960343s: {'status': 'failed'}                                       │
│ loaders [2022-09-23 08:15:06,469: INFO/MainProcess] Task swh.loader.package.golang.tasks.LoadGolang[4d61ebd6-a8fb-4af8-b54e-c777e48e276b] received                                                                                          │
│ loaders [2022-09-23 08:15:26,138: INFO/ForkPoolWorker-1] Task swh.loader.package.golang.tasks.LoadGolang[2e523a68-8eca-44d6-b6fb-07789bcc77d7] succeeded in 19.67315485799918s: {'status': 'eventful', 'snapshot_id': '533b0f65c5af9d458d01 │

[4] https://sentry.softwareheritage.org/organizations/swh/issues/?project=9&query=is%3Aunresolved+is%3Afor_review+assigned_or_suggested%3A%5Bme%2C+none%5D+golang&statsPeriod=14d

Revisions and Commits

rDSCH Scheduling utilities
D8524rDSCH6d0b1d1e1ab5 backend: Prevent query exception when lister ids is empty

Related Objects
Search...

StatusAssignedTask
Unknown Object (Maniphest Task)
 Migratedgitlab-migrationT4124 Golang support
 Migratedgitlab-migrationT4079 Extend archive coverage
 Migratedgitlab-migrationT4545 staging: Deploy golang stack

Event Timeline

ardumont triaged this task as Normal priority.Sep 20 2022, 12:51 PM
ardumont created this task.
ardumont mentioned this in T4124: Golang support.
ardumont mentioned this in R260:64cd15e93464: staging: Deploy golang stack.Sep 22 2022, 6:15 PM
ardumont updated the task description. (Show Details)Sep 22 2022, 6:20 PM
ardumont updated the task description. (Show Details)Sep 22 2022, 6:29 PM
ardumont updated the task description. (Show Details)
ardumont added a revision: D8524: backend: Prevent query exception when lister ids is empty.Sep 23 2022, 9:50 AM
ardumont mentioned this in R260:4c967b9ed085: staging: Deploy golang stack.Sep 23 2022, 10:11 AM
ardumont changed the task status from Open to Work in Progress.Sep 23 2022, 10:17 AM
ardumont added a project: System administration.
ardumont updated the task description. (Show Details)
ardumont moved this task from Backlog to in-progress on the System administration board.
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.
ardumont added a commit: rDSCH6d0b1d1e1ab5: backend: Prevent query exception when lister ids is empty.Sep 23 2022, 10:21 AM
ardumont added a comment.Sep 28 2022, 7:20 PM

Ingestion ok

19:18:28 swh-scheduler@db1:5432=> select lo.url, lo.visit_type, last_seen, last_scheduled, last_successful, last_visit, last_visit_status from listed_origins lo inner join origin_visit_stats o on o.url=lo.url where lo.lister_id = ( select id from listers where name='golang') and last_visit_status='successful' order by last_successful desc limit 10;
+------------------------------------------------------------------+------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------+
|                               url                                | visit_type |           last_seen           |        last_scheduled         |        last_successful        |          last_visit           | last_visit_status |
+------------------------------------------------------------------+------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------+
| https://pkg.go.dev/github.com/oktalz/go-modules-demo/v2          | golang     | 2022-09-22 16:28:49.095502+00 | 2022-09-28 15:41:35.123821+00 | 2022-09-28 17:18:50.632741+00 | 2022-09-28 17:18:50.632741+00 | successful        |
| https://pkg.go.dev/github.com/kubernetes-sigs/testing_frameworks | golang     | 2022-09-22 16:28:49.095502+00 | 2022-09-28 15:41:35.123821+00 | 2022-09-28 17:15:27.283314+00 | 2022-09-28 17:15:27.283314+00 | successful        |
| https://pkg.go.dev/github.com/subtlepseudonym/version            | golang     | 2022-09-22 16:28:49.095502+00 | 2022-09-28 15:41:35.123821+00 | 2022-09-28 17:15:00.661478+00 | 2022-09-28 17:15:00.661478+00 | successful        |
| https://pkg.go.dev/github.com/subchord/go-metrics                | golang     | 2022-09-22 16:28:48.036138+00 | 2022-09-28 15:31:57.539751+00 | 2022-09-28 17:14:32.759955+00 | 2022-09-28 17:14:32.759955+00 | successful        |
| https://pkg.go.dev/github.com/ambbarees/structs                  | golang     | 2022-09-22 16:28:48.036138+00 | 2022-09-28 15:31:57.539751+00 | 2022-09-28 17:13:25.614877+00 | 2022-09-28 17:13:25.614877+00 | successful        |
| https://pkg.go.dev/github.com/csjablonkay/go-hello               | golang     | 2022-09-22 16:28:48.036138+00 | 2022-09-28 15:27:23.161176+00 | 2022-09-28 17:02:26.087178+00 | 2022-09-28 17:02:26.087178+00 | successful        |
| https://pkg.go.dev/github.com/innius/sdk                         | golang     | 2022-09-22 16:28:48.036138+00 | 2022-09-28 15:27:23.161176+00 | 2022-09-28 17:01:29.113135+00 | 2022-09-28 17:01:29.113135+00 | successful        |
| https://pkg.go.dev/github.com/dollarkillerx/obfuscation          | golang     | 2022-09-22 16:28:48.036138+00 | 2022-09-28 15:27:10.72817+00  | 2022-09-28 17:00:06.56387+00  | 2022-09-28 17:00:06.56387+00  | successful        |
| https://pkg.go.dev/github.com/pechorka/tomap                     | golang     | 2022-09-22 16:28:48.036138+00 | 2022-09-28 15:27:10.72817+00  | 2022-09-28 16:58:55.622485+00 | 2022-09-28 16:58:55.622485+00 | successful        |
| https://pkg.go.dev/github.com/googx/fdfs_client                  | golang     | 2022-09-22 16:28:48.036138+00 | 2022-09-28 15:22:47.760867+00 | 2022-09-28 16:58:22.392457+00 | 2022-09-28 16:58:22.392457+00 | successful        |
+------------------------------------------------------------------+------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------+
(10 rows)

Time: 17773.099 ms (00:17.773)
ardumont closed this task as Resolved.Sep 28 2022, 7:21 PM
ardumont claimed this task.
ardumont updated the task description. (Show Details)
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:08 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

bchauvet added a parent task: T4079: Extend archive coverage.Nov 3 2022, 10:17 AM
About · Developer information · Legal