Page MenuHomeSoftware Heritage

Unstuck running save code now origins
Closed, ResolvedPublic

Description

The origins whose status is running (and scheduled a long time ago) may be stuck, we need to reschedule their associated task id:

11:36:59 swh-web@belvedere:5432=> select request_date, origin_url, status, loading_task_status, loading_task_id from save_origin_request where loading_task_status='running' and request_date < now() - interval '1 month';
select request_date, origin_url, status, loading_task_status, loading_task_id from save_origin_request where loading_task_status='running' and request_date < now() - interval '1 month';
+-------------------------------+--------------------------------------------------------------+----------+---------------------+-----------------+
|         request_date          |                          origin_url                          |  status  | loading_task_status | loading_task_id |
+-------------------------------+--------------------------------------------------------------+----------+---------------------+-----------------+
| 2021-03-22 07:03:38.046+00    | https://github.com/kusl/wgeteveryday                         | accepted | running             |       378218248 |
| 2021-04-01 16:57:50.205+00    | https://scm.gforge.inria.fr/anonscm/git/simty/simty.git      | accepted | running             |       379069991 |
| 2021-04-20 08:52:13.105092+00 | https://github.com/coreutils/gnulib                          | accepted | running             |       380606658 |
| 2021-04-20 11:40:21.612664+00 | https://github.com/unitystation/unitystation                 | accepted | running             |       380617630 |
| 2021-09-26 15:25:37.040968+00 | http://svn.code.sf.net/p/sauerbraten/code                    | accepted | running             |       400154016 |
| 2021-10-19 11:43:55.370658+00 | https://gitlab.com/inkscape/inkscape                         | accepted | running             |       401116302 |
| 2021-07-17 22:43:29.870783+00 | https://github.com/keybase/client                            | accepted | running             |       396435047 |
| 2021-05-15 06:09:05.96959+00  | https://git.libreoffice.org/translations/                    | accepted | running             |       381569600 |
| 2021-05-23 12:18:02.213604+00 | https://git.savannah.gnu.org/git/gnulib.git                  | accepted | running             |       381570771 |
| 2021-07-26 12:59:50.038714+00 | https://svn.r-project.org/R-dev-web/trunk/                   | accepted | running             |       396985198 |
| 2021-07-26 12:54:25.784339+00 | svn://svn.code.sf.net/p/codeblocks/code/trunk                | accepted | running             |       396984987 |
| 2021-08-10 09:31:51.290321+00 | https://github.com/CocoaPods/Specs                           | accepted | running             |       397492293 |
| 2021-08-21 08:06:47.263554+00 | https://github.com/404-not-find/client                       | accepted | running             |       398036169 |
| 2021-10-22 13:32:29.289308+00 | http://floppsie.comp.glam.ac.uk/gm2                          | accepted | running             |       401265256 |
| 2021-06-24 18:56:00.441183+00 | https://anonhg.netbsd.org/pkgsrc-public/                     | accepted | running             |       381575556 |
| 2021-06-24 18:55:55.386673+00 | https://anonhg.netbsd.org/pkgsrc-draft/                      | accepted | running             |       381575557 |
| 2021-06-24 18:55:51.53746+00  | https://anonhg.netbsd.org/pkgsrc/                            | accepted | running             |       381575558 |
| 2021-06-24 18:55:41.589663+00 | https://anonhg.netbsd.org/src-public/                        | accepted | running             |       381575560 |
| 2021-06-24 18:55:35.63743+00  | https://anonhg.netbsd.org/src-draft/                         | accepted | running             |       381575561 |
| 2021-06-24 16:53:31.099868+00 | git://git.archlinux.org/svntogit/community.git               | accepted | running             |       381575242 |
| 2021-06-22 21:56:29.522544+00 | https://github.com/jlippold/tweakCompatible                  | accepted | running             |       381575019 |
| 2021-06-29 17:08:37.262648+00 | git://git.archlinux.org/svntogit/packages.git                | accepted | running             |       381575775 |
| 2021-09-09 20:13:05.170087+00 | https://source.puri.sm/Librem5/linux-next.git                | accepted | running             |       399230899 |
| 2020-01-24 00:29:45.32+00     | https://github.com/CambridgeSemiticsLab/BH_time_collocations | accepted | running             |       269843604 |
| 2021-11-10 21:02:01.294182+00 | https://git.texlive.info/texlive                             | accepted | running             |       402275446 |
| 2021-12-10 14:02:24.992009+00 | https://github.com/godotengine/godot                         | accepted | running             |       403530429 |
+-------------------------------+--------------------------------------------------------------+----------+---------------------+-----------------+
(26 rows)

Time: 255.180 ms

Related Objects

Event Timeline

ardumont renamed this task from Unstuck running save code now origin to Unstuck running save code now origins.Thu, Jan 13, 11:34 AM
ardumont triaged this task as High priority.
ardumont created this task.
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont changed the task status from Open to Work in Progress.Mon, Jan 17, 12:22 PM
  • Those tasks were updated with a status failed [1] [2]
  • Their associated scheduler task id where archived recently [3]
  • They have been rescheduled through the save code now cli [4]
  • Their ingestion is ongoing and their associated status should update once done

[1] Those were most likely oom killed since there has been no update on their status.

[2]

12:05:17 swh-web@belvedere:5432=> update save_origin_request set loading_task_status='failed' where id in (71357, 74433, 75668, 92802, 95563, 86401, 78750, 79867, 87538, 87536, 88780, 89569, 95921, 84095, 84094, 84093, 84091, 84090, 84080, 83883, 84583, 91859,  6930, 98106, 01491, 02346);
UPDATE 26
Time: 60.416 ms

[3]

13:48:58 softwareheritage-scheduler@belvedere:5432=> select * from task where id in (378218248, 379069991, 380606658, 380617630, 400154016, 401116302, 396435047, 381569600, 381570771, 396985198, 396984987, 397492293, 398036169, 401265256, 381575556, 381575557, 381575558, 381575560, 381575561, 381575242, 381575019, 381575775, 399230899, 269843604, 402275446, 403530429);
(0 rows)

[4]

$ cat save-code-now.csv | swh web --config-file ~/.config/swh/global.yml save submit-request
[{"origin_url": "https://scm.gforge.inria.fr/anonscm/git/simty/simty.git", "visit_type": "git", "save_request_date": "2022-01-17T11:32:18.234774+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069622, "note": null}, {"origin_url": "https://github.com/unitystation/unitystation", "visit_type": "git", "save_request_date": "2022-01-17T11:32:18.784397+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069623, "note": null}, {"origin_url": "http://svn.code.sf.net/p/sauerbraten/code", "visit_type": "svn", "save_request_date": "2022-01-17T11:32:19.417732+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069624, "note": null}, {"origin_url": "https://gitlab.com/inkscape/inkscape", "visit_type": "git", "save_request_date": "2022-01-17T11:32:19.767342+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069625, "note": null}, {"origin_url": "https://github.com/keybase/client", "visit_type": "git", "save_request_date": "2022-01-17T11:32:20.202335+00:00", "save_request_status": "accepted", "save_task_status": "running", "visit_status": "created", "visit_date": "2022-01-17T11:32:21.457692+00:00", "loading_task_id": 405069626, "note": null}, {"origin_url": "https://git.libreoffice.org/translations", "visit_type": "git", "save_request_date": "2022-01-17T11:32:24.200399+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069629, "note": null}, {"origin_url": "https://git.savannah.gnu.org/git/gnulib.git", "visit_type": "git", "save_request_date": "2022-01-17T11:32:24.551241+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069631, "note": null}, {"origin_url": "https://svn.r-project.org/R-dev-web/trunk", "visit_type": "svn", "save_request_date": "2022-01-17T11:32:25.119235+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069632, "note": null}, {"origin_url": "svn://svn.code.sf.net/p/codeblocks/code/trunk", "visit_type": "svn", "save_request_date": "2022-01-17T11:32:25.331133+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069633, "note": null}, {"origin_url": "https://github.com/CocoaPods/Specs", "visit_type": "git", "save_request_date": "2022-01-17T11:32:25.696584+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069635, "note": null}, {"origin_url": "https://github.com/404-not-find/client", "visit_type": "git", "save_request_date": "2022-01-17T11:32:29.661549+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069638, "note": null}, {"origin_url": "http://floppsie.comp.glam.ac.uk/gm2", "visit_type": "git", "save_request_date": "2022-01-17T11:32:30.554341+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069640, "note": null}, {"origin_url": "https://anonhg.netbsd.org/pkgsrc-public", "visit_type": "hg", "save_request_date": "2022-01-17T11:32:30.915497+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069641, "note": null}, {"origin_url": "https://anonhg.netbsd.org/pkgsrc-draft", "visit_type": "hg", "save_request_date": "2022-01-17T11:32:31.414971+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069642, "note": null}, {"origin_url": "https://anonhg.netbsd.org/pkgsrc", "visit_type": "hg", "save_request_date": "2022-01-17T11:32:31.621251+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069643, "note": null}, {"origin_url": "https://anonhg.netbsd.org/src-public", "visit_type": "hg", "save_request_date": "2022-01-17T11:32:31.785305+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069644, "note": null}, {"origin_url": "https://anonhg.netbsd.org/src-draft", "visit_type": "hg", "save_request_date": "2022-01-17T11:32:32.088883+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069645, "note": null}, {"origin_url": "git://git.archlinux.org/svntogit/community.git", "visit_type": "git", "save_request_date": "2022-01-17T11:32:32.592602+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069646, "note": null}, {"origin_url": "https://github.com/jlippold/tweakCompatible", "visit_type": "git", "save_request_date": "2022-01-17T11:32:33.224794+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069647, "note": null}, {"origin_url": "git://git.archlinux.org/svntogit/packages.git", "visit_type": "git", "save_request_date": "2022-01-17T11:32:34.805022+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069650, "note": null}, {"origin_url": "https://source.puri.sm/Librem5/linux-next.git", "visit_type": "git", "save_request_date": "2022-01-17T11:32:35.713263+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069652, "note": null}, {"origin_url": "https://github.com/CambridgeSemiticsLab/BH_time_collocations", "visit_type": "git", "save_request_date": "2022-01-17T11:32:36.623927+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069653, "note": null}, {"origin_url": "https://git.texlive.info/texlive", "visit_type": "git", "save_request_date": "2022-01-17T11:32:37.568723+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069654, "note": null}, {"origin_url": "https://github.com/godotengine/godot", "visit_type": "git", "save_request_date": "2022-01-17T11:32:37.753028+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069656, "note": null}, {"origin_url": "https://plugins.svn.wordpress.org/wp-activity-log-for-woocommerce", "visit_type": "svn", "save_request_date": "2022-01-17T11:32:39.873069+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_status": null, "visit_date": null, "loading_task_id": 405069657, "note": null}]

[1] P1258: save-code-now.csv

ardumont claimed this task.

Remains 8 origins on running state as they are currently being ingested (and those are large origins).

I'm done with what's possible to do there so closing.