Page MenuHomeSoftware Heritage

Check older pending save code now requests apparently stuck and reschedule those
Closed, ResolvedPublic

Description

Currently 46 origins are in such state [1]

[1] P1008

Event Timeline

ardumont triaged this task as Normal priority.Apr 19 2021, 6:43 PM
ardumont created this task.
ardumont added a subscriber: rdicosmo.
ardumont renamed this task from Make a pass at potential save code now pending and apparently stuck save code now queries to Check older pending save code now requests apparently stuck and reschedule those.Apr 19 2021, 7:07 PM
ardumont updated the task description. (Show Details)

We could reschedule the origins through:
psql ... | xargs swh scheduler task respawn {} \;

but that won't update stats on save code now ui.

We could use the same approach as [1] but i think given it's at least the 3rd time it's required (that i know of).

It makes sense to evolve the swh-web-client to add a save-code-now subcommand in there to allow this.
Bonus point as this cli is already able to deal with authentication token.

[1] https://forge.softwareheritage.org/R238:53ca45741ecc0d1aecd06f0265385f8ee8425574

With [1], the following has been rescheduled [2]

$ psql -t service=swh-web
psql (12.6)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

swh-web=> \copy (select visit_type, origin_url from save_origin_request where loading_task_status='running' order by request_date desc) to '/tmp/reschedule-save-code-now2.csv' delimiter ',';
COPY 40
$ cat /tmp/reschedule-save-code-now2.csv | swh web save-code-now post | jq .
[
  {
    "visit_type": "git",
    "origin_url": "https://github.com/Samsung/TizenFX",
    "save_request_date": "2021-04-20T08:44:18.703294+00:00",
    "save_request_status": "accepted",
    "save_task_status": "running",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://scm.gforge.inria.fr/anonscm/git/simty/simty.git",
    "save_request_date": "2021-04-01T16:57:50.205000+00:00",
    "save_request_status": "accepted",
    "save_task_status": "running",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://git.php.net/repository/web/php-distributions.git",
    "save_request_date": "2021-03-30T09:45:14.138000+00:00",
    "save_request_status": "accepted",
    "save_task_status": "running",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/kusl/wgeteveryday",
    "save_request_date": "2021-03-22T07:03:38.046000+00:00",
    "save_request_status": "accepted",
    "save_task_status": "running",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/HardenedBSD/hardenedBSD-stable",
    "save_request_date": "2021-04-20T08:52:06.772489+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/andrewjherbert/Elliott-900-documentation",
    "save_request_date": "2021-04-20T08:52:07.266501+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/andrewjherbert/Elliott-900-simulator",
    "save_request_date": "2021-04-20T08:52:07.521148+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/andrewjherbert/Elliott-900-archive-web-site",
    "save_request_date": "2021-04-20T08:52:07.728146+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/aosp-mirror/kernel_msm",
    "save_request_date": "2021-04-20T08:52:07.947456+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/kong/docs.konghq.com",
    "save_request_date": "2021-04-20T08:52:08.239671+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/vmware/vsphere-automation-sdk-java",
    "save_request_date": "2021-04-20T08:52:08.477773+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/woocommerce/woocommerce-gutenberg-products-block",
    "save_request_date": "2021-04-20T08:52:08.720696+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/dotnet/docs.es-es",
    "save_request_date": "2021-04-20T08:52:09.035862+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/GoogleCloudPlatform/bank-of-anthos",
    "save_request_date": "2021-04-20T08:52:09.523693+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/openstack-archive/oslo-incubator",
    "save_request_date": "2021-04-20T08:52:09.740042+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/angular/bower-angular-i18n",
    "save_request_date": "2021-04-20T08:52:09.993623+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/envoyproxy/nighthawk",
    "save_request_date": "2021-04-20T08:52:10.672644+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/Awful/Awful.app",
    "save_request_date": "2021-04-20T08:52:10.946862+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/hyperledger/fabric-chaincode-node",
    "save_request_date": "2021-04-20T08:52:11.370112+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/pikelang/Pike",
    "save_request_date": "2021-04-20T08:52:11.661088+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/wikimedia/VisualEditor",
    "save_request_date": "2021-04-20T08:52:12.278048+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/microsoft/vscode-cosmosdb",
    "save_request_date": "2021-04-20T08:52:12.780962+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/coreutils/gnulib",
    "save_request_date": "2021-04-20T08:52:13.105092+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/iliakan/javascript-tutorial-cn-old",
    "save_request_date": "2021-04-20T08:52:13.529833+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/JetBrains/JetBrainsRuntime",
    "save_request_date": "2021-04-20T08:52:13.752772+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/react-native-community/upgrade-support",
    "save_request_date": "2021-04-20T08:52:13.990574+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/microsoft/InnerEye-DeepLearning",
    "save_request_date": "2021-04-20T08:52:14.253191+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/spring-cloud/spring-cloud-dataflow-ui",
    "save_request_date": "2021-04-20T08:52:14.514319+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/oxfordcontrol/COSMO.jl",
    "save_request_date": "2021-04-20T08:52:16.427433+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/pytorch/ios-demo-app",
    "save_request_date": "2021-04-20T08:52:16.688715+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/JohannesBuchner/PyMultiNest",
    "save_request_date": "2021-04-20T08:52:16.944366+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://gitlab.gnome.org/GNOME/tracker",
    "save_request_date": "2021-04-20T08:52:17.357611+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/reMarkable/linux",
    "save_request_date": "2021-04-20T08:52:17.629337+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/oracle/linux-uek",
    "save_request_date": "2021-04-20T08:52:17.865373+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/microsoft/Bing-COVID-19-Data",
    "save_request_date": "2021-04-20T08:52:18.173149+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-aws/+git/focal",
    "save_request_date": "2021-04-20T08:52:18.537706+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/google/bbr",
    "save_request_date": "2021-04-20T08:52:18.841796+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/CocoaPods/Specs",
    "save_request_date": "2021-04-20T08:52:19.126667+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/kickstarter/ios-oss",
    "save_request_date": "2021-04-20T08:52:20.989006+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  },
  {
    "visit_type": "git",
    "origin_url": "https://github.com/CambridgeSemiticsLab/BH_time_collocations",
    "save_request_date": "2021-04-20T08:52:21.366201+00:00",
    "save_request_status": "accepted",
    "save_task_status": "not yet scheduled",
    "visit_date": null
  }
]

[1] D5559

[2] To bypass the rate limit:

$ cat ~/.config/swh/global.yml
swh:
  web:
    client:
      api_url: https://archive.softwareheritage.org/api/1
      bearer_token: <bearer-token>

out of:

swh web auth generate "ardumont"

(mind the quote)

Most probably yes.

Well, I had a look over everything we have so far (grafana will only look to the data it had and it's fairly recent so incomplete):

swh-web=> select loading_task_status, count(*) from save_origin_request group by loading_task_status;
 scheduled           |  2352
 running             |    14
 not yet scheduled   |  1256
 not created         |   103
 succeeded           | 68260
 failed              |  2446

So might as well reschedule the 0.1k not created first, then the not ~1.2k `not yet
scheduled`and at last give ~2k failed ones a change again (some might be transient
errors from the past others probably not found but still some might pass).

ardumont changed the task status from Open to Work in Progress.Apr 20 2021, 11:41 AM
ardumont moved this task from Backlog to in-progress on the System administration board.

ok so the 'not created' ones are apparently failing origins so the issue is about the report
of the state which is somehow wong. [1] There is a task for that (or something near
enough) [2]

So moving along on the plan with the not yet scheduled instead.

[1] P1009

[2] T3266

Rescheduled the ones not yet scheduled.
As expected, some worked, some failed but tl; dr the status are updated...
No more not yet scheduled... But...

Something hit me though during checking the statuses. The status of the save request is
not updated if someone does not go on the web ui so the save requests are actually
updated in the swh-web's db backend. Which is the task i'm set on fixing in the webapp.

Decorellate such status update from the save code no ui. So there we go, that task i'm
speaking about for some time now needs creating and a little bump in priority [2]

[1]

swh-web=> select loading_task_status, count(*) from save_origin_request group by loading_task_status;
 loading_task_status | count
---------------------+-------
 running             |    17
 not created         |   110
 succeeded           | 69119
 failed              |  4262
 scheduled           |  2208
(5 rows)

[2] T3280

Browsing though the save code now ui with increment of 100 per page.

The save code now requests are now up to date in regards to their status.

swh-web=> select loading_task_status, count(*) from save_origin_request group by loading_task_status order by count desc;
 loading_task_status | count
---------------------+-------
 succeeded           | 71316
 failed              |  4270
 not created         |   110
 running             |    17
 scheduled           |     8
(5 rows)

So the cli opened in D5559 has now been rewritten slightly (the diff is now fully tested though). See the diff for details.

The cli usage (names got slightly improved regarding my initial copy/pasted command):

$ swh web save submit-request --help
Usage: swh web save submit-request [OPTIONS]

  Submit new save code now request through cli pipe. The expected format of
  the request     if one csv row ``<visit_type>,<origin>``.

      Example:

          cat list-origins | swh web save submit-request

          echo svn;https://svn-url git;https://git-url | swh web save
          submit-request --delimiter ';'

      Prints:         The output of save code now requests as json output.



Options:
  -d, --delimiter TEXT
  -h, --help            Show this message and exit.

Thanks, this is quite useful indeed.

Pushed and tag swh.web.client v0.3.0 with the new endpoint ^
(ideally we could want at least 2 other endpoints, one to check the status
on one save code now request, another to possibly list in a paginated way
the status of the save code now, a-la ui to be symmetric with that new
endpoint, tasks to open ;)

Closing this now as the gist of this is done and the remaining fix is to be dealt with in another task [1].

[1] T3280

ardumont claimed this task.

Closing this now as the gist of this is done and the remaining fix is to be dealt with in another task [1].

For information, the other part in question, listing separated from update status, is in D5583 now ;)