Currently 46 origins are in such state [1]
[1] P1008
Description
Description
Revisions and Commits
Revisions and Commits
Related Objects
Related Objects
- Mentioned In
- rDWAPPSf92c78719a9a: Separate save code now status refresh routine from the listing ui
D5583: Separate save code now status refresh routine from the listing ui
D5569: Populate the visit status value in save code now request - Mentioned Here
- D5583: Separate save code now status refresh routine from the listing ui
T3280: Separate save code now request updates from the save code now ui
P1009 not created save code now submitted and failing with a more clear error
T3266: Improve save code now failed/uneventful status reporting
D5559: Add new `swh save submit-request` cli to batch save code now requests
R238:53ca45741ecc: Simple script to save the Inria gForge one repo at a time
P1008 Pending running save code now requests
Event Timeline
Comment Actions
We could reschedule the origins through:
psql ... | xargs swh scheduler task respawn {} \;
but that won't update stats on save code now ui.
We could use the same approach as [1] but i think given it's at least the 3rd time it's required (that i know of).
It makes sense to evolve the swh-web-client to add a save-code-now subcommand in there to allow this.
Bonus point as this cli is already able to deal with authentication token.
[1] https://forge.softwareheritage.org/R238:53ca45741ecc0d1aecd06f0265385f8ee8425574
Comment Actions
With [1], the following has been rescheduled [2]
$ psql -t service=swh-web psql (12.6) SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) Type "help" for help. swh-web=> \copy (select visit_type, origin_url from save_origin_request where loading_task_status='running' order by request_date desc) to '/tmp/reschedule-save-code-now2.csv' delimiter ','; COPY 40 $ cat /tmp/reschedule-save-code-now2.csv | swh web save-code-now post | jq . [ { "visit_type": "git", "origin_url": "https://github.com/Samsung/TizenFX", "save_request_date": "2021-04-20T08:44:18.703294+00:00", "save_request_status": "accepted", "save_task_status": "running", "visit_date": null }, { "visit_type": "git", "origin_url": "https://scm.gforge.inria.fr/anonscm/git/simty/simty.git", "save_request_date": "2021-04-01T16:57:50.205000+00:00", "save_request_status": "accepted", "save_task_status": "running", "visit_date": null }, { "visit_type": "git", "origin_url": "https://git.php.net/repository/web/php-distributions.git", "save_request_date": "2021-03-30T09:45:14.138000+00:00", "save_request_status": "accepted", "save_task_status": "running", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/kusl/wgeteveryday", "save_request_date": "2021-03-22T07:03:38.046000+00:00", "save_request_status": "accepted", "save_task_status": "running", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/HardenedBSD/hardenedBSD-stable", "save_request_date": "2021-04-20T08:52:06.772489+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/andrewjherbert/Elliott-900-documentation", "save_request_date": "2021-04-20T08:52:07.266501+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/andrewjherbert/Elliott-900-simulator", "save_request_date": "2021-04-20T08:52:07.521148+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/andrewjherbert/Elliott-900-archive-web-site", "save_request_date": "2021-04-20T08:52:07.728146+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/aosp-mirror/kernel_msm", "save_request_date": "2021-04-20T08:52:07.947456+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/kong/docs.konghq.com", "save_request_date": "2021-04-20T08:52:08.239671+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/vmware/vsphere-automation-sdk-java", "save_request_date": "2021-04-20T08:52:08.477773+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/woocommerce/woocommerce-gutenberg-products-block", "save_request_date": "2021-04-20T08:52:08.720696+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/dotnet/docs.es-es", "save_request_date": "2021-04-20T08:52:09.035862+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/GoogleCloudPlatform/bank-of-anthos", "save_request_date": "2021-04-20T08:52:09.523693+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/openstack-archive/oslo-incubator", "save_request_date": "2021-04-20T08:52:09.740042+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/angular/bower-angular-i18n", "save_request_date": "2021-04-20T08:52:09.993623+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/envoyproxy/nighthawk", "save_request_date": "2021-04-20T08:52:10.672644+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/Awful/Awful.app", "save_request_date": "2021-04-20T08:52:10.946862+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/hyperledger/fabric-chaincode-node", "save_request_date": "2021-04-20T08:52:11.370112+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/pikelang/Pike", "save_request_date": "2021-04-20T08:52:11.661088+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/wikimedia/VisualEditor", "save_request_date": "2021-04-20T08:52:12.278048+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/microsoft/vscode-cosmosdb", "save_request_date": "2021-04-20T08:52:12.780962+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/coreutils/gnulib", "save_request_date": "2021-04-20T08:52:13.105092+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/iliakan/javascript-tutorial-cn-old", "save_request_date": "2021-04-20T08:52:13.529833+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/JetBrains/JetBrainsRuntime", "save_request_date": "2021-04-20T08:52:13.752772+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/react-native-community/upgrade-support", "save_request_date": "2021-04-20T08:52:13.990574+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/microsoft/InnerEye-DeepLearning", "save_request_date": "2021-04-20T08:52:14.253191+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/spring-cloud/spring-cloud-dataflow-ui", "save_request_date": "2021-04-20T08:52:14.514319+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/oxfordcontrol/COSMO.jl", "save_request_date": "2021-04-20T08:52:16.427433+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/pytorch/ios-demo-app", "save_request_date": "2021-04-20T08:52:16.688715+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/JohannesBuchner/PyMultiNest", "save_request_date": "2021-04-20T08:52:16.944366+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://gitlab.gnome.org/GNOME/tracker", "save_request_date": "2021-04-20T08:52:17.357611+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/reMarkable/linux", "save_request_date": "2021-04-20T08:52:17.629337+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/oracle/linux-uek", "save_request_date": "2021-04-20T08:52:17.865373+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/microsoft/Bing-COVID-19-Data", "save_request_date": "2021-04-20T08:52:18.173149+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-aws/+git/focal", "save_request_date": "2021-04-20T08:52:18.537706+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/google/bbr", "save_request_date": "2021-04-20T08:52:18.841796+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/CocoaPods/Specs", "save_request_date": "2021-04-20T08:52:19.126667+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/kickstarter/ios-oss", "save_request_date": "2021-04-20T08:52:20.989006+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null }, { "visit_type": "git", "origin_url": "https://github.com/CambridgeSemiticsLab/BH_time_collocations", "save_request_date": "2021-04-20T08:52:21.366201+00:00", "save_request_status": "accepted", "save_task_status": "not yet scheduled", "visit_date": null } ]
[1] D5559
[2] To bypass the rate limit:
$ cat ~/.config/swh/global.yml swh: web: client: api_url: https://archive.softwareheritage.org/api/1 bearer_token: <bearer-token>
out of:
swh web auth generate "ardumont"
(mind the quote)
Comment Actions
Thanks for looking into this. If I look at https://grafana.softwareheritage.org/d/WXRVVc_Mz/save-code-now?viewPanel=4&orgId=1&from=1617954242247&to=1617975842247&var-environment=production&var-instance=moma.internal.softwareheritage.org&var-status=All&var-load_task_status=All&var-visit_type=All it seems there are also some 255 requests "not yet scheduled". Maybe it's the same issue?
Comment Actions
Most probably yes.
Well, I had a look over everything we have so far (grafana will only look to the data it had and it's fairly recent so incomplete):
swh-web=> select loading_task_status, count(*) from save_origin_request group by loading_task_status; scheduled | 2352 running | 14 not yet scheduled | 1256 not created | 103 succeeded | 68260 failed | 2446
So might as well reschedule the 0.1k not created first, then the not ~1.2k `not yet
scheduled`and at last give ~2k failed ones a change again (some might be transient
errors from the past others probably not found but still some might pass).
Comment Actions
Rescheduled the ones not yet scheduled.
As expected, some worked, some failed but tl; dr the status are updated...
No more not yet scheduled... But...
Something hit me though during checking the statuses. The status of the save request is
not updated if someone does not go on the web ui so the save requests are actually
updated in the swh-web's db backend. Which is the task i'm set on fixing in the webapp.
Decorellate such status update from the save code no ui. So there we go, that task i'm
speaking about for some time now needs creating and a little bump in priority [2]
[1]
swh-web=> select loading_task_status, count(*) from save_origin_request group by loading_task_status; loading_task_status | count ---------------------+------- running | 17 not created | 110 succeeded | 69119 failed | 4262 scheduled | 2208 (5 rows)
[2] T3280
Comment Actions
Browsing though the save code now ui with increment of 100 per page.
The save code now requests are now up to date in regards to their status.
swh-web=> select loading_task_status, count(*) from save_origin_request group by loading_task_status order by count desc; loading_task_status | count ---------------------+------- succeeded | 71316 failed | 4270 not created | 110 running | 17 scheduled | 8 (5 rows)
Comment Actions
So the cli opened in D5559 has now been rewritten slightly (the diff is now fully tested though). See the diff for details.
The cli usage (names got slightly improved regarding my initial copy/pasted command):
$ swh web save submit-request --help Usage: swh web save submit-request [OPTIONS] Submit new save code now request through cli pipe. The expected format of the request if one csv row ``<visit_type>,<origin>``. Example: cat list-origins | swh web save submit-request echo svn;https://svn-url git;https://git-url | swh web save submit-request --delimiter ';' Prints: The output of save code now requests as json output. Options: -d, --delimiter TEXT -h, --help Show this message and exit.
Comment Actions
Pushed and tag swh.web.client v0.3.0 with the new endpoint ^
(ideally we could want at least 2 other endpoints, one to check the status
on one save code now request, another to possibly list in a paginated way
the status of the save code now, a-la ui to be symmetric with that new
endpoint, tasks to open ;)
Closing this now as the gist of this is done and the remaining fix is to be dealt with in another task [1].
[1] T3280
Comment Actions
Closing this now as the gist of this is done and the remaining fix is to be dealt with in another task [1].
For information, the other part in question, listing separated from update status, is in D5583 now ;)