Page MenuHomeSoftware Heritage

Deploy loader.svn v1.1 and analyze remaining errors
Closed, MigratedEdits Locked

Description

When fixes (T3870) land, this will need a new release cycle.

Event Timeline

ardumont triaged this task as Normal priority.Jan 24 2022, 9:28 AM
ardumont created this task.
ardumont updated the task description. (Show Details)
ardumont changed the task status from Open to Work in Progress.Jan 25 2022, 10:51 AM
ardumont moved this task from Weekly backlog to in-progress on the System administration board.

staging:

  • Package python3-swh.loader.svn upgraded
  • Service swh-worker@loader_{svn,high_priority} restarted
ardumont renamed this task from Deploy loader.svn v1.1 to staging: Deploy loader.svn v1.1.Jan 25 2022, 11:29 AM

Subversion origins that failed to load or with a partial visit status when using swh-loader-svn v1.0.0 (T3870#77580)
have been resubmitted on staging to check how many can now be fully loaded using swh-loader-svn v1.1.0.

http://kibana0.internal.softwareheritage.org:5601/app/dashboards#/view/3c126be0-7944-11ec-b7b0-c7cd6a2a1525?_g=h@66049b4

anlambert@carnavalet:~/swh$ cat svn_with_externals_to_reload_with_v1.1.txt | sed s/^/svn,/ | swh web --config-file ~/.config/swh/global.yml save submit-request
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/disy-commons/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gate/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ivs-ros-pkg/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jvcl/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lignedetemps/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/matplotlib/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mediboard/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/restore/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/turbogears1/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ubraa/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ugmdatabase/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ui4w/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/umlmodeler/svncode)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/unixpiger/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/upsurge/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/urbanmediator/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/urt41banguid/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/usboblivion/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/usvn/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/w3af/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/whack-a-mole/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wideimage/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wikipad/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wikiup/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/windsofscrum/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wp-blog/svn)
INFO:root:Submitted origin (svn, https://svn.blender.org/svnroot/bf-translations/)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/aiadsocialnet/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/aoia/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/apertium/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/arp2/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/asicms/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/asunit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ats-lang/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/auber/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/axiomengine/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/azsmrc/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bibdesk/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bilder/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bitpim/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/borgworx/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/boson/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/brlcad/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bsml/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bzflag/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/cesaachen/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/charon-suite/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/chocolate-doom/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/coati-api/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/codeblocks/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/collada-dom/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/crackpot/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/crossfire/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/cs-battletrack/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/csipc2/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/cs-project/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/czretouch/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dafizilla/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/daimonin/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/delphiasiovst/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dlvhex/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/docbook/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dooble/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/doubleagent/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/doubleagent/code/)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/drdivx/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dsao-md/code-code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/emgucv/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/epidoc/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ergatis/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/exponent/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/eyeqhtml5/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ffdshow-tryout/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/fireclipse/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/firemox/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/flightpred/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/flog/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/flylinkdc-client/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/foray/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/fredslibraries/esp-clibs)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/freeorion/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/funambol/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gallery/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/garabagh/code-0)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gar/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/geshi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/getdata/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gexperts/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/giews/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/graphsharp/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/helix/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/homefinance/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/htmlunit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ieatbrainz/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/itelex/code-0)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jamwiki/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jedit/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jgerman/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jsk-ros-pkg/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jython/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/kdenlive/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/libpong/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/librestudio/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lima-vva/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lithron/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/loconetovertcp/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/logilogi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lucifersstand/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lw-framework/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/m2bee/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/madwifi/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/magware/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/makingthings/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/manatee/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/marathon/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/marsyas/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mclarobotics/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mediainfo/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/meshlab/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/microhal/code-0)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mingw-w64/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/misesquipulas/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/modelbuilder/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/moose/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mplayerx-osx/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mwclient/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ndlb/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/nukeshtmlunit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/obi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/olympos/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/oof/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/openautomation/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/openimscore/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/openrocket/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pascalscada/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pcpe/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pencil-planner/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/perp/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/phpaxes/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/phpbbproject/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/phphtmllib/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/plonepm/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pmml2svg/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/potassco/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/prism-api/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/projectorria/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/psycle/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pure-data/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pywebsvcs/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/rand-mh/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/razorbacktm/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/roadnav/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ruby-gnome2/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/sashimi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/seabassshowdown/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/shareaza/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/silex/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/simplectest/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/skanner3d/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/smartweb/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/stemkit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/strawhat/tango-ds)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/sugarcadesnake/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/sumo/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/svnbook/source)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/svnbook-zh/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/texstudio/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tigefa4u/trac)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tikiwiki/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tora/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tortoisesvn/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/trinlibr/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/virtualslight/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wesnoth-umc-dev/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/xreal/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/xvidcap/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/xvrstudio/code)
INFO:root:Submitted origin (svn, http://svn.code.sf.net/p/tortoisesvn/code/trunk)
INFO:root:Submitted origin (svn, http://svn.osdn.net/svnroot/tortoisesvn/)
anlambert renamed this task from staging: Deploy loader.svn v1.1 to staging: Deploy loader.svn v1.1 and analyze remaining errors.Jan 26 2022, 3:27 PM

python3-swh.loader.svn v1.1.1 deployed on staging, production ongoing.

ardumont renamed this task from staging: Deploy loader.svn v1.1 and analyze remaining errors to Deploy loader.svn v1.1 and analyze remaining errors.Jan 28 2022, 12:30 PM

Below are the results of the loading of the subversion origins listed above, I used a filtered version of the webapp database on staging.

13:36 $ sqlite3 swh/web/settings/db.sqlite3 
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> select visit_status, count(*) from save_origin_request group by visit_status;
failed|18
full|76
partial|83

So 76 origins that could not be fully loaded with swh-loader-svn v1.0.0 are now correctly processed with v1.1.0.

They are still failures and partial loading but most of them comes from network errors while dumping subversion
repositories, see example below:

[2022-01-25 17:00:23,243: ERROR/ForkPoolWorker-14] Loading failure, updating to `failed` status
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 335, in load
    self.prepare()
  File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 811, in prepare
    dump_path = self.dump_svn_revisions(self.svn_url, last_loaded_svn_rev)
  File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 771, in dump_svn_revisions
    % (last_dumped_rev, last_loaded_svn_rev)
Exception: Last dumped subversion revision (48) is lesser than the last one loaded into the archive (10498)

This last batch of tests allowed me to detect some small remaining issues that got fixed in swh-loader-svn v1.1.1.

ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.