Page MenuHomeSoftware Heritage
Create Task
Maniphest T3875

Deploy loader.svn v1.1 and analyze remaining errors
Closed, MigratedEdits Locked
Actions

Description

When fixes (T3870) land, this will need a new release cycle.

Revisions and Commits

rDLDSVN Subversion (SVN) loader
D7041rDLDSVN34f61cb58157 svn: Ignore externals parsing related errors when exporting

Related Objects

Event Timeline

ardumont triaged this task as Normal priority.Jan 24 2022, 9:28 AM
ardumont created this task.
ardumont added projects: SVN Loader, System administration.
ardumont updated the task description. (Show Details)
ardumont mentioned this in T3864: staging: Deploy swh.loader.svn v1.0.0.Jan 24 2022, 9:31 AM
ardumont moved this task from Backlog to Weekly backlog on the System administration board.
ardumont changed the task status from Open to Work in Progress.Jan 25 2022, 10:51 AM
ardumont moved this task from Weekly backlog to in-progress on the System administration board.
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.Jan 25 2022, 11:28 AM

staging:

  • Package python3-swh.loader.svn upgraded
  • Service swh-worker@loader_{svn,high_priority} restarted
ardumont renamed this task from Deploy loader.svn v1.1 to staging: Deploy loader.svn v1.1.Jan 25 2022, 11:29 AM
anlambert added a subscriber: anlambert.EditedJan 25 2022, 11:49 AM

Subversion origins that failed to load or with a partial visit status when using swh-loader-svn v1.0.0 (T3870#77580)
have been resubmitted on staging to check how many can now be fully loaded using swh-loader-svn v1.1.0.

http://kibana0.internal.softwareheritage.org:5601/app/dashboards#/view/3c126be0-7944-11ec-b7b0-c7cd6a2a1525?_g=h@66049b4

anlambert@carnavalet:~/swh$ cat svn_with_externals_to_reload_with_v1.1.txt | sed s/^/svn,/ | swh web --config-file ~/.config/swh/global.yml save submit-request
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/disy-commons/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gate/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ivs-ros-pkg/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jvcl/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lignedetemps/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/matplotlib/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mediboard/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/restore/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/turbogears1/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ubraa/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ugmdatabase/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ui4w/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/umlmodeler/svncode)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/unixpiger/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/upsurge/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/urbanmediator/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/urt41banguid/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/usboblivion/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/usvn/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/w3af/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/whack-a-mole/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wideimage/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wikipad/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wikiup/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/windsofscrum/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wp-blog/svn)
INFO:root:Submitted origin (svn, https://svn.blender.org/svnroot/bf-translations/)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/aiadsocialnet/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/aoia/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/apertium/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/arp2/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/asicms/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/asunit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ats-lang/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/auber/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/axiomengine/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/azsmrc/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bibdesk/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bilder/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bitpim/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/borgworx/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/boson/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/brlcad/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bsml/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bzflag/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/cesaachen/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/charon-suite/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/chocolate-doom/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/coati-api/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/codeblocks/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/collada-dom/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/crackpot/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/crossfire/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/cs-battletrack/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/csipc2/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/cs-project/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/czretouch/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dafizilla/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/daimonin/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/delphiasiovst/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dlvhex/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/docbook/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dooble/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/doubleagent/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/doubleagent/code/)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/drdivx/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dsao-md/code-code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/emgucv/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/epidoc/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ergatis/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/exponent/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/eyeqhtml5/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ffdshow-tryout/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/fireclipse/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/firemox/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/flightpred/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/flog/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/flylinkdc-client/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/foray/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/fredslibraries/esp-clibs)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/freeorion/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/funambol/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gallery/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/garabagh/code-0)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gar/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/geshi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/getdata/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gexperts/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/giews/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/graphsharp/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/helix/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/homefinance/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/htmlunit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ieatbrainz/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/itelex/code-0)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jamwiki/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jedit/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jgerman/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jsk-ros-pkg/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jython/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/kdenlive/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/libpong/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/librestudio/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lima-vva/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lithron/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/loconetovertcp/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/logilogi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lucifersstand/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lw-framework/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/m2bee/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/madwifi/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/magware/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/makingthings/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/manatee/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/marathon/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/marsyas/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mclarobotics/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mediainfo/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/meshlab/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/microhal/code-0)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mingw-w64/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/misesquipulas/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/modelbuilder/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/moose/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mplayerx-osx/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mwclient/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ndlb/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/nukeshtmlunit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/obi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/olympos/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/oof/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/openautomation/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/openimscore/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/openrocket/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pascalscada/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pcpe/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pencil-planner/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/perp/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/phpaxes/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/phpbbproject/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/phphtmllib/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/plonepm/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pmml2svg/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/potassco/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/prism-api/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/projectorria/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/psycle/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pure-data/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pywebsvcs/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/rand-mh/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/razorbacktm/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/roadnav/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ruby-gnome2/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/sashimi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/seabassshowdown/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/shareaza/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/silex/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/simplectest/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/skanner3d/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/smartweb/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/stemkit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/strawhat/tango-ds)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/sugarcadesnake/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/sumo/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/svnbook/source)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/svnbook-zh/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/texstudio/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tigefa4u/trac)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tikiwiki/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tora/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tortoisesvn/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/trinlibr/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/virtualslight/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wesnoth-umc-dev/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/xreal/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/xvidcap/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/xvrstudio/code)
INFO:root:Submitted origin (svn, http://svn.code.sf.net/p/tortoisesvn/code/trunk)
INFO:root:Submitted origin (svn, http://svn.osdn.net/svnroot/tortoisesvn/)
anlambert mentioned this in T3870: Analyze svn externals repository ingestion with loader.svn v1.0.Jan 25 2022, 11:50 AM
anlambert renamed this task from staging: Deploy loader.svn v1.1 to staging: Deploy loader.svn v1.1 and analyze remaining errors.Jan 26 2022, 3:27 PM
swh-sentry-integration added a subscriber: swh-sentry-integration.Jan 26 2022, 3:28 PM

Sentry issue: SWH-LOADER-SVN-54

anlambert added a revision: D7041: svn: Ignore externals parsing related errors when exporting.Jan 26 2022, 3:31 PM
anlambert added a commit: rDLDSVN34f61cb58157: svn: Ignore externals parsing related errors when exporting.Jan 26 2022, 4:25 PM
ardumont added a comment.Jan 28 2022, 12:30 PM

python3-swh.loader.svn v1.1.1 deployed on staging, production ongoing.

ardumont renamed this task from staging: Deploy loader.svn v1.1 and analyze remaining errors to Deploy loader.svn v1.1 and analyze remaining errors.Jan 28 2022, 12:30 PM
anlambert added a comment.Jan 28 2022, 1:49 PM

Below are the results of the loading of the subversion origins listed above, I used a filtered version of the webapp database on staging.

13:36 $ sqlite3 swh/web/settings/db.sqlite3 
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> select visit_status, count(*) from save_origin_request group by visit_status;
failed|18
full|76
partial|83

So 76 origins that could not be fully loaded with swh-loader-svn v1.0.0 are now correctly processed with v1.1.0.

They are still failures and partial loading but most of them comes from network errors while dumping subversion
repositories, see example below:

[2022-01-25 17:00:23,243: ERROR/ForkPoolWorker-14] Loading failure, updating to `failed` status
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 335, in load
    self.prepare()
  File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 811, in prepare
    dump_path = self.dump_svn_revisions(self.svn_url, last_loaded_svn_rev)
  File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 771, in dump_svn_revisions
    % (last_dumped_rev, last_loaded_svn_rev)
Exception: Last dumped subversion revision (48) is lesser than the last one loaded into the archive (10498)

This last batch of tests allowed me to detect some small remaining issues that got fixed in swh-loader-svn v1.1.1.

ardumont closed this task as Resolved.Jan 28 2022, 3:34 PM
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:05 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal