Paths

Table of Contentst

Differential D8086

Fix loading of CVS repositories with non valid UTF-8 paths
ClosedPublic
Actions

Authored by anlambert on Jul 6 2022, 4:06 PM.

Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Maniphest Tasks

Restricted Maniphest Task

Commits

rDLDCVSd89f8d13e947: Fix loading of CVS repositories with non valid UTF-8 paths

Summary

Some CVS repositories have paths which are non valid UTF-8 (typically
ISO-8859-1 ones) but the loader implementation assumed all paths can
be safely encoded to UTF-8 and was raising UnicodeEncodeError when
attempting to encode non UTF-8 paths.

That commit modifies the way CVS paths are handled by the loader by
using their raw bytes representation instead of their UTF-8 decoded
string representation.

Also rcsparse.rcsfile constructor has been modified to take bytes path
as argument instead of an unicode one in order to be able to successfully
open non UTF-8 paths.

Such CVS repositories can now be successfully loaded, either using rsync
or pserver protocol.

Related to T3980

Test Plan

I added a new test covering the issue.

I also extracted the list of CVS repositories that could not be loaded
from Sentry and loaded them through a simple bash script, everything
went fine.

cvs_repos_with_path_encoding_issue.txt

rsync://a.cvs.sourceforge.net/cvsroot/medsurvey/medsurvey35
rsync://a.cvs.sourceforge.net/cvsroot/topicmaptools/topicmaptools
rsync://a.cvs.sourceforge.net/cvsroot/beerodalom/Szkeleton
rsync://a.cvs.sourceforge.net/cvsroot/pokerweb/pokerenlaweb
rsync://a.cvs.sourceforge.net/cvsroot/computerwinkel/Nieuwe_UCs
rsync://a.cvs.sourceforge.net/cvsroot/progx/www
rsync://a.cvs.sourceforge.net/cvsroot/nprofile/NProfiler
rsync://a.cvs.sourceforge.net/cvsroot/avantgarde/AvantGarde
rsync://a.cvs.sourceforge.net/cvsroot/can/activex
rsync://a.cvs.sourceforge.net/cvsroot/maxtrixengine/MaxtrixEngine
rsync://a.cvs.sourceforge.net/cvsroot/iaa-pascal/iaa-pascal-1-1
rsync://a.cvs.sourceforge.net/cvsroot/scoutos/scoutos
rsync://a.cvs.sourceforge.net/cvsroot/htmlparser-bio/com.bioinformatixx.htmlparser
rsync://a.cvs.sourceforge.net/cvsroot/oelc/trunk
rsync://a.cvs.sourceforge.net/cvsroot/digir/xforms
rsync://a.cvs.sourceforge.net/cvsroot/xplay/sun
rsync://a.cvs.sourceforge.net/cvsroot/nivelamento/nivelamento
rsync://a.cvs.sourceforge.net/cvsroot/motopain/motopain
rsync://a.cvs.sourceforge.net/cvsroot/gildav/GilDav2
rsync://a.cvs.sourceforge.net/cvsroot/gildav/GilDav
rsync://a.cvs.sourceforge.net/cvsroot/sitefinder/SiteFinder
rsync://a.cvs.sourceforge.net/cvsroot/golem-amiga/wfmh
rsync://a.cvs.sourceforge.net/cvsroot/mousave/mousave.1
rsync://a.cvs.sourceforge.net/cvsroot/cinplp/plp
rsync://a.cvs.sourceforge.net/cvsroot/adconsole/adconsole
rsync://a.cvs.sourceforge.net/cvsroot/wowicon/WowIcon_CVS
rsync://a.cvs.sourceforge.net/cvsroot/slektsweb/prosjektplan
rsync://a.cvs.sourceforge.net/cvsroot/dogfood/dogfood
rsync://a.cvs.sourceforge.net/cvsroot/insamonitor/Server
rsync://a.cvs.sourceforge.net/cvsroot/b2stok/b2stok
rsync://a.cvs.sourceforge.net/cvsroot/arnold/ookcedonc
rsync://a.cvs.sourceforge.net/cvsroot/aline/A-line
rsync://a.cvs.sourceforge.net/cvsroot/php4u/4u
rsync://a.cvs.sourceforge.net/cvsroot/quenottetv/ktv_stable
rsync://a.cvs.sourceforge.net/cvsroot/gervasproject/TGPrecursos
rsync://a.cvs.sourceforge.net/cvsroot/tagua/taguaIslasDeDatos
rsync://a.cvs.sourceforge.net/cvsroot/worldrpg/QCM
rsync://a.cvs.sourceforge.net/cvsroot/pachanga/pachanga
rsync://a.cvs.sourceforge.net/cvsroot/shushividwo/wowfightServer
rsync://a.cvs.sourceforge.net/cvsroot/vectorvisual/files
rsync://a.cvs.sourceforge.net/cvsroot/jrv-dimji/Proyecto
rsync://a.cvs.sourceforge.net/cvsroot/superstore/DB
rsync://a.cvs.sourceforge.net/cvsroot/argonaute/argonaute
rsync://a.cvs.sourceforge.net/cvsroot/geturexe/trunk
rsync://a.cvs.sourceforge.net/cvsroot/rage-engine/Rage3D
rsync://a.cvs.sourceforge.net/cvsroot/iapi/IAPI
rsync://a.cvs.sourceforge.net/cvsroot/jgenea/registres
rsync://a.cvs.sourceforge.net/cvsroot/gervasproject/IA_Guiones
rsync://a.cvs.sourceforge.net/cvsroot/xgeditor/xg
rsync://a.cvs.sourceforge.net/cvsroot/fhfschoolware/backend
rsync://a.cvs.sourceforge.net/cvsroot/nlpfarm/NewsQuest
rsync://a.cvs.sourceforge.net/cvsroot/j-algo/j-algo
rsync://a.cvs.sourceforge.net/cvsroot/miniplanmanager/mpm
rsync://a.cvs.sourceforge.net/cvsroot/sitescooper/sitescooper
rsync://a.cvs.sourceforge.net/cvsroot/id-risiko/risiko
rsync://a.cvs.sourceforge.net/cvsroot/yut/Seowonpill
rsync://a.cvs.sourceforge.net/cvsroot/su-webengine/su5
rsync://a.cvs.sourceforge.net/cvsroot/kursinfo/kursinfo
rsync://a.cvs.sourceforge.net/cvsroot/shushividwo/wowfightClient
rsync://a.cvs.sourceforge.net/cvsroot/qbc-sia/Rational SIA
rsync://a.cvs.sourceforge.net/cvsroot/xblast/levels
rsync://a.cvs.sourceforge.net/cvsroot/ispinterface/DatenbankenProjekt
rsync://a.cvs.sourceforge.net/cvsroot/aeroeuropa/Europa
rsync://a.cvs.sourceforge.net/cvsroot/qknowledge/ccknowledge
rsync://a.cvs.sourceforge.net/cvsroot/tagua/Sirley
rsync://a.cvs.sourceforge.net/cvsroot/sofaa/sofaa
rsync://a.cvs.sourceforge.net/cvsroot/typingtrainer/webhome
rsync://a.cvs.sourceforge.net/cvsroot/iapi/MAI08-202_r_micha
rsync://a.cvs.sourceforge.net/cvsroot/uniquecms/Reseau
rsync://a.cvs.sourceforge.net/cvsroot/magikcmpnts/core_40_swaf
rsync://a.cvs.sourceforge.net/cvsroot/superstore/docs
rsync://a.cvs.sourceforge.net/cvsroot/surfviz/f-s-l-m-
rsync://a.cvs.sourceforge.net/cvsroot/vdr-analogtv/analogtv
rsync://a.cvs.sourceforge.net/cvsroot/jape/obsolete
rsync://a.cvs.sourceforge.net/cvsroot/arnold/ookce
rsync://a.cvs.sourceforge.net/cvsroot/tierramedia/db
rsync://a.cvs.sourceforge.net/cvsroot/computerwinkel/Online Computer Winkel
rsync://a.cvs.sourceforge.net/cvsroot/qingu/qingu
rsync://a.cvs.sourceforge.net/cvsroot/cuttingproblem/Text
rsync://a.cvs.sourceforge.net/cvsroot/yut/PK
rsync://a.cvs.sourceforge.net/cvsroot/cncms/phparticle212
rsync://a.cvs.sourceforge.net/cvsroot/histmondiale/Histoire
rsync://a.cvs.sourceforge.net/cvsroot/emu/speech_tools
rsync://a.cvs.sourceforge.net/cvsroot/perola/Perola
rsync://a.cvs.sourceforge.net/cvsroot/xmledit/diplomareport
rsync://a.cvs.sourceforge.net/cvsroot/ocellaris/ocellaris
rsync://a.cvs.sourceforge.net/cvsroot/pg520/docs
rsync://a.cvs.sourceforge.net/cvsroot/shushividwo/My_Video
rsync://a.cvs.sourceforge.net/cvsroot/computerwinkel/MOP_DCD+ID
rsync://a.cvs.sourceforge.net/cvsroot/blacksheep/BlackSheep
rsync://a.cvs.sourceforge.net/cvsroot/tcc/tcc
rsync://a.cvs.sourceforge.net/cvsroot/pdp0x2001/PDP-0x2001
rsync://a.cvs.sourceforge.net/cvsroot/sconman/sconman
rsync://a.cvs.sourceforge.net/cvsroot/weso/docs
rsync://a.cvs.sourceforge.net/cvsroot/tagua/Sole
rsync://a.cvs.sourceforge.net/cvsroot/clipshare/F6S
rsync://a.cvs.sourceforge.net/cvsroot/openuss/sandbox
rsync://a.cvs.sourceforge.net/cvsroot/morphosambient/ambient
rsync://a.cvs.sourceforge.net/cvsroot/mailstarsf/Internal_Docs
rsync://a.cvs.sourceforge.net/cvsroot/miniplanmanager/datenVerwalter
rsync://a.cvs.sourceforge.net/cvsroot/aigle/aigle
rsync://a.cvs.sourceforge.net/cvsroot/squeak/squeak
rsync://a.cvs.sourceforge.net/cvsroot/quenottetv/ktv_13xx
rsync://a.cvs.sourceforge.net/cvsroot/jjteacher/JazzTeacher
rsync://a.cvs.sourceforge.net/cvsroot/dynamicdraw/Plugin
rsync://a.cvs.sourceforge.net/cvsroot/ziphttpd/server
rsync://a.cvs.sourceforge.net/cvsroot/swift-engine/Swift
rsync://a.cvs.sourceforge.net/cvsroot/problem-xml/problem.xml
rsync://a.cvs.sourceforge.net/cvsroot/c2iexplorer/c2iexhtm
rsync://a.cvs.sourceforge.net/cvsroot/tosmo/Multi_Lingual
rsync://a.cvs.sourceforge.net/cvsroot/asterion/asterion
rsync://a.cvs.sourceforge.net/cvsroot/cvs-repository/sdk
rsync://a.cvs.sourceforge.net/cvsroot/bbooprevalence/Bamboo.Prevalence
rsync://a.cvs.sourceforge.net/cvsroot/mywwwatcher/mywwwatcher
rsync://a.cvs.sourceforge.net/cvsroot/crepuscule/cvsrepo
rsync://a.cvs.sourceforge.net/cvsroot/supertruco/Duck
rsync://a.cvs.sourceforge.net/cvsroot/myg/myg
rsync://a.cvs.sourceforge.net/cvsroot/edgenius/edgenius2
rsync://a.cvs.sourceforge.net/cvsroot/computerwinkel/Nieuwe_SSDs
rsync://a.cvs.sourceforge.net/cvsroot/epgsukapa/epgret
rsync://a.cvs.sourceforge.net/cvsroot/yatikiri/yatikiri
rsync://a.cvs.sourceforge.net/cvsroot/netsitemais/SoftPhone
rsync://a.cvs.sourceforge.net/cvsroot/lefthand/lefthand_1.x
rsync://a.cvs.sourceforge.net/cvsroot/ninjabattles/docs
rsync://a.cvs.sourceforge.net/cvsroot/maklerdesign/workflow
rsync://a.cvs.sourceforge.net/cvsroot/shushividwo/Video
rsync://a.cvs.sourceforge.net/cvsroot/su-webengine/SUWE
rsync://a.cvs.sourceforge.net/cvsroot/yabb/YaBB2Languages
rsync://a.cvs.sourceforge.net/cvsroot/odb/water
rsync://a.cvs.sourceforge.net/cvsroot/source2html/source2html
rsync://a.cvs.sourceforge.net/cvsroot/ogrepprocessor/ORP
rsync://a.cvs.sourceforge.net/cvsroot/eclipse-library/InnerManageProject
rsync://a.cvs.sourceforge.net/cvsroot/adresse/AideDecision
rsync://a.cvs.sourceforge.net/cvsroot/dws/dws16
rsync://a.cvs.sourceforge.net/cvsroot/sntool/sntool
rsync://a.cvs.sourceforge.net/cvsroot/jdf/source
rsync://a.cvs.sourceforge.net/cvsroot/magikcmpnts/core_41_swaf
rsync://a.cvs.sourceforge.net/cvsroot/sophy/Ciclo 1
rsync://a.cvs.sourceforge.net/cvsroot/jext/src
rsync://a.cvs.sourceforge.net/cvsroot/mathenjeu/mathenjeu-www
rsync://a.cvs.sourceforge.net/cvsroot/sntool/sntool-dev
rsync://a.cvs.sourceforge.net/cvsroot/jabak/jabak
rsync://a.cvs.sourceforge.net/cvsroot/transsib/TransSibGame
rsync://a.cvs.sourceforge.net/cvsroot/karhukopla/Grafiikka
rsync://a.cvs.sourceforge.net/cvsroot/nocmaker/src
rsync://a.cvs.sourceforge.net/cvsroot/lejos-osek/lejos_osek
rsync://a.cvs.sourceforge.net/cvsroot/pizzahorta/POS
rsync://a.cvs.sourceforge.net/cvsroot/atom3d/ATOM3D.NET
rsync://a.cvs.sourceforge.net/cvsroot/easysitebuilder/www
rsync://a.cvs.sourceforge.net/cvsroot/radar-opengl/todo
rsync://a.cvs.sourceforge.net/cvsroot/maklerdesign/ER_Modell
rsync://a.cvs.sourceforge.net/cvsroot/winlibre/innosetup_distrib
rsync://a.cvs.sourceforge.net/cvsroot/rezothello/Preparation
rsync://a.cvs.sourceforge.net/cvsroot/pists/pi
rsync://a.cvs.sourceforge.net/cvsroot/planetterritory/PlanetTerritory
rsync://a.cvs.sourceforge.net/cvsroot/pyopengl/PyOpenGL1
rsync://a.cvs.sourceforge.net/cvsroot/spsvtool/spsvtool
rsync://a.cvs.sourceforge.net/cvsroot/tierramedia/Sistema
rsync://a.cvs.sourceforge.net/cvsroot/freeknowledge/BESTdotNET
rsync://a.cvs.sourceforge.net/cvsroot/inetpeering/ipm
rsync://a.cvs.sourceforge.net/cvsroot/purple/Purple01
rsync://a.cvs.sourceforge.net/cvsroot/superstore/UML
rsync://a.cvs.sourceforge.net/cvsroot/easytomcat/EasyTomcat v0.31
rsync://a.cvs.sourceforge.net/cvsroot/sparewha/SourceCode
rsync://a.cvs.sourceforge.net/cvsroot/mtclips/directoryname
rsync://a.cvs.sourceforge.net/cvsroot/zact/kmsys_spike
rsync://a.cvs.sourceforge.net/cvsroot/jbanana/Documentator_Documentation
rsync://a.cvs.sourceforge.net/cvsroot/orion3d/adtf
rsync://a.cvs.sourceforge.net/cvsroot/emu/emu
rsync://a.cvs.sourceforge.net/cvsroot/m-engine/me
rsync://a.cvs.sourceforge.net/cvsroot/jlibrary/org.jlibrary.server
rsync://a.cvs.sourceforge.net/cvsroot/basic-ide/basic-ide
rsync://a.cvs.sourceforge.net/cvsroot/shushividwo/CXKTV
rsync://a.cvs.sourceforge.net/cvsroot/religious/base
rsync://a.cvs.sourceforge.net/cvsroot/pyramidworkshop/libcoding-0.0.0
rsync://a.cvs.sourceforge.net/cvsroot/dex/dex
rsync://a.cvs.sourceforge.net/cvsroot/horen/general files
rsync://a.cvs.sourceforge.net/cvsroot/ccmtools/ccmtools
rsync://a.cvs.sourceforge.net/cvsroot/firm1control/firm2control-alpha
rsync://a.cvs.sourceforge.net/cvsroot/alsub/projeto
rsync://a.cvs.sourceforge.net/cvsroot/ao-ready/toolkit
rsync://a.cvs.sourceforge.net/cvsroot/tacpp/Outils
rsync://a.cvs.sourceforge.net/cvsroot/pocketinsanity/pocketuae
rsync://a.cvs.sourceforge.net/cvsroot/shushividwo/Medical
rsync://a.cvs.sourceforge.net/cvsroot/transsib/TransMap
rsync://a.cvs.sourceforge.net/cvsroot/lanstore/Documents
rsync://a.cvs.sourceforge.net/cvsroot/lexiphone/Lexiphone-TH
rsync://a.cvs.sourceforge.net/cvsroot/ajax-dev-lib/ADL_CVS
rsync://a.cvs.sourceforge.net/cvsroot/upxshell/BackUp
rsync://a.cvs.sourceforge.net/cvsroot/opengoliardia/opengoliardia
rsync://a.cvs.sourceforge.net/cvsroot/igmus/igmus
rsync://a.cvs.sourceforge.net/cvsroot/phprpz/trunk
rsync://a.cvs.sourceforge.net/cvsroot/gervasproject/Fotos
rsync://a.cvs.sourceforge.net/cvsroot/flyticket/FlyTicket
rsync://a.cvs.sourceforge.net/cvsroot/sqlcockpit/sqlcockpit
rsync://a.cvs.sourceforge.net/cvsroot/tagua/ANVV
rsync://a.cvs.sourceforge.net/cvsroot/cognix/modulename
rsync://a.cvs.sourceforge.net/cvsroot/pyramidworkshop/libdatatypes-0.0.0
rsync://a.cvs.sourceforge.net/cvsroot/sgcm/Sgcm
rsync://a.cvs.sourceforge.net/cvsroot/mpodo-project/MPODOTHEGAME
rsync://a.cvs.sourceforge.net/cvsroot/yut/KUL
rsync://a.cvs.sourceforge.net/cvsroot/uengine/uEngine
rsync://a.cvs.sourceforge.net/cvsroot/orion3d/articles
rsync://a.cvs.sourceforge.net/cvsroot/mathenjeu/client_flash_beta_sept07
rsync://a.cvs.sourceforge.net/cvsroot/newscoaster/newscoaster
rsync://a.cvs.sourceforge.net/cvsroot/quenottetv/CVS
rsync://a.cvs.sourceforge.net/cvsroot/tinyos/tinyos-0.4.x
rsync://a.cvs.sourceforge.net/cvsroot/bulmages/bulmages
rsync://a.cvs.sourceforge.net/cvsroot/fillets/fillets_data
rsync://a.cvs.sourceforge.net/cvsroot/geturexe/SandBox
rsync://a.cvs.sourceforge.net/cvsroot/refactory/bodega
rsync://a.cvs.sourceforge.net/cvsroot/omniathena/omniathena
rsync://a.cvs.sourceforge.net/cvsroot/chinesepython/chinesepython
rsync://a.cvs.sourceforge.net/cvsroot/mobidict/MobiDict
rsync://a.cvs.sourceforge.net/cvsroot/mosdev/letters
rsync://a.cvs.sourceforge.net/cvsroot/tactic/Main
rsync://a.cvs.sourceforge.net/cvsroot/motu/cocaine
rsync://a.cvs.sourceforge.net/cvsroot/msntranslator/msntranslator
rsync://a.cvs.sourceforge.net/cvsroot/euphony-project/euphony
rsync://a.cvs.sourceforge.net/cvsroot/smartlang/SmartLang
rsync://a.cvs.sourceforge.net/cvsroot/shushividwo/51Book
rsync://a.cvs.sourceforge.net/cvsroot/fourever/fourever-vmodell
rsync://a.cvs.sourceforge.net/cvsroot/govorilka/client
rsync://a.cvs.sourceforge.net/cvsroot/exanet/exanet2
rsync://a.cvs.sourceforge.net/cvsroot/tenem/doc
rsync://a.cvs.sourceforge.net/cvsroot/knoda/docbook
rsync://a.cvs.sourceforge.net/cvsroot/pontoconf/pontoConf
rsync://a.cvs.sourceforge.net/cvsroot/lookmediaplayer/Gui
rsync://a.cvs.sourceforge.net/cvsroot/hotwork/hotwork-site
rsync://a.cvs.sourceforge.net/cvsroot/dynamicdraw/DynamicDraw
rsync://a.cvs.sourceforge.net/cvsroot/dlog4j/DLOG4J_JAVAYOU
rsync://a.cvs.sourceforge.net/cvsroot/biloba/biloba
rsync://a.cvs.sourceforge.net/cvsroot/laminehantee/TER
rsync://a.cvs.sourceforge.net/cvsroot/wwwserver/langsel
rsync://a.cvs.sourceforge.net/cvsroot/starclient/de.starclient.projectstuff
rsync://a.cvs.sourceforge.net/cvsroot/nassp/projectApolloTextures
rsync://a.cvs.sourceforge.net/cvsroot/redkernel/ArteBangBang
rsync://a.cvs.sourceforge.net/cvsroot/igmus/igmus_studio
rsync://a.cvs.sourceforge.net/cvsroot/xmallcharter/xmallcharter
rsync://a.cvs.sourceforge.net/cvsroot/tagua/Mario
rsync://a.cvs.sourceforge.net/cvsroot/pdpsantos/psantos
rsync://a.cvs.sourceforge.net/cvsroot/nemesis-forum/photoalbum
rsync://a.cvs.sourceforge.net/cvsroot/raumschach/alife
rsync://a.cvs.sourceforge.net/cvsroot/muhasebeci/muhasebeci
rsync://a.cvs.sourceforge.net/cvsroot/pg520/sources
rsync://a.cvs.sourceforge.net/cvsroot/galeriayogurt/yogurt
rsync://a.cvs.sourceforge.net/cvsroot/pyramidworkshop/libBTPC-0.0.0
rsync://a.cvs.sourceforge.net/cvsroot/fourever/projektassistent
rsync://a.cvs.sourceforge.net/cvsroot/sisdec/sisdec
rsync://a.cvs.sourceforge.net/cvsroot/b-news/b-news
rsync://a.cvs.sourceforge.net/cvsroot/pyramidworkshop/libstreamIO-0.0.0
rsync://a.cvs.sourceforge.net/cvsroot/eclipsework/WebWork-Example
rsync://a.cvs.sourceforge.net/cvsroot/usb-led-control/usb-led-control
rsync://a.cvs.sourceforge.net/cvsroot/ajax-dev-lib/docs
rsync://a.cvs.sourceforge.net/cvsroot/suma/suma
rsync://a.cvs.sourceforge.net/cvsroot/metasystem/text
rsync://a.cvs.sourceforge.net/cvsroot/i3m/i3mclient
rsync://a.cvs.sourceforge.net/cvsroot/webpm/jworks
rsync://a.cvs.sourceforge.net/cvsroot/cap4e/CAP
rsync://a.cvs.sourceforge.net/cvsroot/link3/hjms
rsync://a.cvs.sourceforge.net/cvsroot/flisp-j/flisp-j
rsync://a.cvs.sourceforge.net/cvsroot/jext/bin
rsync://a.cvs.sourceforge.net/cvsroot/uwrugbyhupe/uwrugbyhupe
rsync://a.cvs.sourceforge.net/cvsroot/gervasproject/sonido
rsync://a.cvs.sourceforge.net/cvsroot/jrv-dimji/Modulo Movimiento Personaje
rsync://a.cvs.sourceforge.net/cvsroot/hxgrid/hxgrid
rsync://a.cvs.sourceforge.net/cvsroot/firebird/firebird2
rsync://a.cvs.sourceforge.net/cvsroot/jediedit/V1
rsync://a.cvs.sourceforge.net/cvsroot/jbanana/DocumentadorSistemas
rsync://a.cvs.sourceforge.net/cvsroot/beerodalom/grafikus
rsync://a.cvs.sourceforge.net/cvsroot/wikisip/Wiki
rsync://a.cvs.sourceforge.net/cvsroot/lib/LIB
rsync://a.cvs.sourceforge.net/cvsroot/elynx/eLynx
rsync://a.cvs.sourceforge.net/cvsroot/liberum/helpdesk
rsync://a.cvs.sourceforge.net/cvsroot/giratempoweb/Jumping
rsync://a.cvs.sourceforge.net/cvsroot/memodivx/memodivx
rsync://a.cvs.sourceforge.net/cvsroot/ibaccess/ibaccess
rsync://a.cvs.sourceforge.net/cvsroot/sheetster/Sheetster_Server
rsync://a.cvs.sourceforge.net/cvsroot/swingkore/SwingKore
rsync://a.cvs.sourceforge.net/cvsroot/carijo/Bolao
rsync://a.cvs.sourceforge.net/cvsroot/tumourextract/DOC
rsync://a.cvs.sourceforge.net/cvsroot/gmone/gmone
rsync://a.cvs.sourceforge.net/cvsroot/shushividwo/FightToghter
rsync://a.cvs.sourceforge.net/cvsroot/notes2/notes
rsync://a.cvs.sourceforge.net/cvsroot/calangos/calangos
rsync://a.cvs.sourceforge.net/cvsroot/openflyers/openflyers
rsync://a.cvs.sourceforge.net/cvsroot/gervasproject/IS
rsync://a.cvs.sourceforge.net/cvsroot/gervasproject/Graficos
rsync://a.cvs.sourceforge.net/cvsroot/pat/PAT
rsync://a.cvs.sourceforge.net/cvsroot/snap/Process
rsync://a.cvs.sourceforge.net/cvsroot/projectgaffel/docs
rsync://a.cvs.sourceforge.net/cvsroot/opensi/opensi-serveur
rsync://a.cvs.sourceforge.net/cvsroot/life3d/avatar
rsync://a.cvs.sourceforge.net/cvsroot/ased/ased3.0
rsync://a.cvs.sourceforge.net/cvsroot/yoksel/TestsFor1S
rsync://a.cvs.sourceforge.net/cvsroot/wjb/wjb_vb
rsync://a.cvs.sourceforge.net/cvsroot/filex/filex
rsync://a.cvs.sourceforge.net/cvsroot/qbc-sia/trunk
rsync://a.cvs.sourceforge.net/cvsroot/anaxagora/integration
rsync://a.cvs.sourceforge.net/cvsroot/pachanga/pachanguito
rsync://a.cvs.sourceforge.net/cvsroot/pentagesys/pentagesys
rsync://a.cvs.sourceforge.net/cvsroot/yut/myyutclient
rsync://a.cvs.sourceforge.net/cvsroot/sysmaniax/galaxy
rsync://a.cvs.sourceforge.net/cvsroot/ikonmagikos/core
rsync://a.cvs.sourceforge.net/cvsroot/konus/com.nsu.konus.documentation
rsync://a.cvs.sourceforge.net/cvsroot/jsenna/JSennaView
rsync://a.cvs.sourceforge.net/cvsroot/netpath/NetPathDesigner
rsync://a.cvs.sourceforge.net/cvsroot/keepalivexp/kZmieniacz
rsync://a.cvs.sourceforge.net/cvsroot/easyp2p/Iteracion5
rsync://a.cvs.sourceforge.net/cvsroot/phpsomvc/phpsomvc
rsync://a.cvs.sourceforge.net/cvsroot/yakushabb/ysm
rsync://a.cvs.sourceforge.net/cvsroot/elvesvsorcs/evo
rsync://a.cvs.sourceforge.net/cvsroot/esat/esat
rsync://a.cvs.sourceforge.net/cvsroot/dianti/Project
rsync://a.cvs.sourceforge.net/cvsroot/livrarianavi/LivrariaNavi
rsync://a.cvs.sourceforge.net/cvsroot/shushividwo/CVS_Demo
rsync://a.cvs.sourceforge.net/cvsroot/phpgwpl/phpgroupware
rsync://a.cvs.sourceforge.net/cvsroot/insamonitor/Docs
rsync://a.cvs.sourceforge.net/cvsroot/jspackager/JSPackager
rsync://a.cvs.sourceforge.net/cvsroot/firm1control/firm2control-win
rsync://a.cvs.sourceforge.net/cvsroot/jlibrary/org.jlibrary.client.nl1
rsync://a.cvs.sourceforge.net/cvsroot/math-game/math-game.dbfree
rsync://a.cvs.sourceforge.net/cvsroot/uengine/uEngine_Tomcat_Lep44
rsync://a.cvs.sourceforge.net/cvsroot/flyticket/FlyTicket_documentacao
rsync://a.cvs.sourceforge.net/cvsroot/uengine/uEngine-Standalone
rsync://a.cvs.sourceforge.net/cvsroot/gervasproject/documentos
rsync://a.cvs.sourceforge.net/cvsroot/crimewatch/CrimeWatch
rsync://a.cvs.sourceforge.net/cvsroot/dvx/dvxupdate
rsync://a.cvs.sourceforge.net/cvsroot/lefthand/lefthand
rsync://a.cvs.sourceforge.net/cvsroot/webman-cms/source
rsync://a.cvs.sourceforge.net/cvsroot/phpmybibli/PhpMyBibli
rsync://a.cvs.sourceforge.net/cvsroot/maccvspro/maccvs25
rsync://a.cvs.sourceforge.net/cvsroot/dacco/Publisher
rsync://a.cvs.sourceforge.net/cvsroot/icpac/organizational
rsync://a.cvs.sourceforge.net/cvsroot/upxshell/upxshell old v3.0
rsync://a.cvs.sourceforge.net/cvsroot/carnaval/Devel
rsync://a.cvs.sourceforge.net/cvsroot/himock/himock
rsync://a.cvs.sourceforge.net/cvsroot/bba/BuzzerBeaterAmor
rsync://a.cvs.sourceforge.net/cvsroot/crepuscule/dusk
rsync://a.cvs.sourceforge.net/cvsroot/amster/amster
rsync://a.cvs.sourceforge.net/cvsroot/ticketing/service
rsync://a.cvs.sourceforge.net/cvsroot/odb/eclipse
rsync://a.cvs.sourceforge.net/cvsroot/estudy/eStudy
rsync://a.cvs.sourceforge.net/cvsroot/remotedebugsys/rds-php
rsync://a.cvs.sourceforge.net/cvsroot/crpm/Gestion immo
rsync://a.cvs.sourceforge.net/cvsroot/omniathena/docs
rsync://a.cvs.sourceforge.net/cvsroot/nbt/nbt
rsync://a.cvs.sourceforge.net/cvsroot/jrv-dimji/Modulo Sonido
rsync://a.cvs.sourceforge.net/cvsroot/ag-cling/ag-cling
rsync://a.cvs.sourceforge.net/cvsroot/jamiga/src
rsync://a.cvs.sourceforge.net/cvsroot/crepuscule/checkin'
rsync://a.cvs.sourceforge.net/cvsroot/asphyxia/SistemaCaja
rsync://a.cvs.sourceforge.net/cvsroot/brouzer/proj
rsync://a.cvs.sourceforge.net/cvsroot/nbt/source
rsync://a.cvs.sourceforge.net/cvsroot/soclab/soclab-dev
rsync://a.cvs.sourceforge.net/cvsroot/weso/test
rsync://a.cvs.sourceforge.net/cvsroot/wcpr/trunk
rsync://a.cvs.sourceforge.net/cvsroot/laconicacms/Project
rsync://a.cvs.sourceforge.net/cvsroot/opensocks/Socks
rsync://a.cvs.sourceforge.net/cvsroot/eclipse-library/tapestry.components.contrib.test
rsync://a.cvs.sourceforge.net/cvsroot/visualmaths/visualMaths
rsync://a.cvs.sourceforge.net/cvsroot/freadcakeserver/myproject
rsync://a.cvs.sourceforge.net/cvsroot/sophy/Documentacion
rsync://a.cvs.sourceforge.net/cvsroot/bugtraking/docs
rsync://a.cvs.sourceforge.net/cvsroot/gervasproject/TheGervasProject
rsync://a.cvs.sourceforge.net/cvsroot/virtl/virtl
rsync://a.cvs.sourceforge.net/cvsroot/ltag/ParserLopez
rsync://a.cvs.sourceforge.net/cvsroot/valen/valenki
rsync://a.cvs.sourceforge.net/cvsroot/adeopensite/Docs
rsync://a.cvs.sourceforge.net/cvsroot/homeplayer/HomePlayer
rsync://a.cvs.sourceforge.net/cvsroot/govorilka/server
rsync://a.cvs.sourceforge.net/cvsroot/pyramidworkshop/libgenerators-0.0.0
rsync://a.cvs.sourceforge.net/cvsroot/pyramidworkshop/libPPM-0.0.0
rsync://a.cvs.sourceforge.net/cvsroot/computerwinkel/MOP_DMD+SSD
rsync://a.cvs.sourceforge.net/cvsroot/focuspoint/focuspoint
rsync://a.cvs.sourceforge.net/cvsroot/project-x/Project-X
rsync://a.cvs.sourceforge.net/cvsroot/catholiclibrary/webroot
rsync://a.cvs.sourceforge.net/cvsroot/upxshell/upxshell
rsync://a.cvs.sourceforge.net/cvsroot/macperl/perl
rsync://a.cvs.sourceforge.net/cvsroot/uebi/TamTamTamTam
rsync://a.cvs.sourceforge.net/cvsroot/maklerdesign/use cases
rsync://a.cvs.sourceforge.net/cvsroot/activex/activex
rsync://a.cvs.sourceforge.net/cvsroot/orionsbelt/alnitak
rsync://a.cvs.sourceforge.net/cvsroot/medslt/MedSLT2
rsync://a.cvs.sourceforge.net/cvsroot/rns/RNS
rsync://a.cvs.sourceforge.net/cvsroot/keepalivexp/kDodatki2
rsync://a.cvs.sourceforge.net/cvsroot/dinosys/dinosys
rsync://a.cvs.sourceforge.net/cvsroot/math-game/documents
rsync://a.cvs.sourceforge.net/cvsroot/image-stamper/SGT
rsync://a.cvs.sourceforge.net/cvsroot/mousave/mousave.2
rsync://a.cvs.sourceforge.net/cvsroot/math-game/math-game
rsync://a.cvs.sourceforge.net/cvsroot/itaforum/ITAF2
rsync://a.cvs.sourceforge.net/cvsroot/alternatif/tmp
rsync://a.cvs.sourceforge.net/cvsroot/showtheflow/Flow31
rsync://a.cvs.sourceforge.net/cvsroot/macbbdemux/bbDEMUX
rsync://a.cvs.sourceforge.net/cvsroot/gervasproject/TheGervasProjectCompleto
rsync://a.cvs.sourceforge.net/cvsroot/aka/aka
rsync://a.cvs.sourceforge.net/cvsroot/hmi-gesmon3/Athenea
rsync://a.cvs.sourceforge.net/cvsroot/mathou/moostik
rsync://a.cvs.sourceforge.net/cvsroot/bioma/webphp
rsync://a.cvs.sourceforge.net/cvsroot/peabase/org.peabase.dokuman
rsync://a.cvs.sourceforge.net/cvsroot/iaa-pascal/edytor interpretera
rsync://a.cvs.sourceforge.net/cvsroot/gusi/GUSI2
rsync://a.cvs.sourceforge.net/cvsroot/warsurfer/wsdocs
rsync://a.cvs.sourceforge.net/cvsroot/openarc/openarczerodottwo
rsync://a.cvs.sourceforge.net/cvsroot/underpop/projetos_lua_am6
rsync://a.cvs.sourceforge.net/cvsroot/moodle-kb/harjoittelualue
rsync://a.cvs.sourceforge.net/cvsroot/uaedev/uae
rsync://a.cvs.sourceforge.net/cvsroot/tacpp/documents
rsync://a.cvs.sourceforge.net/cvsroot/officeadmin/officeadmin
rsync://a.cvs.sourceforge.net/cvsroot/kdomino/kdomino
rsync://a.cvs.sourceforge.net/cvsroot/remotestart/kak-gorod-update
rsync://a.cvs.sourceforge.net/cvsroot/theaddressbook/104-beta
rsync://a.cvs.sourceforge.net/cvsroot/phpgb/main
rsync://a.cvs.sourceforge.net/cvsroot/pixtd/PixTD
rsync://a.cvs.sourceforge.net/cvsroot/reaper3d/reaper
rsync://a.cvs.sourceforge.net/cvsroot/mute-net/MUTE
rsync://a.cvs.sourceforge.net/cvsroot/xoops/modules
rsync://a.cvs.sourceforge.net/cvsroot/bacookbook/cookbook
rsync://a.cvs.sourceforge.net/cvsroot/freetris/freeTris
rsync://a.cvs.sourceforge.net/cvsroot/tagua/egimenez
rsync://a.cvs.sourceforge.net/cvsroot/pdexpenv/pdexpenv
rsync://a.cvs.sourceforge.net/cvsroot/babytalk/BabyTALK
rsync://a.cvs.sourceforge.net/cvsroot/tudomais/SoftPhone
rsync://a.cvs.sourceforge.net/cvsroot/aojava/aoserver
rsync://a.cvs.sourceforge.net/cvsroot/harbourboleto/harbourboleto
rsync://a.cvs.sourceforge.net/cvsroot/semantic-search/Text Classfication
rsync://a.cvs.sourceforge.net/cvsroot/imagein/Image-In
rsync://a.cvs.sourceforge.net/cvsroot/maccvspro/maccvs22
rsync://a.cvs.sourceforge.net/cvsroot/softmusic/softmusic
rsync://a.cvs.sourceforge.net/cvsroot/facturalux/modulos
rsync://a.cvs.sourceforge.net/cvsroot/freadcakeserver/FriedCakeServer
rsync://a.cvs.sourceforge.net/cvsroot/bitweaver/_bit_phpgedview
rsync://a.cvs.sourceforge.net/cvsroot/jbanana/JoltVideos
rsync://a.cvs.sourceforge.net/cvsroot/mindx/kard
rsync://a.cvs.sourceforge.net/cvsroot/enjoythetime/Documents
rsync://a.cvs.sourceforge.net/cvsroot/kevora/kevora-1
rsync://a.cvs.sourceforge.net/cvsroot/exte/exte
rsync://a.cvs.sourceforge.net/cvsroot/echoedit/echoedit
rsync://a.cvs.sourceforge.net/cvsroot/kdomino/recursos
rsync://a.cvs.sourceforge.net/cvsroot/medialib/MEDIANET
rsync://a.cvs.sourceforge.net/cvsroot/bawsuspectrecog/SuspectRecognition
rsync://a.cvs.sourceforge.net/cvsroot/cpuinformer/cpuinformer
rsync://a.cvs.sourceforge.net/cvsroot/j2me2dgamelib/loopeer_site
rsync://a.cvs.sourceforge.net/cvsroot/bookalyze/bookalyze
rsync://a.cvs.sourceforge.net/cvsroot/gervasproject/doblaje
rsync://a.cvs.sourceforge.net/cvsroot/openglyph/openglyph
rsync://a.cvs.sourceforge.net/cvsroot/jiuston/Jiuston
rsync://a.cvs.sourceforge.net/cvsroot/yacr/exdigim.copy
rsync://a.cvs.sourceforge.net/cvsroot/pbb-player/POO
rsync://a.cvs.sourceforge.net/cvsroot/musmap/musmap
rsync://a.cvs.sourceforge.net/cvsroot/aka/aka-dev
rsync://a.cvs.sourceforge.net/cvsroot/dex/test_dex
rsync://a.cvs.sourceforge.net/cvsroot/arkidb/Arki_DB Development
rsync://a.cvs.sourceforge.net/cvsroot/whisper2/Whisper
rsync://a.cvs.sourceforge.net/cvsroot/oo-cs/OOoFlyer
rsync://a.cvs.sourceforge.net/cvsroot/computerwinkel/Nieuwe_IDs
rsync://a.cvs.sourceforge.net/cvsroot/myseelite/myseelite
rsync://a.cvs.sourceforge.net/cvsroot/sisdec/HCEWeb
rsync://a.cvs.sourceforge.net/cvsroot/yoksel/Tests1Cv8
rsync://a.cvs.sourceforge.net/cvsroot/pyramidworkshop/libmemory-0.0.0
rsync://a.cvs.sourceforge.net/cvsroot/pnptime/zeiterfassung
rsync://a.cvs.sourceforge.net/cvsroot/mtgplay/src
rsync://a.cvs.sourceforge.net/cvsroot/future-tv/Future-TV
rsync://a.cvs.sourceforge.net/cvsroot/pg520/data
rsync://a.cvs.sourceforge.net/cvsroot/balie/BaLIE
rsync://a.cvs.sourceforge.net/cvsroot/phpkrond/phpkrond
rsync://a.cvs.sourceforge.net/cvsroot/olitext/olitext
rsync://a.cvs.sourceforge.net/cvsroot/tableview/htdocs
rsync://a.cvs.sourceforge.net/cvsroot/visualos/VisualOS
rsync://a.cvs.sourceforge.net/cvsroot/g3gc/Test
rsync://a.cvs.sourceforge.net/cvsroot/firemonger/firemonger
rsync://a.cvs.sourceforge.net/cvsroot/dazibo/game
rsync://a.cvs.sourceforge.net/cvsroot/erlyaws/yaws
rsync://a.cvs.sourceforge.net/cvsroot/cap4e/CAP.ui
rsync://a.cvs.sourceforge.net/cvsroot/dnsmail/projeler
rsync://a.cvs.sourceforge.net/cvsroot/warsurfer/warsurfer
rsync://a.cvs.sourceforge.net/cvsroot/emofilt/emofilt
rsync://a.cvs.sourceforge.net/cvsroot/asphyxia/SistemaLiquidaciones
rsync://a.cvs.sourceforge.net/cvsroot/remind/MIND_Publication
rsync://a.cvs.sourceforge.net/cvsroot/dnncb/epContentBuilder
rsync://a.cvs.sourceforge.net/cvsroot/tacpp/sources
rsync://a.cvs.sourceforge.net/cvsroot/uniquecms/Reseau2
rsync://a.cvs.sourceforge.net/cvsroot/ppo-erd/SAAS
rsync://a.cvs.sourceforge.net/cvsroot/lookmediaplayer/Lecteur
rsync://a.cvs.sourceforge.net/cvsroot/littleprince/LP_System
rsync://a.cvs.sourceforge.net/cvsroot/image2xaml/image2xaml
rsync://a.cvs.sourceforge.net/cvsroot/supertruco/dumb
rsync://a.cvs.sourceforge.net/cvsroot/pyramidworkshop/PyramidWorkshop-0.0.0
rsync://a.cvs.sourceforge.net/cvsroot/beamer/V1
rsync://a.cvs.sourceforge.net/cvsroot/resumidor/resumidor
rsync://a.cvs.sourceforge.net/cvsroot/fosup/foscrm
rsync://a.cvs.sourceforge.net/cvsroot/cvswebclient/htdocs
rsync://a.cvs.sourceforge.net/cvsroot/redkernel/BangBang
rsync://a.cvs.sourceforge.net/cvsroot/dazero/WDZ-1
rsync://a.cvs.sourceforge.net/cvsroot/afly/Tools
rsync://a.cvs.sourceforge.net/cvsroot/algviewer/algviewer
rsync://a.cvs.sourceforge.net/cvsroot/fcdsgamelibrary/pacman
rsync://a.cvs.sourceforge.net/cvsroot/weso/homePage
rsync://a.cvs.sourceforge.net/cvsroot/cardgame/CardGame
rsync://a.cvs.sourceforge.net/cvsroot/c2iexplorer/c2iexplr
rsync://a.cvs.sourceforge.net/cvsroot/maxtrixengine/maxtrixenginetest
rsync://a.cvs.sourceforge.net/cvsroot/ibwebadmin/ibwebadmin
rsync://a.cvs.sourceforge.net/cvsroot/jathena/jathena
rsync://a.cvs.sourceforge.net/cvsroot/ontea/SocialNetworks
rsync://a.cvs.sourceforge.net/cvsroot/littleprince/Felipe
rsync://a.cvs.sourceforge.net/cvsroot/nemoz/Nemoz
rsync://a.cvs.sourceforge.net/cvsroot/unicalculus/unignuplot
rsync://a.cvs.sourceforge.net/cvsroot/sector37/_Material_
rsync://a.cvs.sourceforge.net/cvsroot/thereflector/reflector
rsync://a.cvs.sourceforge.net/cvsroot/paediatricrcp/gr.osmosis.paediatric

check_cvs_repos_with_path_encoding_issue_loading.sh

#!/bin/bash

set -e

input="cvs_repos_with_path_encoding_issue.txt"
while IFS= read -r line
do
  swh -l DEBUG loader -C ~/.config/swh/loader/cvs.yml run cvs "$line"
done < "$input"

Diff Detail

Repository

rDLDCVS CVS Loader

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Jul 6 2022, 4:06 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJul 6 2022, 4:06 PM

anlambert edited the test plan for this revision. (Show Details)Jul 6 2022, 4:07 PM

Build is green

Patch application report for D8086 (id=29189)

Rebasing onto b35a9769a0...

Current branch diff-target is up to date.

Changes applied before test

commit d89f8d13e9470c7cea81590e8425370fa6647fa7
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Jul 1 12:15:53 2022 +0200

    Fix loading of CVS repositories with non valid UTF-8 paths
    
    Some CVS repositories have paths which are non valid UTF-8 (typically
    ISO-8859-1 ones) but the loader implementation assumed all paths can
    be safely encoded to UTF-8 and was raising UnicodeEncodeError when
    attempting to encode non UTF-8 paths.
    
    That commit modifies the way CVS paths are handled by the loader by
    using their raw bytes representation instead of their UTF-8 decoded
    string representation.
    
    Also rcsparse.rcsfile constructor has been modified to take bytes path
    as argument instead of an unicode one in order to be able to successfully
    open non UTF-8 paths.
    
    Such CVS repositories can now be successfully loaded, either using rsync
    or pserver protocol.
    
    Related to T3980

See https://jenkins.softwareheritage.org/job/DLDCVS/job/tests-on-diff/117/ for more details.

Harbormaster completed remote builds in B30278: Diff 29189.Jul 6 2022, 4:10 PM

anlambert requested review of this revision.Jul 6 2022, 4:10 PM

Have you considered using the opportunity to switch to pathlib?

In D8086#210297, @vlorentz wrote:

Have you considered using the opportunity to switch to pathlib?

No because it was simpler to simply replace str path by bytes path in the loader code and I did not want to modify too much code. What would be the advantage of using pathlib here ?

Its syntax is more convenient and it automatically deals with encoding. eg. wtpath = os.path.join(os.fsencode(self.tempdir_path), path) becomes wtpath = self.tempdir_path / path with pathlib

In D8086#210308, @vlorentz wrote:

Its syntax is more convenient and it automatically deals with encoding. eg. wtpath = os.path.join(os.fsencode(self.tempdir_path), path) becomes wtpath = self.tempdir_path / path with pathlib

I see, maybe we could handle such change in another diff ? If I recall correctly, the subversion loader implementation could also be migrated to pathlib.

Its syntax is more convenient and it automatically deals with encoding. eg. wtpath = os.path.join(os.fsencode(self.tempdir_path), path) becomes wtpath = self.tempdir_path / path with pathlib

I see, maybe we could handle such change in another diff ? If I recall correctly, the subversion loader implementation could also be migrated to pathlib.

yes, totally, another diff ;)

We did not start with pathlib for historical reason, we started with python3.5 and it was not integrated at the time iirc.

vlorentz accepted this revision.Jul 7 2022, 10:36 AM

This revision is now accepted and ready to land.Jul 7 2022, 10:36 AM

Closed by commit rDLDCVSd89f8d13e947: Fix loading of CVS repositories with non valid UTF-8 paths (authored by anlambert). · Explain WhyJul 7 2022, 10:42 AM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDLDCVSd89f8d13e947: Fix loading of CVS repositories with non valid UTF-8 paths.