SVN loader: Add efficient loader based on remote dumps
ClosedPublic
Actions

Authored by anlambert on Sep 24 2018, 2:21 PM.

Details

Reviewers

Group Reviewers

Summary

This diff adds a new loader class SvnLoaderFromRemoteDump enabling
to load svn repositories in a more efficient way. The loader is based
on the creation of dump files generated with the svnrdump tool. Using
dump files allow to greatly speedup the loading process compared
to a client/server based approach.

This is an updated version of D433 (automatically closed as I
pushed it in a separate branch to keep track of the work done
so far), discarding the incremental loading approach through the
use of partial dumps (as numerous tests with real world repositories
end up with errors).

Related T1161

Diff Detail

Repository

rDLDSVN Subversion (SVN) loader

Branch

svnrdump-loader

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 1441
Build 1785: arc lint + arc unit

Event Timeline

anlambert created this revision.Sep 24 2018, 2:21 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptSep 24 2018, 2:21 PM

Harbormaster completed remote builds in B1434: Diff 1349.Sep 24 2018, 2:21 PM

Sounds good.

Having a test for it would be nice.

Not that i would know how to proceed though... Would file:///something/somewhere would be considered as a remote svn repository? I would think so but i'm not 100% certain of it.

swh/loader/svn/loader.py
694	Wondering if that could also be used in the loader that mounts a dump from an archive dump: Retrieve last svn revision seen for it (if any) Truncate the dump (both compressed/uncompressed now that it is opened here ;) Then load

Update diff:

remove unneeded parameter to SvnLoaderFromRemoteDump constructor
add test

ardumont added inline comments.Sep 25 2018, 12:03 PM

swh/loader/svn/loader.py
694	Clarifying. Today, when mounting a compressed archive dump as a repository prior to loading it, we do not check in advance if we have data for it or not. Whatever the reality of it, we mount it completely. I was hoping to reduce the mounting to whatever is necessary. If we were what you do there, checking for last svn revision seen, we could try and truncate the dump starting from that revision only. But maybe i'm mixing information from the initial diff and/or that is not doable ;) Cheers,

ardumont accepted this revision.Sep 25 2018, 12:07 PM

ardumont added inline comments.

swh/loader/svn/tests/test_loader.py
975	\m/

This revision is now accepted and ready to land.Sep 25 2018, 12:07 PM

anlambert added inline comments.Sep 25 2018, 1:11 PM

swh/loader/svn/loader.py
694	Unfortunately, truncating a dump file to a range [revN, revM] where N > 0 can lead to svnadmin unable to mount a repository from it. This is what I got in https://forge.softwareheritage.org/T1161#22461 when trying to incrementally load the apache repo. Besides that, there is no issue to truncate to range [rev0, revN] as the whole repository history can be replayed without errors.

Harbormaster completed remote builds in B1441: Diff 1356.Sep 25 2018, 2:19 PM

Closed by commit rDLDSVN677225bbd57d (I fixed a last minute typo detected by codespell, so original commit id changed).

(I fixed a last minute typo detected by codespell, so original commit id changed).

ok ;)

ardumont added inline comments.Sep 25 2018, 2:50 PM

swh/loader/svn/loader.py
694	Unfortunately, truncating a dump file to a range [revN, revM] where N > 0 can lead to svnadmin unable to mount a repository from it. This is what I got in https://forge.softwareheritage.org/T1161#22461 when trying to incrementally load the apache repo. Right. But that's not where i was aiming. Besides that, there is no issue to truncate to range [rev0, revN] as the whole repository history can be replayed without errors. That's what i wanted to say, sorry for not being so clear today ;) That way, we can replay time and again the same local dump without actually actually always mounting the dump for nothing (since no new stuff exist).