ingestion starting date: 11/01/2017 13:00

Currently running on swh's internal infrastructure workers.

This is not solved yet.

...
This could hint at something to improve the actual workaround.

The road so far...

I am unable to reproduce this behavior in integration tests...
Svn-loader's initial code computes correctly the hashes (I have exported each revision and computed separately the hashes independently to be certain of this fact).

Taking a look at why the export on subvertpy worked, it's because, the 'native_eol' flag is NULL by default in subvertpy [1].
This then goes on and call the svn lib api binding [2].
That svn lib api binding, for that null value for that flag, falls back to using the 'standard eol marker'. Which is LF on Linux platforms.
So ok for why subvertpy exports works out of the box like svn.

It falls down to the revision 23242 which creates a diverging tree.
Files have a CRLF end of line whereas it should be LF (well according to svn export).

This tasks takes care of:

• loader-core
• loader-dir (depends on loader-core)
• loader-tar (depends on loader-dir and loader-core)
• loader-git
• loader-svn

I've also added some other dependencies from other modules (swh-web-ui, swh-fetcher-googlecode...)
https://wiki.softwareheritage.org/index.php?title=Licensing&curid=70&diff=374&oldid=205

Update on this are on the https://sympa.inria.fr/sympa/arc/swh-devel/2016-06/msg00011.html

Tryout information:

• repo: asf via http connection (as mentioned in description)
• machine: worker01 with a local remote storage plugged to softwareheritage-test-svn db.

Done in:
6326932 * Add integration tests about update cases
e70418b * Add integration test about svn repository update
1c73b36 * Rename appropriately test_converters.org to test_loader.org
7eb8a5d * Add IT tests for one svn repository loading through swh or gitsvn policy

cf. T386#6859 entitled 'comparison - with swh-storage'

Sum up

Every comment has a date which is a permalink to the comment, just use that link directly, or add the trailing #something part to the task name, e.g.: T411#6857

(how do we make link to comment on phab?)

Great news!

swh-loader-svn faster now.

Using the same principle as git-svn (remote-access client), we are now faster than git-svn either with or without storage.

IIRC jelmer/subvertpy does but it's python2 --> This is dulwich's author...

My hypothesis for git-svn is that they orchestrate themselves svn export and diffs.

My take is that we do not want keywords expanded when importing an SVN
So, keeping the unexpanded version of the file would be the safest bet for
having some kind of normal form of it.

The first step to take is to decide whether we want keyword expansion or not.

To give visibility about this.

Nothing in docs about it.
pysvn docs: http://pysvn.stage.tigris.org/docs/pysvn_prog_ref.html#pysvn_client_export

1. Same hash tree

Running a batch on the current 5 repositories with data sending to storage inhibited.

Unfortunately no.
I don't know how to compare fairly those tools and have the details.

Follow up on this.

I see a factor ranging from 2 for short version histories to 8 for long
histories w.r.t. git-svn, which is difficult to analyse like this.

I see a factor ranging from 2 for short version histories to 8 for long
histories w.r.t. git-svn, which is difficult to analyse like this.

swh-svn hash been updated with the following behavior.

conclusion: git-svn faster.