Page MenuHomeSoftware Heritage

SvnLoaderFromRemoteDump: Add debug output when svnrdump is running
ClosedPublic

Authored by anlambert on Mar 8 2019, 3:09 PM.

Details

Summary

This diff adds svnrdump stderr output to logs when logging level is set to DEBUG.
It enables to keep track of the svn revisions dumped so far, see below:

[2019-03-08 14:57:07,216: INFO/MainProcess] Received task: swh.loader.svn.tasks.DumpMountAndLoadSvnRepository[13bff320-be15-4ea2-af2a-bcdda8cdf69a]  
[2019-03-08 14:57:07,219: INFO/ForkPoolWorker-4] Loading config file /home/antoine/.config/swh/scheduler/worker.yml
[2019-03-08 14:57:07,226: DEBUG/ForkPoolWorker-4] Creating svn origin for https://svn.code.sf.net/p/gparted/svn/
[2019-03-08 14:57:07,238: DEBUG/ForkPoolWorker-4] Done creating svn origin for https://svn.code.sf.net/p/gparted/svn/
[2019-03-08 14:57:07,239: DEBUG/ForkPoolWorker-4] Creating origin_visit for origin 21 at time 2019-03-08 13:57:07.238901+00:00
[2019-03-08 14:57:07,245: DEBUG/ForkPoolWorker-4] Done Creating origin_visit for origin 21 at time 2019-03-08 13:57:07.238901+00:00
[2019-03-08 14:57:07,268: DEBUG/ForkPoolWorker-4] Executing svnrdump dump https://svn.code.sf.net/p/gparted/svn/
[2019-03-08 14:57:09,118: DEBUG/ForkPoolWorker-4] * Dumped revision 0.
[2019-03-08 14:57:09,271: DEBUG/ForkPoolWorker-4] * Dumped revision 1.
[2019-03-08 14:57:09,425: DEBUG/ForkPoolWorker-4] * Dumped revision 2.
[2019-03-08 14:57:09,578: DEBUG/ForkPoolWorker-4] * Dumped revision 3.
[2019-03-08 14:57:09,733: DEBUG/ForkPoolWorker-4] * Dumped revision 4.
[2019-03-08 14:57:15,147: DEBUG/ForkPoolWorker-4] * Dumped revision 5.
[2019-03-08 14:57:15,318: DEBUG/ForkPoolWorker-4] * Dumped revision 6.
[2019-03-08 14:57:15,487: DEBUG/ForkPoolWorker-4] * Dumped revision 7.
[2019-03-08 14:57:15,642: DEBUG/ForkPoolWorker-4] * Dumped revision 8.
[2019-03-08 14:57:15,820: DEBUG/ForkPoolWorker-4] * Dumped revision 9.
[2019-03-08 14:57:15,984: DEBUG/ForkPoolWorker-4] * Dumped revision 10.
[2019-03-08 14:57:16,145: DEBUG/ForkPoolWorker-4] * Dumped revision 11.
[2019-03-08 14:57:16,347: DEBUG/ForkPoolWorker-4] * Dumped revision 12.
[2019-03-08 14:57:19,266: DEBUG/ForkPoolWorker-4] * Dumped revision 13.
[2019-03-08 14:57:19,424: DEBUG/ForkPoolWorker-4] * Dumped revision 14.
[2019-03-08 14:57:19,589: DEBUG/ForkPoolWorker-4] * Dumped revision 15.
[2019-03-08 14:57:19,742: DEBUG/ForkPoolWorker-4] * Dumped revision 16.
[2019-03-08 14:57:19,900: DEBUG/ForkPoolWorker-4] * Dumped revision 17.
[2019-03-08 14:57:20,057: DEBUG/ForkPoolWorker-4] * Dumped revision 18.
[2019-03-08 14:57:20,509: DEBUG/ForkPoolWorker-4] * Dumped revision 19.
[2019-03-08 14:57:20,666: DEBUG/ForkPoolWorker-4] * Dumped revision 20.
[2019-03-08 14:57:20,868: DEBUG/ForkPoolWorker-4] * Dumped revision 21.
[2019-03-08 14:57:21,026: DEBUG/ForkPoolWorker-4] * Dumped revision 22.
[2019-03-08 14:57:21,181: DEBUG/ForkPoolWorker-4] * Dumped revision 23.
[2019-03-08 14:57:21,332: DEBUG/ForkPoolWorker-4] * Dumped revision 24.
[2019-03-08 14:57:21,496: DEBUG/ForkPoolWorker-4] * Dumped revision 25.
[2019-03-08 14:57:21,655: DEBUG/ForkPoolWorker-4] * Dumped revision 26.
[2019-03-08 14:57:21,812: DEBUG/ForkPoolWorker-4] * Dumped revision 27.
[2019-03-08 14:57:21,975: DEBUG/ForkPoolWorker-4] * Dumped revision 28.
...

Diff Detail

Repository
rDLDSVN Subversion (SVN) loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert retitled this revision from SvnLoaderFromRemoteDump: Add some debug output when svnrdump is running to SvnLoaderFromRemoteDump: Add debug output when svnrdump is running.
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/loader/svn/loader.py
659–666

nitpick:

readable = True
while readable:
    lines, readable = stderr_stream.read_lines() 
    # ...
663–664

Revisions are fetched in order, right?

If so, all but the last line can be skipped, it will make logs less verbose.

swh/loader/svn/utils.py
37–42

You should document that it assumes the stream is opened in text mode.

54

if there are

69

nitpick: return (lines[:-1], True)

74

same

This revision is now accepted and ready to land.Mar 11 2019, 2:57 PM
swh/loader/svn/loader.py
663–664

yes, fetched in order.

anlambert added inline comments.
swh/loader/svn/loader.py
663–664

The idea here is to get an idea of the current dump progress, so I think it is better to print all dumped revision info.

swh/loader/svn/utils.py
37–42

ack

54

well seen

Update: address vlorentz comments

This revision was automatically updated to reflect the committed changes.