SvnLoaderFromRemoteDump: Add debug output when svnrdump is running
ClosedPublic
Actions

Authored by anlambert on Mar 8 2019, 3:09 PM.

Details

Reviewers

ardumont
vlorentz

Group Reviewers

Reviewers

Commits

rDLDSVN90fc56a91444: SvnLoaderFromRemoteDump: Add some debug output when svnrdump is running

Summary

This diff adds svnrdump stderr output to logs when logging level is set to DEBUG.
It enables to keep track of the svn revisions dumped so far, see below:

[2019-03-08 14:57:07,216: INFO/MainProcess] Received task: swh.loader.svn.tasks.DumpMountAndLoadSvnRepository[13bff320-be15-4ea2-af2a-bcdda8cdf69a]  
[2019-03-08 14:57:07,219: INFO/ForkPoolWorker-4] Loading config file /home/antoine/.config/swh/scheduler/worker.yml
[2019-03-08 14:57:07,226: DEBUG/ForkPoolWorker-4] Creating svn origin for https://svn.code.sf.net/p/gparted/svn/
[2019-03-08 14:57:07,238: DEBUG/ForkPoolWorker-4] Done creating svn origin for https://svn.code.sf.net/p/gparted/svn/
[2019-03-08 14:57:07,239: DEBUG/ForkPoolWorker-4] Creating origin_visit for origin 21 at time 2019-03-08 13:57:07.238901+00:00
[2019-03-08 14:57:07,245: DEBUG/ForkPoolWorker-4] Done Creating origin_visit for origin 21 at time 2019-03-08 13:57:07.238901+00:00
[2019-03-08 14:57:07,268: DEBUG/ForkPoolWorker-4] Executing svnrdump dump https://svn.code.sf.net/p/gparted/svn/
[2019-03-08 14:57:09,118: DEBUG/ForkPoolWorker-4] * Dumped revision 0.
[2019-03-08 14:57:09,271: DEBUG/ForkPoolWorker-4] * Dumped revision 1.
[2019-03-08 14:57:09,425: DEBUG/ForkPoolWorker-4] * Dumped revision 2.
[2019-03-08 14:57:09,578: DEBUG/ForkPoolWorker-4] * Dumped revision 3.
[2019-03-08 14:57:09,733: DEBUG/ForkPoolWorker-4] * Dumped revision 4.
[2019-03-08 14:57:15,147: DEBUG/ForkPoolWorker-4] * Dumped revision 5.
[2019-03-08 14:57:15,318: DEBUG/ForkPoolWorker-4] * Dumped revision 6.
[2019-03-08 14:57:15,487: DEBUG/ForkPoolWorker-4] * Dumped revision 7.
[2019-03-08 14:57:15,642: DEBUG/ForkPoolWorker-4] * Dumped revision 8.
[2019-03-08 14:57:15,820: DEBUG/ForkPoolWorker-4] * Dumped revision 9.
[2019-03-08 14:57:15,984: DEBUG/ForkPoolWorker-4] * Dumped revision 10.
[2019-03-08 14:57:16,145: DEBUG/ForkPoolWorker-4] * Dumped revision 11.
[2019-03-08 14:57:16,347: DEBUG/ForkPoolWorker-4] * Dumped revision 12.
[2019-03-08 14:57:19,266: DEBUG/ForkPoolWorker-4] * Dumped revision 13.
[2019-03-08 14:57:19,424: DEBUG/ForkPoolWorker-4] * Dumped revision 14.
[2019-03-08 14:57:19,589: DEBUG/ForkPoolWorker-4] * Dumped revision 15.
[2019-03-08 14:57:19,742: DEBUG/ForkPoolWorker-4] * Dumped revision 16.
[2019-03-08 14:57:19,900: DEBUG/ForkPoolWorker-4] * Dumped revision 17.
[2019-03-08 14:57:20,057: DEBUG/ForkPoolWorker-4] * Dumped revision 18.
[2019-03-08 14:57:20,509: DEBUG/ForkPoolWorker-4] * Dumped revision 19.
[2019-03-08 14:57:20,666: DEBUG/ForkPoolWorker-4] * Dumped revision 20.
[2019-03-08 14:57:20,868: DEBUG/ForkPoolWorker-4] * Dumped revision 21.
[2019-03-08 14:57:21,026: DEBUG/ForkPoolWorker-4] * Dumped revision 22.
[2019-03-08 14:57:21,181: DEBUG/ForkPoolWorker-4] * Dumped revision 23.
[2019-03-08 14:57:21,332: DEBUG/ForkPoolWorker-4] * Dumped revision 24.
[2019-03-08 14:57:21,496: DEBUG/ForkPoolWorker-4] * Dumped revision 25.
[2019-03-08 14:57:21,655: DEBUG/ForkPoolWorker-4] * Dumped revision 26.
[2019-03-08 14:57:21,812: DEBUG/ForkPoolWorker-4] * Dumped revision 27.
[2019-03-08 14:57:21,975: DEBUG/ForkPoolWorker-4] * Dumped revision 28.
...

Diff Detail

Repository

rDLDSVN Subversion (SVN) loader

Branch

svnrdump-debug-log

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 4572
Build 6062: tox-on-jenkins	Jenkins
Build 6061: arc lint + arc unit

Event Timeline

anlambert created this revision.Mar 8 2019, 3:09 PM

anlambert added a reviewer: Reviewers.Mar 8 2019, 3:09 PM

anlambert retitled this revision from SvnLoaderFromRemoteDump: Add some debug output when svnrdump is running to SvnLoaderFromRemoteDump: Add debug output when svnrdump is running.

Build is green
See https://jenkins.softwareheritage.org/job/DLDSVN/job/tox/30/ for more details.

Harbormaster completed remote builds in B4572: Diff 3866.Mar 8 2019, 3:11 PM

vlorentz accepted this revision.Mar 11 2019, 2:57 PM

vlorentz added a subscriber: vlorentz.

vlorentz added inline comments.

swh/loader/svn/loader.py
660–667	nitpick: readable = True while readable: lines, readable = stderr_stream.read_lines() # ...
664–665	Revisions are fetched in order, right? If so, all but the last line can be skipped, it will make logs less verbose.
swh/loader/svn/utils.py
38–43	You should document that it assumes the stream is opened in text mode.
55	if there are
70	nitpick: `return (lines[:-1], True)`
75	same

This revision is now accepted and ready to land.Mar 11 2019, 2:57 PM

ardumont added inline comments.Mar 11 2019, 3:05 PM

swh/loader/svn/loader.py
664–665	yes, fetched in order.

anlambert marked 3 inline comments as done.Mar 11 2019, 3:41 PM

anlambert added inline comments.

swh/loader/svn/loader.py
664–665	The idea here is to get an idea of the current dump progress, so I think it is better to print all dumped revision info.
swh/loader/svn/utils.py
38–43	ack
55	well seen