Page MenuHomeSoftware Heritage
Differential D8787

loader: Compress dump file and rework truncated dump handling
ClosedPublic
Actions

Authored by anlambert on Oct 27 2022, 3:49 PM.

Details

Reviewers
vlorentz
ardumont
Group Reviewers
Reviewers
Commits
rDLDSVNd24ba1a5ccd6: loader: Compress dump file and rework truncated dump handling
Summary

When dumping a subversion repository to file before loading it, compress
that file using gzip while producing it. It enables to save significant
disk space while dumping a large repository.

Also rework the way how truncated dump is handled now dump file is
compressed by providing the expected max revision number to be loaded
by svnadmin. If the number of loaded revisions matches, we can safely
continue the partial loading of the repository.

Prio this change, the size of dump file for svn://tug.org/texlive was
more than 80Gb while its compressed version has a size of 46,9Gb.

Depends on D8786

Diff Detail

Repository
rDLDSVN Subversion (SVN) loader
Branch
svnrdump-gzip
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32626
Build 51110: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 51109: arc lint + arc unit

Event Timeline

anlambert created this revision.Oct 27 2022, 3:49 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 27 2022, 3:49 PM
swh-public-ci added a comment.Oct 27 2022, 3:53 PM

Build is green

Patch application report for D8787 (id=31670)

Rebasing onto 8c709079ce...

Current branch diff-target is up to date.
Changes applied before test
commit b46ee14525187afad80f066adfe9a113c495b37f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 27 12:17:22 2022 +0200

    loader: Compress dump file and rework truncated dump handling
    
    When dumping a subversion repository to file before loading it, compress
    that file using gzip while producing it. It enables to save significant
    disk space while dumping a large repository.
    
    Also rework the way how truncated dump is handled now dump file is
    compressed by providing the expected max revision number to be loaded
    by svnadmin. If the number of loaded revisions matches, we can safely
    continue the partial loading of the repository.

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/345/ for more details.

Harbormaster completed remote builds in B32623: Diff 31670.Oct 27 2022, 3:53 PM
anlambert requested review of this revision.Oct 27 2022, 3:53 PM
anlambert edited the summary of this revision. (Show Details)Oct 27 2022, 4:37 PM
anlambert added a parent revision: D8786: svn: Ensure to quote URLs provided as parameters to client methods.
anlambert updated this revision to Diff 31673.Oct 27 2022, 4:37 PM

Rebase

swh-public-ci added a comment.Oct 27 2022, 4:41 PM

Build is green

Patch application report for D8787 (id=31673)

Could not rebase; Attempt merge onto 8c709079ce...

Updating 8c70907..e0e971d
Fast-forward
 swh/loader/svn/loader.py            | 44 ++++++++++++++-----------------------
 swh/loader/svn/svn.py               | 16 +++++++++-----
 swh/loader/svn/tests/test_loader.py | 27 ++++++++++++++++++++++-
 swh/loader/svn/tests/test_utils.py  | 41 +++++++++++++++++++++++++++++++++-
 swh/loader/svn/utils.py             | 23 ++++++++++++++-----
 5 files changed, 109 insertions(+), 42 deletions(-)
Changes applied before test
commit e0e971ddda8d503d75ae734e76a2fc43157be9e4
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 27 12:17:22 2022 +0200

    loader: Compress dump file and rework truncated dump handling
    
    When dumping a subversion repository to file before loading it, compress
    that file using gzip while producing it. It enables to save significant
    disk space while dumping a large repository.
    
    Also rework the way how truncated dump is handled now dump file is
    compressed by providing the expected max revision number to be loaded
    by svnadmin. If the number of loaded revisions matches, we can safely
    continue the partial loading of the repository.

commit 991e2b4ffced44d5483b115298e3f56d1f34c90b
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 27 11:29:36 2022 +0200

    svn: Ensure to quote URLs provided as parameters to client methods
    
    URLs provided as parameters to subvertpy.client.Client methods must
    be quoted or an assertion will be raised by libsvn otherwise.

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/346/ for more details.

Harbormaster completed remote builds in B32626: Diff 31673.Oct 27 2022, 4:41 PM
vlorentz accepted this revision.Oct 27 2022, 6:54 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/loader/svn/tests/test_utils.py
131

assert dump_lines > 150 to make sure the test is useful

This revision is now accepted and ready to land.Oct 27 2022, 6:54 PM
anlambert updated this revision to Diff 31681.Oct 28 2022, 11:22 AM

Rebase

swh-public-ci added a comment.Oct 28 2022, 11:25 AM

Build is green

Patch application report for D8787 (id=31681)

Could not rebase; Attempt merge onto 8c709079ce...

Updating 8c70907..c9f006e
Fast-forward
 swh/loader/svn/loader.py            | 44 ++++++++++++++-----------------------
 swh/loader/svn/replay.py            |  2 +-
 swh/loader/svn/svn.py               | 14 +++++++-----
 swh/loader/svn/tests/test_loader.py | 27 ++++++++++++++++++++++-
 swh/loader/svn/tests/test_utils.py  | 41 +++++++++++++++++++++++++++++++++-
 swh/loader/svn/utils.py             | 23 ++++++++++++++-----
 6 files changed, 109 insertions(+), 42 deletions(-)
Changes applied before test
commit c9f006e1f4abfce0ad8da4bebcc738dc1a3689f6
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 27 12:17:22 2022 +0200

    loader: Compress dump file and rework truncated dump handling
    
    When dumping a subversion repository to file before loading it, compress
    that file using gzip while producing it. It enables to save significant
    disk space while dumping a large repository.
    
    Also rework the way how truncated dump is handled now dump file is
    compressed by providing the expected max revision number to be loaded
    by svnadmin. If the number of loaded revisions matches, we can safely
    continue the partial loading of the repository.

commit c6d39b7bb70bb5896840a0fc22f8763d5d8b35cf
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 27 11:29:36 2022 +0200

    svn: Ensure to quote URLs provided as parameters to client methods
    
    URLs provided as parameters to subvertpy.client.Client methods must
    be quoted when it contains space characters or an assertion will be
    raised by libsvn otherwise.

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/348/ for more details.

Harbormaster completed remote builds in B32633: Diff 31681.Oct 28 2022, 11:25 AM
anlambert added a child revision: D8793: replay: Ensure copyfrom operations are properly handled.Oct 28 2022, 6:13 PM
ardumont accepted this revision.Oct 31 2022, 2:21 PM
ardumont added a subscriber: ardumont.

lgtm

one non-blocking suggestion to explicit the compressed nature of the dump inline.

swh/loader/svn/loader.py
662
anlambert updated this revision to Diff 31699.Oct 31 2022, 3:06 PM

Update: Address @ardumont and @vlorentz comments

swh-public-ci added a comment.Oct 31 2022, 3:10 PM

Build is green

Patch application report for D8787 (id=31699)

Could not rebase; Attempt merge onto 8c709079ce...

Updating 8c70907..d24ba1a
Fast-forward
 swh/loader/svn/loader.py            | 54 +++++++++++++++----------------------
 swh/loader/svn/replay.py            |  2 +-
 swh/loader/svn/svn.py               | 14 ++++++----
 swh/loader/svn/tests/test_loader.py | 27 ++++++++++++++++++-
 swh/loader/svn/tests/test_utils.py  | 42 ++++++++++++++++++++++++++++-
 swh/loader/svn/utils.py             | 23 +++++++++++-----
 6 files changed, 115 insertions(+), 47 deletions(-)
Changes applied before test
commit d24ba1a5ccd6c825cc362a62d673446c3176ab7c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 27 12:17:22 2022 +0200

    loader: Compress dump file and rework truncated dump handling
    
    When dumping a subversion repository to file before loading it, compress
    that file using gzip while producing it. It enables to save significant
    disk space while dumping a large repository.
    
    Also rework the way how truncated dump is handled now dump file is
    compressed by providing the expected max revision number to be loaded
    by svnadmin. If the number of loaded revisions matches, we can safely
    continue the partial loading of the repository.

commit c6d39b7bb70bb5896840a0fc22f8763d5d8b35cf
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Oct 27 11:29:36 2022 +0200

    svn: Ensure to quote URLs provided as parameters to client methods
    
    URLs provided as parameters to subvertpy.client.Client methods must
    be quoted when it contains space characters or an assertion will be
    raised by libsvn otherwise.

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/353/ for more details.

Harbormaster completed remote builds in B32649: Diff 31699.Oct 31 2022, 3:10 PM
Closed by commit rDLDSVNd24ba1a5ccd6: loader: Compress dump file and rework truncated dump handling (authored by anlambert). · Explain WhyOct 31 2022, 3:16 PM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDLDSVNd24ba1a5ccd6: loader: Compress dump file and rework truncated dump handling.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
loader/
svn/
44 lines88%
Loading...
tests/
2 lines99%
Loading...
41 lines100%
Loading...
23 lines98%
Loading...

Diff 31673

View Options

swh/loader/svn/loader.py

Loading...
View Options

swh/loader/svn/tests/test_loader.py

Loading...
View Options

swh/loader/svn/tests/test_utils.py

Loading...
View Options

swh/loader/svn/utils.py

Loading...
About · Developer information · Legal