Page MenuHomeSoftware Heritage
Differential D5315

Add LevelDB backend for exporter node sets
ClosedPublic
Actions

Authored by seirl on Mar 23 2021, 10:13 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDDATASETe16e9c5bb271: Add LevelDB backend for exporter node sets
rDMOD9523be0552d8: Model test data: add Release with no author/date
Summary

SQLite seems to slow down over time when a lot of data is inserted in
it. This is an attempt at using LevelDB to have more efficient node
sets.

Q: should we make sqlite/leveldb configurable, in case we want to support setups where it's not easy to install leveldb?

Diff Detail

Repository
rDDATASET Datasets
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

seirl requested review of this revision.Mar 23 2021, 10:13 PM
seirl created this revision.
Harbormaster completed remote builds in B20161: Diff 19069.Mar 23 2021, 10:13 PM
seirl edited the summary of this revision. (Show Details)Mar 23 2021, 10:13 PM
seirl added a reviewer: Reviewers.
vlorentz accepted this revision.Mar 26 2021, 11:21 AM
vlorentz added a subscriber: vlorentz.

Sorry for the long delay, I didn't see you diff for some reason...

swh/dataset/utils.py
103–105

Use this instead:

try:
    import plyvel
except ImportError:
    plyvel = None

it spares invoking the import machinery every time you use LevelDBSet

This revision is now accepted and ready to land.Mar 26 2021, 11:21 AM
seirl updated this revision to Diff 19176.Mar 26 2021, 12:07 PM

Remove phabricator garbage

This revision was landed with ongoing or failed builds.Mar 26 2021, 12:07 PM
Closed by commit rDMOD9523be0552d8: Model test data: add Release with no author/date (authored by seirl). · Explain Why
This revision was automatically updated to reflect the committed changes.
seirl added a commit: rDMOD9523be0552d8: Model test data: add Release with no author/date.
swh-public-ci added a comment.Mar 26 2021, 12:08 PM

Build is green

Patch application report for D5315 (id=19176)

Rebasing onto af5e4614c8...

Current branch diff-target is up to date.
Changes applied before test
commit 9523be0552d822be617da77bf0d2ca2f479da572
Author: Antoine Pietri <antoine.pietri1@gmail.com>
Date:   Tue Mar 23 23:42:52 2021 +0000

    Model test data: add Release with no author/date
    
    Some releases don't have authors and date fields, this case should be
    checked in the tests.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/292/ for more details.

Harbormaster completed remote builds in B20257: Diff 19176.Mar 26 2021, 12:08 PM
seirl reopened this revision.Mar 26 2021, 2:27 PM
This revision is now accepted and ready to land.Mar 26 2021, 2:27 PM
seirl updated this revision to Diff 19195.Mar 26 2021, 2:27 PM

Rebase + fix phabricator incorrect ID

Harbormaster completed remote builds in B20277: Diff 19195.Mar 26 2021, 2:27 PM
Closed by commit rDDATASETe16e9c5bb271: Add LevelDB backend for exporter node sets (authored by seirl). · Explain WhyMar 26 2021, 2:29 PM
This revision was automatically updated to reflect the committed changes.
seirl added a commit: rDDATASETe16e9c5bb271: Add LevelDB backend for exporter node sets.

Revision Contents
Changeset List

PathSize
3 lines
1 line
swh/
dataset/
2 lines
8 lines
test/
2 lines
14 lines
41 lines

Diff 19196

View Options

mypy.ini

Loading...
View Options

requirements.txt

Loading...
View Options

swh/dataset/cli.py

Loading...
View Options

swh/dataset/journalprocessor.py

Loading...
View Options

swh/dataset/test/test_edges.py

Loading...
View Options

swh/dataset/test/test_utils.py

Loading...
View Options

swh/dataset/utils.py

Loading...
About · Developer information · Legal