Page MenuHomeSoftware Heritage
Differential D5723

Add endpoint directory_get_entries, to quickly list a directory's entries
ClosedPublic
Actions

Authored by vlorentz on May 10 2021, 4:12 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Commits
rDSTOe3cbd5ee425c: Add endpoint directory_get_entries, to quickly list a directory's entries
Summary

It spares a join with the content table, which should hopefully make
the vault (and possibly other users) faster when they don't need this
join.

Diff Detail

Repository
rDSTO Storage manager
Branch
dir-get-entries
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 21463
Build 33344: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 33343: arc lint + arc unit

Event Timeline

vlorentz created this revision.May 10 2021, 4:12 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMay 10 2021, 4:12 PM
swh-public-ci added a comment.May 10 2021, 4:21 PM

Build is green

Patch application report for D5723 (id=20446)

Could not rebase; Attempt merge onto 761709957a...

Updating 76170995..474a94c4
Fast-forward
 swh/storage/cassandra/cql.py         | 11 +++++
 swh/storage/cassandra/storage.py     | 24 +++++++++++
 swh/storage/in_memory.py             | 11 +++++
 swh/storage/interface.py             | 28 +++++++++++++
 swh/storage/postgresql/db.py         | 11 ++++-
 swh/storage/postgresql/storage.py    | 27 ++++++++++++
 swh/storage/retry.py                 |  3 +-
 swh/storage/sql/40-funcs.sql         | 28 +++++++++++++
 swh/storage/tests/storage_tests.py   | 38 +++++++++++++++++
 swh/storage/tests/test_cassandra.py  | 80 +++++++++++++++++++++++++++++++++++-
 swh/storage/tests/test_postgresql.py | 10 ++++-
 11 files changed, 267 insertions(+), 4 deletions(-)
Changes applied before test
commit 474a94c40b3a944ce832e05c3a5444b6b8b987f4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon May 10 16:12:05 2021 +0200

    Add endpoint directory_get_entries, to quickly list a directory's entries
    
    It spares a join with the content table, which should hopefully make
    the vault (and possibly other users) faster when they don't need this
    join.

commit 4d3eeb2edd5b1413a968a30b1b0f585be4dcf4e0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon May 10 14:13:20 2021 +0200

    cassandra: Add tests checking directory_add and snapshot_add are atomic.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1320/ for more details.

Harbormaster completed remote builds in B21423: Diff 20446.May 10 2021, 4:21 PM
vlorentz requested review of this revision.May 10 2021, 4:21 PM
vlorentz updated this revision to Diff 20461.May 10 2021, 8:57 PM

remove stupid code

swh-public-ci added a comment.May 10 2021, 9:04 PM

Build is green

Patch application report for D5723 (id=20461)

Could not rebase; Attempt merge onto b487a21f27...

Merge made by the 'recursive' strategy.
 swh/storage/cassandra/cql.py         | 11 +++++
 swh/storage/cassandra/storage.py     | 24 +++++++++++
 swh/storage/in_memory.py             | 11 +++++
 swh/storage/interface.py             | 28 +++++++++++++
 swh/storage/postgresql/db.py         | 11 ++++-
 swh/storage/postgresql/storage.py    | 26 ++++++++++++
 swh/storage/proxies/retry.py         |  3 +-
 swh/storage/sql/40-funcs.sql         | 28 +++++++++++++
 swh/storage/tests/storage_tests.py   | 38 +++++++++++++++++
 swh/storage/tests/test_cassandra.py  | 80 +++++++++++++++++++++++++++++++++++-
 swh/storage/tests/test_postgresql.py | 10 ++++-
 11 files changed, 266 insertions(+), 4 deletions(-)
Changes applied before test
commit e8fb21269afdbe84db50236c940e37b15be9c47c
Merge: b487a21f a6a28319
Author: Jenkins user <jenkins@localhost>
Date:   Mon May 10 18:57:11 2021 +0000

    Merge branch 'diff-target' into HEAD

commit a6a283195782b7f8d2c33b24e8328b1cbbdd599b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon May 10 16:12:05 2021 +0200

    Add endpoint directory_get_entries, to quickly list a directory's entries
    
    It spares a join with the content table, which should hopefully make
    the vault (and possibly other users) faster when they don't need this
    join.

commit 4d3eeb2edd5b1413a968a30b1b0f585be4dcf4e0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon May 10 14:13:20 2021 +0200

    cassandra: Add tests checking directory_add and snapshot_add are atomic.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1321/ for more details.

Harbormaster completed remote builds in B21436: Diff 20461.May 10 2021, 9:04 PM
vlorentz added a parent revision: D5708: git_bare: When possible, use swh-graph instead of swh-storage to query revision history.May 10 2021, 9:53 PM
olasd accepted this revision.May 11 2021, 11:46 AM
olasd added a subscriber: olasd.
olasd added inline comments.
swh/storage/interface.py
445–448

copy and waste of the wrong docs here :)

swh/storage/retry.py
43–44 ↗(On Diff #20461)

Yes! But please put this in a separate commit (no need for review)

swh/storage/tests/storage_tests.py
912

Maybe deserves a _pagination suffix to make the intent clearer?

934

And this one would get the suffix dropped.

938–947

Don't we already have a helper function to do that? :)

This revision is now accepted and ready to land.May 11 2021, 11:46 AM
vlorentz added inline comments.May 11 2021, 11:54 AM
swh/storage/retry.py
43–44 ↗(On Diff #20461)

oops

vlorentz added inline comments.May 11 2021, 11:55 AM
swh/storage/interface.py
445–448

nah, copied from the right docs, but I just forgot to run the regexp on it

445–448

hmm but I did change the interface to page_token after writing the doc tho

vlorentz updated this revision to Diff 20490.May 11 2021, 12:01 PM

apply comments:

  • update docstring
  • remove change to retry proxy
  • rename tests
  • dedup code using stream_results
swh-public-ci added a comment.May 11 2021, 12:08 PM

Build is green

Patch application report for D5723 (id=20490)

Rebasing onto f140f634b6...

Current branch diff-target is up to date.
Changes applied before test
commit e3cbd5ee425cefa1e290a34cd889256036a06db0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon May 10 16:12:05 2021 +0200

    Add endpoint directory_get_entries, to quickly list a directory's entries
    
    It spares a join with the content table, which should hopefully make
    the vault (and possibly other users) faster when they don't need this
    join.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1324/ for more details.

Harbormaster completed remote builds in B21463: Diff 20490.May 11 2021, 12:09 PM
Closed by commit rDSTOe3cbd5ee425c: Add endpoint directory_get_entries, to quickly list a directory's entries (authored by vlorentz). · Explain WhyMay 11 2021, 12:31 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSTOe3cbd5ee425c: Add endpoint directory_get_entries, to quickly list a directory's entries.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
storage/
cassandra/
11 lines98%
Loading...
24 lines97%
Loading...
11 lines99%
Loading...
26 lines70%
Loading...
postgresql/
11 lines97%
Loading...
26 lines95%
Loading...
sql/
28 lines--
tests/
32 lines99%
Loading...
1 line92%
Loading...
10 lines99%
Loading...

Diff 20490

View Options

swh/storage/cassandra/cql.py

Loading...
View Options

swh/storage/cassandra/storage.py

Loading...
View Options

swh/storage/in_memory.py

Loading...
View Options

swh/storage/interface.py

Loading...
View Options

swh/storage/postgresql/db.py

Loading...
View Options

swh/storage/postgresql/storage.py

Loading...
View Options

swh/storage/sql/40-funcs.sql

Loading...
View Options

swh/storage/tests/storage_tests.py

Loading...
View Options

swh/storage/tests/test_cassandra.py

Loading...
View Options

swh/storage/tests/test_postgresql.py

Loading...
About · Developer information · Legal