Page MenuHomeSoftware Heritage
Differential D2085

swh.storage.buffer: Add buffering proxy storage implementation
ClosedPublic
Actions

Authored by ardumont on Oct 8 2019, 2:46 PM.

Details

Reviewers
olasd
vlorentz
douardda
Group Reviewers
Reviewers
Commits
rDSTOC03d5a2cf7ca8: swh.storage.buffer: Add buffering proxy storage implementation
rDSTO03d5a2cf7ca8: swh.storage.buffer: Add buffering proxy storage implementation
Summary

Depends on D2084

Test Plan

tox

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Oct 8 2019, 2:46 PM
Harbormaster failed remote builds in B8173: Diff 7003!Oct 8 2019, 2:47 PM
swh-public-ci added a comment.Oct 8 2019, 2:47 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/669/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/669/console

vlorentz requested changes to this revision.EditedOct 8 2019, 3:00 PM

s/threshold/min_batch_size/

s/Sequence/Iterable/ because we don't do random access (see definition of sequences here: https://docs.python.org/3/glossary.html#term-sequence )

I think any call to a non-add method should trigger a flush of related objects before forwarding the call (eg. revision_missing flushes revisions).

swh/storage/buffer.py
20

.. code-block:: yaml

This revision now requires changes to proceed.Oct 8 2019, 3:00 PM
vlorentz added a comment.Oct 8 2019, 3:01 PM

Also, I understand how that's currently useful, but FYI, batching content has absolutely no effect for Cassandra (the Cassandra backend breaks batches into individual records)

ardumont added a comment.Oct 8 2019, 3:03 PM

Also, I understand how that's currently useful, but FYI, batching content has absolutely no effect for Cassandra (the Cassandra backend breaks batches into individual records)

that's the beauty of it!
That's used through configuration so we will be able to simply remove the configuration about the buffering storage when we migrate to cassandra!

swh/storage/buffer.py
20

I tried, that's too smart for me...

swh-public-ci added a comment.Oct 8 2019, 3:03 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/672/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/672/console

vlorentz added inline comments.Oct 8 2019, 3:10 PM
swh/storage/tests/test_buffer.py
19

this should check the content was not sent to the backend.

Same comment for other instances below

23

len(content[0]['data']) + len(content[1]['data']).

Same comment for other instances below

ardumont updated this revision to Diff 7009.Oct 8 2019, 3:54 PM
  • Fix code-block syntax
  • Add some more assertions on storage writing or not
swh-public-ci added a comment.Oct 8 2019, 3:55 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/674/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/674/console

Harbormaster failed remote builds in B8179: Diff 7009!Oct 8 2019, 3:55 PM
ardumont added a comment.Oct 8 2019, 3:57 PM

BUILD has failed

./swh/storage/buffer.py:82:16: E999 SyntaxError: invalid syntax

Well i fixed my local mypy but the ci mypy (different version is not happy about it)...
That's annoying.

ardumont updated this revision to Diff 7010.Oct 8 2019, 4:00 PM

Fix ci mypy which is not happy, local mypy was ok

swh-public-ci added a comment.Oct 8 2019, 4:05 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/675/ for more details.

Harbormaster completed remote builds in B8180: Diff 7010.Oct 8 2019, 4:05 PM
ardumont updated this revision to Diff 7015.Oct 8 2019, 4:16 PM

Rebase on latest development

swh-public-ci added a comment.Oct 8 2019, 4:29 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/680/ for more details.

Harbormaster completed remote builds in B8185: Diff 7015.Oct 8 2019, 4:29 PM
vlorentz requested changes to this revision.Oct 8 2019, 4:31 PM

You missed this comment:

In D2085#48361, @vlorentz wrote:

s/threshold/min_batch_size/

s/Sequence/Iterable/ because we don't do random access (see definition of sequences here: https://docs.python.org/3/glossary.html#term-sequence )

I think any call to a non-add method should trigger a flush of related objects before forwarding the call (eg. revision_missing flushes revisions).

This revision now requires changes to proceed.Oct 8 2019, 4:31 PM
ardumont added a comment.Oct 8 2019, 4:36 PM

You missed this comment:

I saw and then i forgot.

ardumont updated this revision to Diff 7016.Oct 8 2019, 4:41 PM

Adapt according to last missing review points

This revision was not accepted when it landed; it landed in state Needs Review.Oct 8 2019, 4:42 PM
This revision was landed with ongoing or failed builds.
Closed by commit rDSTO03d5a2cf7ca8: swh.storage.buffer: Add buffering proxy storage implementation (authored by ardumont). · Explain Why
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSTO03d5a2cf7ca8: swh.storage.buffer: Add buffering proxy storage implementation.
ardumont added a comment.Oct 8 2019, 4:44 PM

This revision was not accepted when it landed; it landed in state Needs Review.

damn! I messed up.
i checked out one commit prior to my latest master... to push only to that level...
i git push-ed but that pushed my master anyway...

oh well, i'll start back using branch...

swh-public-ci added a comment.Oct 8 2019, 4:46 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/681/ for more details.

Harbormaster completed remote builds in B8186: Diff 7016.Oct 8 2019, 4:46 PM
ardumont mentioned this in rDLDBASE1543ec5315eb: Use main filtering/buffering storage implementations.Oct 8 2019, 5:37 PM
ardumont added a commit: rDSTOC03d5a2cf7ca8: swh.storage.buffer: Add buffering proxy storage implementation.Oct 30 2019, 5:22 PM

Revision Contents
Changeset List

PathSize
swh/
storage/
7 lines
106 lines
tests/
179 lines
6 lines

Diff 7019

View Options

swh/storage/__init__.py

Loading...
View Options

swh/storage/buffer.py

Loading...
View Options

swh/storage/tests/test_buffer.py

Loading...
View Options

swh/storage/tests/test_init.py

Loading...
About · Developer information · Legal