Depends on D3239.
Details
Details
Diff Detail
Diff Detail
- Repository
- rDSTO Storage manager
- Branch
- origin-metadata-pagination
- Lint
No Linters Available - Unit
No Unit Test Coverage - Build Status
Buildable 12692 Build 19295: Phabricator diff pipeline on jenkins Jenkins console · Jenkins Build 19294: arc lint + arc unit
| Time | Test | |
|---|---|---|
| 4 ms | Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_retry::test_retrying_proxy_storage_origin_metadata_add swh_storage = <swh.storage.validate.ValidatingProxyStorage object at 0x7f0efb0af908>
sample_data = {'authority': [{'metadata': {'location': 'France'}, 'type': 'deposit', 'url': 'http://hal.inria.example.com/'}], 'cont...384>, 'target': b'12345678901234567890', 'type': 'dir'}), 'id': b'4\x013B2S1\x000\xf51\xe62\xa73\xff7\xc3\xa90'}], ...}
| |
| 6 ms | Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content | |
| 2 ms | Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin | |
| 4 ms | Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin_visit | |
| 4 ms | Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_revision | |
| View Full Test Results (1 Failed · 765 Passed · 17 Skipped) |
Event Timeline
Comment Actions
Build has FAILED
Patch application report for D3240 (id=11487)
Could not rebase; Attempt merge onto dcef916e5e...
Auto-merging swh/storage/tests/test_storage.py Auto-merging swh/storage/tests/storage_data.py Auto-merging swh/storage/storage.py Auto-merging swh/storage/interface.py Auto-merging swh/storage/in_memory.py Auto-merging swh/storage/cassandra/storage.py Merge made by the 'recursive' strategy. docs/extrinsic-metadata-specification.rst | 24 +++-- sql/upgrades/153.sql | 13 +++ swh/storage/cassandra/cql.py | 32 ++++++- swh/storage/cassandra/storage.py | 54 ++++++++++-- swh/storage/db.py | 25 ++++-- swh/storage/in_memory.py | 91 ++++++++++++++++--- swh/storage/interface.py | 14 ++- swh/storage/sql/60-swh-indexes.sql | 2 +- swh/storage/storage.py | 50 +++++++++-- swh/storage/tests/storage_data.py | 6 +- swh/storage/tests/test_in_memory.py | 16 ++++ swh/storage/tests/test_storage.py | 141 ++++++++++++++++++++++++++---- 12 files changed, 400 insertions(+), 68 deletions(-) create mode 100644 sql/upgrades/153.sql
Changes applied before test
commit 768617c41f72f792cc6187491852036a7a138229
Merge: dcef916 340cb5b
Author: Jenkins user <jenkins@localhost>
Date: Mon Jun 8 10:45:33 2020 +0000
Merge branch 'diff-target' into HEAD
commit 340cb5b298199a4427794d7dc17878b3bfea1494
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Mon Jun 8 11:53:08 2020 +0200
Add pagination to origin_metadata_get.
commit 96ce9f199a948590ffcce4b12d827cace989916f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Thu Jun 4 17:06:26 2020 +0200
Add SortedList.iter_after.
Strict version of iter_from. I'll need it for pagination.
commit ea2d8fab4a9d6ed2b75db7aea7dfed072669f1d0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Thu Jun 4 12:33:17 2020 +0200
Deduplicate origin-metadata when they have the same authority + discovery_date + fetcher.
By replacing the old value with the new one.
This will allow an easy implementation of pagination, using the fetcher
id as an opaque page_token.
Plus, it did not make sense logically to have different metadata from
the same authority at the same time (especially with the same fetcher).Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/241/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/241/console
Comment Actions
Looks good but there is still a test to fix: test_retrying_proxy_storage_origin_metadata_add.
Looks like you simply need to assert on origin_metadata["results"] after the changes introduced in that diff.
Comment Actions
Build is green
Patch application report for D3240 (id=11493)
Could not rebase; Attempt merge onto dcef916e5e...
Auto-merging swh/storage/tests/test_storage.py Auto-merging swh/storage/tests/storage_data.py Auto-merging swh/storage/storage.py Auto-merging swh/storage/interface.py Auto-merging swh/storage/in_memory.py Auto-merging swh/storage/cassandra/storage.py Merge made by the 'recursive' strategy. docs/extrinsic-metadata-specification.rst | 24 +++-- sql/upgrades/153.sql | 13 +++ swh/storage/cassandra/cql.py | 32 ++++++- swh/storage/cassandra/storage.py | 54 ++++++++++-- swh/storage/db.py | 25 ++++-- swh/storage/in_memory.py | 91 ++++++++++++++++--- swh/storage/interface.py | 14 ++- swh/storage/sql/60-swh-indexes.sql | 2 +- swh/storage/storage.py | 50 +++++++++-- swh/storage/tests/storage_data.py | 6 +- swh/storage/tests/test_in_memory.py | 16 ++++ swh/storage/tests/test_retry.py | 3 +- swh/storage/tests/test_storage.py | 141 ++++++++++++++++++++++++++---- 13 files changed, 402 insertions(+), 69 deletions(-) create mode 100644 sql/upgrades/153.sql
Changes applied before test
commit b0be3f245b08c2c3aaa80169873c617b292f04f6
Merge: dcef916 637e747
Author: Jenkins user <jenkins@localhost>
Date: Mon Jun 8 13:11:15 2020 +0000
Merge branch 'diff-target' into HEAD
commit 637e747dfa74ac9a0442c1fa31f8b69dafd47ce7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Mon Jun 8 11:53:08 2020 +0200
Add pagination to origin_metadata_get.
commit 96ce9f199a948590ffcce4b12d827cace989916f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Thu Jun 4 17:06:26 2020 +0200
Add SortedList.iter_after.
Strict version of iter_from. I'll need it for pagination.
commit ea2d8fab4a9d6ed2b75db7aea7dfed072669f1d0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Thu Jun 4 12:33:17 2020 +0200
Deduplicate origin-metadata when they have the same authority + discovery_date + fetcher.
By replacing the old value with the new one.
This will allow an easy implementation of pagination, using the fetcher
id as an opaque page_token.
Plus, it did not make sense logically to have different metadata from
the same authority at the same time (especially with the same fetcher).See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/243/ for more details.
Comment Actions
Build is green
Patch application report for D3240 (id=11496)
Rebasing onto dcef916e5e...
Current branch diff-target is up to date.
Changes applied before test
commit 7eb44d412be7f8078ed0d62d9b02532c672c768f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Mon Jun 8 11:53:08 2020 +0200
Add pagination to origin_metadata_get.
commit 26a8d4fe9e3ecd03b867ea29be85275b02b60546
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Thu Jun 4 17:06:26 2020 +0200
Add SortedList.iter_after.
Strict version of iter_from. I'll need it for pagination.
commit 6ebdc2f76e294c888c9b121a222d4d360df4507c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date: Thu Jun 4 12:33:17 2020 +0200
Deduplicate origin-metadata when they have the same authority + discovery_date + fetcher.
By replacing the old value with the new one.
This will allow an easy implementation of pagination, using the fetcher
id as an opaque page_token.
Plus, it did not make sense logically to have different metadata from
the same authority at the same time (especially with the same fetcher).See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/246/ for more details.