Page MenuHomeSoftware Heritage

directory_ls: Don't return None for status/length/sha1/... if the content is known but skipped.
ClosedPublic

Authored by vlorentz on Sep 3 2020, 3:24 PM.

Diff Detail

Repository
rDSTO Storage manager
Branch
directory_ls_skipped_contents
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 14915
Build 22986: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 22985: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D3874 (id=13668)

Could not rebase; Attempt merge onto 36d284c730...

Updating 36d284c7..401ad3b5
Fast-forward
 sql/upgrades/162.sql               | 52 ++++++++++++++++++++++++++++++++++++++
 swh/storage/cassandra/cql.py       | 20 +++++++++++++++
 swh/storage/cassandra/storage.py   | 11 ++++++++
 swh/storage/in_memory.py           |  8 ++++++
 swh/storage/postgresql/db.py       |  2 +-
 swh/storage/sql/30-swh-schema.sql  |  2 +-
 swh/storage/sql/40-swh-func.sql    | 18 ++++++++++---
 swh/storage/tests/storage_tests.py | 47 ++++++++++++++++++++++++++++++++++
 8 files changed, 154 insertions(+), 6 deletions(-)
 create mode 100644 sql/upgrades/162.sql
Changes applied before test
commit 401ad3b55ccf353ad365af4a7444fc6228bf8d8e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 3 15:23:22 2020 +0200

    directory_ls: Don't return None for status/length/sha1/... if the content is known but skipped.

commit a2435c67e1154dcb40aea82806528c7cbdd0d062
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 3 15:15:08 2020 +0200

    Add a test for directory_ls when the contents are missing.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/901/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/901/console

  • convert MAGIC_NULL_PK back to None

Build is green

Patch application report for D3874 (id=13689)

Could not rebase; Attempt merge onto 356eacd763...

Auto-merging swh/storage/tests/storage_tests.py
Auto-merging swh/storage/cassandra/storage.py
Merge made by the 'recursive' strategy.
 sql/upgrades/162.sql               | 52 ++++++++++++++++++++++++++++++++++++++
 swh/storage/cassandra/cql.py       | 37 +++++++++++++++++----------
 swh/storage/cassandra/model.py     | 15 +++++++++++
 swh/storage/cassandra/storage.py   | 11 ++++++++
 swh/storage/in_memory.py           |  8 ++++++
 swh/storage/postgresql/db.py       |  2 +-
 swh/storage/sql/30-swh-schema.sql  |  2 +-
 swh/storage/sql/40-swh-func.sql    | 18 ++++++++++---
 swh/storage/tests/storage_tests.py | 47 ++++++++++++++++++++++++++++++++++
 9 files changed, 173 insertions(+), 19 deletions(-)
 create mode 100644 sql/upgrades/162.sql
Changes applied before test
commit c7ac4f956eb5f2c6a0ba122744a526216de553e5
Merge: 356eacd7 a4c3662a
Author: Jenkins user <jenkins@localhost>
Date:   Fri Sep 4 08:41:28 2020 +0000

    Merge branch 'diff-target' into HEAD

commit a4c3662a55aec3a8dccdbd0891383e7b09e6c3c3
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 3 15:23:22 2020 +0200

    directory_ls: Don't return None for status/length/sha1/... if the content is known but skipped.

commit a2435c67e1154dcb40aea82806528c7cbdd0d062
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 3 15:15:08 2020 +0200

    Add a test for directory_ls when the contents are missing.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/903/ for more details.

ardumont added a subscriber: ardumont.

looks good, I got small remarks ;)

unsure about the performance though (a priori should not be a big deal as the skipped
content table is small but there is the question of this being a recursive function... so
¯\_(ツ)_/¯).

swh/storage/sql/40-swh-func.sql
318

misindented i think.

swh/storage/tests/storage_tests.py
780

lol

782

there may exist skipped-content already in the sample-data fixture.

This revision is now accepted and ready to land.Sep 4 2020, 4:19 PM

Build is green

Patch application report for D3874 (id=13718)

Rebasing onto 374e01cf36...

Current branch diff-target is up to date.
Changes applied before test
commit 90eda987521caa51c4c19a68de19ba860bef6a80
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 3 15:23:22 2020 +0200

    directory_ls: Don't return None for status/length/sha1/... if the content is known but skipped.

commit 3198e11d6978dd53f738391fefcc340b5f943559
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Sep 3 15:15:08 2020 +0200

    Add a test for directory_ls when the contents are missing.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/910/ for more details.