Details

Reviewers

Group Reviewers

Commits

rDSTO2b029b764612: Split 'content_add' method into 'content_add' and 'skipped_content_add'.

Summary

Respectively to add present content and skipped content.

This simplifies the logic of both methods, and is a necessary step to
typing / using swh-model objects everywhere, as contents have quite
different attributes depending on whether they are present or missing.

For the moment, this diff only implements for the in-mem backend, not
pg nor cql

Diff Detail

Repository

rDSTO Storage manager

Branch

split-content-add

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 10450
Build 15566: tox-on-jenkins	Jenkins
Build 15565: arc lint + arc unit

Event Timeline

vlorentz created this revision.Feb 5 2020, 12:18 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/960/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/960/console

Harbormaster failed remote builds in B10447: Diff 9387!Feb 5 2020, 12:24 PM

BUILD has failed

expected as this needs a new swh.model release

ardumont accepted this revision.Feb 5 2020, 12:27 PM

This revision is now accepted and ready to land.Feb 5 2020, 12:27 PM

implement for all backends and proxies

vlorentz requested review of this revision.Feb 5 2020, 3:43 PM

vlorentz retitled this revision from [WIP] Split 'content_add' method into 'content_add' and 'skipped_content_add'. to Split 'content_add' method into 'content_add' and 'skipped_content_add'..

vlorentz added a reviewer: Reviewers.

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/961/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/961/console

Harbormaster failed remote builds in B10450: Diff 9390!Feb 5 2020, 3:48 PM

add 'origin' column to cassandra

remove debug prints

Build was aborted

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/962/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/962/console

Harbormaster failed remote builds in B10451: Diff 9391!Feb 5 2020, 3:54 PM

Harbormaster failed remote builds in B10452: Diff 9392!

fix failing test

Not finished the review but i'm side-tracked (indexer storage failing for one) by other stuff so submitting remarks for now, only small ones.

swh/storage/in_memory.py
293	why not directly iterate over skipped_content_missing (removing the explicit call to list in previous line)?
swh/storage/interface.py
304	by the backend.
306	some contents

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/963/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/963/console

Harbormaster failed remote builds in B10455: Diff 9395!Feb 5 2020, 5:59 PM

oh yeah, i accepted the review in a first drafted version...

vlorentz added a child revision: D2628: Unify exception raised by invalid input to API endpoints..Feb 5 2020, 6:40 PM

apply comments

vlorentz added inline comments.Feb 6 2020, 11:25 AM

swh/storage/in_memory.py
293	probably because I wanted to print it while debugging

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/966/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/966/console

Harbormaster failed remote builds in B10462: Diff 9401!Feb 6 2020, 11:25 AM

ardumont accepted this revision.Feb 6 2020, 12:33 PM

ardumont added inline comments.

swh/storage/in_memory.py
293	heh, happens too me as well.
swh/storage/storage.py
445	can't we avoid one loop? now = datetime.datetime.now(tz=datetime.timezone.utc) content = [dict(c.items(), ctime=now) for c in content]