Page MenuHomeSoftware Heritage
Differential D2203

Add support for origin_visit['origin'] with type 'str'.
ClosedPublic
Actions

Authored by vlorentz on Oct 30 2019, 3:31 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Commits
rDJNLd329c92b3431: Add support for origin_visit['origin'] with type 'str'.
Summary

In addition to 'dict' (legacy schema).

Also removes support for origin ids.

Diff Detail

Repository
rDJNL Journal infrastructure
Branch
new-visit-schema
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 8720
Build 12701: tox-on-jenkinsJenkins
Build 12700: arc lint + arc unit

Event Timeline

vlorentz created this revision.Oct 30 2019, 3:31 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 30 2019, 3:31 PM
vlorentz updated this revision to Diff 7512.Oct 30 2019, 3:33 PM

update tox.ini

vlorentz edited the summary of this revision. (Show Details)Oct 30 2019, 3:33 PM
swh-public-ci added a comment.Oct 30 2019, 3:35 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/294/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/294/console

Harbormaster failed remote builds in B8716: Diff 7511!Oct 30 2019, 3:35 PM
vlorentz added a child revision: D2185: Consume messages from Kafka in batches of 20..Oct 30 2019, 3:37 PM
swh-public-ci added a comment.Oct 30 2019, 3:38 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/295/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/295/console

Harbormaster failed remote builds in B8717: Diff 7512!Oct 30 2019, 3:38 PM
vlorentz updated this revision to Diff 7515.Oct 30 2019, 3:42 PM

fix schema version detection

swh-public-ci added a comment.Oct 30 2019, 3:47 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/298/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/298/console

Harbormaster failed remote builds in B8720: Diff 7515!Oct 30 2019, 3:47 PM
vlorentz updated this revision to Diff 7517.Oct 30 2019, 3:50 PM

fix test

swh-public-ci added a comment.Oct 30 2019, 3:53 PM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/300/ for more details.

Harbormaster completed remote builds in B8722: Diff 7517.Oct 30 2019, 3:53 PM
olasd accepted this revision.Oct 30 2019, 3:56 PM
This revision is now accepted and ready to land.Oct 30 2019, 3:56 PM
Closed by commit rDJNLd329c92b3431: Add support for origin_visit['origin'] with type 'str'. (authored by vlorentz). · Explain WhyOct 30 2019, 4:04 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDJNLd329c92b3431: Add support for origin_visit['origin'] with type 'str'..

Revision Contents
Changeset List

PathSize
2 lines
swh/
journal/
28 lines
tests/
4 lines
7 lines
51 lines
2 lines
13 lines

Diff 7515

View Options

requirements-swh.txt

swh.core[db,http] >= 0.0.60 swh.core[db,http] >= 0.0.60
swh.model >= 0.0.40 swh.model >= 0.0.40
swh.storage >= 0.0.147 swh.storage >= 0.0.156
View Options

swh/journal/replay.py

Show First 20 Lines • Show All 83 Lines • ▼ Show 20 Lines for rev in revisions:
good_revisions.append(rev) good_revisions.append(rev)
return good_revisions return good_revisions
def _fix_origin_visits(visits): def _fix_origin_visits(visits):
good_visits = [] good_visits = []
for visit in visits: for visit in visits:
visit = visit.copy() visit = visit.copy()
if isinstance(visit['origin'], str):
# note that it will crash with the pg and
# in-mem storages if the origin is not already known,
# but there is no other choice because we can't add an
# origin without knowing its type. Non-pg storages
# don't use a numeric FK internally,
visit['origin'] = {'url': visit['origin']}
else:
if 'type' not in visit: if 'type' not in visit:
if isinstance(visit['origin'], dict) and 'type' in visit['origin']:
# Very old version of the schema: visits did not have a type,
# but their 'origin' field was a dict with a 'type' key.
visit['type'] = visit['origin']['type'] visit['type'] = visit['origin']['type']
else:
# Very very old version of the schema: 'type' is missing,
# so there is nothing we can do to fix it.
raise ValueError('Got an origin_visit too old to be replayed.')
if isinstance(visit['origin'], dict):
# Old version of the schema: visit['origin'] was a dict.
visit['origin'] = visit['origin']['url']
good_visits.append(visit) good_visits.append(visit)
return good_visits return good_visits
def fix_objects(object_type, objects): def fix_objects(object_type, objects):
"""Converts a possibly old object from the journal to its current """Converts a possibly old object from the journal to its current
expected format. expected format.
▲ Show 20 Lines • Show All 68 Lines • ▼ Show 20 Linesdef fix_objects(object_type, objects):
... 'author': {'email': '', 'fullname': b'', 'name': b''}, ... 'author': {'email': '', 'fullname': b'', 'name': b''},
... 'committer': {'email': '', 'fullname': b'', 'name': b''}, ... 'committer': {'email': '', 'fullname': b'', 'name': b''},
... 'date': date, ... 'date': date,
... 'committer_date': invalid_date3, ... 'committer_date': invalid_date3,
... }]) ... }])
[] []
`visit['origin']` is an URL instead of a dict: `visit['origin']` is a dict instead of an URL:
>>> fix_objects('origin_visit', [{'origin': 'http://foo'}]) >>> fix_objects('origin_visit', [{'origin': {'url': 'http://foo'}}])
[{'origin': {'url': 'http://foo'}}] [{'origin': 'http://foo'}]
`visit['type']` is missing , but `origin['visit']['type']` exists: `visit['type']` is missing , but `origin['visit']['type']` exists:
>>> pprint(fix_objects( >>> pprint(fix_objects(
... 'origin_visit', ... 'origin_visit',
... [{'origin': {'type': 'hg', 'url': 'http://foo'}}])) ... [{'origin': {'type': 'hg', 'url': 'http://foo'}}]))
[{'origin': {'type': 'hg', 'url': 'http://foo'}, 'type': 'hg'}] [{'origin': {'type': 'hg', 'url': 'http://foo'}, 'type': 'hg'}]
""" # noqa """ # noqa
Show All 17 Linesdef _insert_objects(object_type, objects, storage):
elif object_type in ('directory', 'revision', 'release', elif object_type in ('directory', 'revision', 'release',
'snapshot', 'origin'): 'snapshot', 'origin'):
# TODO: split batches that are too large for the storage # TODO: split batches that are too large for the storage
# to handle? # to handle?
method = getattr(storage, object_type + '_add') method = getattr(storage, object_type + '_add')
method(objects) method(objects)
elif object_type == 'origin_visit': elif object_type == 'origin_visit':
for visit in objects: for visit in objects:
storage.origin_add_one(visit['origin']) storage.origin_add_one({'url': visit['origin']})
if 'metadata' not in visit: if 'metadata' not in visit:
visit['metadata'] = None visit['metadata'] = None
storage.origin_visit_upsert(objects) storage.origin_visit_upsert(objects)
else: else:
logger.warning('Received a series of %s, this should not happen', logger.warning('Received a series of %s, this should not happen',
object_type) object_type)
▲ Show 20 Lines • Show All 161 Lines • Show Last 20 Lines
View Options

swh/journal/tests/conftest.py

Show First 20 Lines • Show All 122 Lines • ▼ Show 20 LinesORIGINS = [
}, },
{ {
'url': 'https://overtherainbow.org/fox/den', 'url': 'https://overtherainbow.org/fox/den',
} }
] ]
ORIGIN_VISITS = [ ORIGIN_VISITS = [
{ {
'origin': ORIGINS[0], 'origin': ORIGINS[0]['url'],
'date': '2013-05-07 04:20:39.369271+00:00', 'date': '2013-05-07 04:20:39.369271+00:00',
'snapshot': None, # TODO 'snapshot': None, # TODO
'status': 'ongoing', # TODO 'status': 'ongoing', # TODO
'metadata': {'foo': 'bar'}, 'metadata': {'foo': 'bar'},
'type': 'git', 'type': 'git',
}, },
{ {
'origin': ORIGINS[0], 'origin': ORIGINS[0]['url'],
'date': '2018-11-27 17:20:39+00:00', 'date': '2018-11-27 17:20:39+00:00',
'snapshot': None, # TODO 'snapshot': None, # TODO
'status': 'ongoing', # TODO 'status': 'ongoing', # TODO
'metadata': {'baz': 'qux'}, 'metadata': {'baz': 'qux'},
'type': 'git', 'type': 'git',
} }
] ]
▲ Show 20 Lines • Show All 88 Lines • Show Last 20 Lines
View Options

swh/journal/tests/test_kafka_writer.py

Show First 20 Lines • Show All 117 Lines • ▼ Show 20 Lines for (object_type, (_, objects)) in OBJECT_TYPE_KEYS.items():
'snapshot', 'origin'): 'snapshot', 'origin'):
if object_type == 'content': if object_type == 'content':
objects = [{**obj, 'data': b''} for obj in objects] objects = [{**obj, 'data': b''} for obj in objects]
method(objects) method(objects)
expected_messages += len(objects) expected_messages += len(objects)
elif object_type in ('origin_visit',): elif object_type in ('origin_visit',):
for object_ in objects: for object_ in objects:
object_ = object_.copy() object_ = object_.copy()
origin_id = storage.origin_add_one(object_.pop('origin')) origin_url = object_.pop('origin')
visit = method(origin=origin_id, date=object_.pop('date'), storage.origin_add_one({'url': origin_url})
visit = method(origin=origin_url, date=object_.pop('date'),
type=object_.pop('type')) type=object_.pop('type'))
expected_messages += 1 expected_messages += 1
visit_id = visit['visit'] visit_id = visit['visit']
storage.origin_visit_update(origin_id, visit_id, **object_) storage.origin_visit_update(origin_url, visit_id, **object_)
expected_messages += 1 expected_messages += 1
else: else:
assert False, object_type assert False, object_type
assert_written(consumer, kafka_prefix, expected_messages) assert_written(consumer, kafka_prefix, expected_messages)
View Options

swh/journal/tests/test_replay.py

Loading...
View Options

swh/journal/tests/test_write_replay.py

Loading...
View Options

tox.ini

Loading...
About · Developer information · Legal