Page MenuHomeSoftware Heritage

cli: fix brokers argument handling (+ a pile of small refactorings)
ClosedPublic

Authored by douardda on Jun 12 2019, 10:56 AM.

Details

Summary
  • JournalClient: change the prefix init argument dafault value to None and give this None value the special meaning of 'use the default value', i.e. 'swh.journal.objects' (specifically, the value stored as swh.journal.DEFAULT_PREFIX).

    This is required to make it easier to handle cli arguments and config values read from the config file. Let the None value be the default (from the cli config/argument handling) as long as possible and decide only in the JournalClient what "default" really means.
  • replay: improve logging in process_replay_objects_content() catch exceptions and log human readable ids of failed objects.
  • cli: do only listen to required kafka topic in content-replay command

    Also make the JournalClient's constructor argument object_types default value resolved at execution time (to ease a bit default value handling from cli).
  • cli: fix handling of get_journal_client's 'brokers' argument

    it can be an empty tuple (when built from a click option with multiple=1).
  • Bump dependency on swh.storage to v0.0.141 It is required to make tests pass.

Diff Detail

Repository
rDJNL Journal infrastructure
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 6150
Build 8479: tox-on-jenkinsJenkins
Build 8478: arc lint + arc unit

Event Timeline

douardda created this revision.Jun 12 2019, 10:56 AM
vlorentz added inline comments.
swh/journal/replay.py
62–63

logger.exception('Failed to copy %s', hash_to_hex(obj_id))

vlorentz requested changes to this revision.Jun 12 2019, 11:00 AM
This revision now requires changes to proceed.Jun 12 2019, 11:00 AM
douardda updated this revision to Diff 5184.Jun 12 2019, 11:42 AM

log an exception if process_replay_objects_content() fails

as asked by vlorentz.

Also reword the commit message of the impacted revision.

vlorentz accepted this revision.Jun 12 2019, 1:16 PM
This revision is now accepted and ready to land.Jun 12 2019, 1:16 PM
This revision was automatically updated to reflect the committed changes.