Page MenuHomeSoftware Heritage

cli: fix brokers argument handling (+ a pile of small refactorings)
ClosedPublic

Authored by douardda on Jun 12 2019, 10:56 AM.

Details

Summary
  • JournalClient: change the prefix init argument dafault value to None and give this None value the special meaning of 'use the default value', i.e. 'swh.journal.objects' (specifically, the value stored as swh.journal.DEFAULT_PREFIX).

    This is required to make it easier to handle cli arguments and config values read from the config file. Let the None value be the default (from the cli config/argument handling) as long as possible and decide only in the JournalClient what "default" really means.
  • replay: improve logging in process_replay_objects_content() catch exceptions and log human readable ids of failed objects.
  • cli: do only listen to required kafka topic in content-replay command

    Also make the JournalClient's constructor argument object_types default value resolved at execution time (to ease a bit default value handling from cli).
  • cli: fix handling of get_journal_client's 'brokers' argument

    it can be an empty tuple (when built from a click option with multiple=1).
  • Bump dependency on swh.storage to v0.0.141 It is required to make tests pass.

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Jun 12 2019, 10:56 AM
vlorentz added inline comments.
swh/journal/replay.py
62–63 ↗(On Diff #5182)

logger.exception('Failed to copy %s', hash_to_hex(obj_id))

vlorentz requested changes to this revision.Jun 12 2019, 11:00 AM
This revision now requires changes to proceed.Jun 12 2019, 11:00 AM
douardda updated this revision to Diff 5184.Jun 12 2019, 11:42 AM

log an exception if process_replay_objects_content() fails

as asked by vlorentz.

Also reword the commit message of the impacted revision.

vlorentz accepted this revision.Jun 12 2019, 1:16 PM
This revision is now accepted and ready to land.Jun 12 2019, 1:16 PM
This revision was automatically updated to reflect the committed changes.