Add support for a priviledged "channel" of topics for non-anonymized objects
AbandonedPublic
Actions

Authored by douardda on May 18 2020, 1:47 PM.

Details

Reviewers

None

Group Reviewers

Summary

This is another approach for implementing anonymized topics. It replaces (with its upcoming counterpart in swh-storage) D3149 and D3150.

The idea is to publish on a "public" prefix anonymized objects and have a
priviledged prefix for non-anonymized ones. Currently only used for
Release and Revision objects.

This adds a "privileged" argument to KafkaJournalWriter.write_addition(s)
methods. If True, the topic used to send the message will be

"{prefix}_privileged.{object_type}"

instead of the default

"{prefix}.{object_type}"

On the client side, if both the privileged and unprivileged topics are
visible at subscritopn time, only the privileged topics will be
subscribed to for object types existing on both "channels".

Diff Detail

Repository

rDJNL Journal infrastructure

Branch

priviledged

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 12419
Build 18846: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 18845: arc lint + arc unit

Event Timeline

douardda created this revision.May 18 2020, 1:47 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMay 18 2020, 1:47 PM

Build is green

Patch application report for D3160 (id=11222)

Rebasing onto 89cd8f7bea...

Current branch diff-target is up to date.

Changes applied before test

commit 119fa5f57e146852bfb5e3319ba884d0eb9369fe
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon May 18 11:35:22 2020 +0200

    Add support for a priviledged "channel" of topics for non-anonymized objects
    
    The idea is to publish on a "public" prefix anonymized objects and have a
    priviledged prefix for non-anonymized ones. Currently only used for
    Release and Revision objects.
    
    This adds a "privileged" argument to KafkaJournalWriter.write_addition(s)
    methods. If True, the topic used to send the message will be
    
      "{prefix}_privileged.{object_type}"
    
    instead of the default
    
      "{prefix}.{object_type}"
    
    On the client side, if both the privileged and unprivileged topics are
    visible at subscritopn time, only the privileged topics will be
    subscribed to for object types existing on both "channels".

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/71/ for more details.

Harbormaster completed remote builds in B12409: Diff 11222.May 18 2020, 1:48 PM

douardda mentioned this in D3161: Add support for anonymized journal topics.May 18 2020, 1:53 PM

typo

Build was aborted

Patch application report for D3160 (id=11232)

Rebasing onto 89cd8f7bea...

Current branch diff-target is up to date.

Changes applied before test

commit e5e88d89d90a7ad0b837cde11ed34174de9e3242
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon May 18 11:35:22 2020 +0200

    Add support for a priviledged "channel" of topics for non-anonymized objects
    
    The idea is to publish on a "public" prefix anonymized objects and have a
    priviledged prefix for non-anonymized ones. Currently only used for
    Release and Revision objects.
    
    This adds a "privileged" argument to KafkaJournalWriter.write_addition(s)
    methods. If True, the topic used to send the message will be
    
      "{prefix}_privileged.{object_type}"
    
    instead of the default
    
      "{prefix}.{object_type}"
    
    On the client side, if both the privileged and unprivileged topics are
    visible at subscritopn time, only the privileged topics will be
    subscribed to for object types existing on both "channels".

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/72/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/72/console

Harbormaster failed remote builds in B12419: Diff 11232!May 18 2020, 5:00 PM

douardda mentioned this in D3171: Add support for model object anonymization.May 20 2020, 11:20 AM

douardda mentioned this in D3172: Add support for a priviledged "channel" of topics for non-anonymized objects.

Replaced by D3171 + D3172