Page MenuHomeSoftware Heritage

Add support for a priviledged "channel" of topics for non-anonymized objects
AbandonedPublic

Authored by douardda on May 18 2020, 1:47 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Summary

This is another approach for implementing anonymized topics. It replaces (with its upcoming counterpart in swh-storage) D3149 and D3150.

The idea is to publish on a "public" prefix anonymized objects and have a
priviledged prefix for non-anonymized ones. Currently only used for
Release and Revision objects.

This adds a "privileged" argument to KafkaJournalWriter.write_addition(s)
methods. If True, the topic used to send the message will be

"{prefix}_privileged.{object_type}"

instead of the default

"{prefix}.{object_type}"

On the client side, if both the privileged and unprivileged topics are
visible at subscritopn time, only the privileged topics will be
subscribed to for object types existing on both "channels".

Diff Detail

Event Timeline

Build is green

Patch application report for D3160 (id=11222)

Rebasing onto 89cd8f7bea...

Current branch diff-target is up to date.
Changes applied before test
commit 119fa5f57e146852bfb5e3319ba884d0eb9369fe
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon May 18 11:35:22 2020 +0200

    Add support for a priviledged "channel" of topics for non-anonymized objects
    
    The idea is to publish on a "public" prefix anonymized objects and have a
    priviledged prefix for non-anonymized ones. Currently only used for
    Release and Revision objects.
    
    This adds a "privileged" argument to KafkaJournalWriter.write_addition(s)
    methods. If True, the topic used to send the message will be
    
      "{prefix}_privileged.{object_type}"
    
    instead of the default
    
      "{prefix}.{object_type}"
    
    On the client side, if both the privileged and unprivileged topics are
    visible at subscritopn time, only the privileged topics will be
    subscribed to for object types existing on both "channels".

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/71/ for more details.

Build was aborted

Patch application report for D3160 (id=11232)

Rebasing onto 89cd8f7bea...

Current branch diff-target is up to date.
Changes applied before test
commit e5e88d89d90a7ad0b837cde11ed34174de9e3242
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon May 18 11:35:22 2020 +0200

    Add support for a priviledged "channel" of topics for non-anonymized objects
    
    The idea is to publish on a "public" prefix anonymized objects and have a
    priviledged prefix for non-anonymized ones. Currently only used for
    Release and Revision objects.
    
    This adds a "privileged" argument to KafkaJournalWriter.write_addition(s)
    methods. If True, the topic used to send the message will be
    
      "{prefix}_privileged.{object_type}"
    
    instead of the default
    
      "{prefix}.{object_type}"
    
    On the client side, if both the privileged and unprivileged topics are
    visible at subscritopn time, only the privileged topics will be
    subscribed to for object types existing on both "channels".

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/72/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/72/console