Page MenuHomeSoftware Heritage
Differential D3150

Add support for anonymized journal topics
AbandonedPublic
Actions

Authored by douardda on May 13 2020, 12:23 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Summary

This adds 2 new topics (if activated by config), namely

  • `{kafka_prefix}.release:anonymized` and
  • `{kafka_prefix}.revision:anonymized`

These topics are filled aside from their original non-anonymized version.
The anonymization process consists simply in forging a Person with the
`fullname` being a hash of the triplet (fullname, name, email) of the
original Person in Release and Revision entities.

So the replayer process can be used as is (just have to not replay both
standard and anonymized topics at once).

Depends on D3140.

Diff Detail

Repository
rDSTO Storage manager
Branch
skipped_content
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 12371
Build 18765: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 18764: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
4,080 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_kafka_writer::test_storage_direct_writer
kafka_prefix = 'yclriuzlwn', kafka_server = '127.0.0.1:49293' consumer = <cimpl.Consumer object at 0x7f8972852598>
2,006 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_kafka_writer::test_storage_direct_writer_anonymized
kafka_prefix = 'rrnxafkqlg', kafka_server = '127.0.0.1:49293' consumer = <cimpl.Consumer object at 0x7f8972852400>
3,005 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_replay::test_storage_play_anonymized
kafka_prefix = 'ldilvdqmiv', kafka_consumer_group = 'test-consumer-ldilvdqmiv' kafka_server = '127.0.0.1:49293' caplog = <_pytest.logging.LogCaptureFixture object at 0x7f89727dada0>
6 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin
View Full Test Results (3 Failed · 914 Passed · 24 Skipped)

Event Timeline

douardda created this revision.May 13 2020, 12:23 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMay 13 2020, 12:23 PM
swh-public-ci added a comment.May 13 2020, 12:34 PM

Build has FAILED

Patch application report for D3150 (id=11179)

Could not rebase; Attempt merge onto 306aa69b26...

Updating 306aa69..be5c939
Fast-forward
 swh/storage/replay.py                  |   9 +-
 swh/storage/tests/test_kafka_writer.py | 105 ++++++++++++++++++++++--
 swh/storage/tests/test_replay.py       | 145 ++++++++++++++++++++++++++++++++-
 swh/storage/writer.py                  |  70 ++++++++++------
 4 files changed, 292 insertions(+), 37 deletions(-)
Changes applied before test
commit be5c93901012700455776fcfcc7dc7c616d06744
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu May 7 15:33:35 2020 +0200

    Add support for anonymized journal topics
    
    This adds 2 new topics (if activated by config), namely
    - ``{kafka_prefix}.release:anonymized`` and
    - ``{kafka_prefix}.revision:anonymized``
    
    These topics are filled aside from their original non-anonymized version.
    The anonymization process consists simply in forging a Person with the
    ``fullename`` being a hash of the triplet (fullname, name, email) of the
    original Person in Release and Revision entities.
    
    So the replayer process can be used as is (just have to not replay both
    standard and anonymized topics at once).

commit e0ad4f46a50bb45ceb679d176068f5e478624163
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 13 11:55:44 2020 +0200

    replay: add support for "flags" in topics
    
    simply ignore anything after a ":" in the object_type part of the topic.

commit 87f7bee6935b831a3c300cadd9afb5ce890f2292
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Apr 7 16:20:35 2020 +0200

    journal: add a skipped_content topic dedicated to SkippedContent objects
    
    instead of mixing them with Content in the content topic.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/169/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/169/console

Harbormaster failed remote builds in B12361: Diff 11179!May 13 2020, 12:35 PM
douardda mentioned this in D3149: Add support for "flags" in object_type used for kafka topics.May 13 2020, 3:02 PM
ardumont edited the summary of this revision. (Show Details)May 13 2020, 3:33 PM
ardumont added a subscriber: ardumont.May 13 2020, 3:39 PM

Looks promising ;)

1 typos to fix in the commit message fullename instead of fullname (i fixed
the diff description).

Remaining comments in the diff.

swh/storage/tests/test_replay.py
526

no need for the while

swh/storage/writer.py
22

recursively

douardda updated this revision to Diff 11189.May 13 2020, 4:21 PM

typos (thx ardumont)

swh-public-ci added a comment.May 13 2020, 4:42 PM

Build was aborted

Patch application report for D3150 (id=11189)

Could not rebase; Attempt merge onto 306aa69b26...

Updating 306aa69..13351cd
Fast-forward
 swh/storage/replay.py                  |   9 ++-
 swh/storage/tests/test_kafka_writer.py | 104 ++++++++++++++++++++++--
 swh/storage/tests/test_replay.py       | 143 ++++++++++++++++++++++++++++++++-
 swh/storage/writer.py                  |  70 ++++++++++------
 4 files changed, 289 insertions(+), 37 deletions(-)
Changes applied before test
commit 13351cdb37a87a3597b8de044db8fe30aac6a4e4
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu May 7 15:33:35 2020 +0200

    Add support for anonymized journal topics
    
    This adds 2 new topics (if activated by config), namely
    - ``{kafka_prefix}.release:anonymized`` and
    - ``{kafka_prefix}.revision:anonymized``
    
    These topics are filled aside from their original non-anonymized version.
    The anonymization process consists simply in forging a Person with the
    ``fullname`` being a hash of the triplet (fullname, name, email) of the
    original Person in Release and Revision entities.
    
    So the replayer process can be used as is (just have to not replay both
    standard and anonymized topics at once).

commit e0ad4f46a50bb45ceb679d176068f5e478624163
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 13 11:55:44 2020 +0200

    replay: add support for "flags" in topics
    
    simply ignore anything after a ":" in the object_type part of the topic.

commit 87f7bee6935b831a3c300cadd9afb5ce890f2292
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Apr 7 16:20:35 2020 +0200

    journal: add a skipped_content topic dedicated to SkippedContent objects
    
    instead of mixing them with Content in the content topic.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/170/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/170/console

Harbormaster failed remote builds in B12371: Diff 11189!May 13 2020, 4:42 PM
swh-public-ci added a comment.May 13 2020, 5:40 PM

Build has FAILED

Patch application report for D3150 (id=11189)

Could not rebase; Attempt merge onto 306aa69b26...

Updating 306aa69..13351cd
Fast-forward
 swh/storage/replay.py                  |   9 ++-
 swh/storage/tests/test_kafka_writer.py | 104 ++++++++++++++++++++++--
 swh/storage/tests/test_replay.py       | 143 ++++++++++++++++++++++++++++++++-
 swh/storage/writer.py                  |  70 ++++++++++------
 4 files changed, 289 insertions(+), 37 deletions(-)
Changes applied before test
commit 13351cdb37a87a3597b8de044db8fe30aac6a4e4
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu May 7 15:33:35 2020 +0200

    Add support for anonymized journal topics
    
    This adds 2 new topics (if activated by config), namely
    - ``{kafka_prefix}.release:anonymized`` and
    - ``{kafka_prefix}.revision:anonymized``
    
    These topics are filled aside from their original non-anonymized version.
    The anonymization process consists simply in forging a Person with the
    ``fullname`` being a hash of the triplet (fullname, name, email) of the
    original Person in Release and Revision entities.
    
    So the replayer process can be used as is (just have to not replay both
    standard and anonymized topics at once).

commit e0ad4f46a50bb45ceb679d176068f5e478624163
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed May 13 11:55:44 2020 +0200

    replay: add support for "flags" in topics
    
    simply ignore anything after a ":" in the object_type part of the topic.

commit 87f7bee6935b831a3c300cadd9afb5ce890f2292
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Apr 7 16:20:35 2020 +0200

    journal: add a skipped_content topic dedicated to SkippedContent objects
    
    instead of mixing them with Content in the content topic.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/171/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/171/console

douardda mentioned this in D3160: Add support for a priviledged "channel" of topics for non-anonymized objects.May 18 2020, 1:47 PM
douardda mentioned this in D3161: Add support for anonymized journal topics.May 18 2020, 1:53 PM
douardda abandoned this revision.May 20 2020, 11:16 AM

Revision Contents
Changeset List

PathSize
swh/
storage/
1 line
tests/
104 lines
64 lines
68 lines

Diff 11189

View Options

swh/storage/replay.py

Loading...
View Options

swh/storage/tests/test_kafka_writer.py

Loading...
View Options

swh/storage/tests/test_replay.py

Loading...
View Options

swh/storage/writer.py

Loading...
About · Developer information · Legal