Changeset View
Changeset View
Standalone View
Standalone View
swh/storage/filter.py
Show All 13 Lines | """Filtering Storage implementation. This is in charge of transparently | ||||
filtering out known objects prior to adding them to storage. | filtering out known objects prior to adding them to storage. | ||||
Sample configuration use case for filtering storage: | Sample configuration use case for filtering storage: | ||||
.. code-block: yaml | .. code-block: yaml | ||||
storage: | storage: | ||||
cls: filter | cls: filter | ||||
args: | |||||
storage: | storage: | ||||
cls: remote | cls: remote | ||||
args: http://storage.internal.staging.swh.network:5002/ | url: http://storage.internal.staging.swh.network:5002/ | ||||
""" | """ | ||||
def __init__(self, storage): | def __init__(self, storage): | ||||
self.storage = get_storage(**storage) | self.storage = get_storage(**storage) | ||||
self.objects_seen = { | self.objects_seen = { | ||||
'content': set(), # set of content hashes (sha256) seen | 'content': set(), # set of content hashes (sha256) seen | ||||
'directory': set(), | 'directory': set(), | ||||
'revision': set(), | 'revision': set(), | ||||
▲ Show 20 Lines • Show All 83 Lines • Show Last 20 Lines |