Page MenuHomeSoftware Heritage

storage: Implement visit types filtering in origin_search method
ClosedPublic

Authored by anlambert on Thu, Feb 18, 4:33 PM.

Details

Summary

Enable to filter searched origins by visit types.

Add a new optional visit_types parameter to origin_search method in
StorageInterface.

Implement visit types filtering in storage backends, an origin wil be
returned if it has any of the requested visit types.

This is clearly not designed to be used in production due to performance
issues but rather in testing environments with small archive dataset.

Related to T2869

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D5110 (id=18252)

Rebasing onto 7b4c1247ec...

Current branch diff-target is up to date.
Changes applied before test
commit 484630ec5fa226bf2b753ea38c2b0e609484c58f
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Thu Feb 11 11:23:58 2021 +0100

    storage: Implement visit types filtering in origin_search method
    
    Enable to filter searched origins by visit types.
    
    Add a new optional visit_types parameter to origin_search method in
    StorageInterface.
    
    Implement visit types filtering in storage backends, an origin wil be
    returned if it has any of the requested visit types.
    
    This is clearly not designed to be used in production due to performance
    issues but rather in testing environments with small archive dataset.
    
    Related to T2869

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1156/ for more details.

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/storage/cassandra/storage.py
839–848
This revision is now accepted and ready to land.Fri, Feb 19, 9:41 AM
swh/storage/cassandra/storage.py
839–848

awesome, thanks !

Simplify origin visits iteration code.

Build is green

Patch application report for D5110 (id=18294)

Rebasing onto 7b4c1247ec...

Current branch diff-target is up to date.
Changes applied before test
commit f3ef6e6a5a12a9018a5a3865676b41a8e7974d5d
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Thu Feb 11 11:23:58 2021 +0100

    storage: Implement visit types filtering in origin_search method
    
    Enable to filter searched origins by visit types.
    
    Add a new optional visit_types parameter to origin_search method in
    StorageInterface.
    
    Implement visit types filtering in storage backends, an origin wil be
    returned if it has any of the requested visit types.
    
    This is clearly not designed to be used in production due to performance
    issues but rather in testing environments with small archive dataset.
    
    Related to T2869

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1157/ for more details.