Page MenuHomeSoftware Heritage

storage: Implement visit types filtering in origin_search method
ClosedPublic

Authored by anlambert on Feb 18 2021, 4:33 PM.

Details

Summary

Enable to filter searched origins by visit types.

Add a new optional visit_types parameter to origin_search method in
StorageInterface.

Implement visit types filtering in storage backends, an origin wil be
returned if it has any of the requested visit types.

This is clearly not designed to be used in production due to performance
issues but rather in testing environments with small archive dataset.

Related to T2869

Diff Detail

Repository
rDSTO Storage manager
Branch
visit-type-search
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 19348
Build 30003: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 30002: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D5110 (id=18252)

Rebasing onto 7b4c1247ec...

Current branch diff-target is up to date.
Changes applied before test
commit 484630ec5fa226bf2b753ea38c2b0e609484c58f
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Thu Feb 11 11:23:58 2021 +0100

    storage: Implement visit types filtering in origin_search method
    
    Enable to filter searched origins by visit types.
    
    Add a new optional visit_types parameter to origin_search method in
    StorageInterface.
    
    Implement visit types filtering in storage backends, an origin wil be
    returned if it has any of the requested visit types.
    
    This is clearly not designed to be used in production due to performance
    issues but rather in testing environments with small archive dataset.
    
    Related to T2869

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1156/ for more details.

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/storage/cassandra/storage.py
839–848
This revision is now accepted and ready to land.Feb 19 2021, 9:41 AM
swh/storage/cassandra/storage.py
839–848

awesome, thanks !

Simplify origin visits iteration code.

Build is green

Patch application report for D5110 (id=18294)

Rebasing onto 7b4c1247ec...

Current branch diff-target is up to date.
Changes applied before test
commit f3ef6e6a5a12a9018a5a3865676b41a8e7974d5d
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Thu Feb 11 11:23:58 2021 +0100

    storage: Implement visit types filtering in origin_search method
    
    Enable to filter searched origins by visit types.
    
    Add a new optional visit_types parameter to origin_search method in
    StorageInterface.
    
    Implement visit types filtering in storage backends, an origin wil be
    returned if it has any of the requested visit types.
    
    This is clearly not designed to be used in production due to performance
    issues but rather in testing environments with small archive dataset.
    
    Related to T2869

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1157/ for more details.