Page MenuHomeSoftware Heritage

Storage: Add origin_count method
ClosedPublic

Authored by anlambert on Feb 4 2019, 4:47 PM.

Details

Summary

Add a method to count the number of origins whose urls contain a given string pattern.
Its purpose is to be used in swh-web to count and display the number origins associated
to each source code provider referenced in the archive coverage list.

Related T1463

Diff Detail

Repository
rDSTO Storage manager
Branch
origin-count
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 4173
Build 5501: tox-on-jenkinsJenkins
Build 5500: arc lint + arc unit

Event Timeline

anlambert created this revision.Feb 4 2019, 4:47 PM
vlorentz added inline comments.
swh/storage/db.py
688–690

offset and limit should be removed from the doc.

696–710

Query building should be factorized with origin_search.

swh/storage/in_memory.py
900–901

What's the difference with the former code?

swh/storage/tests/test_storage.py
2256

Should have a test with regexp=False too

anlambert edited the summary of this revision. (Show Details)Feb 5 2019, 2:06 PM
anlambert marked 4 inline comments as done.Feb 5 2019, 2:43 PM
anlambert added inline comments.
swh/storage/db.py
688–690

oops, forgot these ones

696–710

Agreed, will work something out

swh/storage/in_memory.py
900–901

I wanted to remove the limit restriction for counting all found origins but I found a better way without changing
that line. This will be removed in next update.

swh/storage/tests/test_storage.py
2256

ack

anlambert updated this revision to Diff 3395.Feb 5 2019, 3:13 PM

Update: Address vlorentz comments

vlorentz accepted this revision.Feb 5 2019, 4:38 PM
vlorentz added inline comments.
swh/storage/db.py
630–636

nitpick: I find this to be more readable:

if count:
    origin_cols = 'COUNT(*)'
else:
    origin_cols = ','.join(self.origin_cols)
642–643

this can be a single line

644–646

so can this

This revision is now accepted and ready to land.Feb 5 2019, 4:38 PM
anlambert updated this revision to Diff 3537.Feb 12 2019, 2:06 PM

Update: Rebase and address latest vlorentz comments

You need to rebase on master to fix the build failure.

Oh wait, my mistake. I tagged a new version of swh.core that broke that particular test.

I already rebased to master.

Ok I let you handle the fix, I will rebase again afterwards if needed.

This revision was automatically updated to reflect the committed changes.