Page MenuHomeSoftware Heritage

Storage: Add origin_count method
ClosedPublic

Authored by anlambert on Mon, Feb 4, 4:47 PM.

Details

Summary

Add a method to count the number of origins whose urls contain a given string pattern.
Its purpose is to be used in swh-web to count and display the number origins associated
to each source code provider referenced in the archive coverage list.

Related T1463

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Mon, Feb 4, 4:47 PM
vlorentz added inline comments.
swh/storage/db.py
688–690

offset and limit should be removed from the doc.

696–710

Query building should be factorized with origin_search.

swh/storage/in_memory.py
900–901

What's the difference with the former code?

swh/storage/tests/test_storage.py
2256

Should have a test with regexp=False too

anlambert edited the summary of this revision. (Show Details)Tue, Feb 5, 2:06 PM
anlambert marked 4 inline comments as done.Tue, Feb 5, 2:43 PM
anlambert added inline comments.
swh/storage/db.py
688–690

oops, forgot these ones

696–710

Agreed, will work something out

swh/storage/in_memory.py
900–901

I wanted to remove the limit restriction for counting all found origins but I found a better way without changing
that line. This will be removed in next update.

swh/storage/tests/test_storage.py
2256

ack

anlambert updated this revision to Diff 3395.Tue, Feb 5, 3:13 PM

Update: Address vlorentz comments

vlorentz accepted this revision.Tue, Feb 5, 4:38 PM
vlorentz added inline comments.
swh/storage/db.py
630–636

nitpick: I find this to be more readable:

if count:
    origin_cols = 'COUNT(*)'
else:
    origin_cols = ','.join(self.origin_cols)
642–643

this can be a single line

644–646

so can this

This revision is now accepted and ready to land.Tue, Feb 5, 4:38 PM
anlambert updated this revision to Diff 3537.Tue, Feb 12, 2:06 PM

Update: Rebase and address latest vlorentz comments

You need to rebase on master to fix the build failure.

Oh wait, my mistake. I tagged a new version of swh.core that broke that particular test.

I already rebased to master.

Ok I let you handle the fix, I will rebase again afterwards if needed.

This revision was automatically updated to reflect the committed changes.