Page MenuHomeSoftware Heritage
Differential D1075

Storage: Add origin_count method
ClosedPublic
Actions

Authored by anlambert on Feb 4 2019, 4:47 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDSTOCe2c4d3897207: storage: Add origin_count method
R65:e2c4d3897207: storage: Add origin_count method
rDSTOe2c4d3897207: storage: Add origin_count method
Summary

Add a method to count the number of origins whose urls contain a given string pattern.
Its purpose is to be used in swh-web to count and display the number origins associated
to each source code provider referenced in the archive coverage list.

Related T1463

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Feb 4 2019, 4:47 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 4 2019, 4:47 PM
swh-public-ci added a subscriber: swh-public-ci.Feb 4 2019, 4:51 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/175/ for more details.

Harbormaster completed remote builds in B4008: Diff 3388.Feb 4 2019, 4:51 PM
anlambert added a child revision: D1076: swh-web/coverage: Add origin count for each referenced code provider.Feb 4 2019, 4:53 PM
vlorentz added a subscriber: vlorentz.Feb 4 2019, 5:08 PM
vlorentz added inline comments.
swh/storage/db.py
687–689

offset and limit should be removed from the doc.

695–709

Query building should be factorized with origin_search.

swh/storage/in_memory.py
900–901

What's the difference with the former code?

swh/storage/tests/test_storage.py
2255

Should have a test with regexp=False too

anlambert edited the summary of this revision. (Show Details)Feb 5 2019, 2:06 PM
anlambert marked 4 inline comments as done.Feb 5 2019, 2:43 PM
anlambert added inline comments.
swh/storage/db.py
687–689

oops, forgot these ones

695–709

Agreed, will work something out

swh/storage/in_memory.py
900–901

I wanted to remove the limit restriction for counting all found origins but I found a better way without changing
that line. This will be removed in next update.

swh/storage/tests/test_storage.py
2255

ack

anlambert updated this revision to Diff 3395.Feb 5 2019, 3:13 PM

Update: Address vlorentz comments

swh-public-ci added a comment.Feb 5 2019, 3:16 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/176/ for more details.

Harbormaster completed remote builds in B4015: Diff 3395.Feb 5 2019, 3:16 PM
vlorentz accepted this revision.Feb 5 2019, 4:38 PM
vlorentz added inline comments.
swh/storage/db.py
630–636

nitpick: I find this to be more readable:

if count:
    origin_cols = 'COUNT(*)'
else:
    origin_cols = ','.join(self.origin_cols)
642–643

this can be a single line

643–645

so can this

This revision is now accepted and ready to land.Feb 5 2019, 4:38 PM
anlambert updated this revision to Diff 3537.Feb 12 2019, 2:06 PM

Update: Rebase and address latest vlorentz comments

swh-public-ci added a comment.Feb 12 2019, 2:09 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/180/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/180/console

Harbormaster failed remote builds in B4169: Diff 3537!Feb 12 2019, 2:09 PM
vlorentz added a comment.Feb 12 2019, 2:11 PM

You need to rebase on master to fix the build failure.

vlorentz added a comment.Feb 12 2019, 2:12 PM

Oh wait, my mistake. I tagged a new version of swh.core that broke that particular test.

anlambert added a comment.Feb 12 2019, 2:13 PM

I already rebased to master.

anlambert added a comment.Feb 12 2019, 2:14 PM

Ok I let you handle the fix, I will rebase again afterwards if needed.

vlorentz added a comment.Feb 12 2019, 2:19 PM

D1118

anlambert updated this revision to Diff 3542.Feb 12 2019, 2:26 PM

Rebase

swh-public-ci added a comment.Feb 12 2019, 2:29 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/194/ for more details.

Harbormaster completed remote builds in B4173: Diff 3542.Feb 12 2019, 2:29 PM
Closed by commit rDSTOe2c4d3897207: storage: Add origin_count method (authored by anlambert). · Explain WhyFeb 12 2019, 2:38 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
swh/
storage/
api/
5 lines
5 lines
70 lines
21 lines
19 lines
tests/
38 lines

Diff 3543

View Options

swh/storage/api/client.py

Loading...
View Options

swh/storage/api/server.py

Loading...
View Options

swh/storage/db.py

Loading...
View Options

swh/storage/in_memory.py

Loading...
View Options

swh/storage/storage.py

Loading...
View Options

swh/storage/tests/test_storage.py

Loading...
About · Developer information · Legal