Page MenuHomeSoftware Heritage
Differential D1075

Storage: Add origin_count method
ClosedPublic
Actions

Authored by anlambert on Feb 4 2019, 4:47 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDSTOCe2c4d3897207: storage: Add origin_count method
R65:e2c4d3897207: storage: Add origin_count method
rDSTOe2c4d3897207: storage: Add origin_count method
Summary

Add a method to count the number of origins whose urls contain a given string pattern.
Its purpose is to be used in swh-web to count and display the number origins associated
to each source code provider referenced in the archive coverage list.

Related T1463

Diff Detail

Repository
rDSTO Storage manager
Branch
origin-count
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 4008
Build 5254: tox-on-jenkinsJenkins
Build 5253: arc lint + arc unit

Event Timeline

anlambert created this revision.Feb 4 2019, 4:47 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 4 2019, 4:47 PM
swh-public-ci added a subscriber: swh-public-ci.Feb 4 2019, 4:51 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/175/ for more details.

Harbormaster completed remote builds in B4008: Diff 3388.Feb 4 2019, 4:51 PM
anlambert added a child revision: D1076: swh-web/coverage: Add origin count for each referenced code provider.Feb 4 2019, 4:53 PM
vlorentz added a subscriber: vlorentz.Feb 4 2019, 5:08 PM
vlorentz added inline comments.
swh/storage/db.py
673–675

offset and limit should be removed from the doc.

681–695

Query building should be factorized with origin_search.

swh/storage/in_memory.py
870–873

What's the difference with the former code?

swh/storage/tests/test_storage.py
2222

Should have a test with regexp=False too

anlambert edited the summary of this revision. (Show Details)Feb 5 2019, 2:06 PM
anlambert marked 4 inline comments as done.Feb 5 2019, 2:43 PM
anlambert added inline comments.
swh/storage/db.py
673–675

oops, forgot these ones

681–695

Agreed, will work something out

swh/storage/in_memory.py
870–873

I wanted to remove the limit restriction for counting all found origins but I found a better way without changing
that line. This will be removed in next update.

swh/storage/tests/test_storage.py
2222

ack

anlambert updated this revision to Diff 3395.Feb 5 2019, 3:13 PM

Update: Address vlorentz comments

swh-public-ci added a comment.Feb 5 2019, 3:16 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/176/ for more details.

Harbormaster completed remote builds in B4015: Diff 3395.Feb 5 2019, 3:16 PM
vlorentz accepted this revision.Feb 5 2019, 4:38 PM
vlorentz added inline comments.
swh/storage/db.py
641–642

nitpick: I find this to be more readable:

if count:
    origin_cols = 'COUNT(*)'
else:
    origin_cols = ','.join(self.origin_cols)
649–650

this can be a single line

651–653

so can this

This revision is now accepted and ready to land.Feb 5 2019, 4:38 PM
anlambert updated this revision to Diff 3537.Feb 12 2019, 2:06 PM

Update: Rebase and address latest vlorentz comments

swh-public-ci added a comment.Feb 12 2019, 2:09 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/180/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/180/console

Harbormaster failed remote builds in B4169: Diff 3537!Feb 12 2019, 2:09 PM
vlorentz added a comment.Feb 12 2019, 2:11 PM

You need to rebase on master to fix the build failure.

vlorentz added a comment.Feb 12 2019, 2:12 PM

Oh wait, my mistake. I tagged a new version of swh.core that broke that particular test.

anlambert added a comment.Feb 12 2019, 2:13 PM

I already rebased to master.

anlambert added a comment.Feb 12 2019, 2:14 PM

Ok I let you handle the fix, I will rebase again afterwards if needed.

vlorentz added a comment.Feb 12 2019, 2:19 PM

D1118

anlambert updated this revision to Diff 3542.Feb 12 2019, 2:26 PM

Rebase

swh-public-ci added a comment.Feb 12 2019, 2:29 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/194/ for more details.

Harbormaster completed remote builds in B4173: Diff 3542.Feb 12 2019, 2:29 PM
Closed by commit rDSTOe2c4d3897207: storage: Add origin_count method (authored by anlambert). · Explain WhyFeb 12 2019, 2:38 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
swh/
storage/
api/
5 lines
5 lines
36 lines
24 lines
19 lines
tests/
34 lines

Diff 3388

View Options

swh/storage/api/client.py

Loading...
View Options

swh/storage/api/server.py

Loading...
View Options

swh/storage/db.py

Loading...
View Options

swh/storage/in_memory.py

Loading...
View Options

swh/storage/storage.py

Loading...
View Options

swh/storage/tests/test_storage.py

Loading...
About · Developer information · Legal