Paths

Table of Contentst

Differential D2759

indexer.storage: Make add/del endpoints sum up added objects count
ClosedPublic
Actions

Authored by ardumont on Mar 3 2020, 4:18 PM.

Tags

None

Subscribers

Details

Reviewers

Group Reviewers

Summary

This will allow to send metrics at some point.

Test Plan

tox

Diff Detail

Repository

rDCIDX Metadata indexer

Branch

make-add-del-endpoints-summarize

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 10888
Build 16370: tox-on-jenkins	Jenkins
Build 16369: arc lint + arc unit

Event Timeline

ardumont created this revision.Mar 3 2020, 4:18 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMar 3 2020, 4:18 PM

ardumont added a child revision: D2760: swh.indexer.storage: Add metrics to add/del endpoints.Mar 3 2020, 4:19 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/701/ for more details.

Harbormaster completed remote builds in B10881: Diff 9822.Mar 3 2020, 4:20 PM

vlorentz requested changes to this revision.Mar 3 2020, 4:28 PM

vlorentz added a subscriber: vlorentz.

vlorentz added inline comments.

swh/indexer/sql/40-swh-func.sql
450	does this mean the postgres server is sending back all the data? Can't you make it return a count, or at worst just the ids? (same comment for other occurrences above)
swh/indexer/storage/__init__.py
171	`-> Dict[str, int]` (same comment for other occurrences below)

This revision now requires changes to proceed.Mar 3 2020, 4:28 PM

Simplify query counts

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/704/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/704/console

Harbormaster failed remote builds in B10887: Diff 9829!Mar 3 2020, 7:39 PM

Fix flake8 issues

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/705/ for more details.

Harbormaster completed remote builds in B10888: Diff 9830.Mar 3 2020, 7:56 PM

vlorentz accepted this revision.Mar 3 2020, 8:05 PM

vlorentz added inline comments.

swh/indexer/sql/40-swh-func.sql
452–453	wow now that's some black magic!

This revision is now accepted and ready to land.Mar 3 2020, 8:05 PM

Landed in f816a80

ardumont added inline comments.Mar 5 2020, 1:31 PM

sql/upgrades/131.sql
120	ouch... we do end up currently doing twice the query... without returning any result as the second query will do nothing... thus why i see 'uneventful' task fossology license even though this is currently increasing slowly.

Revision Contents
Changeset List

Path

Size

sql/

upgrades/

223 lines

swh/

indexer/

sql/

30-swh-schema.sql

2 lines

40-swh-func.sql

135 lines

storage/

97 lines

53 lines

82 lines

54 lines

tests/

storage/

test_storage.py

66 lines

Diff 9830

sql/upgrades/131.sql

Loading...

swh/indexer/sql/30-swh-schema.sql

Loading...

swh/indexer/sql/40-swh-func.sql

Loading...

swh/indexer/storage/init.py

Loading...

swh/indexer/storage/db.py

Loading...

swh/indexer/storage/in_memory.py

Loading...

swh/indexer/storage/interface.py

Loading...

swh/indexer/tests/storage/test_storage.py

Loading...