Paths

Table of Contentst

Differential D2761

Make indexers return a summary of their actions
ClosedPublic
Actions

Authored by ardumont on Mar 3 2020, 4:21 PM.

Tags

None

Subscribers

Details

Reviewers

Group Reviewers

Summary

This also tries to type a tad better the indexers.

Depends on D2760

Test Plan

tox

Diff Detail

Repository

rDCIDX Metadata indexer

Branch

make-indexers-summarize

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 10893
Build 16380: tox-on-jenkins	Jenkins
Build 16379: arc lint + arc unit

Event Timeline

ardumont created this revision.Mar 3 2020, 4:21 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMar 3 2020, 4:21 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/703/ for more details.

Harbormaster completed remote builds in B10883: Diff 9824.Mar 3 2020, 4:23 PM

vlorentz requested changes to this revision.Mar 3 2020, 4:37 PM

vlorentz added a subscriber: vlorentz.

vlorentz added inline comments.

swh/indexer/fossology_license.py
98	Why is `data` optional?
99	Why `type: ignore`?
127	Why `type: ignore`?
swh/indexer/indexer.py
287–302	Why did you remove it?
swh/indexer/metadata.py
92	`Dict[str, int]` (same comment below)
337	also here
swh/indexer/tasks.py
18–39	Does anything currently use these results?

This revision now requires changes to proceed.Mar 3 2020, 4:37 PM

ardumont mentioned this in D2760: swh.indexer.storage: Add metrics to add/del endpoints.Mar 3 2020, 5:12 PM

ardumont added inline comments.Mar 3 2020, 5:18 PM

swh/indexer/fossology_license.py
98	because we have a gazillion inconsistent indexer... RevisionIndexer does not pass the data along...
99	because i don't want to spend my week satisfying more mypy. without it it says, that the class has no `tool`... which is somewhat true because it's not initialized in the `__init__` method but in the `prepare` for valid reasons which i forgot (most possibly be able to deal with initialization in tess).
127	same. Also, note that i won't refactor more the indexers right now... I want to be able to graph what's happening right now. The modifications i'm doing is for getting there. I'm trying to move as few cogs as possible...
swh/indexer/indexer.py
287–302	forgot to post my diff comment which explains it.
302	because there is no unified parameter call in the subclasses. The only consistency is that indexers are calling the `run` method at some point.

ardumont added inline comments.Mar 3 2020, 5:21 PM

swh/indexer/tasks.py
18–39	the status is used by the scheduler. the other summaries will simply be displayed in logs. so yes (for all tasks) the specific tasks you are targetting by this comment no longer runs for now.

Rebase

Fix commit range

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/707/ for more details.

Harbormaster completed remote builds in B10890: Diff 9832.Mar 3 2020, 7:59 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/708/ for more details.

Harbormaster completed remote builds in B10891: Diff 9833.Mar 3 2020, 8:01 PM

vlorentz accepted this revision.Mar 3 2020, 8:09 PM

vlorentz added inline comments.

swh/indexer/fossology_license.py
99	you could add `tool: Any` in the class declaration

This revision is now accepted and ready to land.Mar 3 2020, 8:09 PM

ardumont added inline comments.Mar 3 2020, 8:11 PM

swh/indexer/fossology_license.py
99	thanks for that hint, i did not realize, i'll check that tomorrow ;)

Improve typing

Use correct range of commits :/

Use correctly the correct range of commits T.T

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/710/ for more details.

Harbormaster completed remote builds in B10893: Diff 9835.Mar 4 2020, 10:07 AM

Harbormaster failed remote builds in B10894: Diff 9836!

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/711/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/711/console

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/712/ for more details.

Harbormaster completed remote builds in B10895: Diff 9837.Mar 4 2020, 10:09 AM

Landed in eed3531

Revision Contents
Changeset List

Path

Size

sql/

upgrades/

223 lines

swh/

indexer/

13 lines

fossology_license.py

34 lines

144 lines

31 lines

10 lines

4 lines

23 lines

sql/

30-swh-schema.sql

2 lines

40-swh-func.sql

135 lines

storage/

130 lines

53 lines

87 lines

58 lines

79 lines

23 lines

tests/

storage/

test_metrics.py

53 lines

test_storage.py

66 lines

test_fossology_license.py

4 lines

9 lines

Diff 9835

sql/upgrades/131.sql

Loading...

swh/indexer/ctags.py

Loading...

swh/indexer/fossology_license.py

Loading...

swh/indexer/indexer.py

Loading...

swh/indexer/metadata.py

Loading...

swh/indexer/mimetype.py

Loading...

swh/indexer/origin_head.py

Loading...

swh/indexer/rehash.py

Loading...

swh/indexer/sql/30-swh-schema.sql

Loading...

swh/indexer/sql/40-swh-func.sql

Loading...

swh/indexer/storage/init.py

Loading...

swh/indexer/storage/db.py

Loading...

swh/indexer/storage/in_memory.py

Loading...

swh/indexer/storage/interface.py

Loading...

swh/indexer/storage/metrics.py

Loading...

swh/indexer/tasks.py

Loading...

swh/indexer/tests/storage/test_metrics.py

Loading...

swh/indexer/tests/storage/test_storage.py

Loading...

swh/indexer/tests/test_fossology_license.py

Loading...

swh/indexer/tests/utils.py

Loading...