Paths

Table of Contentst

Differential D608

Use swh.scheduler instead of celery in the OriginHeadIndexer.
ClosedPublic
Actions

Authored by vlorentz on Oct 26 2018, 10:43 AM.

Tags

None

Subscribers

Details

Reviewers

Group Reviewers

Maniphest Tasks

T1290: Indexers: Use swh.scheduler instead of directly relying on Celery
T1291: indexer: serialization issue on latest deployed indexer

Commits

rDCIDX45c2e035a99f: Use swh.scheduler instead of celery in the OriginHeadIndexer.

Summary

Resolves T1291. Combined with D606, it resolves T1290.

Diff Detail

Repository

rDCIDX Metadata indexer

Branch

origin-head-without-celery

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 1946
Build 2343: arc lint + arc unit

Event Timeline

vlorentz created this revision.Oct 26 2018, 10:43 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptOct 26 2018, 10:43 AM

Harbormaster completed remote builds in B1946: Diff 1858.Oct 26 2018, 10:43 AM

vlorentz added a parent revision: D605: Add SchedulerTestFixture..Oct 26 2018, 10:43 AM

ardumont added a subscriber: ardumont.Oct 26 2018, 11:24 AM

ardumont added inline comments.

swh/indexer/indexer.py
349	I would expect [dict] to mean a list of dict. same for [str] the line just before (which is already there).
swh/indexer/origin_head.py
74	jsyk, there is a `swh.scheduler.utils.create_task_dict` (granted, right now it misses the `retries_left`). Not that i require to change that here ;) In that regard, you might be interested in a discussion to create generically task from the scheduler T1197. Note: T898 might also be related
94	As i see this snippet popping up twice now... (in the same diff). Maybe it'd be worth having an @property on a base class somewhere which does that?

vlorentz marked 2 inline comments as done.Oct 26 2018, 11:33 AM

vlorentz added inline comments.

swh/indexer/origin_head.py
74	jsyk, there is a swh.scheduler.utils.create_task_dict Perfect In that regard, you might be interested in a discussion to create generically task from the scheduler T1197. I don't think you meant that task
94	yeah, but I don't see any place where it makes sense to have it

Fix type in the docstrings.
Use swh.scheduler.utils.create_task_dict.

Harbormaster completed remote builds in B1949: Diff 1861.Oct 26 2018, 11:37 AM

vlorentz marked 2 inline comments as done.Oct 26 2018, 11:41 AM

Resolves T1291. Combined with D606, it resolves T1290.

That's not entirely complete with this diff ;) [1]

[1] https://forge.softwareheritage.org/source/swh-indexer/browse/master/swh/indexer/mimetype.py$145

ardumont added inline comments.Oct 26 2018, 1:57 PM

swh/indexer/origin_head.py
74	I don't think you meant that task Right T1157! (we cannot amend remark, /me is sad)

Sounds good.

Impacts:

swh-site configuration
new task_type(s) on swh-scheduler db

ardumont accepted this revision.Oct 26 2018, 7:06 PM

This revision is now accepted and ready to land.Oct 26 2018, 7:06 PM

ardumont added inline comments.Oct 26 2018, 7:52 PM

swh/indexer/origin_head.py
94	mmm, right. I would have thought BaseIndexer. But i see that we could also share this in the BaseOrchestratorIndexer... so nvm for now ;)

ardumont mentioned this in D613: swh.scheduler.utils: Allow to add more task information.Oct 26 2018, 8:36 PM

Closed by commit rDCIDX45c2e035a99f: Use swh.scheduler instead of celery in the OriginHeadIndexer. (authored by vlorentz). · Explain WhyOct 29 2018, 9:55 AM

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Path

Size

swh/

indexer/

49 lines

19 lines

56 lines

tests/

test_metadata.py

2 lines

test_origin_metadata.py

42 lines

Diff 1858

swh/indexer/indexer.py

Loading...

swh/indexer/metadata.py

Loading...

swh/indexer/origin_head.py

Loading...

swh/indexer/tests/test_metadata.py

Loading...

swh/indexer/tests/test_origin_metadata.py

Loading...