Diffusion Listers 0ad37740d9d7

pattern: Make lister flush regularly origins to scheduler
0ad37740d9d7
Actions

Tags

None

Subscribers

None

Description

pattern: Make lister flush regularly origins to scheduler

As origins is a generator, the previous behavior would try to consume the overall
generator to send the records.

This groups and sends batch of 100 origins to the scheduler for writing.

Related to T3003

Details

Provenance

ardumont	Authored on Jan 28 2021, 4:51 PM
ardumont	Pushed on Jan 28 2021, 5:11 PM

Differential Revision

D4965: pattern: Make lister flush regularly origins to scheduler

Parents

rDLSf86200470025: launchpad: Reimplement lister using new Lister API

Branches

Unknown

Tags

Unknown

Tasks

T3003: next gen lister: Make lister flush their visit state regularly

Build Status

Buildable 18861
Build 29221: test-and-build	Jenkins console · Jenkins

Event Timeline

ardumont committed rDLS0ad37740d9d7: pattern: Make lister flush regularly origins to scheduler (authored by ardumont).Jan 28 2021, 4:52 PM

ardumont added a task: T3003: next gen lister: Make lister flush their visit state regularly.Jan 28 2021, 5:11 PM

ardumont added an edge: D4965: pattern: Make lister flush regularly origins to scheduler.

Harbormaster completed building B18861: rDLS0ad37740d9d7: pattern: Make lister flush regularly origins to scheduler.Jan 28 2021, 5:15 PM

swh-public-ci mentioned this in D4964: launchpad/tasks: Fix ping task function name.Jan 28 2021, 5:39 PM

Changes (1)

Path

Size

swh/

lister/

rDLS0ad37740d9d7

swh/lister/pattern.py

Loading...