Page MenuHomeSoftware Heritage

Scheduling utilitiesFolder
ActivePublic

Members

  • This project does not have any members.

Watchers

  • This project does not have any watchers.

Recent Activity

Tue, Dec 4

ardumont closed T1227: General improvments of the indexer: Schedule indexer tasks as Resolved.
Tue, Dec 4, 11:47 AM · Indexer, Scheduling utilities
ardumont removed a subtask for T1227: General improvments of the indexer: Schedule indexer tasks: T1386: Refactor indexers' initialization step.
Tue, Dec 4, 11:47 AM · Indexer, Scheduling utilities
ardumont removed a parent task for T1386: Refactor indexers' initialization step: T1227: General improvments of the indexer: Schedule indexer tasks.
Tue, Dec 4, 11:47 AM · Indexer, Scheduling utilities
vlorentz added a parent task for T1386: Refactor indexers' initialization step: T1410: Kill implicit configuration.
Tue, Dec 4, 11:29 AM · Indexer, Scheduling utilities

Fri, Nov 30

vlorentz added a comment to T1401: Make celery workers print the content of their stack when logging an exception.

Sorry, I meant the content of local variables in the stack.

Fri, Nov 30, 2:57 PM · Scheduling utilities
anlambert added a comment to T1401: Make celery workers print the content of their stack when logging an exception.

Unless I misunderstood, this is already the case.

Fri, Nov 30, 2:52 PM · Scheduling utilities
vlorentz renamed T1401: Make celery workers print the content of their stack when logging an exception from Make celery worker print the content of their stack when logging an exception to Make celery workers print the content of their stack when logging an exception.
Fri, Nov 30, 2:15 PM · Scheduling utilities
vlorentz triaged T1401: Make celery workers print the content of their stack when logging an exception as Normal priority.
Fri, Nov 30, 2:15 PM · Scheduling utilities
vlorentz triaged T1400: Make celery workers print their arguments in the logs when they crash as Normal priority.
Fri, Nov 30, 2:14 PM · Scheduling utilities

Wed, Nov 28

ardumont closed T1374: content indexer: Determine the identifier ranges to use to schedule those as Resolved.

As we got around ~5b contents for now, i went for 100000 ranges of ~50k each
It's been scheduled and content indexers are now consuming again.

Wed, Nov 28, 12:33 PM · Indexer, Scheduling utilities
ardumont closed T1374: content indexer: Determine the identifier ranges to use to schedule those, a subtask of T1227: General improvments of the indexer: Schedule indexer tasks, as Resolved.
Wed, Nov 28, 12:33 PM · Indexer, Scheduling utilities
ardumont closed T818: indexer DB should not use bytea for mimetype and encoding columns, a subtask of T1374: content indexer: Determine the identifier ranges to use to schedule those, as Resolved.
Wed, Nov 28, 12:26 PM · Indexer, Scheduling utilities

Tue, Nov 27

ardumont triaged T1386: Refactor indexers' initialization step as Normal priority.
Tue, Nov 27, 11:58 AM · Indexer, Scheduling utilities
ardumont added a subtask for T1374: content indexer: Determine the identifier ranges to use to schedule those: T818: indexer DB should not use bytea for mimetype and encoding columns.
Tue, Nov 27, 11:53 AM · Indexer, Scheduling utilities
vlorentz closed T1324: Deploy metadata indexers in production, a subtask of T1227: General improvments of the indexer: Schedule indexer tasks, as Resolved.
Tue, Nov 27, 11:44 AM · Indexer, Scheduling utilities

Wed, Nov 21

ardumont triaged T1374: content indexer: Determine the identifier ranges to use to schedule those as Normal priority.
Wed, Nov 21, 3:25 PM · Indexer, Scheduling utilities
ardumont closed T1310: Simplify indexer design: move away from the pipeline approach as Resolved.
Wed, Nov 21, 3:23 PM · Indexer, Scheduling utilities
ardumont closed T1310: Simplify indexer design: move away from the pipeline approach, a subtask of T1227: General improvments of the indexer: Schedule indexer tasks, as Resolved.
Wed, Nov 21, 3:23 PM · Indexer, Scheduling utilities
ardumont closed T991: Indexers: Send range of ids instead of list of ids, a subtask of T1310: Simplify indexer design: move away from the pipeline approach, as Resolved.
Wed, Nov 21, 3:23 PM · Indexer, Scheduling utilities
ardumont closed T1312: indexer: Adapt textual content indexer to actually filter textual content themselves as Resolved.

Only fossology_license_indexer needs that.

Wed, Nov 21, 9:41 AM · Indexer, Scheduling utilities
ardumont closed T1312: indexer: Adapt textual content indexer to actually filter textual content themselves, a subtask of T1310: Simplify indexer design: move away from the pipeline approach, as Resolved.
Wed, Nov 21, 9:41 AM · Indexer, Scheduling utilities

Tue, Nov 13

ardumont added a subtask for T1227: General improvments of the indexer: Schedule indexer tasks: T1324: Deploy metadata indexers in production.
Tue, Nov 13, 1:43 PM · Indexer, Scheduling utilities
ardumont removed a subtask for T1227: General improvments of the indexer: Schedule indexer tasks: T991: Indexers: Send range of ids instead of list of ids.
Tue, Nov 13, 1:42 PM · Indexer, Scheduling utilities
ardumont added a subtask for T1310: Simplify indexer design: move away from the pipeline approach: T991: Indexers: Send range of ids instead of list of ids.
Tue, Nov 13, 1:42 PM · Indexer, Scheduling utilities
ardumont closed T1311: indexer: Remove orchestrators, a subtask of T1310: Simplify indexer design: move away from the pipeline approach, as Resolved.
Tue, Nov 13, 12:17 PM · Indexer, Scheduling utilities
ardumont closed T1311: indexer: Remove orchestrators as Resolved.
Tue, Nov 13, 12:17 PM · Indexer, Scheduling utilities

Nov 9 2018

ardumont claimed T1310: Simplify indexer design: move away from the pipeline approach.
Nov 9 2018, 11:38 AM · Indexer, Scheduling utilities
ardumont claimed T1311: indexer: Remove orchestrators.
Nov 9 2018, 11:37 AM · Indexer, Scheduling utilities
ardumont claimed T1227: General improvments of the indexer: Schedule indexer tasks.
Nov 9 2018, 11:36 AM · Indexer, Scheduling utilities

Nov 7 2018

ardumont triaged T1312: indexer: Adapt textual content indexer to actually filter textual content themselves as Normal priority.
Nov 7 2018, 10:12 AM · Indexer, Scheduling utilities
ardumont triaged T1311: indexer: Remove orchestrators as Normal priority.
Nov 7 2018, 10:11 AM · Indexer, Scheduling utilities
ardumont triaged T1310: Simplify indexer design: move away from the pipeline approach as Normal priority.
Nov 7 2018, 10:10 AM · Indexer, Scheduling utilities
ardumont added a comment to T1229: Indexers: Make orchestrators use swh-scheduler for scheduling.

So in the end, we will remove the orchestrator anyway.
That will simplify the indexer scheduling (which could not really happen easily in the current state).

Nov 7 2018, 10:04 AM · Indexer, Scheduling utilities

Oct 29 2018

ardumont closed T1229: Indexers: Make orchestrators use swh-scheduler for scheduling as Resolved.

Yes, it should.

Oct 29 2018, 10:14 AM · Indexer, Scheduling utilities
ardumont closed T1229: Indexers: Make orchestrators use swh-scheduler for scheduling, a subtask of T1227: General improvments of the indexer: Schedule indexer tasks, as Resolved.
Oct 29 2018, 10:14 AM · Indexer, Scheduling utilities
vlorentz closed T1290: Indexers: Use swh.scheduler instead of directly relying on Celery as Resolved by committing rDCIDX45c2e035a99f: Use swh.scheduler instead of celery in the OriginHeadIndexer..
Oct 29 2018, 9:55 AM · Indexer, Scheduling utilities
vlorentz closed T1290: Indexers: Use swh.scheduler instead of directly relying on Celery, a subtask of T1229: Indexers: Make orchestrators use swh-scheduler for scheduling, as Resolved.
Oct 29 2018, 9:55 AM · Indexer, Scheduling utilities
vlorentz added a comment to T1229: Indexers: Make orchestrators use swh-scheduler for scheduling.

So this task will be solved by D606, right?

Oct 29 2018, 9:12 AM · Indexer, Scheduling utilities

Oct 26 2018

ardumont added a parent task for T1290: Indexers: Use swh.scheduler instead of directly relying on Celery: T1291: indexer: serialization issue on latest deployed indexer.
Oct 26 2018, 9:35 PM · Indexer, Scheduling utilities
ardumont renamed T1229: Indexers: Make orchestrators use swh-scheduler for scheduling from Indexers: Analysis further the possiblity to remove the orchestrator layer to the benefit of the scheduler to Indexers: Make orchestrators use swh-scheduler for scheduling.
Oct 26 2018, 8:05 PM · Indexer, Scheduling utilities
vlorentz added a revision to T1290: Indexers: Use swh.scheduler instead of directly relying on Celery: D608: Use swh.scheduler instead of celery in the OriginHeadIndexer..
Oct 26 2018, 10:43 AM · Indexer, Scheduling utilities

Oct 25 2018

vlorentz placed T1229: Indexers: Make orchestrators use swh-scheduler for scheduling up for grabs.
Oct 25 2018, 3:58 PM · Indexer, Scheduling utilities
vlorentz renamed T1290: Indexers: Use swh.scheduler instead of directly relying on Celery from Use swh.scheduler instead of directly relying on Celery to Indexers: Use swh.scheduler instead of directly relying on Celery.
Oct 25 2018, 3:57 PM · Indexer, Scheduling utilities
vlorentz triaged T1290: Indexers: Use swh.scheduler instead of directly relying on Celery as Normal priority.
Oct 25 2018, 3:57 PM · Indexer, Scheduling utilities
ardumont added a comment to T1229: Indexers: Make orchestrators use swh-scheduler for scheduling.

Indexers can be made configurable as well

Oct 25 2018, 3:52 PM · Indexer, Scheduling utilities

Oct 24 2018

vlorentz added a comment to T1229: Indexers: Make orchestrators use swh-scheduler for scheduling.

Indexers can be made configurable as well

Oct 24 2018, 2:30 PM · Indexer, Scheduling utilities
ardumont added a comment to T1229: Indexers: Make orchestrators use swh-scheduler for scheduling.

My plan is to move the logic from the orchestrator into the indexers themselves

Oct 24 2018, 1:26 PM · Indexer, Scheduling utilities
vlorentz added a comment to T1229: Indexers: Make orchestrators use swh-scheduler for scheduling.

My plan is to move the logic from the orchestrator into the indexers themselves

Oct 24 2018, 12:27 PM · Indexer, Scheduling utilities
ardumont updated subscribers of T1229: Indexers: Make orchestrators use swh-scheduler for scheduling.

@vlorentz @olasd Beware though, the orchestrator is supposed to per indexer (as in its configuration, e.g. P235):

Oct 24 2018, 12:02 PM · Indexer, Scheduling utilities

Oct 22 2018

vlorentz claimed T1229: Indexers: Make orchestrators use swh-scheduler for scheduling.
Oct 22 2018, 2:05 PM · Indexer, Scheduling utilities