Page MenuHomeSoftware Heritage

Scheduling utilitiesFolder
ActivePublic

Members

  • This project does not have any members.

Watchers

  • This project does not have any watchers.

Recent Activity

Yesterday

zack added a parent task for T1537: semi-automated addition of new "forges": T1538: save "forge" now.
Thu, Feb 21, 8:21 PM · Scheduling utilities, Web app
zack added a comment to T1537: semi-automated addition of new "forges".

As a test case, we've been asked to archive this small (for now) GitLab instance: https://gitlab.etsit.urjc.es/
We can easily find tons of other small public instances for further testing.

Thu, Feb 21, 8:17 PM · Scheduling utilities, Web app
zack added a project to T1537: semi-automated addition of new "forges": Scheduling utilities.
Thu, Feb 21, 8:16 PM · Scheduling utilities, Web app
vlorentz claimed T1536: Add a CLI tool to run a task on all origins.
Thu, Feb 21, 11:12 AM · Scheduling utilities
vlorentz triaged T1536: Add a CLI tool to run a task on all origins as Normal priority.
Thu, Feb 21, 11:11 AM · Scheduling utilities

Wed, Feb 20

vlorentz closed T1495: Move swh.storage.db.BaseDb class in swh.core, a subtask of T1494: swh-scheduler API does not really support concurrency , as Resolved.
Wed, Feb 20, 11:45 AM · Scheduling utilities
ardumont closed T1491: Connection problems with latest celery (4.2.1; docker env, production) as Resolved.

status:

  • runner: fine (no crash, no restart)
  • listener: fine (same)
  • workers: fine (same)
Wed, Feb 20, 10:00 AM · Docker environment, Scheduling utilities

Tue, Feb 19

ardumont changed the status of T1491: Connection problems with latest celery (4.2.1; docker env, production) from Open to Work in Progress.

WIP as the new version has been deployed.
Let's see if the occurrences still occur.

Tue, Feb 19, 2:38 PM · Docker environment, Scheduling utilities
olasd added a parent task for T1460: Add task related metrics to swh-scheduler: T1535: Deploy prometheus-statsd-exporter to gather per-worker metrics.
Tue, Feb 19, 2:25 PM · Scheduling utilities, Metrics/monitoring
ardumont added a comment to T1491: Connection problems with latest celery (4.2.1; docker env, production).

I've done the upgrade on saatchi and restarted both listener and runner. I've removed the runner restart from the saatchi crontab.

Tue, Feb 19, 11:33 AM · Docker environment, Scheduling utilities
olasd added a comment to T1491: Connection problems with latest celery (4.2.1; docker env, production).

I've pushed an updated kombu to our repository.

Tue, Feb 19, 9:55 AM · Docker environment, Scheduling utilities

Sun, Feb 17

ardumont added a comment to T1491: Connection problems with latest celery (4.2.1; docker env, production).

bunch of celery workers (loader*, lister*) indeed have a ConnectionResetError stacktrace (not necessarily the same):

Sun, Feb 17, 9:54 AM · Docker environment, Scheduling utilities

Sat, Feb 16

ardumont renamed T1491: Connection problems with latest celery (4.2.1; docker env, production) from Connection problems with celery in the docker env to Connection problems with latest celery (4.2.1; docker env, production).
Sat, Feb 16, 4:22 PM · Docker environment, Scheduling utilities
ardumont added a parent task for T1491: Connection problems with latest celery (4.2.1; docker env, production): T1530: Deploy all latest development in production, then refactor deprecated configuration/code.
Sat, Feb 16, 4:21 PM · Docker environment, Scheduling utilities
ardumont added a comment to T1491: Connection problems with latest celery (4.2.1; docker env, production).

As per our pair-programming yesterday, I think we reproduce this in production now (with the runner at least).

Sat, Feb 16, 3:00 AM · Docker environment, Scheduling utilities

Thu, Feb 14

vlorentz triaged T1527: Have comments on all columns of all databases as Normal priority.
Thu, Feb 14, 11:08 AM · Storage manager, Scheduling utilities, Indexer

Thu, Feb 7

vlorentz triaged T1521: Explore Faust as a possible kafka-based replacement for celery as Normal priority.
Thu, Feb 7, 4:38 PM · Scheduling utilities
douardda added a comment to T1521: Explore Faust as a possible kafka-based replacement for celery.

Also give a look at Dramatiq https://dramatiq.io/

Thu, Feb 7, 4:17 PM · Scheduling utilities
douardda created T1521: Explore Faust as a possible kafka-based replacement for celery.
Thu, Feb 7, 4:15 PM · Scheduling utilities

Wed, Feb 6

olasd closed T1460: Add task related metrics to swh-scheduler as Resolved by committing rDSCHc29d3834978d: Add basic stats to tasks.
Wed, Feb 6, 1:12 PM · Scheduling utilities, Metrics/monitoring

Thu, Jan 31

douardda closed T1494: swh-scheduler API does not really support concurrency as Resolved.

via cebc11d1752e

Thu, Jan 31, 3:49 PM · Scheduling utilities
ardumont added a comment to T1508: Rationalize task type names.

As a minor improvement I suggest switching from nouns to verb, so: load-git, load-hg-archive, list-gitlab-full, etc. Rationale: "do this" is the semantics associated to a message.

Thu, Jan 31, 3:33 PM · Scheduling utilities
zack added a comment to T1508: Rationalize task type names.

+1 for this need, and +1 also to the initial draft by @ardumont.

Thu, Jan 31, 3:11 PM · Scheduling utilities
ardumont added a comment to T1508: Rationalize task type names.

That'd be great.

Thu, Jan 31, 2:53 PM · Scheduling utilities
douardda triaged T1508: Rationalize task type names as Low priority.
Thu, Jan 31, 2:22 PM · Scheduling utilities

Tue, Jan 29

ardumont closed T1469: clean up stale status="ongoing" visits as Resolved.

We should clean them up.

Tue, Jan 29, 12:02 PM · Scheduling utilities
douardda merged task T904: Factor-out common DB wrappers in swh.core into T1495: Move swh.storage.db.BaseDb class in swh.core.
Tue, Jan 29, 10:22 AM · Core & foundations, Storage manager, Scheduling utilities, Vault
douardda added a comment to T904: Factor-out common DB wrappers in swh.core.

I agree we must be careful with not bloating swh.core, but the current subject, it really makes sense to me to put this basic db access wrapper as a core functionality.

Tue, Jan 29, 10:14 AM · Core & foundations, Storage manager, Scheduling utilities, Vault

Fri, Jan 25

vlorentz added revisions to T1386: Refactor indexers' initialization step: D990: First step towards a simplification of the indexer's code, D993: Simplify a bit unit tests.
Fri, Jan 25, 10:22 AM · Indexer, Scheduling utilities

Thu, Jan 24

douardda added a comment to T1491: Connection problems with latest celery (4.2.1; docker env, production).

I confirm that I do not see ConnectionResetError: [Errno 104] Connection reset by peer and BrokenPipeError: [Errno 32] Broken pipe so far in the runner logs with kombu from git's master.

Thu, Jan 24, 3:40 PM · Docker environment, Scheduling utilities
douardda added a project to T1494: swh-scheduler API does not really support concurrency : Scheduling utilities.
Thu, Jan 24, 3:37 PM · Scheduling utilities
vlorentz triaged T1491: Connection problems with latest celery (4.2.1; docker env, production) as High priority.
Thu, Jan 24, 11:49 AM · Docker environment, Scheduling utilities
ardumont renamed T1492: Requests to the scheduler API take up to 6 seconds in prod from Requests to the scheduler AI take up to 6 seconds in prod to Requests to the scheduler API take up to 6 seconds in prod.
Thu, Jan 24, 10:39 AM · System administration, Scheduling utilities

Jan 23 2019

vlorentz triaged T1492: Requests to the scheduler API take up to 6 seconds in prod as High priority.
Jan 23 2019, 1:18 PM · System administration, Scheduling utilities

Jan 18 2019

douardda closed T1479: Factorize the debug logging of a task in the SWHTask class as Resolved.
Jan 18 2019, 3:10 PM · Scheduling utilities

Jan 17 2019

douardda added a revision to T1479: Factorize the debug logging of a task in the SWHTask class: D967: Add debug logging in the SWHTask class.
Jan 17 2019, 11:29 AM · Scheduling utilities
douardda triaged T1479: Factorize the debug logging of a task in the SWHTask class as Normal priority.
Jan 17 2019, 11:27 AM · Scheduling utilities

Jan 15 2019

vlorentz added a project to T1473: Refactor the swh_scheduler fixture: Scheduling utilities.
Jan 15 2019, 4:39 PM · Scheduling utilities
ardumont triaged T1472: scheduler: Update schema to reference origin informations as Normal priority.
Jan 15 2019, 2:36 PM · Scheduling utilities
ardumont changed the status of T1469: clean up stale status="ongoing" visits from Open to Work in Progress.
Jan 15 2019, 2:09 PM · Scheduling utilities
ardumont claimed T1469: clean up stale status="ongoing" visits.
Jan 15 2019, 11:50 AM · Scheduling utilities

Jan 14 2019

zack triaged T1469: clean up stale status="ongoing" visits as Normal priority.
Jan 14 2019, 12:04 PM · Scheduling utilities

Jan 8 2019

olasd claimed T1460: Add task related metrics to swh-scheduler.
Jan 8 2019, 11:46 AM · Scheduling utilities, Metrics/monitoring
olasd added a project to T1460: Add task related metrics to swh-scheduler: Scheduling utilities.
Jan 8 2019, 11:44 AM · Scheduling utilities, Metrics/monitoring

Dec 18 2018

vlorentz lowered the priority of T1401: Make celery workers print the content of their stack when logging an exception from Normal to Low.
Dec 18 2018, 4:54 PM · Easy hack, Scheduling utilities
vlorentz added a project to T1401: Make celery workers print the content of their stack when logging an exception: Easy hack.

Tip: we can do something based on the collect_extra_debug_data defined here: https://github.com/ProgVal/Limnoria/blob/master/src/utils/python.py

Dec 18 2018, 4:49 PM · Easy hack, Scheduling utilities
vlorentz added a parent task for T898: Automatic registration of tasks in the scheduler: T1405: Make it easy to run a complete swh instance.
Dec 18 2018, 4:11 PM · Scheduling utilities
vlorentz added a project to T1400: Make celery workers print their arguments in the logs when they crash: Easy hack.
Dec 18 2018, 3:38 PM · Easy hack, Scheduling utilities

Dec 14 2018

ardumont added a comment to T725: Scheduler API server.

Right. Thanks.

Dec 14 2018, 1:10 PM · Scheduling utilities

Dec 13 2018

olasd closed T725: Scheduler API server as Resolved.

I think that's been done by @ardumont

Dec 13 2018, 3:41 PM · Scheduling utilities