	Include stories about projects I am a member of.

Jan 18 2022

vsellier changed the status of T3857: unexpected long time between a task is received and the loading start for few tasks from Open to Work in Progress.

Jan 18 2022, 11:53 AM · System administration, Scheduling utilities

vsellier added a comment to T3857: unexpected long time between a task is received and the loading start for few tasks.

After analyzing further the logs, it seems the celery process is accepting the task eb038b42-713e-4c23-a655-17d04f23870a whereas a previous loading is still running
A more contextual logs:

Previous task is received and started:

Jan 17 16:19:31 worker14 python3[2180500]: [2022-01-17 16:19:31,868: INFO/MainProcess] Received task: swh.loader.git.tasks.UpdateGitRepository[70c13ea8-19f4-4ac7-ae4f-ea71f80fc25a]  
Jan 17 16:19:32 worker14 python3[2721731]: [2022-01-17 16:19:32,467: INFO/ForkPoolWorker-133] Load origin 'https://git.libreoffice.org/translations' with type 'git'

The new task is accepted whereas the previous one is not yet finished:

Jan 17 16:26:10 worker14 python3[2180500]: [2022-01-17 16:26:10,683: INFO/MainProcess] Received task: swh.loader.git.tasks.UpdateGitRepository[eb038b42-713e-4c23-a655-17d04f23870a]

the first task ends:

Jan 17 19:48:20 worker14 python3[2721731]: [51B blob data]
Jan 17 19:48:20 worker14 python3[2721731]: [3.6K blob data]
Jan 17 19:49:57 worker14 python3[2721731]: [2022-01-17 19:49:57,085: ERROR/ForkPoolWorker-133] Loading failure, updating to `failed` status
                                           Traceback (most recent call last):
                                             File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 338, in load
                                               more_data_to_fetch = self.fetch_data()
                                             File "/usr/lib/python3/dist-packages/swh/loader/git/loader.py", line 277, in fetch_data
                                               self.origin.url, base_repo, do_progress
                                             File "/usr/lib/python3/dist-packages/swh/loader/git/loader.py", line 213, in fetch_pack_from_origin
                                               progress=do_activity,
                                             File "/usr/lib/python3/dist-packages/dulwich/client.py", line 2001, in fetch_pack
                                               progress,
                                             File "/usr/lib/python3/dist-packages/dulwich/client.py", line 845, in _handle_upload_pack_tail
                                               SIDE_BAND_CHANNEL_PROGRESS: progress,
                                             File "/usr/lib/python3/dist-packages/dulwich/client.py", line 604, in _read_side_band64k_data
                                               cb(pkt)
                                             File "/usr/lib/python3/dist-packages/swh/loader/git/loader.py", line 201, in do_pack
                                               f"Pack file too big for repository {origin_url}, "
                                           OSError: Pack file too big for repository https://git.libreoffice.org/translations, limit is 4294967296 bytes, current size is 4294966867, would write 65515
Jan 17 19:49:59 worker14 python3[2721731]: [2022-01-17 19:49:59,468: INFO/ForkPoolWorker-133] Task swh.loader.git.tasks.UpdateGitRepository[70c13ea8-19f4-4ac7-ae4f-ea71f80fc25a] succeeded in 12627.452697345056s: {'status': 'failed'}

the second task loading starts after 3 hours:

Jan 17 19:50:02 worker14 python3[2736039]: [2022-01-17 19:50:02,768: INFO/ForkPoolWorker-134] Load origin 'https://github.com/webtorrent/webtorrent-desktop' with type 'git'

Jan 18 2022, 11:53 AM · System administration, Scheduling utilities

vsellier triaged T3857: unexpected long time between a task is received and the loading start for few tasks as Normal priority.

Jan 18 2022, 11:46 AM · System administration, Scheduling utilities

Jan 13 2022

ardumont closed T3837: clean up swh-scheduler model db as Resolved.

Jan 13 2022, 11:25 AM · Scheduling utilities, Data Model, System administration

ardumont moved T3837: clean up swh-scheduler model db from deployed/landed/monitoring to done on the System administration board.

Jan 13 2022, 11:24 AM · Scheduling utilities, Data Model, System administration

Jan 12 2022

ardumont moved T3837: clean up swh-scheduler model db from in-progress to deployed/landed/monitoring on the System administration board.

Jan 12 2022, 3:07 PM · Scheduling utilities, Data Model, System administration

ardumont added a comment to T3837: clean up swh-scheduler model db.

And same goes for production [1].

Jan 12 2022, 3:07 PM · Scheduling utilities, Data Model, System administration

ardumont added a comment to T3837: clean up swh-scheduler model db.

Triggered the migration (using a transaction as the dataset is not that huge) and all
went well.

Jan 12 2022, 12:13 PM · Scheduling utilities, Data Model, System administration

ardumont added a comment to T3837: clean up swh-scheduler model db.

So script is ok and landed.
I'll trigger it on staging first.

Jan 12 2022, 11:42 AM · Scheduling utilities, Data Model, System administration

ardumont added a revision to T3837: clean up swh-scheduler model db: D6921: sql: Clean up task/task_run data model.

Jan 12 2022, 11:04 AM · Scheduling utilities, Data Model, System administration

Jan 11 2022

ardumont changed the status of T3837: clean up swh-scheduler model db from Open to Work in Progress.

Jan 11 2022, 9:58 AM · Scheduling utilities, Data Model, System administration

Jan 10 2022

ardumont updated the task description for T3837: clean up swh-scheduler model db.

Jan 10 2022, 4:15 PM · Scheduling utilities, Data Model, System administration

ardumont added a comment to T3837: clean up swh-scheduler model db.

wip script [1]

Jan 10 2022, 3:53 PM · Scheduling utilities, Data Model, System administration

ardumont triaged T3837: clean up swh-scheduler model db as Normal priority.

Jan 10 2022, 10:48 AM · Scheduling utilities, Data Model, System administration

Dec 13 2021

olasd closed T3748: Date overflow error in scheduler journal client for some visit type as Resolved.

I believe this has now been deployed in staging and production by @vsellier (thanks!)

Dec 13 2021, 3:11 PM · Scheduling utilities

Dec 9 2021

ardumont closed T3782: Drop tasks archival service as Resolved.

Dec 9 2021, 11:22 AM · System administration, Scheduling utilities

ardumont moved T3782: Drop tasks archival service from in-progress to deployed/landed/monitoring on the System administration board.

Dec 9 2021, 11:22 AM · System administration, Scheduling utilities

ardumont changed the status of T3782: Drop tasks archival service from Open to Work in Progress.

Dec 9 2021, 11:22 AM · System administration, Scheduling utilities

ardumont added a project to T3782: Drop tasks archival service: System administration.

Dec 9 2021, 11:21 AM · System administration, Scheduling utilities

ardumont added a comment to T3782: Drop tasks archival service.

Cleaned up code pushed.
Puppet code clean up pushed and deployed.
This task can be closed.

Dec 9 2021, 11:21 AM · System administration, Scheduling utilities

ardumont added a revision to T3782: Drop tasks archival service: D6807: Drop no longer running scheduler_archive_tasks service.

Dec 9 2021, 11:01 AM · System administration, Scheduling utilities

ardumont renamed T3782: Drop tasks archival service from Drop task archival service to Drop tasks archival service.

Dec 9 2021, 10:53 AM · System administration, Scheduling utilities

ardumont renamed T3782: Drop tasks archival service from Drop task archival setup to Drop task archival service.

Dec 9 2021, 10:53 AM · System administration, Scheduling utilities

ardumont added a revision to T3782: Drop tasks archival service: D6772: Clean up old scheduler archival task related services.

Dec 9 2021, 10:52 AM · System administration, Scheduling utilities

ardumont triaged T3782: Drop tasks archival service as Normal priority.

Dec 9 2021, 10:52 AM · System administration, Scheduling utilities

Dec 7 2021

olasd added a revision to T3748: Date overflow error in scheduler journal client for some visit type: D6777: Make next_visit_queue_position an integer.

Dec 7 2021, 5:43 PM · Scheduling utilities

Nov 22 2021

olasd added a comment to T3748: Date overflow error in scheduler journal client for some visit type.

diff --git a/swh/scheduler/journal_client.py b/swh/scheduler/journal_client.py
index c001a48..e2a18ad 100644
--- a/swh/scheduler/journal_client.py
+++ b/swh/scheduler/journal_client.py
@@ -4,7 +4,7 @@
 # See top-level LICENSE file for more information

Nov 22 2021, 8:35 PM · Scheduling utilities

olasd added a comment to T3748: Date overflow error in scheduler journal client for some visit type.

I had hot-patched this to clamp dates to datetime.datetime.max, if an overflow was detected. Of course I never actually committed the change. D'oh.

Nov 22 2021, 7:26 PM · Scheduling utilities

olasd added a project to T3748: Date overflow error in scheduler journal client for some visit type: Scheduling utilities.

Nov 22 2021, 7:25 PM · Scheduling utilities

vlorentz closed T3727: Show the progress of vault tasks, a subtask of T3550: Compute and show ETA for vault tasks, as Resolved.

Nov 22 2021, 10:46 AM · Scheduling utilities, Web app, Vault

Nov 15 2021

vlorentz added a comment to T3550: Compute and show ETA for vault tasks.

(marking T2220 as dependency, because we need an up-to-date graph to show an ETA on objects loaded in the last ~year)

Nov 15 2021, 9:56 AM · Scheduling utilities, Web app, Vault

vlorentz added a subtask for T3550: Compute and show ETA for vault tasks: T2220: swh-graph in production.

Nov 15 2021, 9:55 AM · Scheduling utilities, Web app, Vault

vlorentz updated the task description for T3550: Compute and show ETA for vault tasks.

Nov 15 2021, 9:50 AM · Scheduling utilities, Web app, Vault

Oct 28 2021

ardumont closed T3667: Orchestrate origins scheduling according to scheduler metrics feedback, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, as Resolved.

Oct 28 2021, 5:13 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont closed T3667: Orchestrate origins scheduling according to scheduler metrics feedback as Resolved.

Oct 28 2021, 5:13 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont moved T3667: Orchestrate origins scheduling according to scheduler metrics feedback from in-progress to deployed/landed/monitoring on the System administration board.

Oct 28 2021, 4:43 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont moved T3667: Orchestrate origins scheduling according to scheduler metrics feedback from Backlog to in-progress on the System administration board.

Oct 28 2021, 4:43 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont added a project to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: System administration.

Oct 28 2021, 4:42 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont added a comment to T3667: Orchestrate origins scheduling according to scheduler metrics feedback.

Deployed in production.

Oct 28 2021, 4:42 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont changed the status of T3667: Orchestrate origins scheduling according to scheduler metrics feedback, a subtask of T2345: Improve handling of recurrent loading tasks in scheduler, from Open to Work in Progress.

Oct 28 2021, 4:34 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont changed the status of T3667: Orchestrate origins scheduling according to scheduler metrics feedback from Open to Work in Progress.

Deployed in staging.

Oct 28 2021, 4:34 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

olasd added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6583: scheduler: Add service to schedule recurrent visits.

Oct 28 2021, 4:16 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6574: scheduler: Add schedule recurrent tasks service.

Oct 28 2021, 11:35 AM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

Oct 26 2021

ardumont closed T3632: Investigate the ContentDisallowed exception as Invalid.

Sentry never reported any regression since it got marked as resolved.
So that issue was no longer relevant when i opened it, closing it as invalid.

Oct 26 2021, 12:37 PM · Scheduling utilities

Oct 25 2021

ardumont closed T3675: Investigate why cran origins are marked as non-visited in the scheduler metrics as Resolved.

Oct 25 2021, 1:11 PM · Scheduling utilities

ardumont added a comment to T3675: Investigate why cran origins are marked as non-visited in the scheduler metrics.

Remaining coverage fix is deployed.

Oct 25 2021, 1:10 PM · Scheduling utilities

Oct 22 2021

ardumont added a comment to T3675: Investigate why cran origins are marked as non-visited in the scheduler metrics.

Cran loader worked hard and fast, metrics got updated and now are consistent with reality [1]
So we can land D6544, deploy and be done with this issue.

Oct 22 2021, 5:32 PM · Scheduling utilities

ardumont added a revision to T3675: Investigate why cran origins are marked as non-visited in the scheduler metrics: D6544: coverage: Drop the cran origins computation workaround.

Oct 22 2021, 5:29 PM · Scheduling utilities

ardumont added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6543: docker: Install scheduler runner which deals with recurrent tasks.

Oct 22 2021, 5:19 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

anlambert added a comment to T3675: Investigate why cran origins are marked as non-visited in the scheduler metrics.

drop the computation done in the archive coverage part for those origins.

Oct 22 2021, 3:36 PM · Scheduling utilities

ardumont added a comment to T3675: Investigate why cran origins are marked as non-visited in the scheduler metrics.

Plan to fix this:

land diff D6539
package and deploy it (within swh.lister v2.2.0)
update the origins in the scheduler backend (tables listed_origins, scheduler metrics, origin_visit_stats) [1] [2]
ensure cran origins are scheduled (scheduler runner in place)
ensure the swh-worker@loader_cran service does its job
Schedule a recent listing after all fixes
D6544: drop the computation done in the archive coverage part for those origins.

Oct 22 2021, 2:56 PM · Scheduling utilities

ardumont updated subscribers of T3675: Investigate why cran origins are marked as non-visited in the scheduler metrics.

Ah, @anlambert found the issue, tar type instead of cran type. The result of the
following queries [1] should be the opposite... no tar origins, only cran ones.

Oct 22 2021, 2:51 PM · Scheduling utilities

ardumont changed the status of T3675: Investigate why cran origins are marked as non-visited in the scheduler metrics from Open to Work in Progress.

Oct 22 2021, 2:48 PM · Scheduling utilities

anlambert added a revision to T3675: Investigate why cran origins are marked as non-visited in the scheduler metrics: D6539: cran: Fix ListedOrigin visit type.

Oct 22 2021, 2:44 PM · Scheduling utilities

ardumont updated the task description for T3667: Orchestrate origins scheduling according to scheduler metrics feedback.

Oct 22 2021, 1:51 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont updated the task description for T3667: Orchestrate origins scheduling according to scheduler metrics feedback.

Oct 22 2021, 12:21 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

Oct 21 2021

ardumont updated the task description for T3684: swh-scheduler: Document the design of the scheduler in its documentation.

Oct 21 2021, 5:55 PM · Scheduling utilities

Oct 20 2021

ardumont added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6520: Add a new cli endpoint to schedule recurrent visits in Celery.

Oct 20 2021, 4:50 PM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont updated the task description for T3675: Investigate why cran origins are marked as non-visited in the scheduler metrics.

Oct 20 2021, 3:39 PM · Scheduling utilities

ardumont added a project to T3675: Investigate why cran origins are marked as non-visited in the scheduler metrics: Scheduling utilities.

Oct 20 2021, 3:38 PM · Scheduling utilities

ardumont added a subtask for T2345: Improve handling of recurrent loading tasks in scheduler: T3674: Determine how to reference manually listed/ingested origins in the scheduler metrics.

Oct 20 2021, 12:19 PM · Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont added a parent task for T3674: Determine how to reference manually listed/ingested origins in the scheduler metrics: T2345: Improve handling of recurrent loading tasks in scheduler.

Oct 20 2021, 12:19 PM · Scheduling utilities

ardumont renamed T3674: Determine how to reference manually listed/ingested origins in the scheduler metrics from Somehow reference old manually ingested origins in the scheduler metrics to Determine how to reference manually listed/ingested origins in the scheduler metrics.

Oct 20 2021, 12:19 PM · Scheduling utilities

ardumont updated the task description for T3674: Determine how to reference manually listed/ingested origins in the scheduler metrics.

Oct 20 2021, 12:17 PM · Scheduling utilities

ardumont triaged T3674: Determine how to reference manually listed/ingested origins in the scheduler metrics as Normal priority.

Oct 20 2021, 12:15 PM · Scheduling utilities

ardumont added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6515: Drop deprecated listener module.

Oct 20 2021, 10:45 AM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities

ardumont added a revision to T3667: Orchestrate origins scheduling according to scheduler metrics feedback: D6514: Add docstring to runner and listener modules.

Oct 20 2021, 10:44 AM · System administration, Sprint 2021 01, Archive coverage, Scheduling utilities