Page MenuHomeSoftware Heritage

Reschedule pack file too big failing loading task to dedicated queue consumed by large enough workers
Closed, MigratedEdits Locked

Description

Current git loading tasks can fail to be ingested with a [1] 'pack file too big' message.
They are then disabled by the scheduler after a number of tryouts (3).
Those could be redirected to another queue [2] whose consumption can happen only with large enough workers [3].
(with proper setup like pack file limit set to 32g or something).

[1]

raise IOError(
    f"Pack file too big for repository {origin_url}, "
    f"limit is {size_limit} bytes, current size is {cur_size}, "
    f"would write {would_write}"
)

[3] worker17-18 currently

[2] oneshot:swh.loader.git.tasks.UpdateGitRepository

Event Timeline

ardumont renamed this task from Reschedule pack file too big loading to a dedicated queue consumed by large enough workers to Reschedule pack file too big failing loading task to dedicated queue consumed by large enough workers.Jul 12 2022, 11:56 AM
ardumont triaged this task as Normal priority.
ardumont created this task.
ardumont updated the task description. (Show Details)