Page MenuHomeSoftware Heritage

Make long running task stop fast when warm shutdown is triggered
Closed, MigratedEdits Locked

Description

This would currently help to stop/restart fast swh-worker services (as currently deployed in our stack).
That would also help with elastic workers to downscale fast (if need be) or to time frame some loaders. [1]

As it is not currently the case, a stop or restart is as fast as the current
ingestion ongoing will take. For some hg or git repositories, that can take time (hours, well
there is now a timeout but still).

This could be implemented for example with a signal handler to react to the warm shutdown
signal. For loaders, the signal handler would do a partial snapshot with what it had ingested so far.
For listers, it can flush its visited state for the ongoing forge (on incremental).

[1] Related to T3592 (but as described not only that)