cannot process Mercurial data anymore.

well, not only mercurial, any other stuff that needs at least disk to work (loader svn comes to mind easily but not limited to that).

14M swh.loader.mercurial.10o7dn_1

Those folders are the expected ones.
Well, they should be cleaned up.

Those are hosting the repository uncompressed from the archive.

12G sqldict6a19bb
17G sqldict89fc16
786M sqldict8f09fa
634M sqldictc9c674

That must be the disk-based secondary storage cache used from within the mercurial loader @fiendish referred to.

This is quite large indeed!

I did not expect them at that location though.

So, now i've made the deposit/svn/mercurial loaders use private tmp mounting.
So next time this happens, we should only need restarting/stopping the service.
And this will clean up the mess left after this situation arose.

doc: https://www.freedesktop.org/software/systemd/man/systemd.exec.html#PrivateTmp=

Now to avoid the OOM to happen too late and kill other services as well, i thought of quota in the memory usage (systemd permits this).

The memory usage limitation won't be enough for the mercurial/svn loaders to kill other stuff though.

For example, the mercurial loader uses a secondary disk cache when the memory reached a certain point... So that could expand a lot...
Another example would be the svn loader on large repository which could expand a lot on disk as well...

I did not find a way to limit the disk usage (from systemd at least).

In the mean time, i'll reschedule filtering out the huge one (= archive size > 200Mib).
I'll keep a listing of those not rescheduled.

In the mean time, i'll reschedule filtering out the huge one (= archive size > 200Mib).

Well, first, i tried to reduce the disk usage (cf. commits just before that comment).

And second, there are other stuff to clean up anyway (T976).

If cache files are sticking around, then of course the code should make sure that they go away when done or aborted. But I think that a few G used during processing of extremely large repos should be acceptable. :/

If cache files are sticking around, then of course the code should make sure that they go away when done

It does...

or aborted.

...up to the point where they cannot (OOM killed). This is the current predicament.

But I think that a few G used during processing of extremely large repos should be acceptable. :/

It is.

It's just that we need to find a proper way to also clean up outside the loaders when that situation arose.

The thing is we are in a current position where our workers are doing multiple stuff