Change Details

I've tried to load the [[ https://github.com/torvalds/linux | linux kernel repo ]] in a locally running swh stack under the [[ https://docs.softwareheritage.org/devel/getting-started.html#getting-started | docker development environment ]]. That used to work just fine and I've used it in the past to have a non-entirely trivial archive locally available for experimentation (this time it is in the context of T2600). It no longer works for me, because the git loader got OOM killed after reaching ~30 GB of VIRT mem and ~20 GB of RES mem. It happened twice (2nd time when the scheduler retried the same job), so I don't think it's a temporary glitch. I'm filing this partly because it will be nice to allow people to load a "serious repo" locally in their test. But also because if it takes (at least) that much RAM to load the linux kernel repo, it might be the case that there are other non-artificial repos out there that make our workers explode. (Unless this is specific to some settings of the docker dev repo, in which case it would be nice to tune them.)

I've tried to load the [[ https://github.com/torvalds/linux | linux kernel repo ]] in a locally running swh stack under the [[ https://docs.softwareheritage.org/devel/getting-started.html#getting-started | docker development environment ]]. That used to work just fine and I've used it in the past to have a non-entirely trivial archive locally available for experimentation (this time it is in the context of T2600). It no longer works for me, because the git loader got OOM killed after reaching ~30 GB of VIRT mem and ~20 GB of RES mem. It happened twice (2nd time when the scheduler retried the same job), so I don't think it's a temporary glitch. I'm filing this partly because it would be nice to allow people to load a "serious repo" locally in their test, even if that is not a goal per se. But also because if it takes (at least) that much RAM to load the linux kernel repo, it might be the case that there are other non-artificial repos out there that make our workers explode. (Unless this is specific to some settings of the docker dev repo, in which case it would still be nice to tune them to avoid OOM killing.)

I've tried to load the [[ https://github.com/torvalds/linux | linux kernel repo ]] in a locally running swh stack under the [[ https://docs.softwareheritage.org/devel/getting-started.html#getting-started | docker development environment ]]. That used to work just fine and I've used it in the past to have a non-entirely trivial archive locally available for experimentation (this time it is in the context of T2600). It no longer works for me, because the git loader got OOM killed after reaching ~30 GB of VIRT mem and ~20 GB of RES mem. It happened twice (2nd time when the scheduler retried the same job), so I don't think it's a temporary glitch. I'm filing this partly because it willwould be nice to allow people to load a "serious repo" locally in their test, even if that is not a goal per se. But also because if it takes (at least) that much RAM to load the linux kernel repo, it might be the case that there are other non-artificial repos out there that make our workers explode. (Unless this is specific to some settings of the docker dev repo, in which case it would still be nice to tune them to avoid OOM killing.)