Change Details

Recent runs actually showed that the loader git: - [x] T3625: uses too much memory - [ ] T3635: does too much work (regarding forks, it does not deduplicate computations for in the end noops) - [ ] T3640: is too slow to finish which renders maintenance a pain - [ ] T3652: Does not properly handle some origins due to a packfile too large So in given too much concurrency, it could actually be not that stable. This task is actually a parent task to track the work done to actually improve those points, make the loader git: - uses less memory - do less unneeded work - stops fast when required to - actually able to ingest "larger" origins The perimeter of this task is frozen to this 4 points (it's not a neverending task ;)

Recent runs actually showed that the loader git: - [x] T3625: uses too much memory - [ ] T3635: does too much work (regarding forks, it does not deduplicate computations for in the end noops) - [ ] T3640: is too slow to finish which renders maintenance a pain - [ ] T3652: Does not properly handle some origins due to a packfile too large So for example, given too much concurrency, the first point actually make it unstable. Another example, too many forks (at the same time or not) actually make the loader git overall slow. This task is actually a parent task to track the work done to actually fix those points, make the loader git: - uses less memory - do less unneeded work - stops fast when required to - actually able to ingest "larger" origins The perimeter of this task is frozen to this 4 points (it's not a neverending task ;)

Recent runs actually showed that the loader git: - [x] T3625: uses too much memory - [ ] T3635: does too much work (regarding forks, it does not deduplicate computations for in the end noops) - [ ] T3640: is too slow to finish which renders maintenance a pain - [ ] T3652: Does not properly handle some origins due to a packfile too large So inSo for example, given too much concurrencyency, the first point actually make it unstable. Another example, it couldtoo many forks (at the same time or not) actually be not that stablemake the loader git overall slow. This task is actually a parent task to track the work done to actually improvefix those points, make the loader git: - uses less memory - do less unneeded work - stops fast when required to - actually able to ingest "larger" origins The perimeter of this task is frozen to this 4 points (it's not a neverending task ;)