Page MenuHomeSoftware Heritage

[WIP] server: setup: use RandomAccessFile instead of temporary array
AbandonedPublic

Authored by haltode on Fri, Aug 2, 1:47 PM.

Details

Reviewers
zack
seirl
Summary

The temporary array to re-order elements and output them sequentially costs too
much memory on the entire graph. Using RandomAccessFile is slower but enables
Setup to run on any VM.


*THIS IS WIP*

I still need to measure *how much* slower this method is. On RIOC previous
version with the temporary array used 2TB of RAM (!) and around 20h to compute.
Right now tests are running on monster (but IO is really not great...), because
all others VM are already in use.

Diff Detail

Repository
rDGRPH Graph service
Branch
less-mem-mapping
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 7117
Build 10035: arc lint + arc unit

Event Timeline

haltode created this revision.Fri, Aug 2, 1:47 PM

Some updates on the running experiments, I tried with both monster and our new azure VM.

On monster:

Pre-computing node id maps...
12:14:31.113 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 0 seconds.
14:58:13.781 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 9822 seconds.
17:55:13.544 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 10619 seconds.
20:55:53.102 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 10839 seconds.
23:45:19.549 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 10166 seconds.
01:51:13.195 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 7553 seconds.
03:22:06.684 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 5453 seconds.
03:55:25.232 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1998 seconds.
05:09:10.894 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 4425 seconds.
05:30:44.284 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1293 seconds.
06:04:57.636 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 2053 seconds.
06:30:48.789 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1551 seconds.
07:01:09.959 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 1821 seconds.
08:23:28.140 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 4938 seconds.
08:48:31.343 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 1503 seconds.
09:24:08.794 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 2137 seconds.
11:30:55.555 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 7606 seconds.
16:06:02.588 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 16507 seconds.
20:46:59.794 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 16857 seconds.
01:27:30.944 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 16831 seconds.
06:11:20.119 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 17029 seconds.
10:17:32.034 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 14771 seconds.
15:05:28.386 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 17276 seconds.
20:03:42.294 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 17893 seconds.
01:09:21.419 [main] DEBUG org.softwareheritage.graph.backend.Setup - 2% done in 18339 seconds.
06:13:10.182 [main] DEBUG org.softwareheritage.graph.backend.Setup - 2% done in 18228 seconds.

On azure VM:

10:51:56.631 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 0 seconds.
13:11:05.805 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 8349 seconds.
18:10:44.062 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 17978 seconds.
23:58:59.122 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 20895 seconds.
08:37:11.809 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 31092 seconds.
19:06:19.708 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 37747 seconds.
06:15:01.895 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 40122 seconds.

These timings are really not doable to dump the full mappings. Right now using RIOC with the old method seems our best bet. The old method consists of storing PIDs in an array to then write sequentially into the file. This was way faster (the whole mapping took ~20h) but used 2TB of RAM.

Last solution I tried: using a custom mmap output file instead of the temporary array. On monster, results seem better!

12:40:30.665 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 0 seconds.
19:36:06.751 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 24936 seconds.
23:11:16.928 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 12910 seconds.
02:11:33.575 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 10816 seconds.
04:03:06.678 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 6693 seconds.
04:13:26.771 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 620 seconds.
04:19:20.912 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 354 seconds.
04:31:58.387 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 757 seconds.
04:52:21.467 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1223 seconds.
05:10:04.905 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1063 seconds.
05:32:26.112 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1341 seconds.
05:49:46.124 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1040 seconds.
06:05:37.965 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 951 seconds.
06:20:34.442 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 896 seconds.
06:33:50.246 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 795 seconds.
06:46:48.352 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 778 seconds.
09:10:15.571 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 8607 seconds.
09:43:14.907 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 1979 seconds.
09:57:26.257 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 851 seconds.
10:08:30.305 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 664 seconds.
10:18:36.791 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 606 seconds.

The VM is under some constant load from other users but the general timings are much quicker. This can be the solution to the RAM problem, I will try to start it on the azure VM to have more stable timings.

haltode planned changes to this revision.Thu, Aug 8, 4:30 PM
haltode abandoned this revision.Sat, Aug 10, 9:20 AM

Results from Azure with random write using custom mmap:

14:52:41.849 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 0 seconds.
17:11:15.300 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 8313 seconds.
22:02:24.055 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 17468 seconds.
03:00:15.982 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 17871 seconds.
08:09:29.299 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 18553 seconds.
13:36:09.910 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 19600 seconds.
20:46:00.483 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 25790 seconds.
05:57:29.513 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 33089 seconds.

So yeah, let's focus on T1944 first and things will improve on this task.