Differential D1802
[WIP] server: setup: use RandomAccessFile instead of temporary array Authored by haltode on Aug 2 2019, 1:47 PM. Tags None Subscribers None
Details The temporary array to re-order elements and output them sequentially costs too *THIS IS WIP* I still need to measure *how much* slower this method is. On RIOC previous
Diff Detail
Event TimelineComment Actions Some updates on the running experiments, I tried with both monster and our new azure VM. On monster: Pre-computing node id maps... 12:14:31.113 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 0 seconds. 14:58:13.781 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 9822 seconds. 17:55:13.544 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 10619 seconds. 20:55:53.102 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 10839 seconds. 23:45:19.549 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 10166 seconds. 01:51:13.195 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 7553 seconds. 03:22:06.684 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 5453 seconds. 03:55:25.232 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1998 seconds. 05:09:10.894 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 4425 seconds. 05:30:44.284 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1293 seconds. 06:04:57.636 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 2053 seconds. 06:30:48.789 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1551 seconds. 07:01:09.959 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 1821 seconds. 08:23:28.140 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 4938 seconds. 08:48:31.343 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 1503 seconds. 09:24:08.794 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 2137 seconds. 11:30:55.555 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 7606 seconds. 16:06:02.588 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 16507 seconds. 20:46:59.794 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 16857 seconds. 01:27:30.944 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 16831 seconds. 06:11:20.119 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 17029 seconds. 10:17:32.034 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 14771 seconds. 15:05:28.386 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 17276 seconds. 20:03:42.294 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 17893 seconds. 01:09:21.419 [main] DEBUG org.softwareheritage.graph.backend.Setup - 2% done in 18339 seconds. 06:13:10.182 [main] DEBUG org.softwareheritage.graph.backend.Setup - 2% done in 18228 seconds. On azure VM: 10:51:56.631 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 0 seconds. 13:11:05.805 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 8349 seconds. 18:10:44.062 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 17978 seconds. 23:58:59.122 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 20895 seconds. 08:37:11.809 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 31092 seconds. 19:06:19.708 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 37747 seconds. 06:15:01.895 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 40122 seconds. These timings are really not doable to dump the full mappings. Right now using RIOC with the old method seems our best bet. The old method consists of storing PIDs in an array to then write sequentially into the file. This was way faster (the whole mapping took ~20h) but used 2TB of RAM. Comment Actions Last solution I tried: using a custom mmap output file instead of the temporary array. On monster, results seem better! 12:40:30.665 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 0 seconds. 19:36:06.751 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 24936 seconds. 23:11:16.928 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 12910 seconds. 02:11:33.575 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 10816 seconds. 04:03:06.678 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 6693 seconds. 04:13:26.771 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 620 seconds. 04:19:20.912 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 354 seconds. 04:31:58.387 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 757 seconds. 04:52:21.467 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1223 seconds. 05:10:04.905 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1063 seconds. 05:32:26.112 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1341 seconds. 05:49:46.124 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 1040 seconds. 06:05:37.965 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 951 seconds. 06:20:34.442 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 896 seconds. 06:33:50.246 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 795 seconds. 06:46:48.352 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 778 seconds. 09:10:15.571 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 8607 seconds. 09:43:14.907 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 1979 seconds. 09:57:26.257 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 851 seconds. 10:08:30.305 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 664 seconds. 10:18:36.791 [main] DEBUG org.softwareheritage.graph.backend.Setup - 1% done in 606 seconds. The VM is under some constant load from other users but the general timings are much quicker. This can be the solution to the RAM problem, I will try to start it on the azure VM to have more stable timings. Comment Actions Results from Azure with random write using custom mmap: 14:52:41.849 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 0 seconds. 17:11:15.300 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 8313 seconds. 22:02:24.055 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 17468 seconds. 03:00:15.982 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 17871 seconds. 08:09:29.299 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 18553 seconds. 13:36:09.910 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 19600 seconds. 20:46:00.483 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 25790 seconds. 05:57:29.513 [main] DEBUG org.softwareheritage.graph.backend.Setup - 0% done in 33089 seconds. So yeah, let's focus on T1944 first and things will improve on this task. |