Page MenuHomeSoftware Heritage
Paste P556

swh/graph/tests/test_cli.py::TestCompress::test_pipeline
ActivePublic
Actions

Authored by anlambert on Nov 4 2019, 12:01 PM.
swh/graph/tests/test_cli.py::TestCompress::test_pipeline
-------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------
webgraph.py 233 INFO starting compression
webgraph.py 242 INFO starting compression step MPH (1/11)
webgraph.py 153 INFO running: java it.unimi.dsi.sux4j.mph.GOVMinimalPerfectHashFunction --zipped /tmp/tmp465gdf9e.swh-graph-test/example.mph --temp-dir /tmp/tmp465gdf9e.swh-graph-test/tmp /home/antoine/swh/swh-environment/swh-graph/swh/graph/tests/dataset/example.nodes.csv.gz
webgraph.py 161 INFO Picked up JAVA_TOOL_OPTIONS: -Xmx16764243968 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp465gdf9e.swh-graph-test/tmp/logback.xml
webgraph.py 161 INFO 2019-11-04 11:58:42,900 360 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Adding elements...
webgraph.py 161 INFO 2019-11-04 11:58:42,908 368 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Completed.
webgraph.py 161 INFO 2019-11-04 11:58:42,912 372 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Elapsed: 6ms [21 keys, 4,200.00 keys/s, 238.10 µs/key]; used/avail/free/total/max mem: 21.27M/16.74G/242.97M/264.24M/16.76G
webgraph.py 161 INFO 2019-11-04 11:58:42,912 372 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Analysing buckets...
webgraph.py 161 INFO 2019-11-04 11:58:43,095 555 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Unsolvable systems: 1/2 (50.00%)
webgraph.py 161 INFO 2019-11-04 11:58:43,095 555 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Unorientable systems: 2/4 (50.00%)
webgraph.py 161 INFO 2019-11-04 11:58:43,095 555 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Completed.
webgraph.py 161 INFO 2019-11-04 11:58:43,096 556 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Elapsed: 184ms [1 buckets, 5.46 buckets/s, 183.00 ms/bucket]; used/avail/free/total/max mem: 61.11M/16.70G/203.13M/264.24M/16.76G
webgraph.py 161 INFO 2019-11-04 11:58:43,096 556 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Completed.
webgraph.py 161 INFO 2019-11-04 11:58:43,096 556 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Actual bit cost per key: 8.380952380952381
webgraph.py 161 INFO 2019-11-04 11:58:43,117 577 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Saved.
webgraph.py 247 INFO completed compression step MPH (1/11) in 0:00:00.716118
webgraph.py 242 INFO starting compression step BV (2/11)
webgraph.py 153 INFO running: java it.unimi.dsi.big.webgraph.ScatteredArcsASCIIGraph --function /tmp/tmp465gdf9e.swh-graph-test/example.mph --temp-dir /tmp/tmp465gdf9e.swh-graph-test/tmp --zipped /tmp/tmp465gdf9e.swh-graph-test/example-bv
webgraph.py 161 INFO Picked up JAVA_TOOL_OPTIONS: -Xmx16764243968 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp465gdf9e.swh-graph-test/tmp/logback.xml
webgraph.py 161 INFO 2019-11-04 11:58:43,499 259 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Creating sorted batches...
webgraph.py 161 INFO 2019-11-04 11:58:43,551 311 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Completed.
webgraph.py 161 INFO 2019-11-04 11:58:43,555 315 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Elapsed: 52ms [23 arcs, 450.98 arcs/s, 2.22 ms/arc]
webgraph.py 161 INFO 2019-11-04 11:58:43,556 316 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Created 1 batches using 9.39 bits/arc.
webgraph.py 161 INFO 2019-11-04 11:58:43,570 330 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Storing...
webgraph.py 161 INFO 2019-11-04 11:58:43,574 334 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Completed.
webgraph.py 161 INFO 2019-11-04 11:58:43,575 335 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Elapsed: 6ms [21 nodes, 4,200.00 nodes/s, 238.10 µs/node]
webgraph.py 247 INFO completed compression step BV (2/11) in 0:00:00.479274
webgraph.py 242 INFO starting compression step BV_OBL (3/11)
webgraph.py 153 INFO running: java it.unimi.dsi.big.webgraph.BVGraph --list /tmp/tmp465gdf9e.swh-graph-test/example-bv
webgraph.py 161 INFO Picked up JAVA_TOOL_OPTIONS: -Xmx16764243968 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp465gdf9e.swh-graph-test/tmp/logback.xml
webgraph.py 247 INFO completed compression step BV_OBL (3/11) in 0:00:00.470132
webgraph.py 242 INFO starting compression step BFS (4/11)
webgraph.py 153 INFO running: java it.unimi.dsi.law.big.graph.BFS /tmp/tmp465gdf9e.swh-graph-test/example-bv /tmp/tmp465gdf9e.swh-graph-test/example.order
webgraph.py 161 INFO Picked up JAVA_TOOL_OPTIONS: -Xmx16764243968 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp465gdf9e.swh-graph-test/tmp/logback.xml
webgraph.py 161 INFO 2019-11-04 11:58:44,445 256 INFO [main] i.u.d.l.b.g.BFS - Starting breadth-first visit...
webgraph.py 161 INFO 2019-11-04 11:58:44,450 261 INFO [main] i.u.d.l.b.g.BFS - Completed.
webgraph.py 161 INFO 2019-11-04 11:58:44,454 265 INFO [main] i.u.d.l.b.g.BFS - Elapsed: 5ms [21 nodes, 5,250.00 nodes/s, 190.48 µs/node]
webgraph.py 247 INFO completed compression step BFS (4/11) in 0:00:00.380812
webgraph.py 242 INFO starting compression step PERMUTE (5/11)
webgraph.py 153 INFO running: java it.unimi.dsi.big.webgraph.Transform mapOffline /tmp/tmp465gdf9e.swh-graph-test/example-bv /tmp/tmp465gdf9e.swh-graph-test/example /tmp/tmp465gdf9e.swh-graph-test/example.order 1000000000 /tmp/tmp465gdf9e.swh-graph-test/tmp
webgraph.py 161 INFO Picked up JAVA_TOOL_OPTIONS: -Xmx16764243968 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp465gdf9e.swh-graph-test/tmp/logback.xml
webgraph.py 161 INFO 2019-11-04 11:58:44,767 192 INFO [main] i.u.d.b.w.Transform - Reading map...
webgraph.py 161 INFO 2019-11-04 11:58:44,777 202 INFO [main] i.u.d.b.w.Transform - Completed.
webgraph.py 161 INFO 2019-11-04 11:58:44,782 207 INFO [main] i.u.d.b.w.Transform - Elapsed: 10ms [21 items, 2,333.33 items/s, 428.57 µs/item]
webgraph.py 161 INFO Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
webgraph.py 161 INFO at it.unimi.dsi.big.webgraph.Transform.mapOffline(Transform.java:978)
webgraph.py 161 INFO at it.unimi.dsi.big.webgraph.Transform.main(Transform.java:1939)
About · Developer information · Legal