Page MenuHomeSoftware Heritage
Paste P992

(An Untitled Masterwork)
ActivePublic
Actions

Authored by anlambert on Mar 26 2021, 12:42 PM.
swh/graph/tests/test_cli.py::test_pipeline
------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------
DEBUG swh.core.config:config.py:88 Loading config file /tmp/tmp2xijfg30.swh-graph-test/config.yml
INFO root:config.py:33 using swh-graph JAR: /home/anlambert/swh/swh-environment/swh-graph/java/target/swh-graph-0.3.0.jar
INFO root:webgraph.py:205 starting compression at 2021-03-26 12:41:38.621097
INFO root:webgraph.py:214 starting compression step MPH (1/11) at 2021-03-26 12:41:38.621203
INFO root:webgraph.py:146 running: java it.unimi.dsi.sux4j.mph.GOVMinimalPerfectHashFunction --temp-dir /tmp/tmp2xijfg30.swh-graph-test/tmp /tmp/tmp2xijfg30.swh-graph-test/example.mph <( zstdcat /home/anlambert/swh/swh-environment/swh-graph/swh/graph/tests/dataset/example.nodes.csv.zst )
INFO root:webgraph.py:156 Picked up JAVA_TOOL_OPTIONS: -Xmx33479344128 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp2xijfg30.swh-graph-test/tmp/logback.xml -Djava.io.tmpdir=/tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:156 2021-03-26 12:41:39,005 306 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Adding elements...
INFO root:webgraph.py:156 2021-03-26 12:41:39,013 314 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:39,016 317 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Elapsed: 5ms [21 keys, 5,250.00 keys/s, 190.48 µs/key]; used/avail/free/total/max mem: 17.99M/33.47G/514.68M/532.68M/33.48G
INFO root:webgraph.py:156 2021-03-26 12:41:39,022 323 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Analysing buckets...
INFO root:webgraph.py:156 2021-03-26 12:41:39,123 424 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Unsolvable systems: 1/2 (50.00%)
INFO root:webgraph.py:156 2021-03-26 12:41:39,124 425 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Unorientable systems: 3/5 (60.00%)
INFO root:webgraph.py:156 2021-03-26 12:41:39,124 425 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:39,124 425 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Elapsed: 103ms [1 buckets, 9.80 buckets/s, 102.00 ms/bucket]; used/avail/free/total/max mem: 228.21M/33.25G/304.47M/532.68M/33.48G
INFO root:webgraph.py:156 2021-03-26 12:41:39,124 425 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:39,127 428 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Actual bit cost per key: 8.380952380952381
INFO root:webgraph.py:156 2021-03-26 12:41:39,142 443 INFO [main] i.u.d.s.m.GOVMinimalPerfectHashFunction - Saved.
INFO root:webgraph.py:221 completed compression step MPH (1/11) at 2021-03-26 12:41:39.158932 in 0:00:00.537729
INFO root:webgraph.py:214 starting compression step BV (2/11) at 2021-03-26 12:41:39.159124
INFO root:webgraph.py:146 running: zstdcat /home/anlambert/swh/swh-environment/swh-graph/swh/graph/tests/dataset/example.edges.csv.zst | cut -d' ' -f1,2 | java it.unimi.dsi.big.webgraph.ScatteredArcsASCIIGraph --temp-dir /tmp/tmp2xijfg30.swh-graph-test/tmp --function /tmp/tmp2xijfg30.swh-graph-test/example.mph /tmp/tmp2xijfg30.swh-graph-test/example-bv
INFO root:webgraph.py:156 Picked up JAVA_TOOL_OPTIONS: -Xmx33479344128 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp2xijfg30.swh-graph-test/tmp/logback.xml -Djava.io.tmpdir=/tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:156 2021-03-26 12:41:39,423 188 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Creating sorted batches...
INFO root:webgraph.py:156 2021-03-26 12:41:39,459 224 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:39,463 228 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Elapsed: 36ms [23 arcs, 657.14 arcs/s, 1.52 ms/arc]
INFO root:webgraph.py:156 2021-03-26 12:41:39,465 230 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Created 1 batches using 9.39 bits/arc.
INFO root:webgraph.py:156 2021-03-26 12:41:39,470 235 INFO [main] i.u.d.b.w.BVGraph - Compressing using 1 threads
INFO root:webgraph.py:156 2021-03-26 12:41:39,477 242 INFO [ProcessingThread-0] i.u.d.b.w.ScatteredArcsASCIIGraph - Storing...
INFO root:webgraph.py:156 2021-03-26 12:41:39,479 244 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:39,479 244 INFO [main] i.u.d.b.w.ScatteredArcsASCIIGraph - Elapsed: 3ms [21 nodes, 10,500.00 nodes/s, 95.24 µs/node]
INFO root:webgraph.py:221 completed compression step BV (2/11) at 2021-03-26 12:41:39.502593 in 0:00:00.343469
INFO root:webgraph.py:214 starting compression step BV_OBL (3/11) at 2021-03-26 12:41:39.502866
INFO root:webgraph.py:146 running: java it.unimi.dsi.big.webgraph.BVGraph --list /tmp/tmp2xijfg30.swh-graph-test/example-bv
INFO root:webgraph.py:156 Picked up JAVA_TOOL_OPTIONS: -Xmx33479344128 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp2xijfg30.swh-graph-test/tmp/logback.xml -Djava.io.tmpdir=/tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:221 completed compression step BV_OBL (3/11) at 2021-03-26 12:41:39.787006 in 0:00:00.284140
INFO root:webgraph.py:214 starting compression step BFS (4/11) at 2021-03-26 12:41:39.787190
INFO root:webgraph.py:146 running: java it.unimi.dsi.law.big.graph.BFS /tmp/tmp2xijfg30.swh-graph-test/example-bv /tmp/tmp2xijfg30.swh-graph-test/example.order
INFO root:webgraph.py:156 Picked up JAVA_TOOL_OPTIONS: -Xmx33479344128 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp2xijfg30.swh-graph-test/tmp/logback.xml -Djava.io.tmpdir=/tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:156 2021-03-26 12:41:40,039 188 INFO [main] i.u.d.l.b.g.BFS - Starting breadth-first visit...
INFO root:webgraph.py:156 2021-03-26 12:41:40,043 192 INFO [main] i.u.d.l.b.g.BFS - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:40,046 195 INFO [main] i.u.d.l.b.g.BFS - Elapsed: 4ms [21 nodes, 7,000.00 nodes/s, 142.86 µs/node]
INFO root:webgraph.py:221 completed compression step BFS (4/11) at 2021-03-26 12:41:40.051610 in 0:00:00.264420
INFO root:webgraph.py:214 starting compression step PERMUTE (5/11) at 2021-03-26 12:41:40.051790
INFO root:webgraph.py:146 running: java it.unimi.dsi.big.webgraph.Transform mapOffline /tmp/tmp2xijfg30.swh-graph-test/example-bv /tmp/tmp2xijfg30.swh-graph-test/example /tmp/tmp2xijfg30.swh-graph-test/example.order 1000 /tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:156 Picked up JAVA_TOOL_OPTIONS: -Xmx33479344128 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp2xijfg30.swh-graph-test/tmp/logback.xml -Djava.io.tmpdir=/tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:156 2021-03-26 12:41:40,282 160 INFO [main] i.u.d.b.w.Transform - Loading graph...
INFO root:webgraph.py:156 2021-03-26 12:41:40,284 162 INFO [main] i.u.d.b.w.Transform - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:40,289 167 INFO [main] i.u.d.b.w.Transform - Elapsed: 2ms [27 bytes, 27,000.00 bytes/s, 37.04 µs/byte]
INFO root:webgraph.py:156 2021-03-26 12:41:40,290 168 INFO [main] i.u.d.b.w.Transform - Loading offsets...
INFO root:webgraph.py:156 2021-03-26 12:41:40,308 186 INFO [main] i.u.d.b.w.Transform - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:40,309 187 INFO [main] i.u.d.b.w.Transform - Elapsed: 19ms [22 deltas, 1,222.22 deltas/s, 818.18 µs/delta]
INFO root:webgraph.py:156 2021-03-26 12:41:40,309 187 INFO [main] i.u.d.b.w.Transform - Pointer bits per node: 16.86
INFO root:webgraph.py:156 2021-03-26 12:41:40,309 187 INFO [main] i.u.d.b.w.Transform - Reading map...
INFO root:webgraph.py:156 2021-03-26 12:41:40,312 190 INFO [main] i.u.d.b.w.Transform - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:40,312 190 INFO [main] i.u.d.b.w.Transform - Elapsed: 4ms [21 deltas, 7,000.00 deltas/s, 142.86 µs/delta]
INFO root:webgraph.py:156 2021-03-26 12:41:40,317 195 INFO [main] i.u.d.b.w.Transform - Creating sorted batches...
INFO root:webgraph.py:156 2021-03-26 12:41:40,328 206 INFO [main] i.u.d.b.w.Transform - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:40,329 207 INFO [main] i.u.d.b.w.Transform - Elapsed: 12ms [21 nodes, 1,909.09 nodes/s, 523.81 µs/node]
INFO root:webgraph.py:156 2021-03-26 12:41:40,330 208 INFO [main] i.u.d.b.w.Transform - Created 1 batches using 8.35 bits/arc.
INFO root:webgraph.py:156 2021-03-26 12:41:40,331 209 INFO [main] i.u.d.b.w.Transform - Transform computation completed.
INFO root:webgraph.py:156 2021-03-26 12:41:40,331 209 INFO [main] i.u.d.b.w.BVGraph - Compressing using 1 threads
INFO root:webgraph.py:156 2021-03-26 12:41:40,342 220 INFO [ProcessingThread-0] i.u.d.b.w.Transform - Storing...
INFO root:webgraph.py:156 2021-03-26 12:41:40,344 222 INFO [main] i.u.d.b.w.Transform - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:40,344 222 INFO [main] i.u.d.b.w.Transform - Elapsed: 2ms [21 nodes, 21,000.00 nodes/s, 47.62 µs/node]
INFO root:webgraph.py:221 completed compression step PERMUTE (5/11) at 2021-03-26 12:41:40.370822 in 0:00:00.319032
INFO root:webgraph.py:214 starting compression step PERMUTE_OBL (6/11) at 2021-03-26 12:41:40.371057
INFO root:webgraph.py:146 running: java it.unimi.dsi.big.webgraph.BVGraph --list /tmp/tmp2xijfg30.swh-graph-test/example
INFO root:webgraph.py:156 Picked up JAVA_TOOL_OPTIONS: -Xmx33479344128 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp2xijfg30.swh-graph-test/tmp/logback.xml -Djava.io.tmpdir=/tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:221 completed compression step PERMUTE_OBL (6/11) at 2021-03-26 12:41:40.635126 in 0:00:00.264069
INFO root:webgraph.py:214 starting compression step STATS (7/11) at 2021-03-26 12:41:40.635296
INFO root:webgraph.py:146 running: java it.unimi.dsi.big.webgraph.Stats /tmp/tmp2xijfg30.swh-graph-test/example
INFO root:webgraph.py:156 Picked up JAVA_TOOL_OPTIONS: -Xmx33479344128 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp2xijfg30.swh-graph-test/tmp/logback.xml -Djava.io.tmpdir=/tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:156 2021-03-26 12:41:40,874 157 INFO [main] ROOT - Scanning...
INFO root:webgraph.py:156 2021-03-26 12:41:40,877 160 INFO [main] ROOT - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:40,880 163 INFO [main] ROOT - Elapsed: 3ms [21 nodes, 10,500.00 nodes/s, 95.24 µs/node]
INFO root:webgraph.py:221 completed compression step STATS (7/11) at 2021-03-26 12:41:40.888375 in 0:00:00.253079
INFO root:webgraph.py:214 starting compression step TRANSPOSE (8/11) at 2021-03-26 12:41:40.888573
INFO root:webgraph.py:146 running: java it.unimi.dsi.big.webgraph.Transform transposeOffline /tmp/tmp2xijfg30.swh-graph-test/example /tmp/tmp2xijfg30.swh-graph-test/example-transposed 1000 /tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:156 Picked up JAVA_TOOL_OPTIONS: -Xmx33479344128 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp2xijfg30.swh-graph-test/tmp/logback.xml -Djava.io.tmpdir=/tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:156 2021-03-26 12:41:41,128 169 INFO [main] i.u.d.b.w.Transform - Loading graph...
INFO root:webgraph.py:156 2021-03-26 12:41:41,129 170 INFO [main] i.u.d.b.w.Transform - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:41,132 173 INFO [main] i.u.d.b.w.Transform - Elapsed: 1ms [24 bytes, ∞ bytes/s, 0.00 ns/byte]
INFO root:webgraph.py:156 2021-03-26 12:41:41,132 173 INFO [main] i.u.d.b.w.Transform - Loading offsets...
INFO root:webgraph.py:156 2021-03-26 12:41:41,150 191 INFO [main] i.u.d.b.w.Transform - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:41,151 192 INFO [main] i.u.d.b.w.Transform - Elapsed: 19ms [22 deltas, 1,222.22 deltas/s, 818.18 µs/delta]
INFO root:webgraph.py:156 2021-03-26 12:41:41,151 192 INFO [main] i.u.d.b.w.Transform - Pointer bits per node: 16.73
INFO root:webgraph.py:156 2021-03-26 12:41:41,155 196 INFO [main] i.u.d.b.w.Transform - Creating sorted batches...
INFO root:webgraph.py:156 2021-03-26 12:41:41,168 209 INFO [main] i.u.d.b.w.Transform - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:41,168 209 INFO [main] i.u.d.b.w.Transform - Elapsed: 14ms [21 nodes, 1,615.38 nodes/s, 619.05 µs/node]
INFO root:webgraph.py:156 2021-03-26 12:41:41,169 210 INFO [main] i.u.d.b.w.Transform - Created 1 batches using 10.09 bits/arc.
INFO root:webgraph.py:156 2021-03-26 12:41:41,170 211 INFO [main] i.u.d.b.w.BVGraph - Compressing using 1 threads
INFO root:webgraph.py:156 2021-03-26 12:41:41,182 223 INFO [ProcessingThread-0] i.u.d.b.w.Transform - Storing...
INFO root:webgraph.py:156 2021-03-26 12:41:41,187 228 INFO [main] i.u.d.b.w.Transform - Completed.
INFO root:webgraph.py:156 2021-03-26 12:41:41,187 228 INFO [main] i.u.d.b.w.Transform - Elapsed: 6ms [21 nodes, 4,200.00 nodes/s, 238.10 µs/node]
INFO root:webgraph.py:221 completed compression step TRANSPOSE (8/11) at 2021-03-26 12:41:41.212442 in 0:00:00.323869
INFO root:webgraph.py:214 starting compression step TRANSPOSE_OBL (9/11) at 2021-03-26 12:41:41.212635
INFO root:webgraph.py:146 running: java it.unimi.dsi.big.webgraph.BVGraph --list /tmp/tmp2xijfg30.swh-graph-test/example-transposed
INFO root:webgraph.py:156 Picked up JAVA_TOOL_OPTIONS: -Xmx33479344128 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp2xijfg30.swh-graph-test/tmp/logback.xml -Djava.io.tmpdir=/tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:221 completed compression step TRANSPOSE_OBL (9/11) at 2021-03-26 12:41:41.464296 in 0:00:00.251661
INFO root:webgraph.py:214 starting compression step MAPS (10/11) at 2021-03-26 12:41:41.464466
INFO root:webgraph.py:146 running: zstdcat /home/anlambert/swh/swh-environment/swh-graph/swh/graph/tests/dataset/example.nodes.csv.zst | java org.softwareheritage.graph.maps.NodeMapBuilder /tmp/tmp2xijfg30.swh-graph-test/example /tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:156 Picked up JAVA_TOOL_OPTIONS: -Xmx33479344128 -XX:PretenureSizeThreshold=512M -XX:MaxNewSize=4G -XX:+UseLargePages -XX:+UseTransparentHugePages -XX:+UseNUMA -XX:+UseTLAB -XX:+ResizeTLAB -Dlogback.configurationFile=/tmp/tmp2xijfg30.swh-graph-test/tmp/logback.xml -Djava.io.tmpdir=/tmp/tmp2xijfg30.swh-graph-test/tmp
INFO root:webgraph.py:156 2021-03-26 12:41:41,646 121 INFO [main] o.s.g.m.NodeMapBuilder - starting maps generation...
INFO root:webgraph.py:156 2021-03-26 12:41:41,650 125 INFO [main] o.s.g.m.NodeMapBuilder - loading MPH function...
INFO root:webgraph.py:156 2021-03-26 12:41:41,694 169 INFO [main] o.s.g.m.NodeMapBuilder - MPH function loaded
INFO root:webgraph.py:156 2021-03-26 12:41:41,698 173 INFO [main] o.s.g.m.NodeMapBuilder - loading BFS order file...
INFO root:webgraph.py:156 2021-03-26 12:41:41,698 173 INFO [main] o.s.g.m.NodeMapBuilder - BFS order file loaded
INFO root:webgraph.py:156 2021-03-26 12:41:41,715 190 INFO [Thread-0] o.s.g.m.NodeMapBuilder - node2swhid: waiting for sort output...
INFO root:webgraph.py:156 2021-03-26 12:41:41,716 191 INFO [main] o.s.g.m.NodeMapBuilder - filling swhid2node map
INFO root:webgraph.py:156 Exception in thread "main" java.lang.ClassCastException: class [B cannot be cast to class java.lang.CharSequence ([B and java.lang.CharSequence are in module java.base of loader 'bootstrap')
INFO root:webgraph.py:156 at it.unimi.dsi.bits.TransformationStrategies$RawUtf16TransformationStrategy.toBitVector(TransformationStrategies.java:245)
INFO root:webgraph.py:156 at it.unimi.dsi.sux4j.mph.GOVMinimalPerfectHashFunction.getLong(GOVMinimalPerfectHashFunction.java:528)
INFO root:webgraph.py:156 at org.softwareheritage.graph.maps.NodeMapBuilder.precomputeNodeIdMap(NodeMapBuilder.java:150)
INFO root:webgraph.py:156 at org.softwareheritage.graph.maps.NodeMapBuilder.main(NodeMapBuilder.java:55)
About · Developer information · Legal