Fix broken ci build.
This now runs the tests with the packaged tests and not from the local workspace.
Related D2082#48581
Diff Detail
Diff Detail
- Repository
- rDGRPH Compressed graph representation
- Branch
- master
- Lint
No Linters Available - Unit
No Unit Test Coverage - Build Status
Buildable 8239 Build 11894: tox-on-jenkins Jenkins Build 11893: arc lint + arc unit
Event Timeline
Comment Actions
You don't need to make diffs for all these changes
Right, i did not so far :)
I will for loader-mercurial (failing), swh-graph (hanging) and swh-indexer (failing). Those are the ones failing/hanging locally for some unclear reasons.
For that one, i'm not feeling confident enough to determine if that hangs for the right reason or not.
So i'm counting on the ci to reassure me a bit.
Comment Actions
So i'm counting on the ci to reassure me a bit.
Well, the reassuring part is that the tests hang the same as the local one...
Comment Actions
Build was aborted
Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tox/69/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tox/69/console
Comment Actions
Build was aborted
Link to build: https://jenkins.softwareheritage.org/job/DGRPH/job/tox/70/
See console output for more information: https://jenkins.softwareheritage.org/job/DGRPH/job/tox/70/console
Comment Actions
ok, so triggering tox with more verbosity helped a bit:
.tox/py3/lib/python3.7/site-packages/swh/graph/tests/test_api_client.py::test_stats Loading graph /home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/tests/dataset/output/example ...
Process GraphServerProcess-1:
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/conftest.py", line 23, in run
with backend:
File "/home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/swh/graph/backend.py", line 66, in __enter__
self.entry.load_graph(self.graph_path)
File "/home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/py4j/java_gateway.py", line 1286, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/py4j/protocol.py", line 328, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o0.load_graph.
: java.io.FileNotFoundException: /home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/tests/dataset/output/example.properties (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
at it.unimi.dsi.big.webgraph.BVGraph.loadInternal(BVGraph.java:1335)
at it.unimi.dsi.big.webgraph.BVGraph.load(BVGraph.java:1208)
at it.unimi.dsi.big.webgraph.BVGraph.load(BVGraph.java:1222)
at it.unimi.dsi.big.webgraph.BVGraph.load(BVGraph.java:1233)
at org.softwareheritage.graph.Graph.<init>(Graph.java:57)
at org.softwareheritage.graph.Entry.load_graph(Entry.java:26)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:834)so apparently my manifest.in modification is not enough...
which i don't get:
$ workon swh $ cd swh-graph $ python setup.py sdist running sdist running egg_info writing swh.graph.egg-info/PKG-INFO writing dependency_links to swh.graph.egg-info/dependency_links.txt writing entry points to swh.graph.egg-info/entry_points.txt writing requirements to swh.graph.egg-info/requires.txt writing top-level names to swh.graph.egg-info/top_level.txt reading manifest file 'swh.graph.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'swh.graph.egg-info/SOURCES.txt' running check creating swh.graph-0.1.0.post1 creating swh.graph-0.1.0.post1/java creating swh.graph-0.1.0.post1/java/server creating swh.graph-0.1.0.post1/java/server/target creating swh.graph-0.1.0.post1/swh creating swh.graph-0.1.0.post1/swh.graph.egg-info creating swh.graph-0.1.0.post1/swh/graph creating swh.graph-0.1.0.post1/swh/graph/server creating swh.graph-0.1.0.post1/swh/graph/tests creating swh.graph-0.1.0.post1/tests creating swh.graph-0.1.0.post1/tests/dataset creating swh.graph-0.1.0.post1/tests/dataset/img creating swh.graph-0.1.0.post1/tests/dataset/output copying files to swh.graph-0.1.0.post1... copying MANIFEST.in -> swh.graph-0.1.0.post1 copying Makefile -> swh.graph-0.1.0.post1 copying README.md -> swh.graph-0.1.0.post1 copying requirements-swh.txt -> swh.graph-0.1.0.post1 copying requirements.txt -> swh.graph-0.1.0.post1 copying setup.py -> swh.graph-0.1.0.post1 copying version.txt -> swh.graph-0.1.0.post1 copying java/server/target/swh-graph-0.0.2-jar-with-dependencies.jar -> swh.graph-0.1.0.post1/java/server/target copying java/server/target/swh-graph-0.1.0.jar -> swh.graph-0.1.0.post1/java/server/target copying swh/__init__.py -> swh.graph-0.1.0.post1/swh copying swh.graph.egg-info/PKG-INFO -> swh.graph-0.1.0.post1/swh.graph.egg-info copying swh.graph.egg-info/SOURCES.txt -> swh.graph-0.1.0.post1/swh.graph.egg-info copying swh.graph.egg-info/dependency_links.txt -> swh.graph-0.1.0.post1/swh.graph.egg-info copying swh.graph.egg-info/entry_points.txt -> swh.graph-0.1.0.post1/swh.graph.egg-info copying swh.graph.egg-info/requires.txt -> swh.graph-0.1.0.post1/swh.graph.egg-info copying swh.graph.egg-info/top_level.txt -> swh.graph-0.1.0.post1/swh.graph.egg-info copying swh/graph/__init__.py -> swh.graph-0.1.0.post1/swh/graph copying swh/graph/backend.py -> swh.graph-0.1.0.post1/swh/graph copying swh/graph/cli.py -> swh.graph-0.1.0.post1/swh/graph copying swh/graph/client.py -> swh.graph-0.1.0.post1/swh/graph copying swh/graph/dot.py -> swh.graph-0.1.0.post1/swh/graph copying swh/graph/graph.py -> swh.graph-0.1.0.post1/swh/graph copying swh/graph/pid.py -> swh.graph-0.1.0.post1/swh/graph copying swh/graph/py.typed -> swh.graph-0.1.0.post1/swh/graph copying swh/graph/server/__init__.py -> swh.graph-0.1.0.post1/swh/graph/server copying swh/graph/server/app.py -> swh.graph-0.1.0.post1/swh/graph/server copying swh/graph/tests/__init__.py -> swh.graph-0.1.0.post1/swh/graph/tests copying swh/graph/tests/conftest.py -> swh.graph-0.1.0.post1/swh/graph/tests copying swh/graph/tests/test_api_client.py -> swh.graph-0.1.0.post1/swh/graph/tests copying swh/graph/tests/test_graph.py -> swh.graph-0.1.0.post1/swh/graph/tests copying swh/graph/tests/test_pid.py -> swh.graph-0.1.0.post1/swh/graph/tests copying tests/dataset/.gitignore -> swh.graph-0.1.0.post1/tests/dataset copying tests/dataset/example.edges.csv -> swh.graph-0.1.0.post1/tests/dataset copying tests/dataset/example.edges.csv.gz -> swh.graph-0.1.0.post1/tests/dataset copying tests/dataset/example.nodes.csv -> swh.graph-0.1.0.post1/tests/dataset copying tests/dataset/example.nodes.csv.gz -> swh.graph-0.1.0.post1/tests/dataset copying tests/dataset/generate_graph.sh -> swh.graph-0.1.0.post1/tests/dataset copying tests/dataset/img/.gitignore -> swh.graph-0.1.0.post1/tests/dataset/img copying tests/dataset/img/Makefile -> swh.graph-0.1.0.post1/tests/dataset/img copying tests/dataset/img/example.dot -> swh.graph-0.1.0.post1/tests/dataset/img copying tests/dataset/output/example-transposed.graph -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example-transposed.obl -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example-transposed.offsets -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example-transposed.properties -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.graph -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.indegree -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.mph -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.node2pid.bin -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.node2pid.csv -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.node2type.map -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.obl -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.offsets -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.order -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.outdegree -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.pid2node.bin -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.pid2node.csv -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.properties -> swh.graph-0.1.0.post1/tests/dataset/output copying tests/dataset/output/example.stats -> swh.graph-0.1.0.post1/tests/dataset/output Writing swh.graph-0.1.0.post1/setup.cfg creating dist Creating tar archive removing 'swh.graph-0.1.0.post1' (and everything under it)
...
Comment Actions
Oh, i think i see.
The tests folder is not included within the swh arborescence (because possibly java convention) and that must clash a bit with python test convention...
It seems that code must be changed to fix this [1]
I don't see anything wrong in the setup.py [2] in regards to tests sample inclusion though.
The folder tests/dataset is included in the sdist but it's not installed in the tox virtualenv. Again possibly, because it's not under the swh tree but i don't see reference to this being forbidden so ¯\_(ツ)_/¯.
Also moving that folder with swh tree seems complicated as it's expected to be there for the java part as well (see src/java/test/GraphTest.java or some such).
[1] https://forge.softwareheritage.org/source/swh-graph/browse/master/swh/graph/tests/conftest.py$12-13
[2] https://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files
I'll continue investigation later.
Comment Actions
I'll continue investigation later.
Discussing with @douardda, the rationale is that we need to either 1. improve the dataset packaging to include it within the python code or 2. completely move it out.
I'm for 1. but i need to discuss with @seirl first to clarify what's what ;)
Note: i was not completely out of my depth in D2100#48856, so \m/.