Page MenuHomeSoftware Heritage

swh-graph: fixing ci: Fix tox.ini's py3 environment to use packaged tests
Needs ReviewPublic

Authored by ardumont on Wed, Oct 9, 3:46 PM.

Details

Reviewers
seirl
zack
vlorentz
Group Reviewers
Reviewers
Summary

Fix broken ci build.

This now runs the tests with the packaged tests and not from the local workspace.

Related D2082#48581

Test Plan

tox

Diff Detail

Repository
rDGRPH Graph service
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 8239
Build 11894: tox-on-jenkinsJenkins
Build 11893: arc lint + arc unit

Event Timeline

ardumont created this revision.Wed, Oct 9, 3:46 PM
ardumont edited the summary of this revision. (Show Details)Wed, Oct 9, 3:47 PM
vlorentz accepted this revision.Wed, Oct 9, 3:48 PM
vlorentz added a subscriber: vlorentz.

You don't need to make diffs for all these changes

This revision is now accepted and ready to land.Wed, Oct 9, 3:48 PM
ardumont added a comment.EditedWed, Oct 9, 3:54 PM

You don't need to make diffs for all these changes

Right, i did not so far :)

I will for loader-mercurial (failing), swh-graph (hanging) and swh-indexer (failing). Those are the ones failing/hanging locally for some unclear reasons.

For that one, i'm not feeling confident enough to determine if that hangs for the right reason or not.
So i'm counting on the ci to reassure me a bit.

So i'm counting on the ci to reassure me a bit.

Well, the reassuring part is that the tests hang the same as the local one...

tests data are not shipped so that might explain a bit...

Build was aborted

I stopped the build.

ardumont updated this revision to Diff 7047.Wed, Oct 9, 4:12 PM

Ship tests dataset

ok, so triggering tox with more verbosity helped a bit:

.tox/py3/lib/python3.7/site-packages/swh/graph/tests/test_api_client.py::test_stats Loading graph /home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/tests/dataset/output/example ...
Process GraphServerProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/swh/graph/tests/conftest.py", line 23, in run
    with backend:
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/swh/graph/backend.py", line 66, in __enter__
    self.entry.load_graph(self.graph_path)
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/py4j/java_gateway.py", line 1286, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o0.load_graph.
: java.io.FileNotFoundException: /home/tony/work/inria/repo/swh/swh-environment/swh-graph/.tox/py3/lib/python3.7/site-packages/tests/dataset/output/example.properties (No such file or directory)
        at java.base/java.io.FileInputStream.open0(Native Method)
        at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
        at it.unimi.dsi.big.webgraph.BVGraph.loadInternal(BVGraph.java:1335)
        at it.unimi.dsi.big.webgraph.BVGraph.load(BVGraph.java:1208)
        at it.unimi.dsi.big.webgraph.BVGraph.load(BVGraph.java:1222)
        at it.unimi.dsi.big.webgraph.BVGraph.load(BVGraph.java:1233)
        at org.softwareheritage.graph.Graph.<init>(Graph.java:57)
        at org.softwareheritage.graph.Entry.load_graph(Entry.java:26)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.base/java.lang.Thread.run(Thread.java:834)

so apparently my manifest.in modification is not enough...

which i don't get:

$ workon swh
$ cd swh-graph
$ python setup.py sdist
running sdist
running egg_info
writing swh.graph.egg-info/PKG-INFO
writing dependency_links to swh.graph.egg-info/dependency_links.txt
writing entry points to swh.graph.egg-info/entry_points.txt
writing requirements to swh.graph.egg-info/requires.txt
writing top-level names to swh.graph.egg-info/top_level.txt
reading manifest file 'swh.graph.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'swh.graph.egg-info/SOURCES.txt'
running check
creating swh.graph-0.1.0.post1
creating swh.graph-0.1.0.post1/java
creating swh.graph-0.1.0.post1/java/server
creating swh.graph-0.1.0.post1/java/server/target
creating swh.graph-0.1.0.post1/swh
creating swh.graph-0.1.0.post1/swh.graph.egg-info
creating swh.graph-0.1.0.post1/swh/graph
creating swh.graph-0.1.0.post1/swh/graph/server
creating swh.graph-0.1.0.post1/swh/graph/tests
creating swh.graph-0.1.0.post1/tests
creating swh.graph-0.1.0.post1/tests/dataset
creating swh.graph-0.1.0.post1/tests/dataset/img
creating swh.graph-0.1.0.post1/tests/dataset/output
copying files to swh.graph-0.1.0.post1...
copying MANIFEST.in -> swh.graph-0.1.0.post1
copying Makefile -> swh.graph-0.1.0.post1
copying README.md -> swh.graph-0.1.0.post1
copying requirements-swh.txt -> swh.graph-0.1.0.post1
copying requirements.txt -> swh.graph-0.1.0.post1
copying setup.py -> swh.graph-0.1.0.post1
copying version.txt -> swh.graph-0.1.0.post1
copying java/server/target/swh-graph-0.0.2-jar-with-dependencies.jar -> swh.graph-0.1.0.post1/java/server/target
copying java/server/target/swh-graph-0.1.0.jar -> swh.graph-0.1.0.post1/java/server/target
copying swh/__init__.py -> swh.graph-0.1.0.post1/swh
copying swh.graph.egg-info/PKG-INFO -> swh.graph-0.1.0.post1/swh.graph.egg-info
copying swh.graph.egg-info/SOURCES.txt -> swh.graph-0.1.0.post1/swh.graph.egg-info
copying swh.graph.egg-info/dependency_links.txt -> swh.graph-0.1.0.post1/swh.graph.egg-info
copying swh.graph.egg-info/entry_points.txt -> swh.graph-0.1.0.post1/swh.graph.egg-info
copying swh.graph.egg-info/requires.txt -> swh.graph-0.1.0.post1/swh.graph.egg-info
copying swh.graph.egg-info/top_level.txt -> swh.graph-0.1.0.post1/swh.graph.egg-info
copying swh/graph/__init__.py -> swh.graph-0.1.0.post1/swh/graph
copying swh/graph/backend.py -> swh.graph-0.1.0.post1/swh/graph
copying swh/graph/cli.py -> swh.graph-0.1.0.post1/swh/graph
copying swh/graph/client.py -> swh.graph-0.1.0.post1/swh/graph
copying swh/graph/dot.py -> swh.graph-0.1.0.post1/swh/graph
copying swh/graph/graph.py -> swh.graph-0.1.0.post1/swh/graph
copying swh/graph/pid.py -> swh.graph-0.1.0.post1/swh/graph
copying swh/graph/py.typed -> swh.graph-0.1.0.post1/swh/graph
copying swh/graph/server/__init__.py -> swh.graph-0.1.0.post1/swh/graph/server
copying swh/graph/server/app.py -> swh.graph-0.1.0.post1/swh/graph/server
copying swh/graph/tests/__init__.py -> swh.graph-0.1.0.post1/swh/graph/tests
copying swh/graph/tests/conftest.py -> swh.graph-0.1.0.post1/swh/graph/tests
copying swh/graph/tests/test_api_client.py -> swh.graph-0.1.0.post1/swh/graph/tests
copying swh/graph/tests/test_graph.py -> swh.graph-0.1.0.post1/swh/graph/tests
copying swh/graph/tests/test_pid.py -> swh.graph-0.1.0.post1/swh/graph/tests
copying tests/dataset/.gitignore -> swh.graph-0.1.0.post1/tests/dataset
copying tests/dataset/example.edges.csv -> swh.graph-0.1.0.post1/tests/dataset
copying tests/dataset/example.edges.csv.gz -> swh.graph-0.1.0.post1/tests/dataset
copying tests/dataset/example.nodes.csv -> swh.graph-0.1.0.post1/tests/dataset
copying tests/dataset/example.nodes.csv.gz -> swh.graph-0.1.0.post1/tests/dataset
copying tests/dataset/generate_graph.sh -> swh.graph-0.1.0.post1/tests/dataset
copying tests/dataset/img/.gitignore -> swh.graph-0.1.0.post1/tests/dataset/img
copying tests/dataset/img/Makefile -> swh.graph-0.1.0.post1/tests/dataset/img
copying tests/dataset/img/example.dot -> swh.graph-0.1.0.post1/tests/dataset/img
copying tests/dataset/output/example-transposed.graph -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example-transposed.obl -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example-transposed.offsets -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example-transposed.properties -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.graph -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.indegree -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.mph -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.node2pid.bin -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.node2pid.csv -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.node2type.map -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.obl -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.offsets -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.order -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.outdegree -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.pid2node.bin -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.pid2node.csv -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.properties -> swh.graph-0.1.0.post1/tests/dataset/output
copying tests/dataset/output/example.stats -> swh.graph-0.1.0.post1/tests/dataset/output
Writing swh.graph-0.1.0.post1/setup.cfg
creating dist
Creating tar archive
removing 'swh.graph-0.1.0.post1' (and everything under it)

...

ardumont added a comment.EditedWed, Oct 9, 11:38 PM

Oh, i think i see.
The tests folder is not included within the swh arborescence (because possibly java convention) and that must clash a bit with python test convention...

It seems that code must be changed to fix this [1]

I don't see anything wrong in the setup.py [2] in regards to tests sample inclusion though.

The folder tests/dataset is included in the sdist but it's not installed in the tox virtualenv. Again possibly, because it's not under the swh tree but i don't see reference to this being forbidden so ¯\_(ツ)_/¯.

Also moving that folder with swh tree seems complicated as it's expected to be there for the java part as well (see src/java/test/GraphTest.java or some such).

[1] https://forge.softwareheritage.org/source/swh-graph/browse/master/swh/graph/tests/conftest.py$12-13

[2] https://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files

I'll continue investigation later.

ardumont retitled this revision from tox.ini: Fix py3 environment to use packaged tests to swh-graph: fixing ci: Fix tox.ini's py3 environment to use packaged tests.Thu, Oct 10, 11:29 AM
ardumont added a subscriber: douardda.

I'll continue investigation later.

Discussing with @douardda, the rationale is that we need to either 1. improve the dataset packaging to include it within the python code or 2. completely move it out.
I'm for 1. but i need to discuss with @seirl first to clarify what's what ;)

Note: i was not completely out of my depth in D2100#48856, so \m/.

ardumont added 1 blocking reviewer(s): seirl.Thu, Oct 10, 11:37 AM
This revision now requires review to proceed.Thu, Oct 10, 11:37 AM