Page MenuHomeSoftware Heritage

docker setup: simplify and uniform to SWH path conventions
ClosedPublic

Authored by zack on Jul 24 2019, 6:49 PM.

Details

Reviewers
seirl
haltode
Summary

Major simplification pass on the docker documentation, Dockerfile, and scripts.

Main aim was to simplify the steps needed (as documented in docker.rst) and
reduce the amount of redundant information that need to be provided by the
user.

As this required a major overhaul, this revamp also includes:

  • porting path conventions to SWH standards (e.g., everything is now under /srv/softwareheritage/graph, even in the container)
  • sanitizing the main shell script

Diff Detail

Repository
rDGRPH Compressed graph representation
Branch
feature/docker-simplification
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 7029
Build 9897: arc lint + arc unit

Event Timeline

  • docker doc: further shorten CLI examples using relative paths
  • docker doc: drop the list of files generated by Setup class
  • docker doc: add --publish to the run invocation

Awesome, thanks a lot for this diff!

Note that you also need to update the java/server/src/test/dataset/generate_graph.sh script since the Docker environment has changed.

zack planned changes to this revision.Jul 29 2019, 4:58 PM

Tnx, will do.

(Don't hesitate to explicitly request changes in cases like this in the future :))

  • tests: update generate_graph.sh to match new docker layout

Note that you also need to update the java/server/src/test/dataset/generate_graph.sh script since the Docker environment has changed.

This is now done.

Note that this diff still makes the test fail after re-running generate_graph.sh, because apparently the mappings are no longer generated. (Although it doesn't seem compress_graph.sh has ever generated them, so I'm not sure it's related to this diff [?]).
Anyway, if that's OK with you, please accept this diff as is (unless other issues are affected by it), and fix generate_graph.sh later on in a subsequent commit, after it lands.

In D1768#41626, @zack wrote:

Note that this diff still makes the test fail after re-running generate_graph.sh, because apparently the mappings are no longer generated. (Although it doesn't seem compress_graph.sh has ever generated them, so I'm not sure it's related to this diff [?]).

The mapping has always been a separate command but we can indeed add it to the generate_graph.sh script.

This revision is now accepted and ready to land.Jul 30 2019, 4:12 PM

landed in a043b0ee04aae50f8c26f6a06aac1e6c9247340a

The mapping has always been a separate command but we can indeed add it to the generate_graph.sh script.

Oh, ok, that explains it.

That means that the README on how to recreate the testdata isn't correct at present (or at least it's incomplete).

Integrate this into generate_graph.sh is a good idea. When it's done, we can also further simplify docs/docker.rst.

In D1768#41639, @zack wrote:

The mapping has always been a separate command but we can indeed add it to the generate_graph.sh script.

Oh, ok, that explains it.

That means that the README on how to recreate the testdata isn't correct at present (or at least it's incomplete).

Integrate this into generate_graph.sh is a good idea. When it's done, we can also further simplify docs/docker.rst.

Ok now I remember why this was separated, I was waiting on shipping the swh-graph.jar so I could add the step inside the compress_graph. I am tracking this issue in T1941.