HomeSoftware Heritage

CLI: add one-stop shop compression "swh graph compress ...."

Description

CLI: add one-stop shop compression "swh graph compress ...."

Implement a pure Python driver for the entire compression toolchain, equivalent
to the former docker script compress_graph.sh, but much more flexible.

In particular it:

  • allows to selectively execute specific steps/ranges of the compression pipeline
  • requires much less configuration, with a lot of sane defaults (e.g., physical RAM, classpaths, etc.)
  • uses the swh-graph*.jar for all java library needs
  • supports proper Python logging and timing via the logging module
  • cleans up BV graph and tmpdir as last step, freeing up space

Details

Provenance
zackAuthored on Nov 2 2019, 11:09 PM
zackPushed on Nov 2 2019, 11:12 PM
Parents
rDGRPH0ef506e4d1b3: java deps: bump fastutil dep to 8.3.0
Branches
Unknown
Tags
Unknown
Build Status
Buildable 8757
Build 12766: test-and-buildJenkins