They are more tuned toward running automatically, as they call each
other as needed, and can be imported by workflows defined in other
modules (eg. the future swh.graph.luigi module).
This massively re-uses the CLI, so most of the code is:
- telling Luigi how to deduplicate + when/how to reuse output of tasks that already ran
- adding stamp files to avoid accidentally using a partially written export (because it was interrupted midway)
- the meta.json file, which acts as a final stamp and provides information about the dataset export itself (required for T2579)
Depends on D8828