Changeset View
Changeset View
Standalone View
Standalone View
docs/compression.rst
Show First 20 Lines • Show All 125 Lines • ▼ Show 20 Lines | |||||
Compression steps | Compression steps | ||||
================= | ================= | ||||
The compression pipeline consists of the following steps: | The compression pipeline consists of the following steps: | ||||
.. figure:: images/compression_steps.png | .. figure:: images/compression_steps.png | ||||
:align: center | :align: center | ||||
:alt: Compression steps | :alt: Compression steps | ||||
:target: _images/compression_steps.png | :scale: 20% | ||||
Compression steps | Compression steps | ||||
Each of these steps is briefly described below. For more details see the | Each of these steps is briefly described below. For more details see the | ||||
original Software Heritage graph compression paper [SWHGraphCompression2020]_, | original Software Heritage graph compression paper [SWHGraphCompression2020]_, | ||||
as well as chapters 9 and 10 of Antoine Pietri's PhD thesis | as well as chapters 9 and 10 of Antoine Pietri's PhD thesis | ||||
[PietriThesis2021]_. | [PietriThesis2021]_. | ||||
▲ Show 20 Lines • Show All 439 Lines • ▼ Show 20 Lines | |||||
- ``graph-labelled.labeloffsets``: the offsets used to access the labels in | - ``graph-labelled.labeloffsets``: the offsets used to access the labels in | ||||
random order. | random order. | ||||
It then does the same with backward edge batches to get the transposed | It then does the same with backward edge batches to get the transposed | ||||
equivalent of these files: | equivalent of these files: | ||||
``graph-transposed-labelled.{properties,labels,labeloffsets}``. | ``graph-transposed-labelled.{properties,labels,labeloffsets}``. | ||||
23. EDGE_LABELS_OBL | |||||
------------------- | |||||
Cache the label offsets of the forward labelled graph to make loading faster. | |||||
The resulting label offset big list is stored in the | |||||
``graph-labelled.labelobl`` file. | |||||
23. EDGE_LABELS_TRANSPOSE_OBL | |||||
----------------------------- | |||||
Same as EDGE_LABELS_OBL, but for the transposed labelled graph. | |||||
The resulting label offset big list is stored in the | |||||
``graph-transposed-labelled.labelobl`` file. | |||||
22. CLEAN_TMP | 24. CLEAN_TMP | ||||
------------- | ------------- | ||||
This step reclaims space by deleting the temporary directory, as well as all | This step reclaims space by deleting the temporary directory, as well as all | ||||
the intermediate outputs that are no longer necessary now that the final graph | the intermediate outputs that are no longer necessary now that the final graph | ||||
has been compressed (shown in gray in the step diagram). | has been compressed (shown in gray in the step diagram). |