Write related ORC files in the same directory using the same UUID
5a8a8a7847f6
Actions

Description

Write related ORC files in the same directory using the same UUID

related ORC files being ORC files involved in the serialization of a given
object type, namely:

snapshot and snapshot_branches,
revision and revision_history,
directory and directory_entry.

Also include the object_type in the generated file name (in place of the
static 'graph').

So the result will typically be like:

output/orc/shapshot/
  snaphot-18a575cb-3a92-4753-9267-e3475fa30857.orc
  snaphot_branch-18a575cb-3a92-4753-9267-e3475fa30857.orc
  snapshot-1f41d206-994a-49bb-917f-e096e40c2856.orc
  snapshot_branch-1f41d206-994a-49bb-917f-e096e40c2856.orc

Details

Provenance

douardda	Authored on Mar 18 2022, 12:09 PM
douardda	Pushed on Mar 29 2022, 5:39 PM

Differential Revision

Parents

rDDATASET729ae64f36cd: Add some user metadata in generated ORC files

Branches

Unknown

Tags

Unknown

Build Status

Buildable 27957
Build 43779: test-and-build	Jenkins console · Jenkins

Event Timeline

douardda committed rDDATASET5a8a8a7847f6: Write related ORC files in the same directory using the same UUID (authored by douardda).Mar 29 2022, 5:35 PM

douardda added an edge: D7385: Write related ORC files in the same directory using the same UUID.Mar 29 2022, 5:39 PM

Harbormaster completed building B27957: rDDATASET5a8a8a7847f6: Write related ORC files in the same directory using the same UUID.Mar 29 2022, 5:41 PM

swh-public-ci mentioned this in D7387: Add the type fields for revision and origin_visit_status ORC table.Mar 29 2022, 5:42 PM

swh-public-ci mentioned this in D7388: Export revision extra headers in a dedicated ORC file.

swh-public-ci mentioned this in D7389: Add the raw_manifest column for revision, release and directory ORC files.

swh-public-ci mentioned this in D7461: Add support for limited row numbers in ORC files.Mar 29 2022, 5:46 PM

swh-public-ci mentioned this in D7462: Move exporter config entries in dedicated sections.

swh-public-ci mentioned this in D7465: Add support for blob in content export.

swh-public-ci mentioned this in D7463: Delay the unsubscribe to the end of handle_messages.

swh-public-ci mentioned this in D7464: Improve the progress reporting look and feel.

douardda mentioned this in D7476: Keep each ORC table in a dedicated directory.Mar 31 2022, 11:57 AM

douardda mentioned this in rDDATASET76eba6595bac: Keep each ORC table in a dedicated directory.Mar 31 2022, 2:45 PM

Changes (2)

Path

Size

swh/

dataset/

exporters/

orc.py

test/

test_orc.py

rDDATASET5a8a8a7847f6

View Options

swh/dataset/exporters/orc.py