Page MenuHomeSoftware Heritage
Differential D7385

Write related ORC files in the same directory using the same UUID
ClosedPublic
Actions

Authored by douardda on Mar 18 2022, 2:12 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDDATASET5a8a8a7847f6: Write related ORC files in the same directory using the same UUID
Summary

related ORC files being ORC files involved in the serialization of a given
object type, namely:

  • snapshot and snapshot_branches,
  • revision and revision_history,
  • directory and directory_entry.

Also include the object_type in the generated file name (in place of the
static 'graph').

So the result will typically be like:

output/orc/shapshot/
  snaphot-18a575cb-3a92-4753-9267-e3475fa30857.orc
  snaphot_branch-18a575cb-3a92-4753-9267-e3475fa30857.orc
  snapshot-1f41d206-994a-49bb-917f-e096e40c2856.orc
  snapshot_branch-1f41d206-994a-49bb-917f-e096e40c2856.orc

Depends on D7384.

Diff Detail

Event Timeline

douardda created this revision.Mar 18 2022, 2:12 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMar 18 2022, 2:12 PM
douardda added a child revision: D7386: Use the same 'id' column name everywhere in ORC files.Mar 18 2022, 2:12 PM
swh-public-ci added a comment.Mar 18 2022, 2:14 PM

Build is green

Patch application report for D7385 (id=26687)

Could not rebase; Attempt merge onto 68f9bd2028...

Updating 68f9bd2..4c90956
Fast-forward
 mypy.ini                        |  6 +++
 requirements-swh.txt            |  6 +--
 requirements.txt                |  3 +-
 swh/dataset/cli.py              | 10 ++++-
 swh/dataset/exporters/orc.py    | 75 +++++++++++++++++++++++-------------
 swh/dataset/journalprocessor.py | 32 ++++++++++------
 swh/dataset/relational.py       | 15 +++++---
 swh/dataset/test/test_orc.py    | 84 ++++++++++++++++++++++-------------------
 8 files changed, 143 insertions(+), 88 deletions(-)
Changes applied before test
commit 4c90956495f50c7fafc2d123378e5fe82d37b65b
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:09:57 2022 +0100

    Write related ORC files in the same directory using the same UUID
    
    related ORC files being ORC files involved in the serialization of a given
    object type, namely:
    
    - snapshot and snapshot_branches,
    - revision and revision_history,
    - directory and directory_entry.
    
    Also include the object_type in the generated file name (in place of the
    static 'graph').
    
    So the result will typically be like:
    
      output/orc/shapshot/
        snaphot-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snaphot_branch-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snapshot-1f41d206-994a-49bb-917f-e096e40c2856.orc
        snapshot_branch-1f41d206-994a-49bb-917f-e096e40c2856.orc

commit 66db13eab592c399350c129bcc03c71ace7196a7
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:00:41 2022 +0100

    Add some user metadata in generated ORC files
    
    add:
    - object type
    - uuid
    - version of swh.model used at file generation time,
    - version of swh.dataset used at file generation time.

commit a05df6abe5cad5cf6b3e207fa9b317ebc0a1c32f
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 16 10:46:37 2022 +0100

    Implement test_orc exporter as a simple function instead of a fixture
    
    and split it in 2 parts (needed for changes to come).

commit 986885abd6d3ab95df76f02746c016e00a53ba2b
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:50:15 2021 +0100

    Make the kafka group_id prefix configurable in the config file
    
    rather than hardcoding it to 'swh-dataset-export-', use the 'group_id'
    value from the 'journal' section of the config file as prefix, if given
    9otherwise default to the former value).
    
    This is needed because current auth policy of swh kafka cluster only allow
    group_id to start with the actual login for authenticated connection.
    So we need to be able to specify this group_id prefix.

commit 048f273838cdba3c8dfbebfb9768fc675f205d74
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:41:28 2021 +0100

    Use a named logger for journalprocessor.py
    
    and add a few more debug logging statements.

commit 8cae6adb5c63af22bc798ebe7072a33978f3037e
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:44:49 2021 +0100

    Update JournalClientOffsetRanges for swh.journal 0.9
    
    deserialize_message() now takes an optional 'object_type' argument.

commit 8c2b5e951c1a1195c9ec3e700cb9da60711a96ab
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 11:46:37 2022 +0100

    Encode TimestampWithTimezone as (sec, usec, offset) in ORC file
    
    instead of using the ORC Timestamp format, since we cannot always encode
    them in this format.
    
    The offset is encoded as binary (byte string), following recent evolutions
    of swh-model.
    
    This makes swh-dataset compatible with swh-model 5.

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/35/ for more details.

Harbormaster completed remote builds in B27567: Diff 26687.Mar 18 2022, 2:14 PM
douardda requested review of this revision.Mar 18 2022, 2:14 PM
vlorentz accepted this revision.Mar 18 2022, 2:44 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/dataset/test/test_orc.py
36

accidentally quadratic! (yeah I know, it's just a test)

This revision is now accepted and ready to land.Mar 18 2022, 2:44 PM
douardda updated this revision to Diff 26708.Mar 18 2022, 3:50 PM

rebase

swh-public-ci added a comment.Mar 18 2022, 3:55 PM

Build is green

Patch application report for D7385 (id=26708)

Could not rebase; Attempt merge onto 68f9bd2028...

Updating 68f9bd2..9a96367
Fast-forward
 requirements-swh.txt            |   6 +--
 swh/dataset/cli.py              |  10 +++-
 swh/dataset/exporters/orc.py    |  78 ++++++++++++++++++----------
 swh/dataset/journalprocessor.py |  34 ++++++++----
 swh/dataset/relational.py       |  15 +++---
 swh/dataset/test/test_orc.py    | 112 ++++++++++++++++++++++++++--------------
 6 files changed, 168 insertions(+), 87 deletions(-)
Changes applied before test
commit 9a96367ae0c0c664dd4415e80479107300dea1bf
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:09:57 2022 +0100

    Write related ORC files in the same directory using the same UUID
    
    related ORC files being ORC files involved in the serialization of a given
    object type, namely:
    
    - snapshot and snapshot_branches,
    - revision and revision_history,
    - directory and directory_entry.
    
    Also include the object_type in the generated file name (in place of the
    static 'graph').
    
    So the result will typically be like:
    
      output/orc/shapshot/
        snaphot-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snaphot_branch-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snapshot-1f41d206-994a-49bb-917f-e096e40c2856.orc
        snapshot_branch-1f41d206-994a-49bb-917f-e096e40c2856.orc

commit 0bab36224ff89b37c539fff4956b830be46eef81
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:00:41 2022 +0100

    Add some user metadata in generated ORC files
    
    add:
    - object type
    - uuid
    - version of swh.model used at file generation time,
    - version of swh.dataset used at file generation time.

commit eb135d55876346b6381e9c0a7965fe2827872322
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 16 10:46:37 2022 +0100

    Implement test_orc exporter as a simple function instead of a fixture
    
    and split it in 2 parts (needed for changes to come).

commit f7fe626c2db82dc25b58d01e84c6f017867da7c9
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:50:15 2021 +0100

    Make the kafka group_id prefix configurable in the config file
    
    rather than hardcoding it to 'swh-dataset-export-', use the 'group_id'
    value from the 'journal' section of the config file as prefix, if given
    9otherwise default to the former value).
    
    This is needed because current auth policy of swh kafka cluster only allow
    group_id to start with the actual login for authenticated connection.
    So we need to be able to specify this group_id prefix.

commit 4f14a95aaadabc1d2036a9c31c18e6a78befb44d
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:41:28 2021 +0100

    Use a named logger for journalprocessor.py
    
    and add a few more debug logging statements.

commit 316d51b6da36719bca767c78ad04402c609d5abe
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:44:49 2021 +0100

    Update JournalClientOffsetRanges for swh.journal 0.9
    
    deserialize_message() now takes an optional 'object_type' argument.

commit ae440431049470ecac6aca0e8cbed4a51cde0c09
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 11:46:37 2022 +0100

    Encode TimestampWithTimezone as (sec, usec, offset) in ORC file
    
    instead of using the ORC Timestamp format, since we cannot always encode
    them in this format.
    
    The offset is encoded as binary (byte string), following recent evolutions
    of swh-model.
    
    This makes swh-dataset compatible with swh-model 5.

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/47/ for more details.

Harbormaster completed remote builds in B27604: Diff 26708.Mar 18 2022, 3:55 PM
douardda updated this revision to Diff 26778.Mar 22 2022, 3:49 PM

rebase

swh-public-ci added a comment.Mar 22 2022, 3:50 PM

Build has FAILED

Patch application report for D7385 (id=26778)

Could not rebase; Attempt merge onto 68f9bd2028...

Updating 68f9bd2..ef90997
Fast-forward
 requirements-swh.txt            |   6 +-
 swh/dataset/cli.py              |  10 +++-
 swh/dataset/exporters/orc.py    | 121 ++++++++++++++++++++++++++++++--------
 swh/dataset/journalprocessor.py |  34 +++++++----
 swh/dataset/relational.py       |  15 +++--
 swh/dataset/test/test_orc.py    | 125 ++++++++++++++++++++++++++--------------
 6 files changed, 223 insertions(+), 88 deletions(-)
Changes applied before test
commit ef90997443aaff8a085a61c0cdde6a4f1f4f681e
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:09:57 2022 +0100

    Write related ORC files in the same directory using the same UUID
    
    related ORC files being ORC files involved in the serialization of a given
    object type, namely:
    
    - snapshot and snapshot_branches,
    - revision and revision_history,
    - directory and directory_entry.
    
    Also include the object_type in the generated file name (in place of the
    static 'graph').
    
    So the result will typically be like:
    
      output/orc/shapshot/
        snaphot-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snaphot_branch-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snapshot-1f41d206-994a-49bb-917f-e096e40c2856.orc
        snapshot_branch-1f41d206-994a-49bb-917f-e096e40c2856.orc

commit fc59bf8cff8722af17652f5c76961e9ef920d5fc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:00:41 2022 +0100

    Add some user metadata in generated ORC files
    
    add:
    - object type
    - uuid
    - version of swh.model used at file generation time,
    - version of swh.dataset used at file generation time.

commit e120502b262b190364a65d6f1af9f23f2f19b8ed
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 16 10:46:37 2022 +0100

    Implement test_orc exporter as a simple function instead of a fixture
    
    and split it in 2 parts (needed for changes to come).

commit c7cc72902575271d5f1dbef77f94faf255f86868
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:50:15 2021 +0100

    Make the kafka group_id prefix configurable in the config file
    
    rather than hardcoding it to 'swh-dataset-export-', use the 'group_id'
    value from the 'journal' section of the config file as prefix, if given
    9otherwise default to the former value).
    
    This is needed because current auth policy of swh kafka cluster only allow
    group_id to start with the actual login for authenticated connection.
    So we need to be able to specify this group_id prefix.

commit c508c673043458a419f2eeb0d5a2fb60b12aa007
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:41:28 2021 +0100

    Use a named logger for journalprocessor.py
    
    and add a few more debug logging statements.

commit d49db10f0bf7174ea4f2742d5d5ac8c8e25b707a
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:44:49 2021 +0100

    Update JournalClientOffsetRanges for swh.journal 0.9
    
    deserialize_message() now takes an optional 'object_type' argument.

commit 69e806698bbb6df42bfa3520681e0203f91d8a65
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 11:46:37 2022 +0100

    Encode TimestampWithTimezone as (timestamp, offset, raw_offset_bytes) in ORC file
    
    ie. use the standard ORC Timestamp format (aka a couple
    (seconds, nanoseconds)) with 2 extra fields for the offset.
    
    The offset is stored as an integer (in minutes), but the raw offset
    value is also present as a binary string representation, following
    recent evolutions of swh-model.
    
    This makes swh-dataset compatible with swh-model 5.

Link to build: https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/58/
See console output for more information: https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/58/console

Harbormaster failed remote builds in B27699: Diff 26778!Mar 22 2022, 3:50 PM
douardda updated this revision to Diff 26800.Mar 22 2022, 5:30 PM

rebase

swh-public-ci added a comment.Mar 22 2022, 5:33 PM

Build is green

Patch application report for D7385 (id=26800)

Could not rebase; Attempt merge onto 68f9bd2028...

Updating 68f9bd2..4aa860d
Fast-forward
 requirements-swh.txt            |   6 +-
 swh/dataset/cli.py              |  10 +++-
 swh/dataset/exporters/orc.py    | 123 ++++++++++++++++++++++++++++++++--------
 swh/dataset/journalprocessor.py |  34 +++++++----
 swh/dataset/relational.py       |  15 +++--
 swh/dataset/test/test_orc.py    | 123 ++++++++++++++++++++++++++--------------
 6 files changed, 224 insertions(+), 87 deletions(-)
Changes applied before test
commit 4aa860deb508f6388c6e46877097435c944a5909
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:09:57 2022 +0100

    Write related ORC files in the same directory using the same UUID
    
    related ORC files being ORC files involved in the serialization of a given
    object type, namely:
    
    - snapshot and snapshot_branches,
    - revision and revision_history,
    - directory and directory_entry.
    
    Also include the object_type in the generated file name (in place of the
    static 'graph').
    
    So the result will typically be like:
    
      output/orc/shapshot/
        snaphot-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snaphot_branch-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snapshot-1f41d206-994a-49bb-917f-e096e40c2856.orc
        snapshot_branch-1f41d206-994a-49bb-917f-e096e40c2856.orc

commit 85d95ac3701a90f7ab7d12de5b7ff3af9bf8f45f
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:00:41 2022 +0100

    Add some user metadata in generated ORC files
    
    add:
    - object type
    - uuid
    - version of swh.model used at file generation time,
    - version of swh.dataset used at file generation time.

commit a92a271fec1c8f99ea8b47fed89d0c3e447de1dc
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 16 10:46:37 2022 +0100

    Implement test_orc exporter as a simple function instead of a fixture
    
    and split it in 2 parts (needed for changes to come).

commit 7c775ce88ebb840d32aa70dbe6feb1a6835db518
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:50:15 2021 +0100

    Make the kafka group_id prefix configurable in the config file
    
    rather than hardcoding it to 'swh-dataset-export-', use the 'group_id'
    value from the 'journal' section of the config file as prefix, if given
    9otherwise default to the former value).
    
    This is needed because current auth policy of swh kafka cluster only allow
    group_id to start with the actual login for authenticated connection.
    So we need to be able to specify this group_id prefix.

commit 55cf5ac5cc0cb818ae23b5df4af416e57794469c
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:41:28 2021 +0100

    Use a named logger for journalprocessor.py
    
    and add a few more debug logging statements.

commit 70d9d3182de1420ba545f2f507ade8f59b2c2f33
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:44:49 2021 +0100

    Update JournalClientOffsetRanges for swh.journal 0.9
    
    deserialize_message() now takes an optional 'object_type' argument.

commit 09d2840dbd4db6e1a3dd976c44b3c628b9174741
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 11:46:37 2022 +0100

    Encode TimestampWithTimezone as (timestamp, offset, raw_offset_bytes) in ORC file
    
    ie. use the standard ORC Timestamp format (aka a couple
    (seconds, nanoseconds)) with 2 extra fields for the offset.
    
    The offset is stored as an integer (in minutes), but the raw offset
    value is also present as a binary string representation, following
    recent evolutions of swh-model.
    
    This makes swh-dataset compatible with swh-model 5.

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/70/ for more details.

Harbormaster completed remote builds in B27721: Diff 26800.Mar 22 2022, 5:33 PM
douardda updated this revision to Diff 26996.Mar 29 2022, 3:16 PM

rebase

swh-public-ci added a comment.Mar 29 2022, 3:22 PM

Build is green

Patch application report for D7385 (id=26996)

Could not rebase; Attempt merge onto 68f9bd2028...

Updating 68f9bd2..225f875
Fast-forward
 requirements-swh.txt            |   6 +-
 swh/dataset/cli.py              |  10 +++-
 swh/dataset/exporters/orc.py    | 123 ++++++++++++++++++++++++++++++++--------
 swh/dataset/journalprocessor.py |  34 +++++++----
 swh/dataset/relational.py       |   3 +
 swh/dataset/test/test_orc.py    | 123 ++++++++++++++++++++++++++--------------
 6 files changed, 218 insertions(+), 81 deletions(-)
Changes applied before test
commit 225f87573d52cfb13a876899bb59ef004fd1af84
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:09:57 2022 +0100

    Write related ORC files in the same directory using the same UUID
    
    related ORC files being ORC files involved in the serialization of a given
    object type, namely:
    
    - snapshot and snapshot_branches,
    - revision and revision_history,
    - directory and directory_entry.
    
    Also include the object_type in the generated file name (in place of the
    static 'graph').
    
    So the result will typically be like:
    
      output/orc/shapshot/
        snaphot-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snaphot_branch-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snapshot-1f41d206-994a-49bb-917f-e096e40c2856.orc
        snapshot_branch-1f41d206-994a-49bb-917f-e096e40c2856.orc

commit 0a593e3478cc982b95e0ec6ac23a0ba52063ae73
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:00:41 2022 +0100

    Add some user metadata in generated ORC files
    
    add:
    - object type
    - uuid
    - version of swh.model used at file generation time,
    - version of swh.dataset used at file generation time.

commit f8211b934774ded5ea83948a02140af446788e9a
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 16 10:46:37 2022 +0100

    Implement test_orc exporter as a simple function instead of a fixture
    
    and split it in 2 parts (needed for changes to come).

commit 4508da3a91ba634e3b5def318f4509a688168c9e
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:50:15 2021 +0100

    Make the kafka group_id prefix configurable in the config file
    
    rather than hardcoding it to 'swh-dataset-export-', use the 'group_id'
    value from the 'journal' section of the config file as prefix, if given
    9otherwise default to the former value).
    
    This is needed because current auth policy of swh kafka cluster only allow
    group_id to start with the actual login for authenticated connection.
    So we need to be able to specify this group_id prefix.

commit de114c20f105c0b888eb92625f4e073f97a94ae8
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:41:28 2021 +0100

    Use a named logger for journalprocessor.py
    
    and add a few more debug logging statements.

commit a8442bcf7c4311a28bea0898a01dc9475889efc7
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:44:49 2021 +0100

    Update JournalClientOffsetRanges for swh.journal 0.9
    
    deserialize_message() now takes an optional 'object_type' argument.

commit f588e20a41af4b1b8042f9b5f0e88a1f1dc91e59
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 11:46:37 2022 +0100

    Encode TimestampWithTimezone as (timestamp, offset, raw_offset_bytes) in ORC file
    
    ie. use the standard ORC Timestamp format (aka a couple
    (seconds, nanoseconds)) with 2 extra fields for the offset.
    
    The offset is stored as an integer (in minutes), but the raw offset
    value is also present as a binary string representation, following
    recent evolutions of swh-model.
    
    This makes swh-dataset compatible with swh-model 5.

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/82/ for more details.

Harbormaster completed remote builds in B27912: Diff 26996.Mar 29 2022, 3:22 PM
douardda updated this revision to Diff 27033.Mar 29 2022, 5:14 PM

rebase

swh-public-ci added a comment.Mar 29 2022, 5:16 PM

Build is green

Patch application report for D7385 (id=27033)

Could not rebase; Attempt merge onto 31081e4121...

Updating 31081e4..c585725
Fast-forward
 requirements-swh.txt            |  2 +-
 swh/dataset/cli.py              | 10 +++++-
 swh/dataset/exporters/orc.py    | 42 ++++++++++++++++++----
 swh/dataset/journalprocessor.py | 34 ++++++++++++------
 swh/dataset/test/test_orc.py    | 80 +++++++++++++++++++++--------------------
 5 files changed, 110 insertions(+), 58 deletions(-)
Changes applied before test
commit c5857252221fb46fd1f1b50f154d4f09f60b5920
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:09:57 2022 +0100

    Write related ORC files in the same directory using the same UUID
    
    related ORC files being ORC files involved in the serialization of a given
    object type, namely:
    
    - snapshot and snapshot_branches,
    - revision and revision_history,
    - directory and directory_entry.
    
    Also include the object_type in the generated file name (in place of the
    static 'graph').
    
    So the result will typically be like:
    
      output/orc/shapshot/
        snaphot-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snaphot_branch-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snapshot-1f41d206-994a-49bb-917f-e096e40c2856.orc
        snapshot_branch-1f41d206-994a-49bb-917f-e096e40c2856.orc

commit 729ae64f36cd4f2d78bbdd0952a801b5cba5b462
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:00:41 2022 +0100

    Add some user metadata in generated ORC files
    
    add:
    - object type
    - uuid
    - version of swh.model used at file generation time,
    - version of swh.dataset used at file generation time.

commit 2298fb3422804688bd7bbcef3155cd0e9a80a00e
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 16 10:46:37 2022 +0100

    Implement test_orc exporter as a simple function instead of a fixture
    
    and split it in 2 parts (needed for changes to come).

commit 68899901c7e596471cb1a9e769504919c6a19881
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:50:15 2021 +0100

    Make the kafka group_id prefix configurable in the config file
    
    rather than hardcoding it to 'swh-dataset-export-', use the 'group_id'
    value from the 'journal' section of the config file as prefix, if given
    9otherwise default to the former value).
    
    This is needed because current auth policy of swh kafka cluster only allow
    group_id to start with the actual login for authenticated connection.
    So we need to be able to specify this group_id prefix.

commit 769b6a77d250123ee25d8576bc1fe4a9340616f4
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:41:28 2021 +0100

    Use a named logger for journalprocessor.py
    
    and add a few more debug logging statements.

commit d7c332e4e7e1d5ee531a914b302f98c11503663e
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Dec 15 16:44:49 2021 +0100

    Update JournalClientOffsetRanges for swh.journal 0.9
    
    deserialize_message() now takes an optional 'object_type' argument.

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/93/ for more details.

Harbormaster completed remote builds in B27949: Diff 27033.Mar 29 2022, 5:16 PM
douardda added a child revision: D7387: Add the type fields for revision and origin_visit_status ORC table.Mar 29 2022, 5:23 PM
douardda updated this revision to Diff 27040.Mar 29 2022, 5:36 PM

forgot vlorentz' comment...

douardda marked an inline comment as done.Mar 29 2022, 5:36 PM
swh-public-ci added a comment.Mar 29 2022, 5:38 PM

Build is green

Patch application report for D7385 (id=27040)

Rebasing onto 729ae64f36...

Current branch diff-target is up to date.
Changes applied before test
commit 5a8a8a7847f6a179eeb5f47521602d4977c1e482
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:09:57 2022 +0100

    Write related ORC files in the same directory using the same UUID
    
    related ORC files being ORC files involved in the serialization of a given
    object type, namely:
    
    - snapshot and snapshot_branches,
    - revision and revision_history,
    - directory and directory_entry.
    
    Also include the object_type in the generated file name (in place of the
    static 'graph').
    
    So the result will typically be like:
    
      output/orc/shapshot/
        snaphot-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snaphot_branch-18a575cb-3a92-4753-9267-e3475fa30857.orc
        snapshot-1f41d206-994a-49bb-917f-e096e40c2856.orc
        snapshot_branch-1f41d206-994a-49bb-917f-e096e40c2856.orc

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/94/ for more details.

Harbormaster completed remote builds in B27956: Diff 27040.Mar 29 2022, 5:38 PM
Closed by commit rDDATASET5a8a8a7847f6: Write related ORC files in the same directory using the same UUID (authored by douardda). · Explain WhyMar 29 2022, 5:39 PM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDDATASET5a8a8a7847f6: Write related ORC files in the same directory using the same UUID.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
dataset/
exporters/
34 lines97%
Loading...
test/
5 lines98%
Loading...

Diff 26687

View Options

swh/dataset/exporters/orc.py

Loading...
View Options

swh/dataset/test/test_orc.py

Loading...
About · Developer information · Legal