Paths

Table of Contentst

Differential D7461

Add support for limited row numbers in ORC files
ClosedPublic
Actions

Authored by douardda on Mar 29 2022, 5:43 PM.

Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Commits

rDDATASET3df08fd71759: Add support for limited row numbers in ORC files

Summary

Make it possible to specify a maximum number of rows a table can store
in a single ORC file. The limit can only be set on main tables for now
(i.e. cannot be specified for tables like revision_history or
directory_entry).

This can be set by configuration only (no extra cli options).

Depends on D7389.

Diff Detail

Repository

rDDATASET Datasets

Branch

siporc

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 27981
Build 43819: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 43818: arc lint + arc unit

Event Timeline

douardda created this revision.Mar 29 2022, 5:43 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMar 29 2022, 5:43 PM

douardda added a child revision: D7462: Move exporter config entries in dedicated sections.Mar 29 2022, 5:44 PM

Build is green

Patch application report for D7461 (id=27046)

Could not rebase; Attempt merge onto 5a8a8a7847...

Updating 5a8a8a7..e8ccb16
Fast-forward
 swh/dataset/exporters/orc.py |  95 ++++++++++++++++++++++++++++++++++------
 swh/dataset/relational.py    |  10 +++++
 swh/dataset/test/test_orc.py | 100 +++++++++++++++++++++++++++++++++++++++----
 3 files changed, 183 insertions(+), 22 deletions(-)

Changes applied before test

commit e8ccb166a6aaa82f5917388f9b995c830499170a
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 23 16:35:52 2022 +0100

    Add support for limited row numbers in ORC files
    
    Make it possible to specify a maximum number of rows a table can store
    in a single ORC file. The limit can only be set on main tables for now
    (i.e. cannot be specified for tables like revision_history or
    directory_entry).
    
    This can be set by configuration only (no extra cli options).

commit fd3f9aa61de374655fd4bc4920d5047eb7d0c4ca
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:24:31 2022 +0100

    Add the raw_manifest column for revision, release and directory ORC files

commit 5c652bb058e2c1b59bafefd6817f392fdc171a20
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:22:47 2022 +0100

    Export revision extra headers in a dedicated ORC file

commit 45c8124b7a310963a868eb6602ea24e240d761e4
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:20:20 2022 +0100

    Add the type fields for revision and origin_visit_status ORC table

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/98/ for more details.

Harbormaster completed remote builds in B27961: Diff 27046.Mar 29 2022, 5:46 PM

douardda requested review of this revision.Mar 29 2022, 5:46 PM

vlorentz accepted this revision.Mar 29 2022, 5:59 PM

vlorentz added a subscriber: vlorentz.

vlorentz added inline comments.

swh/dataset/exporters/orc.py
1	bump the date
137–148	so before this diff, `self.writers` were never closed?
159–163	shouldn't this fail? it will drop data if no one notices the warning...
swh/dataset/test/test_orc.py
6	missing a copyright header btw

This revision is now accepted and ready to land.Mar 29 2022, 5:59 PM

douardda added inline comments.Mar 29 2022, 6:22 PM

swh/dataset/exporters/orc.py
137–148	they would thanks to the ExitStack used in the Exporter class
159–163	yeah that's a concern I have, not sure yet...

douardda marked 3 inline comments as done.Mar 30 2022, 10:18 AM

douardda added inline comments.

swh/dataset/exporters/orc.py
159–163	Actually I don't think this is a big deal here. If we want to fail, it should be a check done before starting to export anything.

check for max_rows validity at ORCExporter class instanciation + copyrights

Build has FAILED

Patch application report for D7461 (id=27065)

Rebasing onto fd3f9aa61d...

Current branch diff-target is up to date.

Changes applied before test

commit 850ee3be47cf3b1e0ab53f8820cc5e4c86b94f38
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 23 16:35:52 2022 +0100

    Add support for limited row numbers in ORC files
    
    Make it possible to specify a maximum number of rows a table can store
    in a single ORC file. The limit can only be set on main tables for now
    (i.e. cannot be specified for tables like revision_history or
    directory_entry).
    
    This can be set by configuration only (no extra cli options).

Link to build: https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/104/
See console output for more information: https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/104/console

Harbormaster failed remote builds in B27981: Diff 27065!Mar 30 2022, 10:45 AM

douardda marked an inline comment as done.Mar 30 2022, 10:59 AM

make it py37 compliant

Build is green

Patch application report for D7461 (id=27074)

Rebasing onto fd3f9aa61d...

Current branch diff-target is up to date.

Changes applied before test

commit 3df08fd71759487e963e6569c8dfd0c502b060de
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 23 16:35:52 2022 +0100

    Add support for limited row numbers in ORC files
    
    Make it possible to specify a maximum number of rows a table can store
    in a single ORC file. The limit can only be set on main tables for now
    (i.e. cannot be specified for tables like revision_history or
    directory_entry).
    
    This can be set by configuration only (no extra cli options).

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/109/ for more details.

Harbormaster completed remote builds in B27990: Diff 27074.Mar 30 2022, 11:12 AM

Closed by commit rDDATASET3df08fd71759: Add support for limited row numbers in ORC files (authored by douardda). · Explain WhyMar 30 2022, 11:34 AM

This revision was automatically updated to reflect the committed changes.

douardda added a commit: rDDATASET3df08fd71759: Add support for limited row numbers in ORC files.