Differential D7520
Add a ORC file loading function Authored by douardda on Apr 7 2022, 12:04 PM.
Details
Diff Detail
Event TimelineComment Actions Build has FAILED Patch application report for D7520 (id=27283)Could not rebase; Attempt merge onto 9d97f0c082... Updating 9d97f0c..0c0df2e Fast-forward swh/dataset/cli.py | 42 ++++--- swh/dataset/orc_loader.py | 254 ++++++++++++++++++++++++++++++++++++++ swh/dataset/test/test_orc_load.py | 26 ++++ 3 files changed, 303 insertions(+), 19 deletions(-) create mode 100644 swh/dataset/orc_loader.py create mode 100644 swh/dataset/test/test_orc_load.py Changes applied before testcommit 0c0df2e01c76aed77a662d7f22481af3d3da0c89
Author: David Douard <david.douard@sdfa3.org>
Date: Fri Mar 18 14:16:39 2022 +0100
Add a ORC file loading function
this generates swh.model objects from ORC files.
This should allow to rebuild a storage from an ORC dataset of the
archive.
Note: not all object types are supported for now (eg. ExtID, metadata
related objects, etc. are not yet supported).
commit 18325cc8e78e99ac35f550687e41b6f21c5d3a9f
Author: David Douard <david.douard@sdfa3.org>
Date: Wed Apr 6 16:16:09 2022 +0200
Reduce cli's loading time by moving import statements in commandsLink to build: https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/125/ Comment Actions Build has FAILED Patch application report for D7520 (id=27320)Rebasing onto 18325cc8e7... Current branch diff-target is up to date. Changes applied before testcommit 8be63a1a7d7430794b2e4e31aa6f8af50a074dd4
Author: David Douard <david.douard@sdfa3.org>
Date: Fri Mar 18 14:16:39 2022 +0100
Add a ORC file loading function
this generates swh.model objects from ORC files.
This should allow to rebuild a storage from an ORC dataset of the
archive.
Note: not all object types are supported for now (eg. ExtID, metadata
related objects, etc. are not yet supported).Link to build: https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/126/ Comment Actions Build is green Patch application report for D7520 (id=27322)Rebasing onto 18325cc8e7... Current branch diff-target is up to date. Changes applied before testcommit dbf1b87b0cb59a8c76a9928f1efdacd87abcf4ad
Author: David Douard <david.douard@sdfa3.org>
Date: Fri Mar 18 14:16:39 2022 +0100
Add a ORC file loading function
this generates swh.model objects from ORC files.
This should allow to rebuild a storage from an ORC dataset of the
archive.
Note: not all object types are supported for now (eg. ExtID, metadata
related objects, etc. are not yet supported).See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/127/ for more details. Comment Actions Have you tried using swh.storage.postgresql.converters.db_to_*? It is very similar to these cvrt_* functions, so you'd only need to convert dates from ISO8601 strings to datetime
| ||||||||||||||||||||||