HomeSoftware Heritage

Add Orchestrator

Description

Add Orchestrator

This is to build a mechanism to write to write the data from clearcode database which has been mapped with swh storage into swh RawExtrensicMetadata, and the data that has not been mapped to
a table unmapped_data. This process of orchestration will run periodically and will only try to map new data that has been entered after the last orchestration process and the data that was
not mapped in last orchestration.

Initialize tables if they don't exist in database. Initialize swh storage and add MetadataAuthority, MetadataFetcher, then map previously unmapped data and get last run date of orchestration
then read data from clearcode and orchestor rows from clearcode DB (if whole row is mapped then in metadataStorage, if partial or no data is matched then store that row in unmapped data table
(for future mapping purpose), if tool of row is fossoloy then skip that row.

Add tests and docstrings

Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>

Details

Provenance
TG1999Authored on Feb 23 2021, 4:48 PM
TG1999Pushed on Feb 23 2021, 5:43 PM
Differential Revision
D5036: Add Orchestrator
Parents
rDMFCD429ca5f59bbb: mapping_utils: Define AUTHORITY and FETCHER properly instead of evolving them…
Branches
Unknown
Tags
Unknown
Build Status
Buildable 19438
Build 30151: test-and-buildJenkins console · Jenkins