diff --git a/README b/README index 0370631..2287724 100644 --- a/README +++ b/README @@ -1,87 +1,95 @@ SWH-loader-dir ============== The Software Heritage Directory Loader is a tool and a library to walk a local directory and inject into the SWH dataset all unknown contained files. -Configuration sample -==================== -### Directory loader +Directory loader +================ -#### Configuration + +### Configuration This is the loader's (or task's) configuration file. loader/dir.ini: [main] # access to swh's storage storage_class = remote_storage storage_args = http://localhost:5000/ # parameters to condition loading into swh storage send_contents = True send_directories = True send_revisions = True send_releases = True send_occurrences = True content_packet_size = 10000 content_packet_size_bytes = 1073741824 directory_packet_size = 25000 revision_packet_size = 100000 release_packet_size = 100000 occurrence_packet_size = 100000 Present in possible locations: - ~/.config/swh/loader/dir.ini - ~/.swh/loader/dir.ini - /etc/softwareheritage/loader/dir.ini -#### toplevel + +#### Toplevel Load directory directly from code or toplevel: from swh.loader.dir.tasks import LoadDirRepository dir_path = '/path/to/directory # Fill in those origin = {} release = None revision = {} occurrence = {} LoadDirRepository().run(dir_path, origin, revision, release, [occurrence]) -#### celery + +#### Celery Load directory using celery. Providing you have a properly configured celery up and running worker.ini needs to be updated with the following keys: task_modules = swh.loader.dir.tasks task_queues = swh_loader_dir cf. https://forge.softwareheritage.org/diffusion/DCORE/browse/master/README.md for more details You can send the following message to the task queue: from swh.loader.dir.tasks import LoadDirRepository # Fill in those origin = {} release = None revision = {} occurrence = {} # Send message to the task queue LoadDirRepository().apply_async(('/path/to/dir, origin, revision, release, [occurrence])) + + +Directory producer +================== + +None