Page MenuHomeSoftware Heritage

Gather CLI tools spread in SWH packages
Closed, ResolvedPublic

Event Timeline

douardda created this task.Apr 18 2019, 9:36 AM
douardda triaged this task as Normal priority.
douardda added a comment.EditedApr 18 2019, 11:49 AM

Currently, we have:

  • swh.archiver
    • director.py: run an Archiver Director (dispatching batch of contents to archiver workers)
    • updater.py: run an Updater (Journal client in charge of updating new contents in the content_archiver db)
    • checker.py: run a Checker (several content integrity checkers)
  • swh.core
    • cli.py: db initialization tool
  • swh.deposit
    • cli/__init__.py: main swh-deposit command
    • cli/deposit.py: deposit (client) tool to upload deposit artifacts
    • cli/admin.py: management tools for the deposit server
  • swh.indexer
    • cli.py: main cli entry point (swh-indexer) with mapping and task management tools plus the API/RPC server command
    • journal_client.py: example tool to log origins and origin_visit received in the journal
    • storage/api/server.py: deprecated (in favor of swh-indexer)
    • metadata_dictionary/__init__.py: not sure what this tool is for
    • origin_head.py: seems like a utility tool to look for origins in the origin indexer DB
  • swh.journal
    • cli.py: main cli entry point (swh-journal) + the replay subcommand (fill a new storage by reading a journal) and the backfill one (manipulate the backfiller)
  • swh.lister
    • cli.py: main swh-lister command (initialize a lister's DB)
    • debian/utils.py: debian specific utilities (list registered distributions, add one, etc.)
  • swh.loader.debian
    • debian/loader.py: (example) tool to load a given debian package in the archive
  • swh.loader.dir
    • dir/loader.py: (example) tool to load a given source code directory in the archive
  • swh.loader.git
    • git/loader.py: (example) tool to load a given remote git repo in the archive
    • git/from_disk.py: (example) tool to load a given local git repo in the archive
  • swh.loader.mercurial
    • mercurial/loader.py: (example) tool to load a given hg repository in the archive
  • swh.loader.pypi
    • pypi/loader.py: (example) tool to load a given Python source package in the archive
  • swh.loader.svn
    • svn/loader.py: (example) tool to load a given svn repository in the archive
  • swh.model
    • cli.py: swh-identity tool (Compute the Software Heritage persistent identifier (PID) for the given source code object(s))
  • swh.objstorage
    • cli.py: main swh-objstorage command with serve, fsck and import subcommands
  • swh.scheduler
    • cli.py: main swh-scheduler command with many server related subcommands (API server, runner, listener, ghtorrent client and updater) and task management (client) related subcommands
    • bin/swh-worker-control
  • swh.storage
    • cli.py: API server
  • swh.vault
    • cli.py: API server
  • swh.web
    • manage.py: django app management tools
zack closed this task as Resolved.May 25 2019, 5:28 PM
zack claimed this task.
zack added a subscriber: zack.

looks like this is done now, as you're deep in the implementation already! closing