diff --git a/docs/index.rst b/docs/index.rst index 799b076..b557f04 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,225 +1,227 @@ .. _swh-docs: Software Heritage - Development Documentation ============================================= Getting started --------------- * :ref:`getting-started` → deploy a local copy of the Software Heritage software stack in less than 5 minutes, or * :ref:`developer-setup` → get a working development setup that allows to hack on the Software Heritage software stack * :ref:`faq` Contributing ------------ * :ref:`patch-submission` → learn how to submit your patches to the Software Heritage codebase * :ref:`code-review` → rules and guidelines to review code in Software Heritage * :ref:`python-style-guide` → how to format the Python code you write Architecture ------------ * :ref:`architecture-overview` → get a glimpse of the Software Heritage software architecture * :ref:`Metadata workflow ` → learn how Software Heritage stores and handles metadata Data Model and Specifications ----------------------------- * :ref:`persistent-identifiers` Specifications of the SoftWare Heritage persistent IDentifiers (SWHID). * :ref:`data-model` Documentation of the main |swh| archive data model. * :ref:`journal-specs` Documentation of the Kafka journal of the |swh| archive. Tutorials --------- * :ref:`testing-guide` * :doc:`/tutorials/issue-debugging-monitoring` * :ref:`Listing the content of your favorite forge ` and :ref:`running a lister in Docker ` * :ref:`Add a new swh package ` * :ref:`doc-contribution` Roadmap ------- * :ref:`roadmap-2021` System Administration --------------------- * :ref:`Network Infrastructure ` * :ref:`swh-sysadm:mirror` → learn what a Software Heritage mirror is and how to set up one * :ref:`Keycloak ` → learn how to use Keycloak, the authentication system used by |swh|'s web interface and public APIs +.. _components: + Components ---------- Here is brief overview of the most relevant software components in the Software Heritage stack, in alphabetical order. For a better introduction to the architecture, see the :ref:`architecture-overview`, which presents each of them in a didactical order. Each component name is linked to the development documentation of the corresponding Python module. :ref:`swh.auth ` low-level library used by modules needing keycloak authentication :ref:`swh.core ` low-level utilities and helpers used by almost all other modules in the stack :ref:`swh.counters ` service providing efficient estimates of the number of objects in the SWH archive, using Redis's Hyperloglog :ref:`swh.dataset ` public datasets and periodic data dumps of the archive released by Software Heritage :ref:`swh.deposit ` push-based deposit of software artifacts to the archive swh.docs developer documentation (used to generate this doc you are reading) :ref:`swh.fuse ` Virtual file system to browse the Software Heritage archive, based on `FUSE `_ :ref:`swh.graph ` Fast, compressed, in-memory representation of the archive, with tooling to generate and query it. :ref:`swh.indexer ` tools and workers used to crawl the content of the archive and extract derived information from any artifact stored in it :ref:`swh.journal ` persistent logger of changes to the archive, with publish-subscribe support :ref:`swh.lister ` collection of listers for all sorts of source code hosting and distribution places (forges, distributions, package managers, etc.) :ref:`swh.loader-core ` low-level loading utilities and helpers used by all other loaders :ref:`swh.loader-bzr ` loader for `Bazaar `_ and `Breezy `_ repositories :ref:`swh.loader-git ` loader for `Git `_ repositories :ref:`swh.loader-mercurial ` loader for `Mercurial `_ repositories :ref:`swh.loader-svn ` loader for `Subversion `_ repositories :ref:`swh.loader-cvs ` loader for `CVS `_ repositories :ref:`swh.model ` implementation of the :ref:`data-model` to archive source code artifacts :ref:`swh.objstorage ` content-addressable object storage :ref:`swh.objstorage.replayer ` Object storage replication tool :ref:`swh.perfecthash ` Low level management for read-only content-addressable object storage indexed with a perfect hash table :ref:`swh.scanner ` source code scanner to analyze code bases and compare them with source code artifacts archived by Software Heritage :ref:`swh.scheduler ` task manager for asynchronous/delayed tasks, used for recurrent (e.g., listing a forge, loading new stuff from a Git repository) and one-off activities (e.g., loading a specific version of a source package) :ref:`swh.scrubber ` Tooling to check integrity of various data stores (swh.journal, swh.objstorage, swh.storage) and fix corrupt objects they contain. :ref:`swh.search ` search engine for the archive :ref:`swh.storage ` abstraction layer over the archive, allowing to access all stored source code artifacts as well as their metadata :ref:`swh.vault ` implementation of the vault service, allowing to retrieve parts of the archive as self-contained bundles (e.g., individual releases, entire repository snapshots, etc.) :ref:`swh.web ` Web application(s) to browse the archive, for both interactive (HTML UI) and mechanized (REST API) use :ref:`swh.web.client ` Python client for :ref:`swh.web ` Dependencies ------------ The dependency relationships among the various modules are depicted below. .. _py-deps-swh: .. figure:: images/py-deps-swh.svg :width: 1024px :align: center Dependencies among top-level Python modules (click to zoom). Archive ------- * :ref:`Archive ChangeLog `: notable changes to the archive over time Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * `URLs index `_ * :ref:`search` * :ref:`glossary` .. ensure sphinx does not complain about index files not being included .. toctree:: :maxdepth: 2 :caption: Contents: :titlesonly: :hidden: getting-started/index architecture/index contributing/index tutorials/index faq/index roadmap/roadmap-2021 api-reference archive-changelog journal diff --git a/docs/tutorials/add-new-package.rst b/docs/tutorials/add-new-package.rst index bede516..5b253f0 100644 --- a/docs/tutorials/add-new-package.rst +++ b/docs/tutorials/add-new-package.rst @@ -1,150 +1,164 @@ .. _tutorial-new-package: Add a new package ================= The following document demonstrates how to create a new swh package in the current `swh phabricator instance`_ and reference it so the `Continuous Integration (CI)`_ is able to build the new package. We will need to (optionally) create a project, then create a repository, reference it in the CI and finally update the documentation repository so the docs get built with that new package. Optionally, we can also create the necessary debian branches for the debian build deployment process to work. That's not something immediately urgent when bootstraping a package though. .. _create-new-project: Create a project ---------------- Create a `new Project`_ (seen also as a ``Tag``) and fill in the form: - Fill in the "name" field (e.g. ``Origin Sourceforge``, ``Loader XYZ``, ...) - Additional hashtags should be filled in with easy to remember hashtags (e.g. ``#sourceforge``, ``#loader-xyz``) - Add a small description about what the project is about Create a repository ------------------- - Create a `new Git repository`_ and fill in the form: - ``name`` of the repository should be human readable (e.g. ``Git Loader``, ``Gitlab Lister``) - ``callsign`` should follow the current `naming convention`_ - ``short-name`` should be lower case and dash separated (e.g. ``swh-loader-git``, ``swh-lister``, ...) - ``Tags`` should be filled with: - the first project/tag :ref:`you created ` - ``Language-Python``: Enable unit tests execution on commit - ``Has debian packaging branches``: Activate debian builds in the CI - ``Sync to github``: (optional) Activate mirroring sync to github - Add the staging area, click in order from ``BUILDS`` (left menu) > ``Staging Area`` > ``Edit staging`` > fill in ``Staging Area URI`` with https://forge.softwareheritage.org/source/staging.git - Finally, activate the repository Add the repo on the swh-environment project ------------------------------------------- (Only if necessary) Unless specified otherwise, the following commands need to run from the base directory ``swh-environment``. - clone the new repository .. code:: bash git clone https://forge.softwareheritage.org/source/swh-my-new-repo.git - launch the command ``bin/init-py-repo`` to initialize the repository with a project template .. code:: bash bin/init-py-repo swh-new-repo +- Within that new repository, replace the ``swh-py-template`` entry in + ``docs/index.rst`` with the new package name ``swh-`` (e.g: + ``swh-scrubber``). + +.. code:: bash + + REPO_NAME=swh-new-repo # edit this part, keep the "swh-" prefix + sed -i -e "s/swh-py-template/$REPO_NAME/g" docs/index.rst + - Edit the default content of the template (`Example `__) - Configure **your local** pre-commit hook - In the ``swh-environment/swh-my-new-repo`` directory, execute: .. code:: bash grep -q pre-commit.com .git/hooks/pre-commit || pre-commit install - Declare the repository on the mr configuration - Edit the ``.mrconfig`` file and declare the new repository (just duplicate one existing entry and adapt with the new package name) - - Commit file modification (`Example `__) + - Commit file modification (`Example + `__) Install CI jobs --------------- - In the swh-jenkins-jobs_ repository, open the ``jobs/swh-packages.yaml`` and add a section for the new repository as for the others (`Example `__) - Configure the `post-receive hook`_ on the phabricator instance - `Setting up the debian jenkins jobs`_ Setting up debian builds ------------------------ As mentioned early in the introduction, this configuration can be delayed for when the package is actually ready to be deployed. If you want to attend immedialy, follow through the `Setting up the debian build`_ documentation. Documentation updates --------------------- - Documentation repository is located in the swh-docs_ repository. -- Add the new package dependency in the top-level ``requirements-swh.txt`` and - ``requirements-swh-dev.txt`` +- Add the package dependency in the top-level ``requirements-swh.txt`` (publication + build) and ``requirements-swh-dev.txt`` (documentation development build). + +- Reference the package in the toc tree located in :ref:`docs/api-reference.rst + `. -- Add the new package entry in ``docs/index.rst`` with a concise description of the - package +- Reference the package in the index with its concise description located in + :ref:`docs/index.rst `. :: :ref:`swh.my_new_repo ` short description of the repository ... # at the end of the index page swh.my_new_repo - ensure this builds fine locally (e.g run `tox`, then `make -C docs`) -- Then open a diff to advertise the new documentation entrypoints (`Example `__) +- Then open a diff to advertise the new documentation entrypoints (`Example + `__) .. _`swh phabricator instance`: https://forge.softwareheritage.org/ .. _`Continuous Integration (CI)`: https://jenkins.softwareheritage.org .. _`new Project`: https://forge.softwareheritage.org/project/edit/form/3/ .. _`new Git repository`: https://forge.softwareheritage.org/diffusion/edit/form/default/?vcs=git .. _`naming convention`: https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention .. _swh-jenkins-jobs: https://forge.softwareheritage.org/source/swh-jenkins-jobs .. _`post-receive hook`: https://wiki.softwareheritage.org/wiki/Debian_packaging#Setting_up_the_repository_on_Phabricator .. _`Setting up the debian jenkins jobs`: https://wiki.softwareheritage.org/wiki/Debian_packaging#Setting_up_the_Jenkins_jobs .. _`Setting up the debian build`: https://wiki.softwareheritage.org/wiki/Debian_packaging#Git_repositories_for_Debian_packages .. _swh-docs: https://forge.softwareheritage.org/source/swh-docs/