diff --git a/docs/index.rst b/docs/index.rst index 853a876..bfe1101 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,238 +1,239 @@ .. _swh-docs: Software Heritage - Development Documentation ============================================= Getting started --------------- * :ref:`getting-started` → deploy a local copy of the Software Heritage software stack in less than 5 minutes, or * :ref:`developer-setup` → get a working development setup that allows to hack on the Software Heritage software stack Contributing ------------ * :ref:`patch-submission` → learn how to submit your patches to the Software Heritage codebase * :ref:`code-review` → rules and guidelines to review code in Software Heritage * :ref:`python-style-guide` → how to format the Python code you write Architecture ------------ * :ref:`architecture-overview` → get a glimpse of the Software Heritage software architecture * :ref:`mirror` → learn what a Software Heritage mirror is and how to set up one * :ref:`Keycloak ` → learn how to use Keycloak, the authentication system used by |swh|'s web interface and public APIs Data Model and Specifications ----------------------------- * :ref:`persistent-identifiers` Specifications of the SoftWare Heritage persistent IDentifiers (SWHID). * :ref:`data-model` Documentation of the main |swh| archive data model. * :ref:`journal-specs` Documentation of the Kafka journal of the |swh| archive. Tutorials --------- * :ref:`testing-guide` * :doc:`/tutorials/issue-debugging-monitoring` * :ref:`Listing the content of your favorite forge ` and :ref:`running a lister in Docker ` +* :ref:`Add a new swh package ` Frequently Asked Questions -------------------------- .. toctree:: :maxdepth: 2 faq/index Roadmap ------- * :ref:`roadmap-2021` Engineering ----------- * :ref:`infrastructure` Components ---------- Here is brief overview of the most relevant software components in the Software Heritage stack, in alphabetical order. For a better introduction to the architecture, see the :ref:`architecture-overview`, which presents each of them in a didactical order. Each component name is linked to the development documentation of the corresponding Python module. :ref:`swh.auth ` low-level library used by modules needing keycloak authentication :ref:`swh.core ` low-level utilities and helpers used by almost all other modules in the stack :ref:`swh.counters ` service providing efficient estimates of the number of objects in the SWH archive, using Redis's Hyperloglog :ref:`swh.dataset ` public datasets and periodic data dumps of the archive released by Software Heritage :ref:`swh.deposit ` push-based deposit of software artifacts to the archive swh.docs developer documentation (used to generate this doc you are reading) :ref:`swh.fuse ` Virtual file system to browse the Software Heritage archive, based on `FUSE `_ :ref:`swh.graph ` Fast, compressed, in-memory representation of the archive, with tooling to generate and query it. :ref:`swh.indexer ` tools and workers used to crawl the content of the archive and extract derived information from any artifact stored in it :ref:`swh.journal ` persistent logger of changes to the archive, with publish-subscribe support :ref:`swh.lister ` collection of listers for all sorts of source code hosting and distribution places (forges, distributions, package managers, etc.) :ref:`swh.loader-core ` low-level loading utilities and helpers used by all other loaders :ref:`swh.loader-git ` loader for `Git `_ repositories :ref:`swh.loader-mercurial ` loader for `Mercurial `_ repositories :ref:`swh.loader-svn ` loader for `Subversion `_ repositories :ref:`swh.loader-cvs ` loader for `CVS `_ repositories :ref:`swh.model ` implementation of the :ref:`data-model` to archive source code artifacts :ref:`swh.objstorage ` content-addressable object storage :ref:`swh.objstorage.replayer ` Object storage replication tool :ref:`swh.scanner ` source code scanner to analyze code bases and compare them with source code artifacts archived by Software Heritage :ref:`swh.scheduler ` task manager for asynchronous/delayed tasks, used for recurrent (e.g., listing a forge, loading new stuff from a Git repository) and one-off activities (e.g., loading a specific version of a source package) :ref:`swh.search ` search engine for the archive :ref:`swh.storage ` abstraction layer over the archive, allowing to access all stored source code artifacts as well as their metadata :ref:`swh.vault ` implementation of the vault service, allowing to retrieve parts of the archive as self-contained bundles (e.g., individual releases, entire repository snapshots, etc.) :ref:`swh.web ` Web application(s) to browse the archive, for both interactive (HTML UI) and mechanized (REST API) use :ref:`swh.web.client ` Python client for :ref:`swh.web ` Dependencies ------------ The dependency relationships among the various modules are depicted below. .. _py-deps-swh: .. figure:: images/py-deps-swh.svg :width: 1024px :align: center Dependencies among top-level Python modules (click to zoom). Archive ------- * :ref:`Archive ChangeLog `: notable changes to the archive over time Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * `URLs index `_ * :ref:`search` * :ref:`glossary` .. ensure sphinx does not complain about index files not being included .. toctree:: :maxdepth: 2 :caption: Contents: :titlesonly: :hidden: getting-started/index architecture/index contributing/index tutorials/index faq/index roadmap/roadmap-2021.rst infrastructure/index swh.auth swh.core swh.counters swh.dataset swh.deposit swh.fuse swh.graph swh.indexer swh.journal swh.lister swh.loader swh.model swh.objstorage swh.objstorage.replayer swh.scanner swh.scheduler swh.search swh.storage swh.vault swh.web swh.web.client archive-changelog journal Python modules autodocumentation diff --git a/docs/tutorials/add-new-package.rst b/docs/tutorials/add-new-package.rst new file mode 100644 index 0000000..bede516 --- /dev/null +++ b/docs/tutorials/add-new-package.rst @@ -0,0 +1,150 @@ +.. _tutorial-new-package: + +Add a new package +================= + +The following document demonstrates how to create a new swh package in the current `swh +phabricator instance`_ and reference it so the `Continuous Integration (CI)`_ is able to +build the new package. + +We will need to (optionally) create a project, then create a repository, reference it in +the CI and finally update the documentation repository so the docs get built with that +new package. Optionally, we can also create the necessary debian branches for the debian +build deployment process to work. That's not something immediately urgent when +bootstraping a package though. + +.. _create-new-project: + +Create a project +---------------- + +Create a `new Project`_ (seen also as a ``Tag``) and fill in the form: + + - Fill in the "name" field (e.g. ``Origin Sourceforge``, ``Loader XYZ``, ...) + + - Additional hashtags should be filled in with easy to remember hashtags (e.g. + ``#sourceforge``, ``#loader-xyz``) + + - Add a small description about what the project is about + +Create a repository +------------------- + +- Create a `new Git repository`_ and fill in the form: + +- ``name`` of the repository should be human readable (e.g. ``Git Loader``, ``Gitlab Lister``) + +- ``callsign`` should follow the current `naming convention`_ + +- ``short-name`` should be lower case and dash separated (e.g. ``swh-loader-git``, + ``swh-lister``, ...) + +- ``Tags`` should be filled with: + + - the first project/tag :ref:`you created ` + + - ``Language-Python``: Enable unit tests execution on commit + + - ``Has debian packaging branches``: Activate debian builds in the CI + + - ``Sync to github``: (optional) Activate mirroring sync to github + +- Add the staging area, click in order from ``BUILDS`` (left menu) > ``Staging Area`` > + ``Edit staging`` > fill in ``Staging Area URI`` with + https://forge.softwareheritage.org/source/staging.git + +- Finally, activate the repository + +Add the repo on the swh-environment project +------------------------------------------- + +(Only if necessary) + +Unless specified otherwise, the following commands need to run from the base directory +``swh-environment``. + +- clone the new repository + +.. code:: bash + + git clone https://forge.softwareheritage.org/source/swh-my-new-repo.git + +- launch the command ``bin/init-py-repo`` to initialize the repository with a project + template + +.. code:: bash + + bin/init-py-repo swh-new-repo + +- Edit the default content of the template (`Example `__) + +- Configure **your local** pre-commit hook + + - In the ``swh-environment/swh-my-new-repo`` directory, execute: + + .. code:: bash + + grep -q pre-commit.com .git/hooks/pre-commit || pre-commit install + +- Declare the repository on the mr configuration + + - Edit the ``.mrconfig`` file and declare the new repository (just + duplicate one existing entry and adapt with the new package name) + + - Commit file modification (`Example `__) + +Install CI jobs +--------------- + +- In the swh-jenkins-jobs_ repository, open the + ``jobs/swh-packages.yaml`` and add a section for the new repository as for the others + (`Example `__) + +- Configure the `post-receive hook`_ on the phabricator instance + +- `Setting up the debian jenkins jobs`_ + +Setting up debian builds +------------------------ + +As mentioned early in the introduction, this configuration can be delayed for when the +package is actually ready to be deployed. + +If you want to attend immedialy, follow through the `Setting up the debian build`_ +documentation. + +Documentation updates +--------------------- + +- Documentation repository is located in the swh-docs_ repository. + +- Add the new package dependency in the top-level ``requirements-swh.txt`` and + ``requirements-swh-dev.txt`` + +- Add the new package entry in ``docs/index.rst`` with a concise description of the + package + +:: + + :ref:`swh.my_new_repo ` + short description of the repository + ... + + # at the end of the index page + swh.my_new_repo + +- ensure this builds fine locally (e.g run `tox`, then `make -C docs`) + +- Then open a diff to advertise the new documentation entrypoints (`Example `__) + + +.. _`swh phabricator instance`: https://forge.softwareheritage.org/ +.. _`Continuous Integration (CI)`: https://jenkins.softwareheritage.org +.. _`new Project`: https://forge.softwareheritage.org/project/edit/form/3/ +.. _`new Git repository`: https://forge.softwareheritage.org/diffusion/edit/form/default/?vcs=git +.. _`naming convention`: https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention +.. _swh-jenkins-jobs: https://forge.softwareheritage.org/source/swh-jenkins-jobs +.. _`post-receive hook`: https://wiki.softwareheritage.org/wiki/Debian_packaging#Setting_up_the_repository_on_Phabricator +.. _`Setting up the debian jenkins jobs`: https://wiki.softwareheritage.org/wiki/Debian_packaging#Setting_up_the_Jenkins_jobs +.. _`Setting up the debian build`: https://wiki.softwareheritage.org/wiki/Debian_packaging#Git_repositories_for_Debian_packages +.. _swh-docs: https://forge.softwareheritage.org/source/swh-docs/ diff --git a/docs/tutorials/index.rst b/docs/tutorials/index.rst index 62186fc..d374c54 100644 --- a/docs/tutorials/index.rst +++ b/docs/tutorials/index.rst @@ -1,11 +1,12 @@ Tutorials ========= .. keep this in sync with the 'Tutorials' section in swh-docs/docs/index.rst .. toctree:: :maxdepth: 2 :titlesonly: testing issue-debugging-monitoring + add-new-package