diff --git a/docs/developer-setup.rst b/docs/developer-setup.rst
index dc63c5c..21cdce2 100644
--- a/docs/developer-setup.rst
+++ b/docs/developer-setup.rst
@@ -1,253 +1,254 @@
.. _developer-setup:
Developer setup
===============
In this guide we describe how to set up a developer environment in which one
can easily navigate the source code, make modifications, write and execute unit
tests.
For this, we will use a `virtualenv`_ in which all the |swh| packages will be
installed in 'develop' mode, this will allow you to navigate the source code,
hack it, and run locally the unit tests.
If you want to test the effect of your modifications in a running |swh|
instance, you should check the swh-environment docker `documentation`_.
.. _`documentation`: https://forge.softwareheritage.org/source/swh-environment/browse/master/docker/README.md?as=remarkup
.. _`virtualenv`: https://pypi.org/project/virtualenv/
Install required dependencies
-----------------------------
Software Heritage requires some dependencies that are usually packaged by your
package manager. On Debian/Ubuntu-based distributions::
sudo apt install lsb-release
sudo wget https://www.postgresql.org/media/keys/ACCC4CF8.asc -O /etc/apt/trusted.gpg.d/postgresql.asc
echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" | sudo tee -a /etc/apt/sources.list.d/pgdg.list
sudo wget https://downloads.apache.org/cassandra/KEYS -O /etc/apt/trusted.gpg.d/cassandra.asc
echo "deb https://downloads.apache.org/cassandra/debian 40x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.list
sudo apt update
sudo apt install \
build-essential pkg-config lzip rsync \
python3 python3-pip python3-venv virtualenvwrapper \
libpython3-dev libsystemd-dev libsvn-dev fuse3 libfuse3-dev \
git myrepos \
postgresql-autodoc graphviz plantuml inkscape \
postgresql libpq-dev cassandra
.. Note:: Python 3.7 or newer is required
This installs basic system utilities, Python library dependencies, development tools,
documentation tools and our main database management systems.
Cassandra and postgresql will be started by tests when they need it, so you
don't need them started globally (this will save you some RAM)::
sudo systemctl stop postgresql
sudo systemctl disable postgresql
sudo systemctl stop cassandra
sudo systemctl disable cassandra
If you intend to hack on the frontend part of |swh| Web Applications, you will also
need to have nodejs >= 12 in your development environment. If the version in your
Debian-based distribution is lower, you can install node 12 using these commands::
sudo wget https://deb.nodesource.com/gpgkey/nodesource.gpg.key -O /etc/apt/trusted.gpg.d/nodesource.asc
echo "deb https://deb.nodesource.com/node_12.x $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list.d/nodesource.list
sudo apt update
sudo apt install nodejs
+.. _checkout-source-code:
Checkout the source code
------------------------
Clone the |swh| environment repository::
~$ git clone https://forge.softwareheritage.org/source/swh-environment.git
[...]
~$ cd swh-environment
~/swh-environment$
Create a virtualenv::
~/swh-environment$ source /usr/share/virtualenvwrapper/virtualenvwrapper.sh
~/swh-environment$ mkvirtualenv -p /usr/bin/python3 -a $PWD swh
[...]
(swh) ~/swh-environment$
Checkout all the swh packages source repositories::
(swh) ~/swh-environment$ pip install pre-commit
(swh) ~/swh-environment$ ./bin/update
In the future you can re-activate the created virtualenv with::
$ workon swh
(swh) ~/swh-environment$
.. Note:: the above assumes you are using virtualenvwrapper_ to manage your
Python virtualenvs, but that is by no means mandatory. You can use plain
virtualenvs, or any other virtualenv management tool (pipenv_ or poetry_ for
example). virtualenvwrapper_ is our preference, but YMMV.
.. _virtualenvwrapper: https://virtualenvwrapper.readthedocs.io/
.. _poetry: https://poetry.eustace.io/
.. _pipenv: https://pipenv.readthedocs.io/
Install all the swh packages (in development mode, with testing dependencies)::
(swh) ~/swh-environment$ bin/install
.. note::
If you experience issues with :program:`pip` dependency resolution, try with
``bin/install --use-deprecated=legacy-resolver`` (the flag will be passed on
to ``pip install``). The same flag can also be set globally in
:file:`~/.config/pip/pip.conf`::
[install]
use-deprecated=legacy-resolver
Executing unit tests
--------------------
Unit tests are using the pytest_ framework, and can be executed directly or via
tox_. The main difference between these 2 test execution environments is:
- When executed via tox_, all the dependencies (including swh ones) are
installed from pypi_: you test your modifications against the latest
published version of every swh package but the current one.
- When you execute pytest_ directly, swh dependencies are used from your
current virtualenv, installed from the git repositories: you test your
modification against the HEAD of every swh package.
For example, running unit tests for the swh-loader-git_ package::
(swh) ~/swh-environment$ cd swh-loader-git
(swh) ~/swh-environment/swh-loader-git$ pytest
=========================== test session starts ============================
platform linux -- Python 3.5.3, pytest-3.8.2, py-1.6.0, pluggy-0.7.1
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/ddouard/src/swh-environment/swh-loader-git/.hypothesis/examples')
rootdir: /home/ddouard/src/swh-environment/swh-loader-git, inifile: pytest.ini
plugins: requests-mock-1.5.2, postgresql-1.3.4, env-0.6.2, django-3.4.7, cov-2.6.0, pylama-7.6.5, hypothesis-3.76.0, celery-4.2.1
collected 25 items
swh/loader/git/tests/test_converters.py ........ [ 32%]
swh/loader/git/tests/test_from_disk.py ..... [ 52%]
swh/loader/git/tests/test_loader.py ...... [ 76%]
swh/loader/git/tests/test_tasks.py ... [ 88%]
swh/loader/git/tests/test_utils.py ... [100%]
============================= warnings summary =============================
[...]
================== 25 passed, 12 warnings in 6.66 seconds ==================
Running the same test, plus code linting and static analysis, using tox::
(swh) ~/swh-environment/swh-loader-git$ tox
GLOB sdist-make: ~/swh-environment/swh-loader-git/setup.py
flake8 create: ~/swh-environment/swh-loader-git/.tox/flake8
flake8 installdeps: flake8
flake8 installed: entrypoints==0.3,flake8==3.7.7,mccabe==0.6.1,pycodestyle==2.5.0,pyflakes==2.1.1,swh.loader.git==0.0.48.post3
flake8 run-test-pre: PYTHONHASHSEED='2028963506'
flake8 runtests: commands[0] | ~/swh-environment/swh-loader-git/.tox/flake8/bin/python -m flake8
py3 create: ~/swh-environment/swh-loader-git/.tox/py3
py3 installdeps: .[testing], pytest-cov
py3 inst: ~/swh-environment/swh-loader-git/.tox/.tmp/package/1/swh.loader.git-0.0.48.post3.zip
py3 installed: aiohttp==3.5.4,amqp==2.4.2,arrow==0.13.1,async-timeout==3.0.1,atomicwrites==1.3.0,attrs==19.1.0,billiard==3.5.0.5,celery==4.2.1,certifi==2018.11.29,chardet==3.0.4,Click==7.0,coverage==4.5.2,decorator==4.3.2,dulwich==0.19.11,elasticsearch==6.3.1,Flask==1.0.2,idna==2.8,idna-ssl==1.1.0,itsdangerous==1.1.0,Jinja2==2.10,kombu==4.4.0,MarkupSafe==1.1.1,more-itertools==6.0.0,msgpack-python==0.5.6,multidict==4.5.2,pathlib2==2.3.3,pluggy==0.9.0,psutil==5.6.0,psycopg2==2.7.7,py==1.8.0,pytest==3.10.1,pytest-cov==2.6.1,python-dateutil==2.8.0,pytz==2018.9,PyYAML==3.13,requests==2.21.0,retrying==1.3.3,six==1.12.0,swh.core==0.0.55,swh.loader.core==0.0.39,swh.loader.git==0.0.48.post3,swh.model==0.0.30,swh.objstorage==0.0.30,swh.scheduler==0.0.49,swh.storage==0.0.129,systemd-python==234,typing-extensions==3.7.2,urllib3==1.24.1,vcversioner==2.16.0.0,vine==1.2.0,Werkzeug==0.14.1,yarl==1.3.0
py3 run-test-pre: PYTHONHASHSEED='2028963506'
py3 runtests: commands[0] | pytest --cov=swh --cov-branch
=========================== test session starts ============================
platform linux -- Python 3.5.3, pytest-3.10.1, py-1.8.0, pluggy-0.9.0
rootdir: ~/swh-environment/swh-loader-git, inifile: pytest.ini
plugins: cov-2.6.1, celery-4.2.1
collected 25 items
swh/loader/git/tests/test_converters.py ........ [ 32%]
swh/loader/git/tests/test_from_disk.py ..... [ 52%]
swh/loader/git/tests/test_loader.py ...... [ 76%]
swh/loader/git/tests/test_tasks.py ... [ 88%]
swh/loader/git/tests/test_utils.py ... [100%]
----------- coverage: platform linux, python 3.5.3-final-0 -----------
Name Stmts Miss Branch BrPart Cover
---------------------------------------------------------------------------
swh/__init__.py 1 0 0 0 100%
swh/loader/__init__.py 1 0 0 0 100%
swh/loader/git/__init__.py 0 0 0 0 100%
swh/loader/git/converters.py 102 10 44 7 86%
swh/loader/git/from_disk.py 157 44 50 6 67%
swh/loader/git/loader.py 271 59 114 17 75%
swh/loader/git/tasks.py 14 0 0 0 100%
swh/loader/git/tests/__init__.py 1 0 0 0 100%
swh/loader/git/tests/conftest.py 4 0 0 0 100%
swh/loader/git/tests/test_converters.py 94 0 6 0 100%
swh/loader/git/tests/test_from_disk.py 100 4 0 0 96%
swh/loader/git/tests/test_loader.py 12 0 0 0 100%
swh/loader/git/tests/test_tasks.py 26 0 0 0 100%
swh/loader/git/tests/test_utils.py 14 0 2 0 100%
swh/loader/git/utils.py 25 8 8 1 61%
---------------------------------------------------------------------------
TOTAL 822 125 224 31 80%
============================= warnings summary =============================
.tox/py3/lib/python3/site-packages/psycopg2/__init__.py:144
~/swh-environment/swh-loader-git/.tox/py3/lib/python3/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
""")
-- Docs: https://docs.pytest.org/en/latest/warnings.html
================== 25 passed, 1 warnings in 7.34 seconds ===================
_________________________________ summary __________________________________
flake8: commands succeeded
py3: commands succeeded
congratulations :)
Beware that some swh packages require a postgresql server properly configured
to execute the tests. In this case, you will want to use pifpaf_, which will
spawn a temporary instance of postgresql, to encapsulate the call to pytest.
For example, running pytest in the swh-core package::
(swh) ~/swh-environment$ cd swh-core
(swh) ~/swh-environment/swh-core$ pifpaf run postgresql -- pytest
=========================== test session starts ============================
platform linux -- Python 3.5.3, pytest-3.8.2, py-1.6.0, pluggy-0.7.1
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/ddouard/src/swh-environment/swh-core/.hypothesis/examples')
rootdir: /home/ddouard/src/swh-environment/swh-core, inifile: pytest.ini
plugins: requests-mock-1.5.2, postgresql-1.3.4, env-0.6.2, django-3.4.7, cov-2.6.0, pylama-7.6.5, hypothesis-3.76.0, celery-4.2.1
collected 79 items
swh/core/tests/test_api.py .. [ 2%]
swh/core/tests/test_config.py .............. [ 20%]
swh/core/tests/test_db.py .... [ 25%]
swh/core/tests/test_logger.py . [ 26%]
swh/core/tests/test_serializers.py ..... [ 32%]
swh/core/tests/test_statsd.py ...................................... [ 81%]
........ [ 91%]
swh/core/tests/test_utils.py ....... [100%]
======================== 79 passed in 6.59 seconds =========================
Sending your changes
--------------------
After you are done making the changes you want, you can send them on our
forge_. The best way to do that is to use Arcanist, so we recommend you read
these two guides if you are not familiar with it:
* https://wiki.softwareheritage.org/wiki/Arcanist_setup
* https://wiki.softwareheritage.org/wiki/Code_review_in_Phabricator
.. _pytest: https://pytest.org
.. _tox: https://tox.readthedocs.io
.. _pypi: https://pypi.org
.. _swh-loader-git: https://forge.softwareheritage.org/source/swh-loader-git
.. _pifpaf: https://github.com/jd/pifpaf
.. _forge: https://forge.softwareheritage.org/
diff --git a/docs/faq/index.rst b/docs/faq/index.rst
index 9c586a0..38f3e34 100644
--- a/docs/faq/index.rst
+++ b/docs/faq/index.rst
@@ -1,252 +1,254 @@
.. _faq:
Frequently Asked Questions
**************************
.. _faq_prerequisites:
Prerequisites for code contributions
====================================
What are the Skills required to be a code contributor?
------------------------------------------------------
-It depends on what area you want to work on. `Internships
-<https://wiki.softwareheritage.org/wiki/Internships>`__ postings list specific skills
+It depends on what area you want to work on. The `Internships postings <interships>`
+list specific skills
required. Generally, only Python and basic Git knowledge are required. Feel free to
contact us on one of the `development channels
<https://www.softwareheritage.org/community/developers/>`__ for details.
What are the minimum system requirements (hardware/software) to run SWH locally?
--------------------------------------------------------------------------------
-Python3.7 or newer is required. See the `developer setup documentation
-<https://docs.softwareheritage.org/devel/developer-setup.html#developer-setup>`__ for
-more details.
+Python3.7 or newer is required. See the :ref:`developer setup documentation
+<developer-setup>` for more details.
.. _faq_getting_started:
Getting Started
===============
What are the must read docs before I start contributing?
--------------------------------------------------------
-We recommend you read the top links listed at from the `documentation home page
-<https://docs.softwareheritage.org/devel/index.html>`__ in order: getting started,
+We recommend you read the top links listed at from the :ref:`documentation home page
+<swh-docs>` in order: getting started,
contributing, and architecture, as well as the data model.
Where can I see the getting started guide for developers?
---------------------------------------------------------
See :ref:`getting-started` tutorial.
How do I find an easy ticket to get started?
--------------------------------------------
We maintain a `list of easy tickets
<https://forge.softwareheritage.org/maniphest/query/WcCLxlHnXok9/>`__ to work on, see
the `Easy hacks page <https://wiki.softwareheritage.org/wiki/Easy_hacks>`__ for more
details.
I am skilled in one specific technology, can I find tickets requiring that skill?
---------------------------------------------------------------------------------
Unfortunately, not at the moment. But you can look at the `Internship list
-<https://wiki.softwareheritage.org/wiki/Internships>`__ to look for something matching
+<interships>`_ to look for something matching
this skill, and this may allow you to find topics to search for in the `bug tracking
-system <https://forge.softwareheritage.org/>`__.
+system`_.
-Either way, feel free to `contact our developers
-<https://www.softwareheritage.org/community/developers/>`__, we would love to work with
+Either way, feel free to contact our developers through any of the
+`development channels`_, we would love to work with
you.
Where should I ask for technical help?
--------------------------------------
You can choose one of the following:
-* `development channels <https://www.softwareheritage.org/community/developers/>`__
-* `contact form for any enquiries <https://www.softwareheritage.org/contact/>`__
+* `development channels`_
+* `contact form` for any enquiries
.. _faq_run_swh:
Running an SWH instance locally
===============================
I have SWH stack running in my local. How do I get some initial data to play around?
------------------------------------------------------------------------------------
-You can setup a job on your local machine, for this you can `schedule a listing task
-<https://docs.softwareheritage.org/devel/getting-started/using-docker.html#inserting-a-new-lister-task>`__
+You can setup a job on your local machine, for this you can
+:ref:`schedule a listing task <docker-schedule-lister-task>`
for example. Doing so on small forge, will allow you to load some repositories.
-Or you can also trigger directly `loading from the cli
-<https://docs.softwareheritage.org/devel/swh-loader-core/package-loader-tutorial.html#with-docker>`__.
+Or you can also trigger directly :ref:`loading from the cli <docker-run-loader-cli>`.
I have a SWH stack running in local, How do I setup a lister/loader job?
------------------------------------------------------------------------
-See the `"Managing tasks" chapter
-<https://docs.softwareheritage.org/devel/getting-started/using-docker.html#managing-tasks>`__
+See the :ref:`"Managing tasks" chapter <docker-manage-tasks>`
in the Docker environment documentation.
How can I create a user in my local instance?
---------------------------------------------
We cannot right now. Stay either anonymous or use the user "test" (password "test") or
the user ambassador (password "ambassador").
Should I run/test the web app in any particular browser?
--------------------------------------------------------
We expect the web app to work on all major browsers. It uses mostly straightforward
HTML/CSS and a little Javascript for search and source code highlighting, so testing in
a single browser is usually enough.
.. _faq_dataset:
Getting sample datasets
=======================
Is there a way to connect to SWH archived (production) database from my local machine?
--------------------------------------------------------------------------------------
-We provide the archive as a dataset on public clouds, see the `swh-dataset
-documentation <https://docs.softwareheritage.org/devel/swh-dataset/index.html>`__. We can
-also provide read access to one of the main databases on request `contact
-us <https://www.softwareheritage.org/contact/>`__.
+We provide the archive as a dataset on public clouds, see the :ref:`swh-dataset
+documentation <swh-dataset>`. We can
+also provide read access to one of the main databases on request, `contact us`_.
.. _faq_error_bugs:
Errors and bugs
===============
I found a bug/improvement in the system, where should I report it?
------------------------------------------------------------------
-Please report it on our `bug tracking system <https://forge.softwareheritage.org/>`__.
+Please report it on our `bug tracking system`_.
First create an account, then create a bug report using the "Create task" button. You
should get some feedback within a week (at least someone triaging your issue). If not,
-`get in touch with us <https://www.softwareheritage.org/community/developers/>`__ to
+`get in touch with us <development channels>`_ to
make sure we did not miss it.
.. _faq_legal:
Legal matters
=============
Do I need to sign a form to contribute code?
--------------------------------------------
Yes, on your first diff, you will have to sign such document.
As long as it's not signed, your diff content won't be visible.
Will my name be added to a CONTRIBUTORS file?
---------------------------------------------
You will be asked during review to add yourself.
.. _faq_code_review:
Code Review
===========
I found a straightforward typo fix, should my fix go through the entire code review process?
--------------------------------------------------------------------------------------------
You are welcome to drop us a message at one of the `development
-channels <https://www.softwareheritage.org/community/developers/>`__, we will pick it up
-and fix it so you don't have to follow the whole `code review
-process <https://docs.softwareheritage.org/devel/contributing/phabricator.html>`__.
+channels`_, we will pick it up
+and fix it so you don't have to follow the whole :ref:`code review process <patch-submission>`.
What tests I should run before committing the code?
---------------------------------------------------
Mostly run `tox` (or `pytest`) to run the unit tests suite. When you will propose a
patch in our forge, the continuous integration factory will trigger a build (using `tox`
as well).
I am getting errors while trying to commit. What is going wrong?
----------------------------------------------------------------
-Ensure you followed the proper guide to `setup your
-environment <https://docs.softwareheritage.org/devel/developer-setup.html#checkout-the-source-code>`__
+Ensure you followed the proper guide to :ref:`setup your
+environment <checkout-source-code>`
and try again. If the error persists, you are welcome to drop us a message at one of the
-`development channels <https://www.softwareheritage.org/community/developers/>`__
+`development channels`_
Is there a format/guideline for writing commit messages?
--------------------------------------------------------
-See the `Git style guide <https://docs.softwareheritage.org/devel/contributing/git-style-guide.html>`__
+See the :ref:`git-style-guide`
Is there some recommended git branching strategy?
-------------------------------------------------
It's left at the developer's discretion. Mostly people hack on their feature, then
propose a diff from a git branch or directly from the master branch. There is no
imperative. The only imperative is that for a feature to be packaged and deployed, it
needs to land first in the master branch.
how should I document the code I contribute to SWH?
---------------------------------------------------
Any new feature should include documentation in the form of comments and/or docstrings.
Ideally, they should also be documented in plain English in the repository's `docs/`
folder if relevant to a single package, or in the main `swh-docs` repository if it is a
transversal feature.
.. _faq_api:
Software Heritage API
=====================
How do I generate API usage credentials?
----------------------------------------
-See the `Authentication guide <https://docs.softwareheritage.org/devel/swh-web-client/index.html#authentication>`__.
+See the :ref:`Authentication guide <swh-web-client-auth>`.
Is there a page where I can see all the API endpoints?
------------------------------------------------------
-See the `API endpoint listing page <https://archive.softwareheritage.org/api/1/>`__.
+See the `API endpoint listing page <API>`_.
What are the usage limits for SWH APIs?
---------------------------------------
Maximum number of permitted requests per hour:
* 120 for anonymous users
* 1200 for authenticated users
-It's described in the `rate limit documentation page
-<https://archive.softwareheritage.org/api/#rate-limiting>`__.
+It's described in the `rate limit documentation page <API-rate-limit>`_.
.. It's temporarily here but it should be moved into its own sphinx instance at some
point in the future.
.. _faq_sysadm:
System Administration
=====================
How does SWH release?
---------------------
Release is mostly done:
- first in docker (somewhat as part of the development process)
- secondly packaged and deployed on staging (mostly)
- thirdly the same package is deployed on production
Is there a release cycle?
-------------------------
When a functionality is ready (tests ok, landed in master, docker run ok), the module is
tagged. The tag is pushed. This triggers a packaging build process. When the package is
ready, depending on the module [1], sysadms deploy the package with the help of puppet.
[1] swh-web module is mostly automatic. Other modules are not yet automatic as some
internal state migration (dbs) often enters the release cycle and due to the data
volume, that may need human intervention.
+
+.. _API: https://archive.softwareheritage.org/api/1/
+.. _API-rate-limit: https://archive.softwareheritage.org/api/#rate-limiting
+.. _bug tracking system: https://forge.softwareheritage.org/
+.. _contact form: https://www.softwareheritage.org/contact/
+.. _contact us: https://www.softwareheritage.org/contact/
+.. _development channels: https://www.softwareheritage.org/community/developers/
+.. _internships: https://wiki.softwareheritage.org/wiki/Internships
diff --git a/docs/index.rst b/docs/index.rst
index 73db53f..7d91bb7 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,235 +1,235 @@
-
+.. _swh-docs:
Software Heritage - Development Documentation
=============================================
Getting started
---------------
* :ref:`getting-started` → deploy a local copy of the Software Heritage
software stack in less than 5 minutes, or
* :ref:`developer-setup` → get a working development setup that allows to hack
on the Software Heritage software stack
Contributing
------------
* :ref:`patch-submission` → learn how to submit your patches to the
Software Heritage codebase
* :ref:`code-review` → rules and guidelines to review code in
Software Heritage
* :ref:`python-style-guide` → how to format the Python code you write
Architecture
------------
* :ref:`architecture-overview` → get a glimpse of the Software Heritage software
architecture
* :ref:`mirror` → learn what a Software Heritage mirror is and how to set up
one
* :ref:`Keycloak <keycloak>` → learn how to use Keycloak,
the authentication system used by |swh|'s web interface and public APIs
Data Model and Specifications
-----------------------------
* :ref:`persistent-identifiers` Specifications of the SoftWare Heritage persistent IDentifiers (SWHID).
* :ref:`data-model` Documentation of the main |swh| archive data model.
* :ref:`journal-specs` Documentation of the Kafka journal of the |swh| archive.
Tutorials
---------
* :ref:`testing-guide`
* :doc:`/tutorials/issue-debugging-monitoring`
* :ref:`Listing the content of your favorite forge <lister-tutorial>`
and :ref:`running a lister in Docker <run-lister-tutorial>`
Frequently Asked Questions
--------------------------
.. toctree::
:maxdepth: 2
faq/index
Roadmap
-------
* :ref:`roadmap-2021`
Engineering
-----------
* :ref:`infrastructure`
Components
----------
Here is brief overview of the most relevant software components in the Software
Heritage stack, in alphabetical order.
For a better introduction to the architecture, see the :ref:`architecture-overview`,
which presents each of them in a didactical order.
Each component name is linked to the development documentation
of the corresponding Python module.
:ref:`swh.auth <swh-auth>`
low-level library used by modules needing keycloak authentication
:ref:`swh.core <swh-core>`
low-level utilities and helpers used by almost all other modules in the
stack
:ref:`swh.counters <swh-counters>`
service providing efficient estimates of the number of objects in the SWH archive,
using Redis's Hyperloglog
:ref:`swh.dataset <swh-dataset>`
public datasets and periodic data dumps of the archive released by Software
Heritage
:ref:`swh.deposit <swh-deposit>`
push-based deposit of software artifacts to the archive
swh.docs
developer documentation (used to generate this doc you are reading)
:ref:`swh.fuse <swh-fuse>`
Virtual file system to browse the Software Heritage archive, based on
`FUSE <https://github.com/libfuse/libfuse>`_
:ref:`swh.graph <swh-graph>`
Fast, compressed, in-memory representation of the archive, with tooling to
generate and query it.
:ref:`swh.indexer <swh-indexer>`
tools and workers used to crawl the content of the archive and extract
derived information from any artifact stored in it
:ref:`swh.journal <swh-journal>`
persistent logger of changes to the archive, with publish-subscribe support
:ref:`swh.lister <swh-lister>`
collection of listers for all sorts of source code hosting and distribution
places (forges, distributions, package managers, etc.)
:ref:`swh.loader-core <swh-loader-core>`
low-level loading utilities and helpers used by all other loaders
:ref:`swh.loader-git <swh-loader-git>`
loader for `Git <https://git-scm.com/>`_ repositories
:ref:`swh.loader-mercurial <swh-loader-mercurial>`
loader for `Mercurial <https://www.mercurial-scm.org/>`_ repositories
:ref:`swh.loader-svn <swh-loader-svn>`
loader for `Subversion <https://subversion.apache.org/>`_ repositories
:ref:`swh.model <swh-model>`
implementation of the :ref:`data-model` to archive source code artifacts
:ref:`swh.objstorage <swh-objstorage>`
content-addressable object storage
:ref:`swh.objstorage.replayer <swh-objstorage-replayer>`
Object storage replication tool
:ref:`swh.scanner <swh-scanner>`
source code scanner to analyze code bases and compare them with source code
artifacts archived by Software Heritage
:ref:`swh.scheduler <swh-scheduler>`
task manager for asynchronous/delayed tasks, used for recurrent (e.g.,
listing a forge, loading new stuff from a Git repository) and one-off
activities (e.g., loading a specific version of a source package)
:ref:`swh.search <swh-search>`
search engine for the archive
:ref:`swh.storage <swh-storage>`
abstraction layer over the archive, allowing to access all stored source
code artifacts as well as their metadata
:ref:`swh.vault <swh-vault>`
implementation of the vault service, allowing to retrieve parts of the
archive as self-contained bundles (e.g., individual releases, entire
repository snapshots, etc.)
:ref:`swh.web <swh-web>`
Web application(s) to browse the archive, for both interactive (HTML UI)
and mechanized (REST API) use
:ref:`swh.web.client <swh-web-client>`
Python client for :ref:`swh.web <swh-web>`
Dependencies
------------
The dependency relationships among the various modules are depicted below.
.. _py-deps-swh:
.. figure:: images/py-deps-swh.svg
:width: 1024px
:align: center
Dependencies among top-level Python modules (click to zoom).
Archive
-------
* :ref:`Archive ChangeLog <archive-changelog>`: notable changes to the archive
over time
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* `URLs index <http-routingtable.html>`_
* :ref:`search`
* :ref:`glossary`
.. ensure sphinx does not complain about index files not being included
.. toctree::
:maxdepth: 2
:caption: Contents:
:titlesonly:
:hidden:
getting-started/index
architecture/index
contributing/index
tutorials/index
faq/index
roadmap/roadmap-2021.rst
infrastructure/index
swh.auth <swh-auth/index>
swh.core <swh-core/index>
swh.counters <swh-counters/index>
swh.dataset <swh-dataset/index>
swh.deposit <swh-deposit/index>
swh.fuse <swh-fuse/index>
swh.graph <swh-graph/index>
swh.indexer <swh-indexer/index>
swh.journal <swh-journal/index>
swh.lister <swh-lister/index>
swh.loader <swh-loader>
swh.model <swh-model/index>
swh.objstorage <swh-objstorage/index>
swh.objstorage.replayer <swh-objstorage-replayer/index>
swh.scanner <swh-scanner/index>
swh.scheduler <swh-scheduler/index>
swh.search <swh-search/index>
swh.storage <swh-storage/index>
swh.vault <swh-vault/index>
swh.web <swh-web/index>
swh.web.client <swh-web-client/index>
archive-changelog
journal
Python modules autodocumentation <apidoc/modules>