diff --git a/README.md b/README.md index e94f256..3e1c8ca 100644 --- a/README.md +++ b/README.md @@ -1,107 +1,124 @@ swh-docs ======== This module contains (the logics for generating) the Software Heritage development documentation. Specifically, it contains some general information about Software Heritage internals (stuff that would not fit in any other specific software component of the Software Heritage stack) and bundle them together component-specific documentation coming from other modules of the stack. All documentation is written and typeset using [Sphinx][1]. General documentation is shipped as part of this module. Module-specific documentation is centralized here via symlinks to the `docs/` dirs of individual modules. Therefore to build the full documentation you need a working and complete [Software Heritage development environment][2]. How to build the doc -------------------- Install the [Software Heritage development environment][2] $ git clone https://forge.softwareheritage.org/source/swh-environment $ cd swh-environment $ ./bin/update # this will clone needed git repos, inc. swh-docs $ cd swh-docs Ensure you have the required tools to generate images ([graphviz][3]'s `dot`, [plantuml][4] and [inkscape][5]). On a Debian system: $ sudo apt install plantuml graphviz These additional packages are required on Debian 10.x systems: - libapr1-dev - libaprutil1-dev - libsvn-dev - postgresql-11 - dia - postgresql-autodoc It is also recommended to build the doc using [tox][6], so ensure you have it installed, eg. on a Debian system: $ sudo apt install tox Then (from the `swh-environment/swh-docs/` directory): $ tox -e sphinx-dev This tox environment will build the documentation from the sources available in the parent directory (`swh-environment`). Behind the scene, this tox environment will run the sphinx documentation building process via [pifpaf][7] to encapsulate the need os Postgresql to generate database schemas. The documentation building process itself consists mainly in 3 steps: ### 1. Generate documentation assets for all modules $ cd swh-environment $ make docs-assets This will *not* build the documentation in each module (there is `make docs` for that). ### 2. Build the api docs for all swh python packages $ cd swh-docs/docs $ make apidoc ### 3. Build the documentation $ cd swh-docs/docs $ make The HTML documentation is now available starting from `_build/html/index.html`. Cleaning up ----------- $ cd docs $ make distclean The former (`make clean`) will only clean the local Sphinx build, without touching other modules. The latter (`make distclean`) will also clean Sphinx builds in all other modules. Publishing the doc ------------------ The publication of the documentation is now managed by the [CI][7]. +Building standalone package documentation +----------------------------------------- + +Each documentation local to a swh package can also be built with [tox][6]. + +For instance to build the standalone documentation of ``swh-web``, proceed as +follows: + + $ cd swh-environment/swh-web + $ tox -e sphinx-dev + +Sphinx warnings related to unresolved references located in other swh packages are suppressed because expected. + +Please also note that Sphinx warnings are turned into errors in that case. + +The HTML documentation is now available starting from `docs/_build/html/index.html`. + [1]: http://www.sphinx-doc.org/ [2]: https://forge.softwareheritage.org/source/swh-environment/ [3]: https://graphviz.org [4]: http://plantuml.com [5]: https://inkscape.org/ [6]: https://tox.readthedocs.io/ -[7]: https://github.com/jd/pifpaf +[7]: https://jenkins.softwareheritage.org/job/DDOC/ diff --git a/swh/docs/sphinx/conf.py b/swh/docs/sphinx/conf.py index 77addec..a00446a 100755 --- a/swh/docs/sphinx/conf.py +++ b/swh/docs/sphinx/conf.py @@ -1,237 +1,283 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- # +import logging import os from typing import Dict from sphinx.ext import autodoc from swh.docs.django_settings import force_django_settings # General information about the project. project = "Software Heritage - Development Documentation" copyright = "2015-2021 The Software Heritage developers" author = "The Software Heritage developers" # -- General configuration ------------------------------------------------ # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ "sphinx.ext.autodoc", "sphinx.ext.napoleon", "sphinxcontrib.httpdomain", "sphinx.ext.extlinks", "sphinxcontrib.images", "sphinxcontrib.programoutput", "sphinx.ext.viewcode", "sphinx_tabs.tabs", "sphinx_rtd_theme", "sphinx.ext.graphviz", "sphinx_click.ext", "myst_parser", "sphinx.ext.todo", "sphinx_reredirects", "swh.docs.sphinx.view_in_phabricator", # swh.scheduler inherits some attribute descriptions from celery that use # custom crossrefs (eg. :setting:`task_ignore_result`) "sphinx_celery.setting_crossref", ] # Add any paths that contain templates here, relative to this directory. templates_path = ["_templates"] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # source_suffix = ".rst" # The master toctree document. master_doc = "index" # A string of reStructuredText that will be included at the beginning of every # source file that is read. # A bit hackish but should work both for each swh package and the whole swh-doc rst_prolog = """ .. include:: /../../swh-docs/docs/swh_substitutions """ # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. version = "" # The full version, including alpha/beta/rc tags. release = "" # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. language = "en" # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This patterns also effect to html_static_path and html_extra_path exclude_patterns = ["_build", "swh-icinga-plugins/index.rst"] # The name of the Pygments (syntax highlighting) style to use. pygments_style = "sphinx" # If true, `todo` and `todoList` produce output, else they produce nothing. todo_include_todos = True # -- Options for HTML output ---------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # html_theme = "sphinx_rtd_theme" html_favicon = "_static/favicon.ico" # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. # html_theme_options = { "collapse_navigation": True, "sticky_navigation": True, } html_logo = "_static/software-heritage-logo-title-motto-vertical-white.png" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ["_static"] # make logo actually appear, avoiding gotcha due to alabaster default conf. # https://github.com/bitprophet/alabaster/issues/97#issuecomment-303722935 html_sidebars = { "**": [ "about.html", "globaltoc.html", "relations.html", "sourcelink.html", "searchbox.html", ] } # If not None, a 'Last updated on:' timestamp is inserted at every page # bottom, using the given strftime format. # The empty string is equivalent to '%b %d, %Y'. html_last_updated_fmt = "%Y-%m-%d %H:%M:%S %Z" # refer to the Python standard library. intersphinx_mapping = {"python": ("https://docs.python.org/3", None)} # Redirects for pages that were moved, so we don't break external links. # Uses sphinx-reredirects redirects = { "swh-deposit/spec-api": "api/api-documentation.html", "swh-deposit/metadata": "api/metadata.html", "swh-deposit/specs/blueprint": "../api/use-cases.html", "swh-deposit/user-manual": "api/user-manual.html", "architecture": "architecture/overview.html", "mirror": "architecture/mirror.html", } # -- autodoc configuration ---------------------------------------------- autodoc_default_flags = [ "members", "undoc-members", "private-members", "special-members", ] autodoc_member_order = "bysource" autodoc_mock_imports = ["rados"] autoclass_content = "both" modindex_common_prefix = ["swh."] # For the todo extension. Todo and todolist produce output only if this is True todo_include_todos = True _swh_web_base_url = "https://archive.softwareheritage.org" # for the extlinks extension, sub-projects should fill that dict extlinks: Dict = { "swh_web": (f"{_swh_web_base_url}/%s", None), "swh_web_api": (f"{_swh_web_base_url}/api/1/%s", None), "swh_web_browse": (f"{_swh_web_base_url}/browse/%s", None), } +# SWH_PACKAGE_DOC_TOX_BUILD environment variable is set in a tox environment +# named sphinx for each swh package (except the swh-docs package itself). +swh_package_doc_tox_build = os.environ.get("SWH_PACKAGE_DOC_TOX_BUILD", False) + +# override some configuration when building a swh package +# documentation with tox to remove warnings and suppress +# those related to unresolved references +if swh_package_doc_tox_build: + swh_substitutions = os.path.join( + os.path.dirname(__file__), "../../../docs/swh_substitutions" + ) + rst_prolog = f".. include:: /{swh_substitutions}" + suppress_warnings = ["ref.ref"] + html_favicon = "" + html_logo = "" + class SimpleDocumenter(autodoc.FunctionDocumenter): """ Custom autodoc directive to inline the docstring of a function in a document without the signature header and with no indentation. Example of use:: .. autosimple:: swh.web.api.views.directory.api_directory """ objtype = "simple" # ensure the priority is lesser than the base FunctionDocumenter # to avoid side effects with autodoc processing priority = -1 # do not indent the content content_indent = "" # do not add a header to the docstring def add_directive_header(self, sig): pass # XXX Kill this ASA this PR is accepted and released # https://github.com/sphinx-contrib/httpdomain/pull/19 def register_routingtable_as_label(app, document): from sphinx.locale import _ # noqa labels = app.env.domaindata["std"]["labels"] labels["routingtable"] = "http-routingtable", "", _("HTTP Routing Table") anonlabels = app.env.domaindata["std"]["anonlabels"] anonlabels["routingtable"] = "http-routingtable", "" # sphinx event handler to set adequate django settings prior reading # apidoc generated rst files when building doc to avoid autodoc errors def set_django_settings(app, env, docname): package_settings = { "auth": "swh.auth.tests.django.app.apptest.settings", "deposit": "swh.deposit.settings.development", "web": "swh.web.settings.development", } for package, settings in package_settings.items(): if any( [pattern in docname for pattern in (f"swh.{package}", f"swh-{package}")] ): force_django_settings(settings) +# when building local package documentation with tox, insert glossary +# content at the end of the index file in order to resolve references +# to the terms it contains +def add_glossary_to_index(app, docname, source): + if docname == "index": + glossary_path = os.path.join( + os.path.dirname(__file__), "../../../docs/glossary.rst" + ) + with open(glossary_path, "r") as glossary: + source[0] += "\n" + glossary.read() + + def setup(app): # env-purge-doc event is fired before source-read app.connect("env-purge-doc", set_django_settings) # add autosimple directive (used in swh-web) app.add_autodocumenter(SimpleDocumenter) # set an environment variable indicating we are currently building # the documentation os.environ["SWH_DOC_BUILD"] = "1" # register routingtable label for sphinxcontrib-httpdomain <= 1.7.0 from distutils.version import StrictVersion # noqa import pkg_resources # noqa httpdomain = pkg_resources.get_distribution("sphinxcontrib-httpdomain") if StrictVersion(httpdomain.version) <= StrictVersion("1.7.0"): app.connect("doctree-read", register_routingtable_as_label) + + if swh_package_doc_tox_build: + # ensure glossary will be available in package doc scope + app.connect("source-read", add_glossary_to_index) + + # suppress httpdomain warnings in non web packages + if "swh-web" not in app.srcdir: + + # filter out httpdomain unresolved reference warnings + # to not consider them as errors when using -W option of sphinx-build + class HttpDomainWarningFilter(logging.Filter): + def filter(self, record: logging.LogRecord) -> bool: + return not record.msg.startswith("Cannot resolve reference to") + + logger = logging.getLogger("sphinx") + # insert a custom filter in the warning log handler of sphinx + logger.handlers[1].filters.insert(0, HttpDomainWarningFilter())