Page MenuHomeSoftware Heritage

tox: Add a sphinx environment to check sane doc build
ClosedPublic

Authored by anlambert on Apr 16 2021, 7:16 PM.

Details

Summary

Now that all sphinx warnings have been fixed by @vlorentz, it would
be great to avoid introducing new ones as much as possible.

This diff showcases how a sphinx environment can be added to tox
to build and check the documentation proper to a swh package.

As some external references could not be resolved in a single package
scope, related sphinx warnings are suppressed for the build to succeed
but other ones are not.

This would allow to catch any rst errors prior landing a diff.

Ideally a new parallel step in our Jenkins pipelines to build and
check the doc should be added. We already have a CI docker image
containing all the required tools to build the doc so this should
not be too complicated to integrate.

Depends on D5571

Diff Detail

Event Timeline

Build is green

Patch application report for D5550 (id=19820)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit 3ad3315a657bb79109daf4610a92d32c18e61b9c
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/211/ for more details.

olasd added inline comments.
tox.ini
70

the leading slash in /../.tox/sphinx/... is weird; not sure the {envdir} substitution will work there?

76

that's a lot of code to add to every package :/

But it's also the easiest solution to solve T3258, so I don't know...

As mentioned on IRC, I think you could remove ref.term from the suppressed warnings by including glossary.rst

that's a lot of code to add to every package :/

But it's also the easiest solution to solve T3258, so I don't know...

As mentioned on IRC, I think you could remove ref.term from the suppressed warnings by including glossary.rst

I have an idea to avoid leaking the overridden sphinx configuration in every tox file and how to include the glossary, I will update that diff once it is tested.

tox.ini
76

I already tried but for some reasons it does not work:

(swh) ✘-1 /tmp/swh-core [tox-sphinx L|✚ 1⚑ 1] 
16:54 $ tox -e sphinx
sphinx develop-inst-noop: /tmp/swh-core
sphinx installed: aiohttp==3.7.4.post0,aiohttp-utils==3.1.1,alabaster==0.7.12,appdirs==1.4.4,async-timeout==3.0.1,attrs==20.3.0,Babel==2.9.0,blinker==1.4,certifi==2020.12.5,cfgv==3.2.0,chardet==4.0.0,click==7.1.2,Deprecated==1.2.12,distlib==0.3.1,docutils==0.16,filelock==3.0.12,Flask==1.1.2,gunicorn==20.1.0,hypothesis==6.10.0,identify==2.2.4,idna==2.10,imagesize==1.2.0,importlib-metadata==4.0.1,iniconfig==1.1.1,iso8601==0.1.14,itsdangerous==1.1.0,Jinja2==2.11.3,markdown-it-py==0.6.2,MarkupSafe==1.1.1,mdit-py-plugins==0.2.6,mirakuru==2.3.0,msgpack==1.0.2,multidict==5.1.0,myst-parser==0.13.6,nodeenv==1.6.0,packaging==20.9,pluggy==0.13.1,port-for==0.4,pre-commit==2.12.1,psutil==5.8.0,psycopg2==2.8.6,py==1.10.0,Pygments==2.8.1,pyparsing==2.4.7,pytest==6.2.3,pytest-mock==3.5.1,pytest-postgresql==2.6.1,python-mimeparse==1.6.0,pytz==2021.1,PyYAML==5.4.1,requests==2.25.1,requests-mock==1.8.0,sentry-sdk==1.0.0,six==1.15.0,snowballstemmer==2.1.0,sortedcontainers==2.3.0,Sphinx==3.5.4,sphinx-celery==2.0.0,sphinx-click==2.7.1,sphinx-reredirects==0.0.0,sphinx-rtd-theme==0.5.2,sphinx-tabs==2.1.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==1.0.3,sphinxcontrib-httpdomain==1.7.0,sphinxcontrib-images==0.9.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-programoutput==0.17,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.4,-e git+https://forge.softwareheritage.org/source/swh-core.git@3ad3315a657bb79109daf4610a92d32c18e61b9c#egg=swh.core,-e git+file:///home/anlambert/swh/swh-environment/swh-docs@ea3ceb6b60fc113a40bd0501c9d1b31a518338e1#egg=swh.docs,systemd-python==234,toml==0.10.2,typing-extensions==3.7.4.3,urllib3==1.26.4,virtualenv==20.4.4,Werkzeug==1.0.1,wrapt==1.12.1,yarl==1.6.3,zipp==3.4.1
sphinx run-test-pre: PYTHONHASHSEED='2758786654'
sphinx run-test: commands[0] | make -I /tmp/swh-core/.tox/sphinx/src/swh-docs/swh/ -C docs
make: Entering directory '/tmp/swh-core/docs'
Makefile:1: ../../swh-docs/Makefile.sphinx: No such file or directory
make: *** No rule to make target '../../swh-docs/Makefile.sphinx'.  Stop.
make: Leaving directory '/tmp/swh-core/docs'
ERROR: InvocationError for command /usr/bin/make -I .tox/sphinx/src/swh-docs/swh -C docs (exited with code 2)
____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
ERROR:   sphinx: commands failed

Update: Simplify sphinx environment definition:

  • Define SWH_PACKAGE_DOC_TOX_BUILD environment variable
  • All overriden sphinx configuration is now handled in the conf.py file of swh-docs based on the value of the above variable

It enables to avoid the duplication of overriden sphinx configuration
and to centralize dedicated sphinx tweaks for tox build in a single place.

Build is green

Patch application report for D5550 (id=19905)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit d33a39e5f3c6e17002266aa39699b5e89fcf30a6
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/212/ for more details.

Build is green

Patch application report for D5550 (id=19907)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit 9c579e2935c6dff72f0e792785cacc5ec1ca4a06
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/213/ for more details.

Fix make include dir, {envdir} tox substitution makes the build failed.

Build is green

Patch application report for D5550 (id=19910)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit 9b5dfe6bbf9564f50bd58688e60872966c9cec57
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/214/ for more details.

Another random idea: what about -e ../swh-docs#egg=swh.docs (or a variant that works, I didn't try it) instead of `-e git+https://forge.softwareheritage.org/source/swh-docs#egg=swh.docs ?

This way it runs faster, spares some bandwidth, works without internet access, and allows local modifications in swh-docs

Another random idea: what about -e ../swh-docs#egg=swh.docs (or a variant that works, I didn't try it) instead of `-e git+https://forge.softwareheritage.org/source/swh-docs#egg=swh.docs ?

This way it runs faster, spares some bandwidth, works without internet access, and allows local modifications in swh-docs

This will not work when building the standalone doc on Jenkins.

How about adding a sphinx-dev tox environment that will fetch swh-docs on the local swh-environment ?

This will not work when building the standalone doc on Jenkins.

Oh, indeed.

How about adding a sphinx-dev tox environment that will fetch swh-docs on the local swh-environment ?

Sounds good! And the improved consistency in environment names is nice too.

This will not work when building the standalone doc on Jenkins.

Oh, indeed.

How about adding a sphinx-dev tox environment that will fetch swh-docs on the local swh-environment ?

Sounds good! And the improved consistency in environment names is nice too.

Let's go for it then ! I will also update swh-docs README in D5571 to use the sphinx-dev env.

Next steps:

  • modifiy tox.ini in all swh packages
  • modify Jenkins pipelines to add a parallel step for building the documentation

Add sphinx-dev tox environment and update commit message.

Build is green

Patch application report for D5550 (id=19968)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit 4383ecf1c1f6ccaed9209711d90434a711a8a61d
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add sphinx environments to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    The sphinx environment is designed to be used in continuous integration.
    
    The sphinx-dev environment is designed to be used inside a full swh
    development environment.
    
    Related to T3258

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/215/ for more details.

Could you add a comment or two, to explain the difference between the two envs?

Build is green

Patch application report for D5550 (id=19972)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit 9ba33def779d70197d40c212d69bc363bcf0b0e8
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add sphinx environments to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    The sphinx environment is designed to be used in continuous integration.
    
    The sphinx-dev environment is designed to be used inside a full swh
    development environment.
    
    Related to T3258

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/217/ for more details.

tox.ini
52–53

Could you also explain this is used on the CI because it does not have ../swh-docs/?

Build is green

Patch application report for D5550 (id=19975)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit df31524e35690816be57c17800e48d019d6575fd
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add sphinx environments to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    The sphinx environment is designed to be used in continuous integration
    in order to prevent breaking documentation build when committing changes.
    
    The sphinx-dev environment is designed to be used inside a full swh
    development environment.
    
    Related to T3258

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/218/ for more details.

tox.ini
52–53

I mentioned this is to build the doc outside swh-environment which is equivalent.

This revision is now accepted and ready to land.Apr 23 2021, 4:27 PM

Landed ! I will handle other swh repositories and update Jenkins pipeline next week.