Page MenuHomeSoftware Heritage
Differential D5550

tox: Add a sphinx environment to check sane doc build
ClosedPublic
Actions

Authored by anlambert on Apr 16 2021, 7:16 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T3258: Build documentation on every diff
Commits
rDCOREdf31524e3569: tox: Add sphinx environments to check sane doc build
Summary

Now that all sphinx warnings have been fixed by @vlorentz, it would
be great to avoid introducing new ones as much as possible.

This diff showcases how a sphinx environment can be added to tox
to build and check the documentation proper to a swh package.

As some external references could not be resolved in a single package
scope, related sphinx warnings are suppressed for the build to succeed
but other ones are not.

This would allow to catch any rst errors prior landing a diff.

Ideally a new parallel step in our Jenkins pipelines to build and
check the doc should be added. We already have a CI docker image
containing all the required tools to build the doc so this should
not be too complicated to integrate.

Depends on D5571

Diff Detail

Repository
rDCORE Foundations and core functionalities
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Apr 16 2021, 7:16 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 16 2021, 7:16 PM
swh-public-ci added a comment.Apr 16 2021, 7:17 PM

Build is green

Patch application report for D5550 (id=19820)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit 3ad3315a657bb79109daf4610a92d32c18e61b9c
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/211/ for more details.

Harbormaster completed remote builds in B20830: Diff 19820.Apr 16 2021, 7:17 PM
anlambert requested review of this revision.Apr 16 2021, 7:17 PM
vlorentz added a task: T3258: Build documentation on every diff.Apr 16 2021, 7:27 PM
olasd added a subscriber: olasd.Apr 16 2021, 7:46 PM
olasd added inline comments.
tox.ini
70

the leading slash in /../.tox/sphinx/... is weird; not sure the {envdir} substitution will work there?

76
vlorentz added a comment.Apr 19 2021, 10:20 AM

that's a lot of code to add to every package :/

But it's also the easiest solution to solve T3258, so I don't know...

As mentioned on IRC, I think you could remove ref.term from the suppressed warnings by including glossary.rst

anlambert planned changes to this revision.Apr 19 2021, 10:58 AM
In D5550#140750, @vlorentz wrote:

that's a lot of code to add to every package :/

But it's also the easiest solution to solve T3258, so I don't know...

As mentioned on IRC, I think you could remove ref.term from the suppressed warnings by including glossary.rst

I have an idea to avoid leaking the overridden sphinx configuration in every tox file and how to include the glossary, I will update that diff once it is tested.

anlambert added inline comments.Apr 21 2021, 4:55 PM
tox.ini
76

I already tried but for some reasons it does not work:

(swh) ✘-1 /tmp/swh-core [tox-sphinx L|✚ 1⚑ 1] 
16:54 $ tox -e sphinx
sphinx develop-inst-noop: /tmp/swh-core
sphinx installed: aiohttp==3.7.4.post0,aiohttp-utils==3.1.1,alabaster==0.7.12,appdirs==1.4.4,async-timeout==3.0.1,attrs==20.3.0,Babel==2.9.0,blinker==1.4,certifi==2020.12.5,cfgv==3.2.0,chardet==4.0.0,click==7.1.2,Deprecated==1.2.12,distlib==0.3.1,docutils==0.16,filelock==3.0.12,Flask==1.1.2,gunicorn==20.1.0,hypothesis==6.10.0,identify==2.2.4,idna==2.10,imagesize==1.2.0,importlib-metadata==4.0.1,iniconfig==1.1.1,iso8601==0.1.14,itsdangerous==1.1.0,Jinja2==2.11.3,markdown-it-py==0.6.2,MarkupSafe==1.1.1,mdit-py-plugins==0.2.6,mirakuru==2.3.0,msgpack==1.0.2,multidict==5.1.0,myst-parser==0.13.6,nodeenv==1.6.0,packaging==20.9,pluggy==0.13.1,port-for==0.4,pre-commit==2.12.1,psutil==5.8.0,psycopg2==2.8.6,py==1.10.0,Pygments==2.8.1,pyparsing==2.4.7,pytest==6.2.3,pytest-mock==3.5.1,pytest-postgresql==2.6.1,python-mimeparse==1.6.0,pytz==2021.1,PyYAML==5.4.1,requests==2.25.1,requests-mock==1.8.0,sentry-sdk==1.0.0,six==1.15.0,snowballstemmer==2.1.0,sortedcontainers==2.3.0,Sphinx==3.5.4,sphinx-celery==2.0.0,sphinx-click==2.7.1,sphinx-reredirects==0.0.0,sphinx-rtd-theme==0.5.2,sphinx-tabs==2.1.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==1.0.3,sphinxcontrib-httpdomain==1.7.0,sphinxcontrib-images==0.9.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-programoutput==0.17,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.4,-e git+https://forge.softwareheritage.org/source/swh-core.git@3ad3315a657bb79109daf4610a92d32c18e61b9c#egg=swh.core,-e git+file:///home/anlambert/swh/swh-environment/swh-docs@ea3ceb6b60fc113a40bd0501c9d1b31a518338e1#egg=swh.docs,systemd-python==234,toml==0.10.2,typing-extensions==3.7.4.3,urllib3==1.26.4,virtualenv==20.4.4,Werkzeug==1.0.1,wrapt==1.12.1,yarl==1.6.3,zipp==3.4.1
sphinx run-test-pre: PYTHONHASHSEED='2758786654'
sphinx run-test: commands[0] | make -I /tmp/swh-core/.tox/sphinx/src/swh-docs/swh/ -C docs
make: Entering directory '/tmp/swh-core/docs'
Makefile:1: ../../swh-docs/Makefile.sphinx: No such file or directory
make: *** No rule to make target '../../swh-docs/Makefile.sphinx'.  Stop.
make: Leaving directory '/tmp/swh-core/docs'
ERROR: InvocationError for command /usr/bin/make -I .tox/sphinx/src/swh-docs/swh -C docs (exited with code 2)
____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
ERROR:   sphinx: commands failed
anlambert updated this revision to Diff 19905.Apr 21 2021, 5:14 PM

Update: Simplify sphinx environment definition:

  • Define SWH_PACKAGE_DOC_TOX_BUILD environment variable
  • All overriden sphinx configuration is now handled in the conf.py file of swh-docs based on the value of the above variable

It enables to avoid the duplication of overriden sphinx configuration
and to centralize dedicated sphinx tweaks for tox build in a single place.

swh-public-ci added a comment.Apr 21 2021, 5:15 PM

Build is green

Patch application report for D5550 (id=19905)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit d33a39e5f3c6e17002266aa39699b5e89fcf30a6
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/212/ for more details.

anlambert edited the summary of this revision. (Show Details)Apr 21 2021, 5:15 PM
anlambert added a parent revision: D5571: sphinx/conf.py: Add standalone package doc configuration for tox build.
Harbormaster completed remote builds in B20910: Diff 19905.Apr 21 2021, 5:15 PM
anlambert updated this revision to Diff 19907.Apr 21 2021, 5:17 PM

Update swh-docs clone URL.

swh-public-ci added a comment.Apr 21 2021, 5:18 PM

Build is green

Patch application report for D5550 (id=19907)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit 9c579e2935c6dff72f0e792785cacc5ec1ca4a06
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/213/ for more details.

Harbormaster completed remote builds in B20912: Diff 19907.Apr 21 2021, 5:19 PM
anlambert updated this revision to Diff 19910.Apr 21 2021, 5:42 PM

Fix make include dir, {envdir} tox substitution makes the build failed.

swh-public-ci added a comment.Apr 21 2021, 5:43 PM

Build is green

Patch application report for D5550 (id=19910)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit 9b5dfe6bbf9564f50bd58688e60872966c9cec57
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/214/ for more details.

Harbormaster completed remote builds in B20915: Diff 19910.Apr 21 2021, 5:43 PM
vlorentz added a comment.EditedApr 23 2021, 12:58 AM

Another random idea: what about -e ../swh-docs#egg=swh.docs (or a variant that works, I didn't try it) instead of `-e git+https://forge.softwareheritage.org/source/swh-docs#egg=swh.docs ?

This way it runs faster, spares some bandwidth, works without internet access, and allows local modifications in swh-docs

anlambert added a comment.Apr 23 2021, 12:26 PM
In D5550#141719, @vlorentz wrote:

Another random idea: what about -e ../swh-docs#egg=swh.docs (or a variant that works, I didn't try it) instead of `-e git+https://forge.softwareheritage.org/source/swh-docs#egg=swh.docs ?

This way it runs faster, spares some bandwidth, works without internet access, and allows local modifications in swh-docs

This will not work when building the standalone doc on Jenkins.

How about adding a sphinx-dev tox environment that will fetch swh-docs on the local swh-environment ?

vlorentz added a comment.Apr 23 2021, 12:38 PM
In D5550#141833, @anlambert wrote:

This will not work when building the standalone doc on Jenkins.

Oh, indeed.

How about adding a sphinx-dev tox environment that will fetch swh-docs on the local swh-environment ?

Sounds good! And the improved consistency in environment names is nice too.

anlambert added a comment.Apr 23 2021, 1:06 PM
In D5550#141846, @vlorentz wrote:
In D5550#141833, @anlambert wrote:

This will not work when building the standalone doc on Jenkins.

Oh, indeed.

How about adding a sphinx-dev tox environment that will fetch swh-docs on the local swh-environment ?

Sounds good! And the improved consistency in environment names is nice too.

Let's go for it then ! I will also update swh-docs README in D5571 to use the sphinx-dev env.

Next steps:

  • modifiy tox.ini in all swh packages
  • modify Jenkins pipelines to add a parallel step for building the documentation
anlambert updated this revision to Diff 19968.Apr 23 2021, 2:15 PM

Add sphinx-dev tox environment and update commit message.

swh-public-ci added a comment.Apr 23 2021, 2:16 PM

Build is green

Patch application report for D5550 (id=19968)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit 4383ecf1c1f6ccaed9209711d90434a711a8a61d
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add sphinx environments to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    The sphinx environment is designed to be used in continuous integration.
    
    The sphinx-dev environment is designed to be used inside a full swh
    development environment.
    
    Related to T3258

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/215/ for more details.

Harbormaster completed remote builds in B20970: Diff 19968.Apr 23 2021, 2:16 PM
vlorentz added a comment.Apr 23 2021, 2:25 PM

Could you add a comment or two, to explain the difference between the two envs?

anlambert updated this revision to Diff 19972.Apr 23 2021, 3:28 PM

Update comments

swh-public-ci added a comment.Apr 23 2021, 3:29 PM

Build is green

Patch application report for D5550 (id=19972)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit 9ba33def779d70197d40c212d69bc363bcf0b0e8
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add sphinx environments to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    The sphinx environment is designed to be used in continuous integration.
    
    The sphinx-dev environment is designed to be used inside a full swh
    development environment.
    
    Related to T3258

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/217/ for more details.

Harbormaster completed remote builds in B20974: Diff 19972.Apr 23 2021, 3:29 PM
vlorentz added inline comments.Apr 23 2021, 3:35 PM
tox.ini
52–53

Could you also explain this is used on the CI because it does not have ../swh-docs/?

anlambert updated this revision to Diff 19975.Apr 23 2021, 4:19 PM

Improve comments

swh-public-ci added a comment.Apr 23 2021, 4:20 PM

Build is green

Patch application report for D5550 (id=19975)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.
Changes applied before test
commit df31524e35690816be57c17800e48d019d6575fd
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add sphinx environments to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    The sphinx environment is designed to be used in continuous integration
    in order to prevent breaking documentation build when committing changes.
    
    The sphinx-dev environment is designed to be used inside a full swh
    development environment.
    
    Related to T3258

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/218/ for more details.

Harbormaster completed remote builds in B20977: Diff 19975.Apr 23 2021, 4:20 PM
anlambert added inline comments.Apr 23 2021, 4:21 PM
tox.ini
52–53

I mentioned this is to build the doc outside swh-environment which is equivalent.

vlorentz accepted this revision.Apr 23 2021, 4:27 PM
This revision is now accepted and ready to land.Apr 23 2021, 4:27 PM
Closed by commit rDCOREdf31524e3569: tox: Add sphinx environments to check sane doc build (authored by anlambert). · Explain WhyApr 23 2021, 5:20 PM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDCOREdf31524e3569: tox: Add sphinx environments to check sane doc build.
anlambert added a comment.Apr 23 2021, 5:21 PM

Landed ! I will handle other swh repositories and update Jenkins pipeline next week.

Revision Contents
Changeset List

PathSize
46 lines

Diff 19980

View Options

tox.ini

Loading...
About · Developer information · Legal