Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Maniphest Tasks

T3258: Build documentation on every diff

Commits

rDCOREdf31524e3569: tox: Add sphinx environments to check sane doc build

Summary

Now that all sphinx warnings have been fixed by @vlorentz, it would
be great to avoid introducing new ones as much as possible.

This diff showcases how a sphinx environment can be added to tox
to build and check the documentation proper to a swh package.

As some external references could not be resolved in a single package
scope, related sphinx warnings are suppressed for the build to succeed
but other ones are not.

This would allow to catch any rst errors prior landing a diff.

Ideally a new parallel step in our Jenkins pipelines to build and
check the doc should be added. We already have a CI docker image
containing all the required tools to build the doc so this should
not be too complicated to integrate.

Depends on D5571

Diff Detail

Repository

rDCORE Foundations and core functionalities

Branch

tox-sphinx

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 20915
Build 32457: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 32456: arc lint + arc unit

Event Timeline

anlambert created this revision.Apr 16 2021, 7:16 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptApr 16 2021, 7:16 PM

Build is green

Patch application report for D5550 (id=19820)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.

Changes applied before test

commit 3ad3315a657bb79109daf4610a92d32c18e61b9c
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/211/ for more details.

Harbormaster completed remote builds in B20830: Diff 19820.Apr 16 2021, 7:17 PM

anlambert requested review of this revision.Apr 16 2021, 7:17 PM

vlorentz added a task: T3258: Build documentation on every diff.Apr 16 2021, 7:27 PM

olasd added a subscriber: olasd.Apr 16 2021, 7:46 PM

olasd added inline comments.

tox.ini
70	the leading slash in `/../.tox/sphinx/...` is weird; not sure the `{envdir}` substitution will work there?
76

that's a lot of code to add to every package :/

But it's also the easiest solution to solve T3258, so I don't know...

As mentioned on IRC, I think you could remove ref.term from the suppressed warnings by including glossary.rst

In D5550#140750, @vlorentz wrote:

that's a lot of code to add to every package :/

But it's also the easiest solution to solve T3258, so I don't know...

As mentioned on IRC, I think you could remove ref.term from the suppressed warnings by including glossary.rst

I have an idea to avoid leaking the overridden sphinx configuration in every tox file and how to include the glossary, I will update that diff once it is tested.

anlambert added inline comments.Apr 21 2021, 4:55 PM

tox.ini

I already tried but for some reasons it does not work:

(swh) ✘-1 /tmp/swh-core [tox-sphinx L|✚ 1⚑ 1] 
16:54 $ tox -e sphinx
sphinx develop-inst-noop: /tmp/swh-core
sphinx installed: aiohttp==3.7.4.post0,aiohttp-utils==3.1.1,alabaster==0.7.12,appdirs==1.4.4,async-timeout==3.0.1,attrs==20.3.0,Babel==2.9.0,blinker==1.4,certifi==2020.12.5,cfgv==3.2.0,chardet==4.0.0,click==7.1.2,Deprecated==1.2.12,distlib==0.3.1,docutils==0.16,filelock==3.0.12,Flask==1.1.2,gunicorn==20.1.0,hypothesis==6.10.0,identify==2.2.4,idna==2.10,imagesize==1.2.0,importlib-metadata==4.0.1,iniconfig==1.1.1,iso8601==0.1.14,itsdangerous==1.1.0,Jinja2==2.11.3,markdown-it-py==0.6.2,MarkupSafe==1.1.1,mdit-py-plugins==0.2.6,mirakuru==2.3.0,msgpack==1.0.2,multidict==5.1.0,myst-parser==0.13.6,nodeenv==1.6.0,packaging==20.9,pluggy==0.13.1,port-for==0.4,pre-commit==2.12.1,psutil==5.8.0,psycopg2==2.8.6,py==1.10.0,Pygments==2.8.1,pyparsing==2.4.7,pytest==6.2.3,pytest-mock==3.5.1,pytest-postgresql==2.6.1,python-mimeparse==1.6.0,pytz==2021.1,PyYAML==5.4.1,requests==2.25.1,requests-mock==1.8.0,sentry-sdk==1.0.0,six==1.15.0,snowballstemmer==2.1.0,sortedcontainers==2.3.0,Sphinx==3.5.4,sphinx-celery==2.0.0,sphinx-click==2.7.1,sphinx-reredirects==0.0.0,sphinx-rtd-theme==0.5.2,sphinx-tabs==2.1.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==1.0.3,sphinxcontrib-httpdomain==1.7.0,sphinxcontrib-images==0.9.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-programoutput==0.17,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.4,-e git+https://forge.softwareheritage.org/source/swh-core.git@3ad3315a657bb79109daf4610a92d32c18e61b9c#egg=swh.core,-e git+file:///home/anlambert/swh/swh-environment/swh-docs@ea3ceb6b60fc113a40bd0501c9d1b31a518338e1#egg=swh.docs,systemd-python==234,toml==0.10.2,typing-extensions==3.7.4.3,urllib3==1.26.4,virtualenv==20.4.4,Werkzeug==1.0.1,wrapt==1.12.1,yarl==1.6.3,zipp==3.4.1
sphinx run-test-pre: PYTHONHASHSEED='2758786654'
sphinx run-test: commands[0] | make -I /tmp/swh-core/.tox/sphinx/src/swh-docs/swh/ -C docs
make: Entering directory '/tmp/swh-core/docs'
Makefile:1: ../../swh-docs/Makefile.sphinx: No such file or directory
make: *** No rule to make target '../../swh-docs/Makefile.sphinx'.  Stop.
make: Leaving directory '/tmp/swh-core/docs'
ERROR: InvocationError for command /usr/bin/make -I .tox/sphinx/src/swh-docs/swh -C docs (exited with code 2)
____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
ERROR:   sphinx: commands failed

Update: Simplify sphinx environment definition:

Define SWH_PACKAGE_DOC_TOX_BUILD environment variable
All overriden sphinx configuration is now handled in the conf.py file of swh-docs based on the value of the above variable

It enables to avoid the duplication of overriden sphinx configuration
and to centralize dedicated sphinx tweaks for tox build in a single place.

Build is green

Patch application report for D5550 (id=19905)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.

Changes applied before test

commit d33a39e5f3c6e17002266aa39699b5e89fcf30a6
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/212/ for more details.

anlambert edited the summary of this revision. (Show Details)Apr 21 2021, 5:15 PM

anlambert added a parent revision: D5571: sphinx/conf.py: Add standalone package doc configuration for tox build.

Harbormaster completed remote builds in B20910: Diff 19905.Apr 21 2021, 5:15 PM

Update swh-docs clone URL.

Build is green

Patch application report for D5550 (id=19907)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.

Changes applied before test

commit 9c579e2935c6dff72f0e792785cacc5ec1ca4a06
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/213/ for more details.

Harbormaster completed remote builds in B20912: Diff 19907.Apr 21 2021, 5:19 PM

Fix make include dir, {envdir} tox substitution makes the build failed.

Build is green

Patch application report for D5550 (id=19910)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.

Changes applied before test

commit 9b5dfe6bbf9564f50bd58688e60872966c9cec57
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add a sphinx environment to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    This could be added as a new parralel step in our Jenkins pipelines
    to ensure a sane documentation.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/214/ for more details.

Harbormaster completed remote builds in B20915: Diff 19910.Apr 21 2021, 5:43 PM

Another random idea: what about -e ../swh-docs#egg=swh.docs (or a variant that works, I didn't try it) instead of `-e git+https://forge.softwareheritage.org/source/swh-docs#egg=swh.docs ?

This way it runs faster, spares some bandwidth, works without internet access, and allows local modifications in swh-docs

In D5550#141719, @vlorentz wrote:

Another random idea: what about -e ../swh-docs#egg=swh.docs (or a variant that works, I didn't try it) instead of `-e git+https://forge.softwareheritage.org/source/swh-docs#egg=swh.docs ?

This way it runs faster, spares some bandwidth, works without internet access, and allows local modifications in swh-docs

This will not work when building the standalone doc on Jenkins.

How about adding a sphinx-dev tox environment that will fetch swh-docs on the local swh-environment ?

In D5550#141833, @anlambert wrote:

This will not work when building the standalone doc on Jenkins.

Oh, indeed.

How about adding a sphinx-dev tox environment that will fetch swh-docs on the local swh-environment ?

Sounds good! And the improved consistency in environment names is nice too.

In D5550#141846, @vlorentz wrote:

In D5550#141833, @anlambert wrote:

This will not work when building the standalone doc on Jenkins.

Oh, indeed.

How about adding a sphinx-dev tox environment that will fetch swh-docs on the local swh-environment ?

Sounds good! And the improved consistency in environment names is nice too.

Let's go for it then ! I will also update swh-docs README in D5571 to use the sphinx-dev env.

Next steps:

modifiy tox.ini in all swh packages
modify Jenkins pipelines to add a parallel step for building the documentation

Add sphinx-dev tox environment and update commit message.

Build is green

Patch application report for D5550 (id=19968)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.

Changes applied before test

commit 4383ecf1c1f6ccaed9209711d90434a711a8a61d
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add sphinx environments to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    The sphinx environment is designed to be used in continuous integration.
    
    The sphinx-dev environment is designed to be used inside a full swh
    development environment.
    
    Related to T3258

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/215/ for more details.

Harbormaster completed remote builds in B20970: Diff 19968.Apr 23 2021, 2:16 PM

Could you add a comment or two, to explain the difference between the two envs?

Update comments

Build is green

Patch application report for D5550 (id=19972)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.

Changes applied before test

commit 9ba33def779d70197d40c212d69bc363bcf0b0e8
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add sphinx environments to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    The sphinx environment is designed to be used in continuous integration.
    
    The sphinx-dev environment is designed to be used inside a full swh
    development environment.
    
    Related to T3258

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/217/ for more details.

Harbormaster completed remote builds in B20974: Diff 19972.Apr 23 2021, 3:29 PM

vlorentz added inline comments.Apr 23 2021, 3:35 PM

tox.ini
52–53	Could you also explain this is used on the CI because it does not have `../swh-docs/`?

Improve comments

Build is green

Patch application report for D5550 (id=19975)

Rebasing onto 80832699e0...

Current branch diff-target is up to date.

Changes applied before test

commit df31524e35690816be57c17800e48d019d6575fd
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Apr 16 19:06:42 2021 +0200

    tox: Add sphinx environments to check sane doc build
    
    Enable to check package documentation can be built without producing
    sphinx warnings.
    
    The sphinx environment is designed to be used in continuous integration
    in order to prevent breaking documentation build when committing changes.
    
    The sphinx-dev environment is designed to be used inside a full swh
    development environment.
    
    Related to T3258

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/218/ for more details.

Harbormaster completed remote builds in B20977: Diff 19975.Apr 23 2021, 4:20 PM

anlambert added inline comments.Apr 23 2021, 4:21 PM

tox.ini
52–53	I mentioned this is to build the doc outside swh-environment which is equivalent.

vlorentz accepted this revision.Apr 23 2021, 4:27 PM

This revision is now accepted and ready to land.Apr 23 2021, 4:27 PM

Closed by commit rDCOREdf31524e3569: tox: Add sphinx environments to check sane doc build (authored by anlambert). · Explain WhyApr 23 2021, 5:20 PM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDCOREdf31524e3569: tox: Add sphinx environments to check sane doc build.

Landed ! I will handle other swh repositories and update Jenkins pipeline next week.

tox: Add a sphinx environment to check sane doc build
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Patch application report for D5550 (id=19820)

Changes applied before test

Patch application report for D5550 (id=19905)

Changes applied before test

Patch application report for D5550 (id=19907)

Changes applied before test

Patch application report for D5550 (id=19910)

Changes applied before test

Patch application report for D5550 (id=19968)

Changes applied before test

Patch application report for D5550 (id=19972)

Changes applied before test

Patch application report for D5550 (id=19975)

Changes applied before test

Revision Contents
Changeset List

Diff 19910

tox.ini

tox: Add a sphinx environment to check sane doc buildClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D5550 (id=19820)

Changes applied before test

Patch application report for D5550 (id=19905)

Changes applied before test

Patch application report for D5550 (id=19907)

Changes applied before test

Patch application report for D5550 (id=19910)

Changes applied before test

Patch application report for D5550 (id=19968)

Changes applied before test

Patch application report for D5550 (id=19972)

Changes applied before test

Patch application report for D5550 (id=19975)

Changes applied before test

Revision ContentsChangeset List

Diff 19910

tox.ini

tox: Add a sphinx environment to check sane doc build
ClosedPublic
Actions

Revision Contents
Changeset List