Details

Reviewers

Group Reviewers

Commits

rDDOCc8137bd2981a: Update the README file for latest (normally tox-based) build system
rDDOCe98950fced4b: Ugly hackish fix to try to make autodoc happy with django
rDDOC1ffdf60f524e: conf: simplify a bit the django hack
rDDOC45e2a28f57f0: conf: exclude swh. part when building the module index
rDDOC040052150641: tox: enforce python3 for all environments and fix sphinx-dev for old tox…
rDDOC0e0c85487798: tox: add a sphinx-dev tox environment
rDDOC891ad17c1a1c: make-doc: refactor doc build system to build the doc as a whole

Summary

back to assets only building in each package
run sphinx-autodoc on the whole swh package at once
add a tox.ini file with 2 sphinx environments:
- sphinx: build the doc reading sources from installed swh subpackages,
- sphinx-dev: build the doc from source swh subpackages in ..

we do not really need a local standalone buildable sphinx project in each swh
subpackage, so running a full 'make html' in each of them is not required any
more.

Since we want to build the doc at once instead of building each swh subpackage
independantly, we need to run apidoc on the whole swh package.

This is done by actually running the apidoc on the pip installed swh
packages instead of the source repositories, since sphinx-apidoc does
not seem to support multiple MODULE_PATH as cmdline argument.

So it's best to use tox to build the doc.

Also note that this evolution of the doc build system requires modifications
in every swh-*/docs/index.rst file to fix the module indexes.

Related to T1330.

Diff Detail

Repository

rDDOC Development documentation

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Nov 22 2018, 2:01 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptNov 22 2018, 2:01 PM

Harbormaster completed remote builds in B2527: Diff 2187.Nov 22 2018, 2:01 PM

douardda mentioned this in D697: doc: update index to match new swh-doc format.Nov 22 2018, 2:03 PM

I would like to test your changes but arc patch tells me the following:

arc patch D696
Created and checked out branch arcpatch-D696.


    This diff is against commit e8d48567534b808d1e03100ead46b0b6d6e3cb71, but
    the commit is nowhere in the working copy. Try to apply it against the
    current working copy state? (4801d9c104c4c176be1df15b55ba4b8f72103bc4)
    [Y/n]

Looks like a rebase is needed

douardda added a parent revision: D668: Add the beginning of a top-level architecture document.Nov 22 2018, 2:42 PM

tox -e sphinx seems to work but the resulting Module index is incomplete (I only have swh.archiver in it)

tox -e sphinx-dev fails with the following output:

GLOB sdist-make: /home/antoine/swh/swh-environment/swh-docs/setup.py
sphinx-dev create: /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev
sphinx-dev installdeps: Django < 2, -r requirements-swh-dev.txt, pifpaf
ERROR: invocation failed (exit code 1), logfile: /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/log/sphinx-dev-1.log
ERROR: actionid: sphinx-dev
msg: getenv
cmdargs: ['/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin/pip', 'install', 'Django < 2', '-r requirements-swh-dev.txt', 'pifpaf']
env: {'SSH_AUTH_SOCK': '/tmp/ssh-EVqLKiWSsEFm/agent.2534', 'XDG_SESSION_DESKTOP': 'KDE', 'GIT_BRANCH': 'arcpatch-D696', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session1', 'PWD': '/home/antoine/swh/swh-environment/swh-docs', 'PYTHONHASHSEED': '3756312444', 'QT_AUTO_SCREEN_SCALE_FACTOR': '0', 'TERM': 'xterm-256color', 'PAM_KWALLET5_LOGIN': '/tmp/kwallet5_antoine.socket', 'LANG': 'fr_FR.UTF-8', 'DEBFULLNAME': 'Antoine Lambert', 'QMLSCENE_DEVICE': '', 'OLDPWD': '/home/antoine', '__GIT_PROMPT_SHOW_UNTRACKED_FILES': 'normal', 'DEBEMAIL': 'antoine.lambert@inria.fr', 'XDG_DATA_DIRS': '/usr/share:/usr/share:/usr/local/share', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'XDG_SESSION_ID': '3', 'PATH': '/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin:/usr/lib/go-1.10/bin:/home/antoine/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/games', 'XDG_CURRENT_DESKTOP': 'KDE', 'KDE_SESSION_VERSION': '5', '__GIT_PROMPT_SHOW_UPSTREAM': '', 'USER': 'antoine', 'KONSOLE_PROFILE_NAME': 'Profile 1', 'KONSOLE_DBUS_SERVICE': ':1.195', 'WINDOWID': '2097158', 'DESKTOP_SESSION': '/usr/share/xsessions/plasma', 'COLORFGBG': '15;0', 'SHELL_SESSION_ID': 'b8985c2db3a44253bdc305f3674a69c9', 'XDG_VTNR': '7', 'KONSOLE_DBUS_SESSION': '/Sessions/5', 'DISPLAY': ':0', 'QSG_RENDER_LOOP': '', 'GTK_IM_MODULE': 'xim', 'SHLVL': '1', 'XAUTHORITY': '/home/antoine/.Xauthority', 'VIRTUAL_ENV': '/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev', 'SSH_AGENT_PID': '2600', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'COLORTERM': 'truecolor', 'PROFILEHOME': '', 'XCURSOR_THEME': 'breeze_cursors', 'XDG_SESSION_TYPE': 'x11', 'LOGNAME': 'antoine', 'SHELL': '/usr/bin/zsh', 'GPG_TTY': '/dev/pts/10', 'GPG_AGENT_INFO': '/run/user/1000/gnupg/S.gpg-agent:0:1', 'GTK_MODULES': 'gail:atk-bridge', 'XDG_SEAT': 'seat0', 'QT_ACCESSIBILITY': '1', 'XDG_SESSION_CLASS': 'user', '__GIT_PROMPT_IGNORE_SUBMODULES': '', 'GS_LIB': '/home/antoine/.fonts', 'QT_LINUX_ACCESSIBILITY_ALWAYS_ON': '1', '__GIT_PROMPT_IGNORE_STASH': '', 'KDE_FULL_SESSION': 'true', 'KONSOLE_DBUS_WINDOW': '/Windows/1', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'SESSION_MANAGER': 'local/guggenheim:@/tmp/.ICE-unix/2704,unix/guggenheim:/tmp/.ICE-unix/2704', 'KDE_SESSION_UID': '1000', 'HOME': '/home/antoine', '__GIT_PROMPT_SHOW_CHANGED_FILES_COUNT': '1', '_': '/usr/bin/tox', 'XMODIFIERS': '@im=uim', 'LANGUAGE': '', 'XDG_RUNTIME_DIR': '/run/user/1000'}

Could not open requirements file: [Errno 2] Aucun fichier ou dossier de ce type: ' requirements-swh-dev.txt'

ERROR: could not install deps [Django < 2, -r requirements-swh-dev.txt, pifpaf]; v = InvocationError('/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin/pip install Django < 2 -r requirements-swh-dev.txt pifpaf (see /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/log/sphinx-dev-1.log)', 1)
___________________________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________________________
ERROR:   sphinx-dev: could not install deps [Django < 2, -r requirements-swh-dev.txt, pifpaf]; v = InvocationError('/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin/pip install Django < 2 -r requirements-swh-dev.txt pifpaf (see /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/log/sphinx-dev-1.log)', 1)

For the record, I have all swh module paths registered in the PYTHONPATH env variable.

Also having dedicated make targets that call tox with the right arguments would be useful.

tox.ini
6	django -> Django
14	django -> Django

I also have the following sphinx warnings when calling tox -e sphinx (with D697 applied)

/home/antoine/swh/swh-environment/swh-docs/docs/swh-core/index.rst:12: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.core'
looking for now-outdated files... none found
pickling environment... fait
checking consistency... /home/antoine/swh/swh-environment/swh-docs/docs/apidoc/modules.rst: WARNING: document isn't included in any toctree

swh/docs/sphinx/conf.py
135–137	The django setup can be moved in the `setup` function. This way it will be called only once instead of each time a rst source file has been read

In D696#14558, @anlambert wrote:

For the record, I have all swh module paths registered in the PYTHONPATH env variable.

I haven't checked with a PYTHONPATH based dev setup indeed. Not sure I really want to, however. Convince me! 😄

Also having dedicated make targets that call tox with the right arguments would be useful.

Indeed.

In D696#14558, @anlambert wrote:

tox -e sphinx seems to work but the resulting Module index is incomplete (I only have swh.archiver in it)

Can you try with your PYTHONPATH unset?

I have the same results when unsetting PYTHONPATH.

Seems that even when it is set, the correct path to swh module is picked.

The SWHPKGDIR Makefile variable is set to /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx/lib/python3.5/site-packages/swh
in both cases.

tox: enforce python3 for all environments

Harbormaster completed remote builds in B2546: Diff 2210.Nov 23 2018, 10:50 AM

and fix sphinx-dev for old tox versions

Harbormaster completed remote builds in B2547: Diff 2211.Nov 23 2018, 11:01 AM

conf: exclude swh. part when building the module index

Harbormaster completed remote builds in B2549: Diff 2213.Nov 23 2018, 11:20 AM

Apart the django settings initialization that should be moved directly in the setup function, LGTM.

This revision is now accepted and ready to land.Nov 23 2018, 11:31 AM

Add a few tweaks and tunings

conf: simplify a bit the django hack
Ugly hackish fix to try to make autodoc happy with django
use swh.objstorage[testing] dep to have libcloud and azure stuff installed

Harbormaster completed remote builds in B2551: Diff 2215.Nov 23 2018, 2:01 PM

rework git history

Harbormaster completed remote builds in B2554: Diff 2218.Nov 23 2018, 2:34 PM

Closed by commit rDDOC891ad17c1a1c: make-doc: refactor doc build system to build the doc as a whole (authored by douardda). · Explain WhyNov 23 2018, 2:50 PM

This revision was automatically updated to reflect the committed changes.

make-doc: refactor doc build system to build the doc as a whole
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 2222

.gitignore

docs/Makefile

requirements-swh.txt

swh/docs/django_settings.py

swh/docs/sphinx/conf.py

tox.ini

make-doc: refactor doc build system to build the doc as a wholeClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 2222

.gitignore

docs/Makefile

requirements-swh.txt

swh/docs/django_settings.py

swh/docs/sphinx/conf.py

tox.ini

make-doc: refactor doc build system to build the doc as a whole
ClosedPublic
Actions

Revision Contents
Changeset List