Page MenuHomeSoftware Heritage

make-doc: refactor doc build system to build the doc as a whole
ClosedPublic

Authored by douardda on Nov 22 2018, 2:01 PM.

Details

Summary
  • back to assets only building in each package
  • run sphinx-autodoc on the whole swh package at once
  • add a tox.ini file with 2 sphinx environments:
    • sphinx: build the doc reading sources from installed swh subpackages,
    • sphinx-dev: build the doc from source swh subpackages in ..

we do not really need a local standalone buildable sphinx project in each swh
subpackage, so running a full 'make html' in each of them is not required any
more.

Since we want to build the doc at once instead of building each swh subpackage
independantly, we need to run apidoc on the whole swh package.

This is done by actually running the apidoc on the pip installed swh
packages instead of the source repositories, since sphinx-apidoc does
not seem to support multiple MODULE_PATH as cmdline argument.

So it's best to use tox to build the doc.

Also note that this evolution of the doc build system requires modifications
in every swh-*/docs/index.rst file to fix the module indexes.

Related to T1330.

Diff Detail

Repository
rDDOC Development documentation
Branch
doc
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 2527
Build 3134: arc lint + arc unit

Event Timeline

I would like to test your changes but arc patch tells me the following:

arc patch D696
Created and checked out branch arcpatch-D696.


    This diff is against commit e8d48567534b808d1e03100ead46b0b6d6e3cb71, but
    the commit is nowhere in the working copy. Try to apply it against the
    current working copy state? (4801d9c104c4c176be1df15b55ba4b8f72103bc4)
    [Y/n]

Looks like a rebase is needed

tox -e sphinx seems to work but the resulting Module index is incomplete (I only have swh.archiver in it)

tox -e sphinx-dev fails with the following output:

GLOB sdist-make: /home/antoine/swh/swh-environment/swh-docs/setup.py
sphinx-dev create: /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev
sphinx-dev installdeps: Django < 2, -r requirements-swh-dev.txt, pifpaf
ERROR: invocation failed (exit code 1), logfile: /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/log/sphinx-dev-1.log
ERROR: actionid: sphinx-dev
msg: getenv
cmdargs: ['/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin/pip', 'install', 'Django < 2', '-r requirements-swh-dev.txt', 'pifpaf']
env: {'SSH_AUTH_SOCK': '/tmp/ssh-EVqLKiWSsEFm/agent.2534', 'XDG_SESSION_DESKTOP': 'KDE', 'GIT_BRANCH': 'arcpatch-D696', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session1', 'PWD': '/home/antoine/swh/swh-environment/swh-docs', 'PYTHONHASHSEED': '3756312444', 'QT_AUTO_SCREEN_SCALE_FACTOR': '0', 'TERM': 'xterm-256color', 'PAM_KWALLET5_LOGIN': '/tmp/kwallet5_antoine.socket', 'LANG': 'fr_FR.UTF-8', 'DEBFULLNAME': 'Antoine Lambert', 'QMLSCENE_DEVICE': '', 'OLDPWD': '/home/antoine', '__GIT_PROMPT_SHOW_UNTRACKED_FILES': 'normal', 'DEBEMAIL': 'antoine.lambert@inria.fr', 'XDG_DATA_DIRS': '/usr/share:/usr/share:/usr/local/share', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'XDG_SESSION_ID': '3', 'PATH': '/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin:/usr/lib/go-1.10/bin:/home/antoine/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/games', 'XDG_CURRENT_DESKTOP': 'KDE', 'KDE_SESSION_VERSION': '5', '__GIT_PROMPT_SHOW_UPSTREAM': '', 'USER': 'antoine', 'KONSOLE_PROFILE_NAME': 'Profile 1', 'KONSOLE_DBUS_SERVICE': ':1.195', 'WINDOWID': '2097158', 'DESKTOP_SESSION': '/usr/share/xsessions/plasma', 'COLORFGBG': '15;0', 'SHELL_SESSION_ID': 'b8985c2db3a44253bdc305f3674a69c9', 'XDG_VTNR': '7', 'KONSOLE_DBUS_SESSION': '/Sessions/5', 'DISPLAY': ':0', 'QSG_RENDER_LOOP': '', 'GTK_IM_MODULE': 'xim', 'SHLVL': '1', 'XAUTHORITY': '/home/antoine/.Xauthority', 'VIRTUAL_ENV': '/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev', 'SSH_AGENT_PID': '2600', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'COLORTERM': 'truecolor', 'PROFILEHOME': '', 'XCURSOR_THEME': 'breeze_cursors', 'XDG_SESSION_TYPE': 'x11', 'LOGNAME': 'antoine', 'SHELL': '/usr/bin/zsh', 'GPG_TTY': '/dev/pts/10', 'GPG_AGENT_INFO': '/run/user/1000/gnupg/S.gpg-agent:0:1', 'GTK_MODULES': 'gail:atk-bridge', 'XDG_SEAT': 'seat0', 'QT_ACCESSIBILITY': '1', 'XDG_SESSION_CLASS': 'user', '__GIT_PROMPT_IGNORE_SUBMODULES': '', 'GS_LIB': '/home/antoine/.fonts', 'QT_LINUX_ACCESSIBILITY_ALWAYS_ON': '1', '__GIT_PROMPT_IGNORE_STASH': '', 'KDE_FULL_SESSION': 'true', 'KONSOLE_DBUS_WINDOW': '/Windows/1', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'SESSION_MANAGER': 'local/guggenheim:@/tmp/.ICE-unix/2704,unix/guggenheim:/tmp/.ICE-unix/2704', 'KDE_SESSION_UID': '1000', 'HOME': '/home/antoine', '__GIT_PROMPT_SHOW_CHANGED_FILES_COUNT': '1', '_': '/usr/bin/tox', 'XMODIFIERS': '@im=uim', 'LANGUAGE': '', 'XDG_RUNTIME_DIR': '/run/user/1000'}

Could not open requirements file: [Errno 2] Aucun fichier ou dossier de ce type: ' requirements-swh-dev.txt'

ERROR: could not install deps [Django < 2, -r requirements-swh-dev.txt, pifpaf]; v = InvocationError('/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin/pip install Django < 2 -r requirements-swh-dev.txt pifpaf (see /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/log/sphinx-dev-1.log)', 1)
___________________________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________________________
ERROR:   sphinx-dev: could not install deps [Django < 2, -r requirements-swh-dev.txt, pifpaf]; v = InvocationError('/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin/pip install Django < 2 -r requirements-swh-dev.txt pifpaf (see /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/log/sphinx-dev-1.log)', 1)

For the record, I have all swh module paths registered in the PYTHONPATH env variable.

Also having dedicated make targets that call tox with the right arguments would be useful.

tox.ini
6

django -> Django

14

django -> Django

I also have the following sphinx warnings when calling tox -e sphinx (with D697 applied)

/home/antoine/swh/swh-environment/swh-docs/docs/swh-core/index.rst:12: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.core'
looking for now-outdated files... none found
pickling environment... fait
checking consistency... /home/antoine/swh/swh-environment/swh-docs/docs/apidoc/modules.rst: WARNING: document isn't included in any toctree
swh/docs/sphinx/conf.py
135

The django setup can be moved in the setup function.

This way it will be called only once instead of each time a rst source file has been read

For the record, I have all swh module paths registered in the PYTHONPATH env variable.

I haven't checked with a PYTHONPATH based dev setup indeed. Not sure I really want to, however. Convince me! 😄

Also having dedicated make targets that call tox with the right arguments would be useful.

Indeed.

tox -e sphinx seems to work but the resulting Module index is incomplete (I only have swh.archiver in it)

Can you try with your PYTHONPATH unset?

I have the same results when unsetting PYTHONPATH.

Seems that even when it is set, the correct path to swh module is picked.

The SWHPKGDIR Makefile variable is set to /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx/lib/python3.5/site-packages/swh
in both cases.

tox: enforce python3 for all environments

and fix sphinx-dev for old tox versions

conf: exclude swh. part when building the module index

Apart the django settings initialization that should be moved directly in the setup function, LGTM.

This revision is now accepted and ready to land.Nov 23 2018, 11:31 AM

Add a few tweaks and tunings

  • conf: simplify a bit the django hack
  • Ugly hackish fix to try to make autodoc happy with django
  • use swh.objstorage[testing] dep to have libcloud and azure stuff installed
This revision was automatically updated to reflect the committed changes.