Page MenuHomeSoftware Heritage

make-doc: refactor doc build system to build the doc as a whole
ClosedPublic

Authored by douardda on Thu, Nov 22, 2:01 PM.

Details

Summary
  • back to assets only building in each package
  • run sphinx-autodoc on the whole swh package at once
  • add a tox.ini file with 2 sphinx environments:
    • sphinx: build the doc reading sources from installed swh subpackages,
    • sphinx-dev: build the doc from source swh subpackages in ..

we do not really need a local standalone buildable sphinx project in each swh
subpackage, so running a full 'make html' in each of them is not required any
more.

Since we want to build the doc at once instead of building each swh subpackage
independantly, we need to run apidoc on the whole swh package.

This is done by actually running the apidoc on the pip installed swh
packages instead of the source repositories, since sphinx-apidoc does
not seem to support multiple MODULE_PATH as cmdline argument.

So it's best to use tox to build the doc.

Also note that this evolution of the doc build system requires modifications
in every swh-*/docs/index.rst file to fix the module indexes.

Related to T1330.

Diff Detail

Repository
rDDOC Development documentation
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Thu, Nov 22, 2:01 PM

I would like to test your changes but arc patch tells me the following:

arc patch D696
Created and checked out branch arcpatch-D696.


    This diff is against commit e8d48567534b808d1e03100ead46b0b6d6e3cb71, but
    the commit is nowhere in the working copy. Try to apply it against the
    current working copy state? (4801d9c104c4c176be1df15b55ba4b8f72103bc4)
    [Y/n]

Looks like a rebase is needed

tox -e sphinx seems to work but the resulting Module index is incomplete (I only have swh.archiver in it)

tox -e sphinx-dev fails with the following output:

GLOB sdist-make: /home/antoine/swh/swh-environment/swh-docs/setup.py
sphinx-dev create: /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev
sphinx-dev installdeps: Django < 2, -r requirements-swh-dev.txt, pifpaf
ERROR: invocation failed (exit code 1), logfile: /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/log/sphinx-dev-1.log
ERROR: actionid: sphinx-dev
msg: getenv
cmdargs: ['/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin/pip', 'install', 'Django < 2', '-r requirements-swh-dev.txt', 'pifpaf']
env: {'SSH_AUTH_SOCK': '/tmp/ssh-EVqLKiWSsEFm/agent.2534', 'XDG_SESSION_DESKTOP': 'KDE', 'GIT_BRANCH': 'arcpatch-D696', 'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session1', 'PWD': '/home/antoine/swh/swh-environment/swh-docs', 'PYTHONHASHSEED': '3756312444', 'QT_AUTO_SCREEN_SCALE_FACTOR': '0', 'TERM': 'xterm-256color', 'PAM_KWALLET5_LOGIN': '/tmp/kwallet5_antoine.socket', 'LANG': 'fr_FR.UTF-8', 'DEBFULLNAME': 'Antoine Lambert', 'QMLSCENE_DEVICE': '', 'OLDPWD': '/home/antoine', '__GIT_PROMPT_SHOW_UNTRACKED_FILES': 'normal', 'DEBEMAIL': 'antoine.lambert@inria.fr', 'XDG_DATA_DIRS': '/usr/share:/usr/share:/usr/local/share', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:', 'XDG_SESSION_ID': '3', 'PATH': '/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin:/usr/lib/go-1.10/bin:/home/antoine/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/games', 'XDG_CURRENT_DESKTOP': 'KDE', 'KDE_SESSION_VERSION': '5', '__GIT_PROMPT_SHOW_UPSTREAM': '', 'USER': 'antoine', 'KONSOLE_PROFILE_NAME': 'Profile 1', 'KONSOLE_DBUS_SERVICE': ':1.195', 'WINDOWID': '2097158', 'DESKTOP_SESSION': '/usr/share/xsessions/plasma', 'COLORFGBG': '15;0', 'SHELL_SESSION_ID': 'b8985c2db3a44253bdc305f3674a69c9', 'XDG_VTNR': '7', 'KONSOLE_DBUS_SESSION': '/Sessions/5', 'DISPLAY': ':0', 'QSG_RENDER_LOOP': '', 'GTK_IM_MODULE': 'xim', 'SHLVL': '1', 'XAUTHORITY': '/home/antoine/.Xauthority', 'VIRTUAL_ENV': '/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev', 'SSH_AGENT_PID': '2600', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'COLORTERM': 'truecolor', 'PROFILEHOME': '', 'XCURSOR_THEME': 'breeze_cursors', 'XDG_SESSION_TYPE': 'x11', 'LOGNAME': 'antoine', 'SHELL': '/usr/bin/zsh', 'GPG_TTY': '/dev/pts/10', 'GPG_AGENT_INFO': '/run/user/1000/gnupg/S.gpg-agent:0:1', 'GTK_MODULES': 'gail:atk-bridge', 'XDG_SEAT': 'seat0', 'QT_ACCESSIBILITY': '1', 'XDG_SESSION_CLASS': 'user', '__GIT_PROMPT_IGNORE_SUBMODULES': '', 'GS_LIB': '/home/antoine/.fonts', 'QT_LINUX_ACCESSIBILITY_ALWAYS_ON': '1', '__GIT_PROMPT_IGNORE_STASH': '', 'KDE_FULL_SESSION': 'true', 'KONSOLE_DBUS_WINDOW': '/Windows/1', 'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0', 'SESSION_MANAGER': 'local/guggenheim:@/tmp/.ICE-unix/2704,unix/guggenheim:/tmp/.ICE-unix/2704', 'KDE_SESSION_UID': '1000', 'HOME': '/home/antoine', '__GIT_PROMPT_SHOW_CHANGED_FILES_COUNT': '1', '_': '/usr/bin/tox', 'XMODIFIERS': '@im=uim', 'LANGUAGE': '', 'XDG_RUNTIME_DIR': '/run/user/1000'}

Could not open requirements file: [Errno 2] Aucun fichier ou dossier de ce type: ' requirements-swh-dev.txt'

ERROR: could not install deps [Django < 2, -r requirements-swh-dev.txt, pifpaf]; v = InvocationError('/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin/pip install Django < 2 -r requirements-swh-dev.txt pifpaf (see /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/log/sphinx-dev-1.log)', 1)
___________________________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________________________
ERROR:   sphinx-dev: could not install deps [Django < 2, -r requirements-swh-dev.txt, pifpaf]; v = InvocationError('/home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/bin/pip install Django < 2 -r requirements-swh-dev.txt pifpaf (see /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx-dev/log/sphinx-dev-1.log)', 1)

For the record, I have all swh module paths registered in the PYTHONPATH env variable.

Also having dedicated make targets that call tox with the right arguments would be useful.

tox.ini
7

django -> Django

15

django -> Django

I also have the following sphinx warnings when calling tox -e sphinx (with D697 applied)

/home/antoine/swh/swh-environment/swh-docs/docs/swh-core/index.rst:12: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.core'
looking for now-outdated files... none found
pickling environment... fait
checking consistency... /home/antoine/swh/swh-environment/swh-docs/docs/apidoc/modules.rst: WARNING: document isn't included in any toctree
swh/docs/sphinx/conf.py
135–137

The django setup can be moved in the setup function.

This way it will be called only once instead of each time a rst source file has been read

For the record, I have all swh module paths registered in the PYTHONPATH env variable.

I haven't checked with a PYTHONPATH based dev setup indeed. Not sure I really want to, however. Convince me! πŸ˜„

Also having dedicated make targets that call tox with the right arguments would be useful.

Indeed.

tox -e sphinx seems to work but the resulting Module index is incomplete (I only have swh.archiver in it)

Can you try with your PYTHONPATH unset?

I have the same results when unsetting PYTHONPATH.

Seems that even when it is set, the correct path to swh module is picked.

The SWHPKGDIR Makefile variable is set to /home/antoine/swh/swh-environment/swh-docs/.tox/sphinx/lib/python3.5/site-packages/swh
in both cases.

douardda updated this revision to Diff 2210.Fri, Nov 23, 10:50 AM

tox: enforce python3 for all environments

douardda updated this revision to Diff 2211.Fri, Nov 23, 11:01 AM

and fix sphinx-dev for old tox versions

douardda updated this revision to Diff 2213.Fri, Nov 23, 11:20 AM

conf: exclude swh. part when building the module index

anlambert accepted this revision.Fri, Nov 23, 11:31 AM

Apart the django settings initialization that should be moved directly in the setup function, LGTM.

This revision is now accepted and ready to land.Fri, Nov 23, 11:31 AM
douardda updated this revision to Diff 2215.Fri, Nov 23, 2:01 PM

Add a few tweaks and tunings

  • conf: simplify a bit the django hack
  • Ugly hackish fix to try to make autodoc happy with django
  • use swh.objstorage[testing] dep to have libcloud and azure stuff installed
douardda updated this revision to Diff 2218.Fri, Nov 23, 2:34 PM

rework git history

This revision was automatically updated to reflect the committed changes.