Page MenuHomeSoftware Heritage

apidoc generation is not performed on swh modules during dev build
Closed, MigratedEdits Locked

Description

The swh dependencies are not installed during the dev build[1] which leads to a build failure.

/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-auth/index.rst:8: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.auth'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-core/index.rst:8: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.core'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-deposit/index.rst:17: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.deposit'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-fuse/index.rst:6: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.fuse'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-graph/index.rst:6: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.graph'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-indexer/index.rst:22: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.indexer'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-journal/index.rst:12: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.journal'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-lister/index.rst:27: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.lister'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-loader-core/index.rst:16: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.loader.core'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-loader-core/index.rst:16: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.loader.package'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-loader-git/index.rst:12: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.loader.git'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-loader-mercurial/index.rst:12: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.loader.mercurial'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-loader-svn/index.rst:12: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.loader.svn'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-model/index.rst:8: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.model'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-objstorage-replayer/index.rst:17: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.objstorage.replayer'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-objstorage/index.rst:12: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.objstorage'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-scanner/index.rst:12: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.scanner'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-scheduler/index.rst:165: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.scheduler'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-search/index.rst:33: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.search'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-storage/index.rst:90: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.storage'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-vault/index.rst:22: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.vault'
/home/jenkins/workspace/DDOC/dev/swh-docs/docs/swh-web-client/index.rst:90: WARNING: toctree contains reference to nonexisting document 'apidoc/swh.web.client'
make: *** [Makefile:70: sphinx/html] Error 1

It seems it's related to the work done during D5525

[1] https://jenkins.softwareheritage.org/job/DDOC/job/dev

Event Timeline

vsellier changed the task status from Open to Work in Progress.May 5 2021, 10:25 AM
vsellier triaged this task as Normal priority.
vsellier created this task.

This is not a dependencies installation issue. When the swh packages are installed in dev mode (pip -e) and the global doc is built, the apidoc files are generated in swh-docs/docs/apidoc, but as those files are linked in the index.rst file of each package, sphinx expects them to be in swh-docs/docs/<swh_package>/apidoc so the warnings.

I think the simplest fix is to add a symbolic link to swh-docs/docs/apidoc in each swh-docs/docs/<swh_package>/ directory when building the global doc and remove it afterwards.

Aparrently, the modules are well installed but the sphinx-apidoc is not detecting them due to the local installation

jenkins@4e5220b923d8:~/workspace/swh-environment/swh-docs$ .tox/sphinx-dev/bin/python3 -m pip list | grep swh
swh.auth                      0.5.4                           /home/jenkins/workspace/swh-environment/swh-auth
swh.core                      0.13.2.dev1+g7d42035            /home/jenkins/workspace/swh-environment/swh-core
swh.counters                  0.7.1.dev1+g6a44a84             /home/jenkins/workspace/swh-environment/swh-counters
swh.deposit                   0.13.6                          /home/jenkins/workspace/swh-environment/swh-deposit
swh.docs                      0.0.1.dev334+g044cb9b.d20210505
swh.fuse                      1.0.3                           /home/jenkins/workspace/swh-environment/swh-fuse
swh.graph                     0.3.2.dev3+g62c2fd3             /home/jenkins/workspace/swh-environment/swh-graph
swh.icinga-plugins            0.3.1.dev1+g8878925             /home/jenkins/workspace/swh-environment/swh-icinga-plugins
swh.indexer                   0.7.1.dev4+g8f1fb0f             /home/jenkins/workspace/swh-environment/swh-indexer
swh.journal                   0.7.2.dev8+g2972c7a             /home/jenkins/workspace/swh-environment/swh-journal
swh.lister                    1.1.0                           /home/jenkins/workspace/swh-environment/swh-lister
swh.loader.core               0.22.1.dev2+g0e4bb4b            /home/jenkins/workspace/swh-environment/swh-loader-core
swh.loader.git                0.9.2.dev1+g15e12fa             /home/jenkins/workspace/swh-environment/swh-loader-git
swh.loader.mercurial          0.5.1.dev4+g8884714             /home/jenkins/workspace/swh-environment/swh-loader-mercurial
swh.loader.svn                0.7.1                           /home/jenkins/workspace/swh-environment/swh-loader-svn
swh.model                     2.4.2.dev1+gdf036ef             /home/jenkins/workspace/swh-environment/swh-model
swh.objstorage                0.2.3                           /home/jenkins/workspace/swh-environment/swh-objstorage
swh.objstorage.replayer       0.2.2                           /home/jenkins/workspace/swh-environment/swh-objstorage-replayer
swh.scanner                   0.4.2.dev1+g30b40cc             /home/jenkins/workspace/swh-environment/swh-scanner
swh.scheduler                 0.13.1.dev5+gbab557e            /home/jenkins/workspace/swh-environment/swh-scheduler
swh.search                    0.8.1                           /home/jenkins/workspace/swh-environment/swh-search
swh.storage                   0.27.5.dev4+g051b7715           /home/jenkins/workspace/swh-environment/swh-storage
swh.vault                     0.5.2.dev2+gf87dd54             /home/jenkins/workspace/swh-environment/swh-vault
swh.web                       0.0.307.dev3+gf8c750b6          /home/jenkins/workspace/swh-environment/swh-web
swh.web.client                0.3.1.dev1+g4b610ad             /home/jenkins/workspace/swh-environment/swh-web-client
sphinx-apidoc \
	--implicit-namespaces \
	--templatedir=../swh/docs/templates/ \
	--maxdepth=3 \
	--ext-viewcode --separate \
	-o apidoc \
	/home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh \
	/home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/tests /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/tests/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/tests/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/*/tests/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/migrations /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/migrations/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/migrations/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/*/migrations/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/wsgi.py /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/wsgi.py /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/*/wsgi.py /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/*/*/*/wsgi.py /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/deposit/settings/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/web/settings/* /home/jenkins/workspace/swh-environment/swh-docs/.tox/sphinx-dev/lib/python3.7/site-packages/swh/dataset/*
Creating file apidoc/swh.rst.
Creating file apidoc/swh.docs.rst.
Creating file apidoc/swh.docs.django_settings.rst.
Creating file apidoc/swh.docs.sphinx.rst.
Creating file apidoc/swh.docs.sphinx.conf.rst.
Creating file apidoc/swh.docs.sphinx.view_in_phabricator.rst.
Creating file apidoc/modules.rst.

@vsellier, you should be able to create the required symbolic links by adding an extra step in the apidoc-stamp make target.

You could add new functions in the ln-sphinx-subprojects file to do so.

thanks a lot @anlambert, I will look in that direction

vsellier renamed this task from swh dependencies not installed during dev build to apidoc generation is not performed on swh modules during dev build.May 5 2021, 11:36 AM

@anlambert I didn't succeed to work with link to the modules, but I found another way with a standard installation (i.e without the -e flag) we can force with a flag on the ci (cf D5681).
I really not sure if it's the good approach or not.