Page MenuHomeSoftware Heritage

doc generation broken with sphinx < 1.5
Closed, ResolvedPublic

Description

Developer documentation no longer builds (this time at all, rather than just throwing ignorable errors), apparently due to swh-deposit.
Here's the traceback:

~/swh-environment/swh-docs/docs $ make
Running Sphinx v1.6.4
loading translations [en]... done
Generating grammar tables from /usr/share/sphinx/pycode/Grammar-py3.txt
loading pickled environment... done
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 11 source files that are out of date
updating environment: [extensions changed] 133 added, 0 changed, 1 removed
DEPOSIT_STATUS True[  9%] swh-deposit/apidoc/modules                                           
content True
directory True
executable_content True
revision True
symlink True
ALGORITHMS True
DEFAULT_ALGORITHMS True
HASH_BLOCK_SIZE True
type True

Exception occurred:
  File "/usr/lib/python3/dist-packages/sphinxcontrib/httpdomain.py", line 439, in http_header_role
    msg = inliner.reporter.error('%s is not unknown HTTP header' % header,
AttributeError: 'NoneType' object has no attribute 'reporter'
The full traceback has been saved in /tmp/sphinx-err-myex0h6h.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Makefile:39: recipe for target 'sphinx/html' failed
make: *** [sphinx/html] Error 1

Event Timeline

zack created this task.Oct 21 2017, 5:13 PM

here is the traceback

Hello,

I don't reproduce... thus hard to troubleshoot.

I tested from the latest commits in swh-docs (which got updated yesterday).
I cleaned up eventual remaining tryout files first (make distclean and make, cf. P184).

From the stacktrace you provided:

  • regarding version, we seem to share the same package version (i use sphinx's debian unstable version 1.6.4).
ii  sphinx-doc                                            1.4.9-2                         all                             documentation generator for Python projects - documentation
ii  python3-recommonmark                                  0.4.0+ds-2                      all                             CommonMark utility for Docutils and Sphinx projects -- Python 3
ii  python3-recommonmark                                  0.4.0+ds-2                      all                             CommonMark utility for Docutils and Sphinx projects -- Python 3
ii  python3-sphinxcontrib.httpdomain                      1.3.0-1                         all                             Sphinx domain for HTTP APIs - Python 3.x
ii  python3-sphinx                                        1.6.4-2                         all                             documentation generator for Python projects (implemented in Python 3)
  • it seems that somehow the error is not understanding the`mailheader` keyword (stacktrace: there is an error about a mailheader key which does not exist which triggers another error)...

swh-deposit does not use that keyword.

Ok, looking for that mailheader in swh-environment, i'm finding references in swh-web as sphinx keyword :mailheader:.
(http://www.sphinx-doc.org/en/1.6.4/markup/inline.html#role-mailheader)

As it works locally, that leads me to think that you may be missing something somehow.

As a note, I recall having seen a new dependency in the latest swh-docs commit regarding the configuration (there sphinx.ext.extlinks, apt-file search says debian's sphinx-docs package)...
I don't think it's related though (since i did not have it prior to running and everything was buiding fine).

Maybe running the cd ../../swh-web/docs; make html could show more?

apparently due to swh-deposit.

The initial sphinx output is misleading. Even though i don't reproduce the error, on my side (cf. P184), the sphinx message output seems to be scrambled when reaching swh-deposit.
Indeed, the percentage stops at swh-deposit (since computing stuff for it prints noise somehow... i don't see why though since i don't see any dangling print statement there...).

Cheers,

zack reassigned this task from ardumont to anlambert.
zack added a subscriber: ardumont.

The initial sphinx output is misleading. Even though i don't reproduce the error, on my side (cf. P184), the sphinx message output seems to be scrambled when reaching swh-deposit.

Your tryout doesn't show if you've updated your entire swh-environment or not. Have you?

But you're right that the output related to swh-deposit is misleading. I've now cornered the issue as related to swh-web. If I remove the symlink to swh-web after all symlinks are created and rebuild from scratch, the build failure goes away. I'm reassigning/tagging as needed.

ardumont added a comment.EditedOct 21 2017, 6:37 PM

ahah

Something in the reasoning were not perfectly sound.
That made me check for any dangling pip3 installed software...
I did have sphinxcontrib-httpdomain-1.5.0 installed...

Removing it, i now reproduced the error:

python3 -c 'import sphinx.apidoc, sys; sys.exit(sphinx.apidoc.main(sys.argv))' --implicit-namespaces -o apidoc ..  . ../setup.py '../swh/*/tests' '../swh/*/tests/*' '../swh/*/*/tests/*' '../swh/*/*/*/tests/*' '../swh/*/migrations' '../swh/*/migrations/*' '../swh/*/*/migrations/*' '../swh/*/*/*/migrations/*' '../bin/*' '../utils/*'
Creating file apidoc/swh.rst.
Creating file apidoc/swh.web.rst.
#   sphinx.ext.extlinks (1.6.4) from /usr/lib/python3/dist-packages/sphinx/ext/extlinks.py
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sphinxcontrib/httpdomain.py", line 589, in resolve_xref
    info = self.data[str(typ)][target]
KeyError: 'mailheader'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/sphinx/cmdline.py", line 306, in main
    app.build(opts.force_all, filenames)
  File "/usr/lib/python3/dist-packages/sphinx/application.py", line 339, in build
    self.builder.build_update()
  File "/usr/lib/python3/dist-packages/sphinx/builders/__init__.py", line 331, in build_update
    'out of date' % len(to_build))
  File "/usr/lib/python3/dist-packages/sphinx/builders/__init__.py", line 397, in build
    self.write(docnames, list(updated_docnames), method)
  File "/usr/lib/python3/dist-packages/sphinx/builders/__init__.py", line 434, in write
    self._write_serial(sorted(docnames))
  File "/usr/lib/python3/dist-packages/sphinx/builders/__init__.py", line 441, in _write_serial
    doctree = self.env.get_and_resolve_doctree(docname, self)
  File "/usr/lib/python3/dist-packages/sphinx/environment/__init__.py", line 910, in get_and_resolve_doctree
    self.apply_post_transforms(doctree, docname)
  File "/usr/lib/python3/dist-packages/sphinx/environment/__init__.py", line 957, in apply_post_transforms
    transformer.apply_transforms()
  File "/usr/lib/python3/dist-packages/sphinx/transforms/__init__.py", line 92, in apply_transforms
    Transformer.apply_transforms(self)
  File "/usr/lib/python3/dist-packages/docutils/transforms/__init__.py", line 171, in apply_transforms
    transform.apply(**kwargs)
  File "/usr/lib/python3/dist-packages/sphinx/transforms/post_transforms/__init__.py", line 89, in apply
    typ, target, node, contnode)
  File "/usr/lib/python3/dist-packages/sphinxcontrib/httpdomain.py", line 595, in resolve_xref
    return http_header_role(None, text, text, None, None)[0][0]
  File "/usr/lib/python3/dist-packages/sphinxcontrib/httpdomain.py", line 439, in http_header_role
    msg = inliner.reporter.error('%s is not unknown HTTP header' % header,
AttributeError: 'NoneType' object has no attribute 'reporter'

So the immediate workaround is to install a latest version of sphinxcontrib-httpdomain version (through pip3 which is at 1.5.0, debian is at 1.3.0)...
The more reasonable fix is to upgrade the python3-sphinxcontrib.httpdomain debian package.

Note that there seems to be something fishy in sphinx though.
I don't see in sphinx documentation the need for that library to use the :mailheader: keyword...

Cheers,

Your tryout doesn't show if you've updated your entire swh-environment or not. Have you?

Yes, i did but should have mentioned it.
The only way to be in sync as a team is to be on the latest (swh/puppet)-environment (reasonably true, all the more on week-end :)

zack added a comment.Oct 22 2017, 12:14 PM

I did have sphinxcontrib-httpdomain-1.5.0 installed...
Removing it, i now reproduced the error:

Nice catch, so that't the issue.

An even saner fix would be to avoid depending on on sphinx >= 1.5, but will see with @anlambert whether he has strongs reasons for that dependencies or if it was just an overlook.

zack renamed this task from doc generation broken: "%s is not unknown HTTP header" to doc generation broken with sphinx < 1.5.Oct 22 2017, 12:14 PM

For the sake of completeness...

As it works locally, that leads me to think that you may be missing something somehow.

Well, that was the 'not perfectly sound' argument.
It could have been the other way around, that is i could have something more in my environment.
As a matter of fact, i did.

Note that there seems to be something fishy in sphinx though.
I don't see in sphinx documentation the need for that library to use the :mailheader: keyword...

No. It's not sphinx which is at fault, it's probably more the sphinxcontrib library...

It's probably the python3-sphinxcontrib.httpversion at version 1.3.0 which does not expect new keywords from a superior version of sphinx (1.6.4).
Indeed, the python3-sphinxcontrib.httpversion's setup.py requires statements is for Sphinx >= 1.0...
So 1.6.4 is ok but they probably did not account for new 'keywords' in there somehow...

Effectively I was using sphinxcontrib.httpdomain 1.5 installed trough pip so I did not catch the issue. But I also got mistaken by httpdomain doc which makes uses of the deprecated :mailheader: directive in the examples ...

I have also found another issue regarding the 'Link' header doc which is not present in httpdomain 1.3.