Page MenuHomeSoftware Heritage

Makefile: Speedup sphinx-apidoc execution significantly
ClosedPublic

Authored by anlambert on Nov 17 2022, 6:25 PM.

Details

Summary

Previously the file patterns that sphinx-apidoc must exclude, stored
in the APIDOC_EXCLUDES variables, were expanded by the shell before
executing the tool resulting in a really long list of files to exclude.
Each time a file was processed by sphinx-apidoc, its path was compared
against that long list of files which was significantly slowing down
the apidoc generation process.

So ensure file exclude patterns are quoted to avoid shell expansion
and greatly speedup sphinx-apidoc execution.

Diff Detail

Repository
rDDOC Development documentation
Branch
speedup-sphinx-apidoc
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32848
Build 51471: Phabricator diff pipeline on Jenkins for swh-docsJenkins console · Jenkins
Build 51470: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D8855 (id=31917)

Rebasing onto 65edd83bf1...

Current branch diff-target is up to date.
Changes applied before test
commit 531487d409af4d2caed5d1c53ae03924f2b628df
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 17 18:17:38 2022 +0100

    Makefile: Speedup sphinx-apidoc execution significantly
    
    Previously the file patterns that sphinx-apidoc must exclude, stored
    in the APIDOC_EXCLUDES variables, were expanded by the shell before
    executing the tool resulting in a really long list of files to exclude.
    Each time a file was processed by sphinx-apidoc, its path was compared
    against that long list of files which was significantly slowing down
    the apidoc generation process.
    
    So ensure file exclude patterns are quoted to avoid shell expansion
    and greatly speedup sphinx-apidoc execution.

Link to build: https://jenkins.softwareheritage.org/job/DDOC/job/build-on-diff/262/
See console output for more information: https://jenkins.softwareheritage.org/job/DDOC/job/build-on-diff/262/console

Harbormaster returned this revision to the author for changes because remote builds failed.Nov 17 2022, 6:27 PM
Harbormaster failed remote builds in B32847: Diff 31917!

Build has FAILED

Patch application report for D8855 (id=31917)

Rebasing onto 65edd83bf1...

Current branch diff-target is up to date.
Changes applied before test
commit 531487d409af4d2caed5d1c53ae03924f2b628df
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 17 18:17:38 2022 +0100

    Makefile: Speedup sphinx-apidoc execution significantly
    
    Previously the file patterns that sphinx-apidoc must exclude, stored
    in the APIDOC_EXCLUDES variables, were expanded by the shell before
    executing the tool resulting in a really long list of files to exclude.
    Each time a file was processed by sphinx-apidoc, its path was compared
    against that long list of files which was significantly slowing down
    the apidoc generation process.
    
    So ensure file exclude patterns are quoted to avoid shell expansion
    and greatly speedup sphinx-apidoc execution.

Link to build: https://jenkins.softwareheritage.org/job/DDOC/job/build-on-diff/263/
See console output for more information: https://jenkins.softwareheritage.org/job/DDOC/job/build-on-diff/263/console

Add missing / in patsubst command

Build is green

Patch application report for D8855 (id=31918)

Rebasing onto 65edd83bf1...

Current branch diff-target is up to date.
Changes applied before test
commit 2f5cbf0cea45cc7c8bb508ada48668322b6fe1e1
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Nov 17 18:17:38 2022 +0100

    Makefile: Speedup sphinx-apidoc execution significantly
    
    Previously the file patterns that sphinx-apidoc must exclude, stored
    in the APIDOC_EXCLUDES variables, were expanded by the shell before
    executing the tool resulting in a really long list of files to exclude.
    Each time a file was processed by sphinx-apidoc, its path was compared
    against that long list of files which was significantly slowing down
    the apidoc generation process.
    
    So ensure file exclude patterns are quoted to avoid shell expansion
    and greatly speedup sphinx-apidoc execution.

See https://jenkins.softwareheritage.org/job/DDOC/job/build-on-diff/264/ for more details.

vlorentz added a subscriber: vlorentz.

Great, thanks!

FWIW I also submitted a PR to Sphinx to fix the pathological runtimes there too: https://github.com/sphinx-doc/sphinx/pull/10981

This revision is now accepted and ready to land.Nov 17 2022, 7:53 PM