Page MenuHomeSoftware Heritage
Differential D3907

cli: speedup the `swh` cli command startup time
ClosedPublic
Actions

Authored by douardda on Sep 10 2020, 3:59 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2575: Investigate if/how we could improve `swh` cli command startup time
Commits
rDSCHf5c815435bf6: cli: speedup the `swh` cli command startup time
Summary

by moving import statements in functions and using conditional
import of typechecking modules (especially StorageInterface which
triggers the loading of 300+ modules).

Related to T2575.

Diff Detail

Unit TestsFailed
View All

TimeTest
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.cli.task::swh.scheduler.cli.task.pretty_print_task
079 Pretty-print a task 080 081 If 'full' is True, also print the status and priority fields.
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.cli.utils::swh.scheduler.cli.utils.parse_options
349 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.es.test_backend_es::test_backend_setup_basic
347 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.es.test_backend_es::test_backend_setup_index
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.scheduler.tests.es.test_backend_es::test_get_elasticsearch
View Full Test Results (1 Failed · 121 Passed · 1 Skipped)

Event Timeline

douardda created this revision.Sep 10 2020, 3:59 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 10 2020, 3:59 PM
swh-public-ci added a comment.Sep 10 2020, 4:01 PM

Build has FAILED

Patch application report for D3907 (id=13766)

Rebasing onto b24be0c78f...

Current branch diff-target is up to date.
Changes applied before test
commit 8f15eb8ab23cfd82dd1763e0340946b69fbb1868
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Sep 10 11:25:08 2020 +0200

    cli: speedup the `swh` cli command startup time
    
    by moving import statements in functions and using conditional
    import of typechecking modules (especially StorageInterface which
    triggers the loading of 300+ modules).
    
    Related to T2575.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/58/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/58/console

Harbormaster failed remote builds in B14993: Diff 13766!Sep 10 2020, 4:01 PM
douardda updated this revision to Diff 13779.Sep 10 2020, 4:23 PM

add warnings

swh-public-ci added a comment.Sep 10 2020, 4:25 PM

Build has FAILED

Patch application report for D3907 (id=13779)

Rebasing onto b24be0c78f...

Current branch diff-target is up to date.
Changes applied before test
commit 8f15eb8ab23cfd82dd1763e0340946b69fbb1868
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Sep 10 11:25:08 2020 +0200

    cli: speedup the `swh` cli command startup time
    
    by moving import statements in functions and using conditional
    import of typechecking modules (especially StorageInterface which
    triggers the loading of 300+ modules).
    
    Related to T2575.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/59/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/59/console

Harbormaster failed remote builds in B15006: Diff 13779!Sep 10 2020, 4:25 PM
vlorentz added a subscriber: vlorentz.Sep 10 2020, 4:40 PM
vlorentz added inline comments.
swh/scheduler/cli/task.py
302–308

you can probably do it with strings, which is cleaner:

def iter_origins(
    storage: "StorageInterface",
    page_token: "Optional[str]" = None,
) -> "Iterator[Origin]":
douardda updated this revision to Diff 13808.Sep 10 2020, 5:32 PM

use string annotation instead of comment one, as suggested by vlorentz

swh-public-ci added a comment.Sep 10 2020, 5:34 PM

Build has FAILED

Patch application report for D3907 (id=13808)

Rebasing onto b24be0c78f...

Current branch diff-target is up to date.
Changes applied before test
commit a5cfc1ca68f3408ec0f73a15dc5f1a95bc170060
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Sep 10 11:25:08 2020 +0200

    cli: speedup the `swh` cli command startup time
    
    by moving import statements in functions and using conditional
    import of typechecking modules (especially StorageInterface which
    triggers the loading of 300+ modules).
    
    Related to T2575.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/60/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/60/console

Harbormaster failed remote builds in B15033: Diff 13808!Sep 10 2020, 5:34 PM
douardda updated this revision to Diff 13812.Sep 10 2020, 5:46 PM

fix a doctest

swh-public-ci added a comment.Sep 10 2020, 5:48 PM

Build is green

Patch application report for D3907 (id=13812)

Rebasing onto b24be0c78f...

Current branch diff-target is up to date.
Changes applied before test
commit f5c815435bf604e409a322dd543ef513c01afc13
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Sep 10 11:25:08 2020 +0200

    cli: speedup the `swh` cli command startup time
    
    by moving import statements in functions and using conditional
    import of typechecking modules (especially StorageInterface which
    triggers the loading of 300+ modules).
    
    Related to T2575.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/61/ for more details.

Harbormaster completed remote builds in B15037: Diff 13812.Sep 10 2020, 5:49 PM
vlorentz accepted this revision.Sep 10 2020, 6:48 PM
This revision is now accepted and ready to land.Sep 10 2020, 6:48 PM
Closed by commit rDSCHf5c815435bf6: cli: speedup the `swh` cli command startup time (authored by douardda). · Explain WhySep 11 2020, 10:33 AM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDSCHf5c815435bf6: cli: speedup the `swh` cli command startup time.

Revision Contents
Changeset List

PathSize
swh/
scheduler/
cli/
2 lines
2 lines
7 lines
46 lines
5 lines
12 lines

Diff 13766

View Options

swh/scheduler/cli/__init__.py

Loading...
View Options

swh/scheduler/cli/admin.py

Loading...
View Options

swh/scheduler/cli/celery_monitor.py

Loading...
View Options

swh/scheduler/cli/task.py

Loading...
View Options

swh/scheduler/cli/task_type.py

Loading...
View Options

swh/scheduler/cli/utils.py

Loading...
About · Developer information · Legal