Page MenuHomeSoftware Heritage
Differential D3907

cli: speedup the `swh` cli command startup time
ClosedPublic
Actions

Authored by douardda on Sep 10 2020, 3:59 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2575: Investigate if/how we could improve `swh` cli command startup time
Commits
rDSCHf5c815435bf6: cli: speedup the `swh` cli command startup time
Summary

by moving import statements in functions and using conditional
import of typechecking modules (especially StorageInterface which
triggers the loading of 300+ modules).

Related to T2575.

Diff Detail

Repository
rDSCH Scheduling utilities
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Sep 10 2020, 3:59 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 10 2020, 3:59 PM
swh-public-ci added a comment.Sep 10 2020, 4:01 PM

Build has FAILED

Patch application report for D3907 (id=13766)

Rebasing onto b24be0c78f...

Current branch diff-target is up to date.
Changes applied before test
commit 8f15eb8ab23cfd82dd1763e0340946b69fbb1868
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Sep 10 11:25:08 2020 +0200

    cli: speedup the `swh` cli command startup time
    
    by moving import statements in functions and using conditional
    import of typechecking modules (especially StorageInterface which
    triggers the loading of 300+ modules).
    
    Related to T2575.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/58/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/58/console

Harbormaster failed remote builds in B14993: Diff 13766!Sep 10 2020, 4:01 PM
douardda updated this revision to Diff 13779.Sep 10 2020, 4:23 PM

add warnings

swh-public-ci added a comment.Sep 10 2020, 4:25 PM

Build has FAILED

Patch application report for D3907 (id=13779)

Rebasing onto b24be0c78f...

Current branch diff-target is up to date.
Changes applied before test
commit 8f15eb8ab23cfd82dd1763e0340946b69fbb1868
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Sep 10 11:25:08 2020 +0200

    cli: speedup the `swh` cli command startup time
    
    by moving import statements in functions and using conditional
    import of typechecking modules (especially StorageInterface which
    triggers the loading of 300+ modules).
    
    Related to T2575.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/59/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/59/console

Harbormaster failed remote builds in B15006: Diff 13779!Sep 10 2020, 4:25 PM
vlorentz added a subscriber: vlorentz.Sep 10 2020, 4:40 PM
vlorentz added inline comments.
swh/scheduler/cli/task.py
303–307

you can probably do it with strings, which is cleaner:

def iter_origins(
    storage: "StorageInterface",
    page_token: "Optional[str]" = None,
) -> "Iterator[Origin]":
douardda updated this revision to Diff 13808.Sep 10 2020, 5:32 PM

use string annotation instead of comment one, as suggested by vlorentz

swh-public-ci added a comment.Sep 10 2020, 5:34 PM

Build has FAILED

Patch application report for D3907 (id=13808)

Rebasing onto b24be0c78f...

Current branch diff-target is up to date.
Changes applied before test
commit a5cfc1ca68f3408ec0f73a15dc5f1a95bc170060
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Sep 10 11:25:08 2020 +0200

    cli: speedup the `swh` cli command startup time
    
    by moving import statements in functions and using conditional
    import of typechecking modules (especially StorageInterface which
    triggers the loading of 300+ modules).
    
    Related to T2575.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/60/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/60/console

Harbormaster failed remote builds in B15033: Diff 13808!Sep 10 2020, 5:34 PM
douardda updated this revision to Diff 13812.Sep 10 2020, 5:46 PM

fix a doctest

swh-public-ci added a comment.Sep 10 2020, 5:48 PM

Build is green

Patch application report for D3907 (id=13812)

Rebasing onto b24be0c78f...

Current branch diff-target is up to date.
Changes applied before test
commit f5c815435bf604e409a322dd543ef513c01afc13
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Sep 10 11:25:08 2020 +0200

    cli: speedup the `swh` cli command startup time
    
    by moving import statements in functions and using conditional
    import of typechecking modules (especially StorageInterface which
    triggers the loading of 300+ modules).
    
    Related to T2575.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/61/ for more details.

Harbormaster completed remote builds in B15037: Diff 13812.Sep 10 2020, 5:49 PM
vlorentz accepted this revision.Sep 10 2020, 6:48 PM
This revision is now accepted and ready to land.Sep 10 2020, 6:48 PM
Closed by commit rDSCHf5c815435bf6: cli: speedup the `swh` cli command startup time (authored by douardda). · Explain WhySep 11 2020, 10:33 AM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDSCHf5c815435bf6: cli: speedup the `swh` cli command startup time.

Revision Contents
Changeset List

PathSize
swh/
scheduler/
cli/
2 lines
2 lines
7 lines
45 lines
5 lines
12 lines

Diff 13823

View Options

swh/scheduler/cli/__init__.py

# Copyright (C) 2019-2020 The Software Heritage developers # Copyright (C) 2019-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import base64 import base64
from typing import Any, Dict, Iterable, Iterator, List, Optional from typing import Any, Dict, Iterable, Iterator, List, Optional
from elasticsearch import Elasticsearch from elasticsearch import Elasticsearch, helpers
from elasticsearch.helpers import bulk, scan
import msgpack import msgpack
from swh.indexer import codemeta from swh.indexer import codemeta
from swh.model import model from swh.model import model
from swh.model.identifiers import origin_identifier from swh.model.identifiers import origin_identifier
from swh.search.interface import PagedResult from swh.search.interface import PagedResult
from swh.search.metrics import send_metric, timed
def _sanitize_origin(origin): def _sanitize_origin(origin):
origin = origin.copy() origin = origin.copy()
# Whitelist fields to be saved in Elasticsearch # Whitelist fields to be saved in Elasticsearch
res = {"url": origin.pop("url")} res = {"url": origin.pop("url")}
for field_name in ("intrinsic_metadata", "has_visits"): for field_name in ("intrinsic_metadata", "has_visits"):
Show All 38 Lines def __init__(self, hosts: List[str], index_prefix=None):
self._backend = Elasticsearch(hosts=hosts) self._backend = Elasticsearch(hosts=hosts)
self.index_prefix = index_prefix self.index_prefix = index_prefix
self.origin_index = "origin" self.origin_index = "origin"
if index_prefix: if index_prefix:
self.origin_index = index_prefix + "_" + self.origin_index self.origin_index = index_prefix + "_" + self.origin_index
@timed
def check(self): def check(self):
return self._backend.ping() return self._backend.ping()
def deinitialize(self) -> None: def deinitialize(self) -> None:
"""Removes all indices from the Elasticsearch backend""" """Removes all indices from the Elasticsearch backend"""
self._backend.indices.delete(index="*") self._backend.indices.delete(index="*")
def initialize(self) -> None: def initialize(self) -> None:
Show All 33 Lines def initialize(self) -> None:
"type": "keyword", "type": "keyword",
} }
}, },
}, },
} }
}, },
) )
@timed
def flush(self) -> None: def flush(self) -> None:
self._backend.indices.refresh(index=self.origin_index) self._backend.indices.refresh(index=self.origin_index)
@timed
def origin_update(self, documents: Iterable[Dict]) -> None: def origin_update(self, documents: Iterable[Dict]) -> None:
documents = map(_sanitize_origin, documents) documents = map(_sanitize_origin, documents)
documents_with_sha1 = ( documents_with_sha1 = (
(origin_identifier(document), document) for document in documents (origin_identifier(document), document) for document in documents
) )
actions = [ actions = [
{ {
"_op_type": "update", "_op_type": "update",
"_id": sha1, "_id": sha1,
"_index": self.origin_index, "_index": self.origin_index,
"doc": {**document, "sha1": sha1,}, "doc": {**document, "sha1": sha1,},
"doc_as_upsert": True, "doc_as_upsert": True,
} }
for (sha1, document) in documents_with_sha1 for (sha1, document) in documents_with_sha1
] ]
bulk(self._backend, actions, index=self.origin_index)
indexed_count, errors = helpers.bulk(
self._backend, actions, index=self.origin_index
)
assert isinstance(errors, List) # Make mypy happy
send_metric("document:index", count=indexed_count, method_name="origin_update")
send_metric(
"document:index_error", count=len(errors), method_name="origin_update"
)
def origin_dump(self) -> Iterator[model.Origin]: def origin_dump(self) -> Iterator[model.Origin]:
results = scan(self._backend, index=self.origin_index) results = helpers.scan(self._backend, index=self.origin_index)
for hit in results: for hit in results:
yield self._backend.termvectors( yield self._backend.termvectors(
index=self.origin_index, id=hit["_id"], fields=["*"] index=self.origin_index, id=hit["_id"], fields=["*"]
) )
@timed
def origin_search( def origin_search(
self, self,
*, *,
url_pattern: Optional[str] = None, url_pattern: Optional[str] = None,
metadata_pattern: Optional[str] = None, metadata_pattern: Optional[str] = None,
with_visit: bool = False, with_visit: bool = False,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: int = 50, limit: int = 50,
▲ Show 20 Lines • Show All 86 Lines • Show Last 20 Lines
View Options

swh/scheduler/cli/admin.py

# Copyright (C) 2019-2020 The Software Heritage developers # Copyright (C) 2019-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import base64 import base64
from typing import Any, Dict, Iterable, Iterator, List, Optional from typing import Any, Dict, Iterable, Iterator, List, Optional
from elasticsearch import Elasticsearch from elasticsearch import Elasticsearch, helpers
from elasticsearch.helpers import bulk, scan
import msgpack import msgpack
from swh.indexer import codemeta from swh.indexer import codemeta
from swh.model import model from swh.model import model
from swh.model.identifiers import origin_identifier from swh.model.identifiers import origin_identifier
from swh.search.interface import PagedResult from swh.search.interface import PagedResult
from swh.search.metrics import send_metric, timed
def _sanitize_origin(origin): def _sanitize_origin(origin):
origin = origin.copy() origin = origin.copy()
# Whitelist fields to be saved in Elasticsearch # Whitelist fields to be saved in Elasticsearch
res = {"url": origin.pop("url")} res = {"url": origin.pop("url")}
for field_name in ("intrinsic_metadata", "has_visits"): for field_name in ("intrinsic_metadata", "has_visits"):
Show All 38 Lines def __init__(self, hosts: List[str], index_prefix=None):
self._backend = Elasticsearch(hosts=hosts) self._backend = Elasticsearch(hosts=hosts)
self.index_prefix = index_prefix self.index_prefix = index_prefix
self.origin_index = "origin" self.origin_index = "origin"
if index_prefix: if index_prefix:
self.origin_index = index_prefix + "_" + self.origin_index self.origin_index = index_prefix + "_" + self.origin_index
@timed
def check(self): def check(self):
return self._backend.ping() return self._backend.ping()
def deinitialize(self) -> None: def deinitialize(self) -> None:
"""Removes all indices from the Elasticsearch backend""" """Removes all indices from the Elasticsearch backend"""
self._backend.indices.delete(index="*") self._backend.indices.delete(index="*")
def initialize(self) -> None: def initialize(self) -> None:
Show All 33 Lines def initialize(self) -> None:
"type": "keyword", "type": "keyword",
} }
}, },
}, },
} }
}, },
) )
@timed
def flush(self) -> None: def flush(self) -> None:
self._backend.indices.refresh(index=self.origin_index) self._backend.indices.refresh(index=self.origin_index)
@timed
def origin_update(self, documents: Iterable[Dict]) -> None: def origin_update(self, documents: Iterable[Dict]) -> None:
documents = map(_sanitize_origin, documents) documents = map(_sanitize_origin, documents)
documents_with_sha1 = ( documents_with_sha1 = (
(origin_identifier(document), document) for document in documents (origin_identifier(document), document) for document in documents
) )
actions = [ actions = [
{ {
"_op_type": "update", "_op_type": "update",
"_id": sha1, "_id": sha1,
"_index": self.origin_index, "_index": self.origin_index,
"doc": {**document, "sha1": sha1,}, "doc": {**document, "sha1": sha1,},
"doc_as_upsert": True, "doc_as_upsert": True,
} }
for (sha1, document) in documents_with_sha1 for (sha1, document) in documents_with_sha1
] ]
bulk(self._backend, actions, index=self.origin_index)
indexed_count, errors = helpers.bulk(
self._backend, actions, index=self.origin_index
)
assert isinstance(errors, List) # Make mypy happy
send_metric("document:index", count=indexed_count, method_name="origin_update")
send_metric(
"document:index_error", count=len(errors), method_name="origin_update"
)
def origin_dump(self) -> Iterator[model.Origin]: def origin_dump(self) -> Iterator[model.Origin]:
results = scan(self._backend, index=self.origin_index) results = helpers.scan(self._backend, index=self.origin_index)
for hit in results: for hit in results:
yield self._backend.termvectors( yield self._backend.termvectors(
index=self.origin_index, id=hit["_id"], fields=["*"] index=self.origin_index, id=hit["_id"], fields=["*"]
) )
@timed
def origin_search( def origin_search(
self, self,
*, *,
url_pattern: Optional[str] = None, url_pattern: Optional[str] = None,
metadata_pattern: Optional[str] = None, metadata_pattern: Optional[str] = None,
with_visit: bool = False, with_visit: bool = False,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: int = 50, limit: int = 50,
▲ Show 20 Lines • Show All 86 Lines • Show Last 20 Lines
View Options

swh/scheduler/cli/celery_monitor.py

# Copyright (C) 2019-2020 The Software Heritage developers # Copyright (C) 2019-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import base64 import base64
from typing import Any, Dict, Iterable, Iterator, List, Optional from typing import Any, Dict, Iterable, Iterator, List, Optional
from elasticsearch import Elasticsearch from elasticsearch import Elasticsearch, helpers
from elasticsearch.helpers import bulk, scan
import msgpack import msgpack
from swh.indexer import codemeta from swh.indexer import codemeta
from swh.model import model from swh.model import model
from swh.model.identifiers import origin_identifier from swh.model.identifiers import origin_identifier
from swh.search.interface import PagedResult from swh.search.interface import PagedResult
from swh.search.metrics import send_metric, timed
def _sanitize_origin(origin): def _sanitize_origin(origin):
origin = origin.copy() origin = origin.copy()
# Whitelist fields to be saved in Elasticsearch # Whitelist fields to be saved in Elasticsearch
res = {"url": origin.pop("url")} res = {"url": origin.pop("url")}
for field_name in ("intrinsic_metadata", "has_visits"): for field_name in ("intrinsic_metadata", "has_visits"):
Show All 38 Lines def __init__(self, hosts: List[str], index_prefix=None):
self._backend = Elasticsearch(hosts=hosts) self._backend = Elasticsearch(hosts=hosts)
self.index_prefix = index_prefix self.index_prefix = index_prefix
self.origin_index = "origin" self.origin_index = "origin"
if index_prefix: if index_prefix:
self.origin_index = index_prefix + "_" + self.origin_index self.origin_index = index_prefix + "_" + self.origin_index
@timed
def check(self): def check(self):
return self._backend.ping() return self._backend.ping()
def deinitialize(self) -> None: def deinitialize(self) -> None:
"""Removes all indices from the Elasticsearch backend""" """Removes all indices from the Elasticsearch backend"""
self._backend.indices.delete(index="*") self._backend.indices.delete(index="*")
def initialize(self) -> None: def initialize(self) -> None:
Show All 33 Lines def initialize(self) -> None:
"type": "keyword", "type": "keyword",
} }
}, },
}, },
} }
}, },
) )
@timed
def flush(self) -> None: def flush(self) -> None:
self._backend.indices.refresh(index=self.origin_index) self._backend.indices.refresh(index=self.origin_index)
@timed
def origin_update(self, documents: Iterable[Dict]) -> None: def origin_update(self, documents: Iterable[Dict]) -> None:
documents = map(_sanitize_origin, documents) documents = map(_sanitize_origin, documents)
documents_with_sha1 = ( documents_with_sha1 = (
(origin_identifier(document), document) for document in documents (origin_identifier(document), document) for document in documents
) )
actions = [ actions = [
{ {
"_op_type": "update", "_op_type": "update",
"_id": sha1, "_id": sha1,
"_index": self.origin_index, "_index": self.origin_index,
"doc": {**document, "sha1": sha1,}, "doc": {**document, "sha1": sha1,},
"doc_as_upsert": True, "doc_as_upsert": True,
} }
for (sha1, document) in documents_with_sha1 for (sha1, document) in documents_with_sha1
] ]
bulk(self._backend, actions, index=self.origin_index)
indexed_count, errors = helpers.bulk(
self._backend, actions, index=self.origin_index
)
assert isinstance(errors, List) # Make mypy happy
send_metric("document:index", count=indexed_count, method_name="origin_update")
send_metric(
"document:index_error", count=len(errors), method_name="origin_update"
)
def origin_dump(self) -> Iterator[model.Origin]: def origin_dump(self) -> Iterator[model.Origin]:
results = scan(self._backend, index=self.origin_index) results = helpers.scan(self._backend, index=self.origin_index)
for hit in results: for hit in results:
yield self._backend.termvectors( yield self._backend.termvectors(
index=self.origin_index, id=hit["_id"], fields=["*"] index=self.origin_index, id=hit["_id"], fields=["*"]
) )
@timed
def origin_search( def origin_search(
self, self,
*, *,
url_pattern: Optional[str] = None, url_pattern: Optional[str] = None,
metadata_pattern: Optional[str] = None, metadata_pattern: Optional[str] = None,
with_visit: bool = False, with_visit: bool = False,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: int = 50, limit: int = 50,
▲ Show 20 Lines • Show All 86 Lines • Show Last 20 Lines
View Options

swh/scheduler/cli/task.py

# Copyright (C) 2019-2020 The Software Heritage developers # Copyright (C) 2019-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
import base64 import base64
from typing import Any, Dict, Iterable, Iterator, List, Optional from typing import Any, Dict, Iterable, Iterator, List, Optional
from elasticsearch import Elasticsearch from elasticsearch import Elasticsearch, helpers
from elasticsearch.helpers import bulk, scan
import msgpack import msgpack
from swh.indexer import codemeta from swh.indexer import codemeta
from swh.model import model from swh.model import model
from swh.model.identifiers import origin_identifier from swh.model.identifiers import origin_identifier
from swh.search.interface import PagedResult from swh.search.interface import PagedResult
from swh.search.metrics import send_metric, timed
def _sanitize_origin(origin): def _sanitize_origin(origin):
origin = origin.copy() origin = origin.copy()
# Whitelist fields to be saved in Elasticsearch # Whitelist fields to be saved in Elasticsearch
res = {"url": origin.pop("url")} res = {"url": origin.pop("url")}
for field_name in ("intrinsic_metadata", "has_visits"): for field_name in ("intrinsic_metadata", "has_visits"):
Show All 38 Lines def __init__(self, hosts: List[str], index_prefix=None):
self._backend = Elasticsearch(hosts=hosts) self._backend = Elasticsearch(hosts=hosts)
self.index_prefix = index_prefix self.index_prefix = index_prefix
self.origin_index = "origin" self.origin_index = "origin"
if index_prefix: if index_prefix:
self.origin_index = index_prefix + "_" + self.origin_index self.origin_index = index_prefix + "_" + self.origin_index
@timed
def check(self): def check(self):
return self._backend.ping() return self._backend.ping()
def deinitialize(self) -> None: def deinitialize(self) -> None:
"""Removes all indices from the Elasticsearch backend""" """Removes all indices from the Elasticsearch backend"""
self._backend.indices.delete(index="*") self._backend.indices.delete(index="*")
def initialize(self) -> None: def initialize(self) -> None:
Show All 33 Lines def initialize(self) -> None:
"type": "keyword", "type": "keyword",
} }
}, },
}, },
} }
}, },
) )
@timed
def flush(self) -> None: def flush(self) -> None:
self._backend.indices.refresh(index=self.origin_index) self._backend.indices.refresh(index=self.origin_index)
@timed
def origin_update(self, documents: Iterable[Dict]) -> None: def origin_update(self, documents: Iterable[Dict]) -> None:
documents = map(_sanitize_origin, documents) documents = map(_sanitize_origin, documents)
documents_with_sha1 = ( documents_with_sha1 = (
(origin_identifier(document), document) for document in documents (origin_identifier(document), document) for document in documents
) )
actions = [ actions = [
{ {
"_op_type": "update", "_op_type": "update",
"_id": sha1, "_id": sha1,
"_index": self.origin_index, "_index": self.origin_index,
"doc": {**document, "sha1": sha1,}, "doc": {**document, "sha1": sha1,},
"doc_as_upsert": True, "doc_as_upsert": True,
} }
for (sha1, document) in documents_with_sha1 for (sha1, document) in documents_with_sha1
] ]
bulk(self._backend, actions, index=self.origin_index)
indexed_count, errors = helpers.bulk(
self._backend, actions, index=self.origin_index
)
assert isinstance(errors, List) # Make mypy happy
send_metric("document:index", count=indexed_count, method_name="origin_update")
send_metric(
"document:index_error", count=len(errors), method_name="origin_update"
)
def origin_dump(self) -> Iterator[model.Origin]: def origin_dump(self) -> Iterator[model.Origin]:
results = scan(self._backend, index=self.origin_index) results = helpers.scan(self._backend, index=self.origin_index)
for hit in results: for hit in results:
yield self._backend.termvectors( yield self._backend.termvectors(
index=self.origin_index, id=hit["_id"], fields=["*"] index=self.origin_index, id=hit["_id"], fields=["*"]
) )
@timed
def origin_search( def origin_search(
self, self,
*, *,
url_pattern: Optional[str] = None, url_pattern: Optional[str] = None,
metadata_pattern: Optional[str] = None, metadata_pattern: Optional[str] = None,
with_visit: bool = False, with_visit: bool = False,
page_token: Optional[str] = None, page_token: Optional[str] = None,
limit: int = 50, limit: int = 50,
▲ Show 20 Lines • Show All 86 Lines • Show Last 20 Lines
View Options

swh/scheduler/cli/task_type.py

Loading...
View Options

swh/scheduler/cli/utils.py

Loading...
About · Developer information · Legal