Page MenuHomeSoftware Heritage
Differential D2948

common/identifiers: Add mypy type annotations
ClosedPublic
Actions

Authored by anlambert on Apr 2 2020, 5:00 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDWAPPS0b54881210a6: common/identifiers: Add mypy type annotations
Summary

Add typing information in swh.web.common.identifiers.

Related to T2342

Diff Detail

Repository
rDWAPPS Web applications
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Apr 2 2020, 5:00 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 2 2020, 5:00 PM
anlambert mentioned this in D2946: common: Move PIDs related functions from utils to dedicated sub-module.Apr 2 2020, 5:01 PM
ardumont accepted this revision.Apr 2 2020, 5:02 PM
ardumont added a subscriber: ardumont.

\o/

This revision is now accepted and ready to land.Apr 2 2020, 5:02 PM
anlambert updated this revision to Diff 10495.Apr 2 2020, 5:02 PM

Update: s/pointed/targeted/ in docstring

anlambert updated this revision to Diff 10496.Apr 2 2020, 5:04 PM

Update: Another s/pointed/targeted/ in docstring

swh-public-ci added a comment.Apr 2 2020, 5:08 PM

Build is green

Patch application report for D2948 (id=10494)

Rebasing onto 8b3e0ce543...

Current branch diff-target is up to date.
Changes applied before test
commit b1b91b6216a06bd8c4f13cc36ad060e9ed8e1938
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Thu Apr 2 16:58:19 2020 +0200

    common/identifiers: Add mypy type annotations
    
    Related to T2342

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/40/ for more details.

Harbormaster completed remote builds in B11586: Diff 10494.Apr 2 2020, 5:08 PM
swh-public-ci added a comment.Apr 2 2020, 5:17 PM

Build has FAILED

Patch application report for D2948 (id=10495)

Rebasing onto 8b3e0ce543...

Current branch diff-target is up to date.
Changes applied before test
commit b49db5042a87d363a91fc5e97417e055a3c692e3
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Thu Apr 2 16:58:19 2020 +0200

    common/identifiers: Add mypy type annotations
    
    Related to T2342

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/41/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/41/console

Harbormaster failed remote builds in B11587: Diff 10495!Apr 2 2020, 5:17 PM
swh-public-ci added a comment.Apr 2 2020, 5:24 PM

Build is green

Patch application report for D2948 (id=10496)

Rebasing onto 8b3e0ce543...

Current branch diff-target is up to date.
Changes applied before test
commit 0b54881210a6bc3a043806e96d13728007a77baa
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Thu Apr 2 16:58:19 2020 +0200

    common/identifiers: Add mypy type annotations
    
    Related to T2342

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/42/ for more details.

swh-public-ci added a comment.Apr 2 2020, 5:32 PM

Build is green

Patch application report for D2948 (id=10496)

Rebasing onto 8b3e0ce543...

Current branch diff-target is up to date.
Changes applied before test
commit 0b54881210a6bc3a043806e96d13728007a77baa
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Thu Apr 2 16:58:19 2020 +0200

    common/identifiers: Add mypy type annotations
    
    Related to T2342

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/43/ for more details.

Harbormaster completed remote builds in B11588: Diff 10496.Apr 2 2020, 5:32 PM
Closed by commit rDWAPPS0b54881210a6: common/identifiers: Add mypy type annotations (authored by anlambert). · Explain WhyApr 2 2020, 5:54 PM
This revision was automatically updated to reflect the committed changes.
anlambert added a commit: rDWAPPS0b54881210a6: common/identifiers: Add mypy type annotations.

Revision Contents
Changeset List

PathSize
swh/
web/
common/
76 lines
10 lines
3 lines
misc/
2 lines

Diff 10498

View Options

swh/web/common/identifiers.py

# Copyright (C) 2020 The Software Heritage developers # Copyright (C) 2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU Affero General Public License version 3, or any later version # License: GNU Affero General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
from typing import Dict, Iterable, List, Optional
from typing_extensions import TypedDict
from django.http import QueryDict from django.http import QueryDict
from swh.model.exceptions import ValidationError from swh.model.exceptions import ValidationError
from swh.model.hashutil import hash_to_bytes from swh.model.hashutil import hash_to_bytes
from swh.model.identifiers import ( from swh.model.identifiers import (
persistent_identifier, parse_persistent_identifier, persistent_identifier, parse_persistent_identifier,
CONTENT, DIRECTORY, ORIGIN, RELEASE, REVISION, SNAPSHOT CONTENT, DIRECTORY, ORIGIN, RELEASE, REVISION, SNAPSHOT,
PersistentId
) )
from swh.web.common.exc import BadInputExc from swh.web.common.exc import BadInputExc
from swh.web.common.typing import QueryParameters
from swh.web.common.utils import reverse from swh.web.common.utils import reverse
def get_swh_persistent_id(object_type, object_id, scheme_version=1): def get_swh_persistent_id(object_type: str, object_id: str,
scheme_version: int = 1) -> str:
""" """
Returns the persistent identifier for a swh object based on: Returns the persistent identifier for a swh object based on:
* the object type * the object type
* the object id * the object id
* the swh identifiers scheme version * the swh identifiers scheme version
Args: Args:
object_type (str): the swh object type object_type: the swh object type
(content/directory/release/revision/snapshot) (content/directory/release/revision/snapshot)
object_id (str): the swh object id (hexadecimal representation object_id: the swh object id (hexadecimal representation
of its hash value) of its hash value)
scheme_version (int): the scheme version of the swh scheme_version: the scheme version of the swh
persistent identifiers persistent identifiers
Returns: Returns:
str: the swh object persistent identifier the swh object persistent identifier
Raises: Raises:
BadInputExc: if the provided parameters do not enable to BadInputExc: if the provided parameters do not enable to
generate a valid identifier generate a valid identifier
""" """
try: try:
swh_id = persistent_identifier(object_type, object_id, scheme_version) swh_id = persistent_identifier(object_type, object_id, scheme_version)
except ValidationError as e: except ValidationError as e:
raise BadInputExc('Invalid object (%s) for swh persistent id. %s' % raise BadInputExc('Invalid object (%s) for swh persistent id. %s' %
(object_id, e)) (object_id, e))
else: else:
return swh_id return swh_id
def resolve_swh_persistent_id(swh_id, query_params=None): ResolvedPersistentId = TypedDict('ResolvedPersistentId', {
'swh_id_parsed': PersistentId,
'browse_url': Optional[str]
})
def resolve_swh_persistent_id(swh_id: str,
query_params: Optional[QueryParameters] = None
) -> ResolvedPersistentId:
""" """
Try to resolve a Software Heritage persistent id into an url for Try to resolve a Software Heritage persistent id into an url for
browsing the pointed object. browsing the targeted object.
Args: Args:
swh_id (str): a Software Heritage persistent identifier swh_id: a Software Heritage persistent identifier
query_params (django.http.QueryDict): optional dict filled with query_params: optional dict filled with
query parameters to append to the browse url query parameters to append to the browse url
Returns: Returns:
dict: a dict with the following keys: a dict with the following keys:
* **swh_id_parsed (swh.model.identifiers.PersistentId)**: * **swh_id_parsed**: the parsed identifier
the parsed identifier * **browse_url**: the url for browsing the targeted object
* **browse_url (str)**: the url for browsing the pointed object
""" """
swh_id_parsed = get_persistent_identifier(swh_id) swh_id_parsed = get_persistent_identifier(swh_id)
object_type = swh_id_parsed.object_type object_type = swh_id_parsed.object_type
object_id = swh_id_parsed.object_id object_id = swh_id_parsed.object_id
browse_url = None browse_url = None
query_dict = QueryDict('', mutable=True) query_dict = QueryDict('', mutable=True)
if query_params and len(query_params) > 0: if query_params and len(query_params) > 0:
for k in sorted(query_params.keys()): for k in sorted(query_params.keys()):
Show All 27 Lines elif object_type == SNAPSHOT:
browse_url = reverse('browse-snapshot', browse_url = reverse('browse-snapshot',
url_args={'snapshot_id': object_id}, url_args={'snapshot_id': object_id},
query_params=query_dict) query_params=query_dict)
elif object_type == ORIGIN: elif object_type == ORIGIN:
raise BadInputExc(('Origin PIDs (Persistent Identifiers) are not ' raise BadInputExc(('Origin PIDs (Persistent Identifiers) are not '
'publicly resolvable because they are for ' 'publicly resolvable because they are for '
'internal usage only')) 'internal usage only'))
return {'swh_id_parsed': swh_id_parsed, return {
'browse_url': browse_url} 'swh_id_parsed': swh_id_parsed,
'browse_url': browse_url
}
def get_persistent_identifier(persistent_id): def get_persistent_identifier(persistent_id: str) -> PersistentId:
"""Check if a persistent identifier is valid. """Check if a persistent identifier is valid.
Args: Args:
persistent_id: A string representing a Software Heritage persistent_id: A string representing a Software Heritage
persistent identifier. persistent identifier.
Raises: Raises:
BadInputExc: if the provided persistent identifier can BadInputExc: if the provided persistent identifier can
not be parsed. not be parsed.
Return: Return:
A persistent identifier object. A persistent identifier object.
""" """
try: try:
pid_object = parse_persistent_identifier(persistent_id) pid_object = parse_persistent_identifier(persistent_id)
except ValidationError as ve: except ValidationError as ve:
raise BadInputExc('Error when parsing identifier: %s' % raise BadInputExc('Error when parsing identifier: %s' %
' '.join(ve.messages)) ' '.join(ve.messages))
else: else:
return pid_object return pid_object
def group_swh_persistent_identifiers(persistent_ids): def group_swh_persistent_identifiers(persistent_ids: Iterable[PersistentId]
) -> Dict[str, List[bytes]]:
""" """
Groups many Software Heritage persistent identifiers into a Groups many Software Heritage persistent identifiers into a
dictionary depending on their type. dictionary depending on their type.
Args: Args:
persistent_ids (list): a list of Software Heritage persistent persistent_ids: an iterable of Software Heritage persistent
identifier objects identifier objects
Returns: Returns:
A dictionary with: A dictionary with:
keys: persistent identifier types keys: persistent identifier types
values: list(bytes) persistent identifiers id values: persistent identifiers id
Raises: Raises:
BadInputExc: if one of the provided persistent identifier can BadInputExc: if one of the provided persistent identifier can
not be parsed. not be parsed.
""" """
pids_by_type = { pids_by_type: Dict[str, List[bytes]] = {
CONTENT: [], CONTENT: [],
DIRECTORY: [], DIRECTORY: [],
REVISION: [], REVISION: [],
RELEASE: [], RELEASE: [],
SNAPSHOT: [] SNAPSHOT: []
} }
for pid in persistent_ids: for pid in persistent_ids:
obj_id = pid.object_id obj_id = pid.object_id
obj_type = pid.object_type obj_type = pid.object_type
pids_by_type[obj_type].append(hash_to_bytes(obj_id)) pids_by_type[obj_type].append(hash_to_bytes(obj_id))
return pids_by_type return pids_by_type
View Options

swh/web/common/typing.py

  • This file was added.
# Copyright (C) 2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU Affero General Public License version 3, or any later version
# See top-level LICENSE file for more information
from typing import Any, Dict, Union
from django.http import QueryDict
QueryParameters = Union[Dict[str, Any], QueryDict]
View Options

swh/web/common/utils.py

Show All 21 Lines
from django.urls import reverse as django_reverse from django.urls import reverse as django_reverse
from django.http import QueryDict, HttpRequest from django.http import QueryDict, HttpRequest
from prometheus_client.registry import CollectorRegistry from prometheus_client.registry import CollectorRegistry
from rest_framework.authentication import SessionAuthentication from rest_framework.authentication import SessionAuthentication
from swh.web.common.exc import BadInputExc from swh.web.common.exc import BadInputExc
from swh.web.common.typing import QueryParameters
from swh.web.config import get_config from swh.web.config import get_config
SWH_WEB_METRICS_REGISTRY = CollectorRegistry(auto_describe=True) SWH_WEB_METRICS_REGISTRY = CollectorRegistry(auto_describe=True)
swh_object_icons = { swh_object_icons = {
'branch': 'fa fa-code-fork', 'branch': 'fa fa-code-fork',
'branches': 'fa fa-code-fork', 'branches': 'fa fa-code-fork',
'content': 'fa fa-file-text', 'content': 'fa fa-file-text',
'directory': 'fa fa-folder', 'directory': 'fa fa-folder',
'person': 'fa fa-user', 'person': 'fa fa-user',
'revisions history': 'fa fa-history', 'revisions history': 'fa fa-history',
'release': 'fa fa-tag', 'release': 'fa fa-tag',
'releases': 'fa fa-tag', 'releases': 'fa fa-tag',
'revision': 'octicon-git-commit', 'revision': 'octicon-git-commit',
'snapshot': 'fa fa-camera', 'snapshot': 'fa fa-camera',
'visits': 'fa fa-calendar', 'visits': 'fa fa-calendar',
} }
def reverse(viewname: str, def reverse(viewname: str,
url_args: Optional[Dict[str, Any]] = None, url_args: Optional[Dict[str, Any]] = None,
query_params: Optional[Dict[str, Any]] = None, query_params: Optional[QueryParameters] = None,
current_app: Optional[str] = None, current_app: Optional[str] = None,
urlconf: Optional[str] = None, urlconf: Optional[str] = None,
request: Optional[HttpRequest] = None) -> str: request: Optional[HttpRequest] = None) -> str:
"""An override of django reverse function supporting query parameters. """An override of django reverse function supporting query parameters.
Args: Args:
viewname: the name of the django view from which to compute a url viewname: the name of the django view from which to compute a url
url_args: dictionary of url arguments indexed by their names url_args: dictionary of url arguments indexed by their names
▲ Show 20 Lines • Show All 281 Lines • Show Last 20 Lines
View Options

swh/web/misc/badges.py

Show First 20 Lines • Show All 92 Lines • ▼ Show 20 Linesdef _swh_badge(request: HttpRequest, object_type: str, object_id: str,
Returns: Returns:
HTTP response with content type *image/svg+xml* containing the SVG HTTP response with content type *image/svg+xml* containing the SVG
badge data. If the provided parameters are invalid, HTTP 400 status badge data. If the provided parameters are invalid, HTTP 400 status
code will be returned. If the object can not be found in the archive, code will be returned. If the object can not be found in the archive,
HTTP 404 status code will be returned. HTTP 404 status code will be returned.
""" """
left_text = 'error' left_text = 'error'
whole_link = '' whole_link = None
try: try:
if object_type == ORIGIN: if object_type == ORIGIN:
service.lookup_origin({'url': object_id}) service.lookup_origin({'url': object_id})
right_text = 'repository' right_text = 'repository'
whole_link = reverse('browse-origin', whole_link = reverse('browse-origin',
url_args={'origin_url': object_id}) url_args={'origin_url': object_id})
else: else:
▲ Show 20 Lines • Show All 61 Lines • Show Last 20 Lines
About · Developer information · Legal