Page MenuHomeSoftware Heritage
Files F9342406

No OneTemporary
Actions

View Options

diff --git a/PKG-INFO b/PKG-INFO
index 519c2f0..94af8e7 100644
--- a/PKG-INFO
+++ b/PKG-INFO
@@ -1,107 +1,107 @@
Metadata-Version: 2.1
Name: swh.loader.git
-Version: 1.2.0
+Version: 1.3.0
Summary: Software Heritage git loader
Home-page: https://forge.softwareheritage.org/diffusion/DLDG/
Author: Software Heritage developers
Author-email: swh-devel@inria.fr
License: UNKNOWN
Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest
Project-URL: Funding, https://www.softwareheritage.org/donate
Project-URL: Source, https://forge.softwareheritage.org/source/swh-loader-git
Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-loader-git/
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 5 - Production/Stable
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Provides-Extra: testing
License-File: LICENSE
License-File: AUTHORS
swh-loader-git
==============
The Software Heritage Git Loader is a tool and a library to walk a local
Git repository and inject into the SWH dataset all contained files that
weren't known before.
The main entry points are:
- :class:`swh.loader.git.loader.GitLoader` for the main loader which can ingest either
local or remote git repository's contents. This is the main implementation deployed in
production.
- :class:`swh.loader.git.from_disk.GitLoaderFromDisk` which ingests only local git clone
repository.
- :class:`swh.loader.git.loader.GitLoaderFromArchive` which ingests a git repository
wrapped in an archive.
License
-------
This program is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation, either version 3 of the License, or (at your
option) any later version.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
Public License for more details.
See top-level LICENSE file for the full text of the GNU General Public
License along with this program.
Dependencies
------------
### Runtime
- python3
- python3-dulwich
- python3-retrying
- python3-swh.core
- python3-swh.model
- python3-swh.storage
- python3-swh.scheduler
### Test
- python3-nose
Requirements
------------
- implementation language, Python3
- coding guidelines: conform to PEP8
- Git access: via dulwich
CLI Run
----------
You can run the loader from a remote origin (*loader*) or from an origin on disk
(*from_disk*) directly by calling:
```
swh loader -C <config-file> run git <git-repository-url>
```
or "git_disk".
## Configuration sample
/tmp/git.yml:
```
storage:
cls: remote
args:
url: http://localhost:5002/
```
diff --git a/swh.loader.git.egg-info/PKG-INFO b/swh.loader.git.egg-info/PKG-INFO
index 519c2f0..94af8e7 100644
--- a/swh.loader.git.egg-info/PKG-INFO
+++ b/swh.loader.git.egg-info/PKG-INFO
@@ -1,107 +1,107 @@
Metadata-Version: 2.1
Name: swh.loader.git
-Version: 1.2.0
+Version: 1.3.0
Summary: Software Heritage git loader
Home-page: https://forge.softwareheritage.org/diffusion/DLDG/
Author: Software Heritage developers
Author-email: swh-devel@inria.fr
License: UNKNOWN
Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest
Project-URL: Funding, https://www.softwareheritage.org/donate
Project-URL: Source, https://forge.softwareheritage.org/source/swh-loader-git
Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-loader-git/
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Development Status :: 5 - Production/Stable
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Provides-Extra: testing
License-File: LICENSE
License-File: AUTHORS
swh-loader-git
==============
The Software Heritage Git Loader is a tool and a library to walk a local
Git repository and inject into the SWH dataset all contained files that
weren't known before.
The main entry points are:
- :class:`swh.loader.git.loader.GitLoader` for the main loader which can ingest either
local or remote git repository's contents. This is the main implementation deployed in
production.
- :class:`swh.loader.git.from_disk.GitLoaderFromDisk` which ingests only local git clone
repository.
- :class:`swh.loader.git.loader.GitLoaderFromArchive` which ingests a git repository
wrapped in an archive.
License
-------
This program is free software: you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation, either version 3 of the License, or (at your
option) any later version.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General
Public License for more details.
See top-level LICENSE file for the full text of the GNU General Public
License along with this program.
Dependencies
------------
### Runtime
- python3
- python3-dulwich
- python3-retrying
- python3-swh.core
- python3-swh.model
- python3-swh.storage
- python3-swh.scheduler
### Test
- python3-nose
Requirements
------------
- implementation language, Python3
- coding guidelines: conform to PEP8
- Git access: via dulwich
CLI Run
----------
You can run the loader from a remote origin (*loader*) or from an origin on disk
(*from_disk*) directly by calling:
```
swh loader -C <config-file> run git <git-repository-url>
```
or "git_disk".
## Configuration sample
/tmp/git.yml:
```
storage:
cls: remote
args:
url: http://localhost:5002/
```
diff --git a/swh/__init__.py b/swh/__init__.py
index 8d9f151..b36383a 100644
--- a/swh/__init__.py
+++ b/swh/__init__.py
@@ -1,4 +1,3 @@
from pkgutil import extend_path
-from typing import List
-__path__: List[str] = extend_path(__path__, __name__)
+__path__ = extend_path(__path__, __name__)
diff --git a/swh/loader/__init__.py b/swh/loader/__init__.py
index 8d9f151..b36383a 100644
--- a/swh/loader/__init__.py
+++ b/swh/loader/__init__.py
@@ -1,4 +1,3 @@
from pkgutil import extend_path
-from typing import List
-__path__: List[str] = extend_path(__path__, __name__)
+__path__ = extend_path(__path__, __name__)
diff --git a/swh/loader/git/dumb.py b/swh/loader/git/dumb.py
index 316169e..00f913d 100644
--- a/swh/loader/git/dumb.py
+++ b/swh/loader/git/dumb.py
@@ -1,207 +1,197 @@
# Copyright (C) 2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
from __future__ import annotations
from collections import defaultdict
import logging
import stat
import struct
from tempfile import SpooledTemporaryFile
from typing import TYPE_CHECKING, Callable, Dict, Iterable, List, Set, cast
+import urllib.parse
-from dulwich.client import HttpGitClient
from dulwich.errors import NotGitRepository
from dulwich.objects import S_IFGITLINK, Commit, ShaFile, Tree
from dulwich.pack import Pack, PackData, PackIndex, load_pack_index_file
-from urllib3.response import HTTPResponse
+import requests
from swh.loader.git.utils import HexBytes
if TYPE_CHECKING:
from .loader import RepoRepresentation
logger = logging.getLogger(__name__)
-class DumbHttpGitClient(HttpGitClient):
- """Simple wrapper around dulwich.client.HTTPGitClient
- """
-
- def __init__(self, base_url: str):
- super().__init__(base_url)
- self.user_agent = "Software Heritage dumb Git loader"
-
- def get(self, url: str) -> HTTPResponse:
- logger.debug("Fetching %s", url)
- response, _ = self._http_request(url, headers={"User-Agent": self.user_agent})
- return response
+HEADERS = {"User-Agent": "Software Heritage dumb Git loader"}
def check_protocol(repo_url: str) -> bool:
"""Checks if a git repository can be cloned using the dumb protocol.
Args:
repo_url: Base URL of a git repository
Returns:
Whether the dumb protocol is supported.
"""
if not repo_url.startswith("http"):
return False
- http_client = DumbHttpGitClient(repo_url)
- url = http_client.get_url("info/refs?service=git-upload-pack")
- response = http_client.get(url)
+ url = urllib.parse.urljoin(repo_url, "info/refs?service=git-upload-pack/")
+ response = requests.get(url, headers=HEADERS)
+ content_type = response.headers.get("Content-Type")
return (
- response.status in (200, 304,)
+ response.status_code in (200, 304,)
# header is not mandatory in protocol specification
- and response.content_type is None
- or not response.content_type.startswith("application/x-git-")
+ and (content_type is None or not content_type.startswith("application/x-git-"))
)
class GitObjectsFetcher:
"""Git objects fetcher using dumb HTTP protocol.
Fetches a set of git objects for a repository according to its archival
state by Software Heritage and provides iterators on them.
Args:
repo_url: Base URL of a git repository
base_repo: State of repository archived by Software Heritage
"""
def __init__(self, repo_url: str, base_repo: RepoRepresentation):
- self.http_client = DumbHttpGitClient(repo_url)
+ self._session = requests.Session()
+ self.repo_url = repo_url
self.base_repo = base_repo
self.objects: Dict[bytes, Set[bytes]] = defaultdict(set)
self.refs = self._get_refs()
self.head = self._get_head() if self.refs else {}
self.packs = self._get_packs()
def fetch_object_ids(self) -> None:
"""Fetches identifiers of git objects to load into the archive.
"""
wants = self.base_repo.determine_wants(self.refs)
# process refs
commit_objects = []
for ref in wants:
ref_object = self._get_git_object(ref)
if ref_object.get_type() == Commit.type_num:
commit_objects.append(cast(Commit, ref_object))
self.objects[b"commit"].add(ref)
else:
self.objects[b"tag"].add(ref)
# perform DFS on commits graph
while commit_objects:
commit = commit_objects.pop()
# fetch tree and blob ids recursively
self._fetch_tree_objects(commit.tree)
for parent in commit.parents:
if (
# commit not already seen in the current load
parent not in self.objects[b"commit"]
# commit not already archived by a previous load
and parent not in self.base_repo.heads
):
commit_objects.append(cast(Commit, self._get_git_object(parent)))
self.objects[b"commit"].add(parent)
def iter_objects(self, object_type: bytes) -> Iterable[ShaFile]:
"""Returns a generator on fetched git objects per type.
Args:
object_type: Git object type, either b"blob", b"commit", b"tag" or b"tree"
Returns:
A generator fetching git objects on the fly.
"""
return map(self._get_git_object, self.objects[object_type])
def _http_get(self, path: str) -> SpooledTemporaryFile:
- url = self.http_client.get_url(path)
- response = self.http_client.get(url)
+ url = urllib.parse.urljoin(self.repo_url, path)
+ response = self._session.get(url, headers=HEADERS)
buffer = SpooledTemporaryFile(max_size=100 * 1024 * 1024)
- buffer.write(response.data)
+ for chunk in response.iter_content(chunk_size=10 * 1024 * 1024):
+ buffer.write(chunk)
buffer.flush()
buffer.seek(0)
return buffer
def _get_refs(self) -> Dict[bytes, HexBytes]:
refs = {}
refs_resp_bytes = self._http_get("info/refs")
for ref_line in refs_resp_bytes.readlines():
ref_target, ref_name = ref_line.replace(b"\n", b"").split(b"\t")
refs[ref_name] = ref_target
return refs
def _get_head(self) -> Dict[bytes, HexBytes]:
head_resp_bytes = self._http_get("HEAD")
_, head_target = head_resp_bytes.readline().replace(b"\n", b"").split(b" ")
return {b"HEAD": head_target}
def _get_pack_data(self, pack_name: str) -> Callable[[], PackData]:
def _pack_data() -> PackData:
pack_data_bytes = self._http_get(f"objects/pack/{pack_name}")
return PackData(pack_name, file=pack_data_bytes)
return _pack_data
def _get_pack_idx(self, pack_idx_name: str) -> Callable[[], PackIndex]:
def _pack_idx() -> PackIndex:
pack_idx_bytes = self._http_get(f"objects/pack/{pack_idx_name}")
return load_pack_index_file(pack_idx_name, pack_idx_bytes)
return _pack_idx
def _get_packs(self) -> List[Pack]:
packs = []
packs_info_bytes = self._http_get("objects/info/packs")
packs_info = packs_info_bytes.read().decode()
for pack_info in packs_info.split("\n"):
if pack_info:
pack_name = pack_info.split(" ")[1]
pack_idx_name = pack_name.replace(".pack", ".idx")
# pack index and data file will be lazily fetched when required
packs.append(
Pack.from_lazy_objects(
self._get_pack_data(pack_name),
self._get_pack_idx(pack_idx_name),
)
)
return packs
def _get_git_object(self, sha: bytes) -> ShaFile:
# try to get the object from a pack file first to avoid flooding
# git server with numerous HTTP requests
for pack in list(self.packs):
try:
if sha in pack:
return pack[sha]
except (NotGitRepository, struct.error):
# missing (dulwich http client raises NotGitRepository on 404)
# or invalid pack index/content, remove it from global packs list
logger.debug("A pack file is missing or its content is invalid")
self.packs.remove(pack)
# fetch it from objects/ directory otherwise
sha_hex = sha.decode()
object_path = f"objects/{sha_hex[:2]}/{sha_hex[2:]}"
return ShaFile.from_file(self._http_get(object_path))
def _fetch_tree_objects(self, sha: bytes) -> None:
if sha not in self.objects[b"tree"]:
tree = cast(Tree, self._get_git_object(sha))
self.objects[b"tree"].add(sha)
for item in tree.items():
if item.mode == S_IFGITLINK:
# skip submodules as objects are not stored in repository
continue
if item.mode & stat.S_IFDIR:
self._fetch_tree_objects(item.sha)
else:
self.objects[b"blob"].add(item.sha)
diff --git a/swh/loader/git/loader.py b/swh/loader/git/loader.py
index 8f100c5..3f6fe79 100644
--- a/swh/loader/git/loader.py
+++ b/swh/loader/git/loader.py
@@ -1,527 +1,531 @@
# Copyright (C) 2016-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
from dataclasses import dataclass
import datetime
import logging
import os
import pickle
import sys
from tempfile import SpooledTemporaryFile
from typing import Any, Callable, Dict, Iterable, Iterator, List, Optional, Set, Type
import dulwich.client
from dulwich.errors import GitProtocolError, NotGitRepository
from dulwich.object_store import ObjectStoreGraphWalker
from dulwich.objects import ShaFile
from dulwich.pack import PackData, PackInflater
from swh.loader.core.loader import DVCSLoader
from swh.loader.exception import NotFound
from swh.model import hashutil
from swh.model.model import (
BaseContent,
Directory,
Origin,
Release,
Revision,
Snapshot,
SnapshotBranch,
TargetType,
)
from swh.storage.algos.snapshot import snapshot_get_latest
from swh.storage.interface import StorageInterface
from . import converters, dumb, utils
from .utils import HexBytes
logger = logging.getLogger(__name__)
class RepoRepresentation:
"""Repository representation for a Software Heritage origin."""
def __init__(
self,
storage,
base_snapshot: Optional[Snapshot] = None,
incremental: bool = True,
):
self.storage = storage
self.incremental = incremental
if base_snapshot and incremental:
self.base_snapshot: Snapshot = base_snapshot
else:
self.base_snapshot = Snapshot(branches={})
self.heads: Set[HexBytes] = set()
def get_parents(self, commit: bytes) -> List[bytes]:
"""This method should return the list of known parents"""
return []
def graph_walker(self) -> ObjectStoreGraphWalker:
return ObjectStoreGraphWalker(self.heads, self.get_parents)
def determine_wants(self, refs: Dict[bytes, HexBytes]) -> List[HexBytes]:
"""Get the list of bytehex sha1s that the git loader should fetch.
This compares the remote refs sent by the server with the base snapshot
provided by the loader.
"""
if not refs:
return []
# Cache existing heads
local_heads: Set[HexBytes] = set()
for branch_name, branch in self.base_snapshot.branches.items():
if not branch or branch.target_type == TargetType.ALIAS:
continue
local_heads.add(hashutil.hash_to_hex(branch.target).encode())
self.heads = local_heads
# Get the remote heads that we want to fetch
remote_heads: Set[HexBytes] = set()
for ref_name, ref_target in refs.items():
if utils.ignore_branch_name(ref_name):
continue
remote_heads.add(ref_target)
logger.debug("local_heads_count=%s", len(local_heads))
logger.debug("remote_heads_count=%s", len(remote_heads))
wanted_refs = list(remote_heads - local_heads)
logger.debug("wanted_refs_count=%s", len(wanted_refs))
return wanted_refs
@dataclass
class FetchPackReturn:
remote_refs: Dict[bytes, HexBytes]
symbolic_refs: Dict[bytes, HexBytes]
pack_buffer: SpooledTemporaryFile
pack_size: int
class GitLoader(DVCSLoader):
"""A bulk loader for a git repository"""
visit_type = "git"
def __init__(
self,
storage: StorageInterface,
url: str,
base_url: Optional[str] = None,
incremental: bool = True,
repo_representation: Type[RepoRepresentation] = RepoRepresentation,
pack_size_bytes: int = 4 * 1024 * 1024 * 1024,
temp_file_cutoff: int = 100 * 1024 * 1024,
save_data_path: Optional[str] = None,
max_content_size: Optional[int] = None,
):
"""Initialize the bulk updater.
Args:
repo_representation: swh's repository representation
which is in charge of filtering between known and remote
data.
...
incremental: If True, the default, this starts from the last known snapshot
(if any) references. Otherwise, this loads the full repository.
"""
super().__init__(
storage=storage,
save_data_path=save_data_path,
max_content_size=max_content_size,
)
self.origin_url = url
self.base_url = base_url
self.incremental = incremental
self.repo_representation = repo_representation
self.pack_size_bytes = pack_size_bytes
self.temp_file_cutoff = temp_file_cutoff
# state initialized in fetch_data
self.remote_refs: Dict[bytes, HexBytes] = {}
self.symbolic_refs: Dict[bytes, HexBytes] = {}
self.ref_object_types: Dict[bytes, Optional[TargetType]] = {}
def fetch_pack_from_origin(
self,
origin_url: str,
base_repo: RepoRepresentation,
do_activity: Callable[[bytes], None],
) -> FetchPackReturn:
"""Fetch a pack from the origin"""
pack_buffer = SpooledTemporaryFile(max_size=self.temp_file_cutoff)
# Hardcode the use of the tcp transport (for GitHub origins)
# Even if the Dulwich API lets us process the packfile in chunks as it's
# received, the HTTP transport implementation needs to entirely allocate
# the packfile in memory *twice*, once in the HTTP library, and once in
# a BytesIO managed by Dulwich, before passing chunks to the `do_pack`
# method Overall this triples the memory usage before we can even try to
# interrupt the loader before it overruns its memory limit.
# In contrast, the Dulwich TCP transport just gives us the read handle
# on the underlying socket, doing no processing or copying of the bytes.
# We can interrupt it as soon as we've received too many bytes.
transport_url = origin_url
- if transport_url.startswith("https://github.com/"):
- transport_url = "git" + transport_url[5:]
+
+ # Temporarily Disabled due to GitHub removing support for the tcp transport. See
+ # https://forge.softwareheritage.org/T3544
+ #
+ # if transport_url.startswith("https://github.com/"):
+ # transport_url = "git" + transport_url[5:]
logger.debug("Transport url to communicate with server: %s", transport_url)
client, path = dulwich.client.get_transport_and_path(
transport_url, thin_packs=False
)
logger.debug("Client %s to fetch pack at %s", client, path)
size_limit = self.pack_size_bytes
def do_pack(data: bytes) -> None:
cur_size = pack_buffer.tell()
would_write = len(data)
if cur_size + would_write > size_limit:
raise IOError(
f"Pack file too big for repository {origin_url}, "
f"limit is {size_limit} bytes, current size is {cur_size}, "
f"would write {would_write}"
)
pack_buffer.write(data)
pack_result = client.fetch_pack(
path,
base_repo.determine_wants,
base_repo.graph_walker(),
do_pack,
progress=do_activity,
)
remote_refs = pack_result.refs or {}
symbolic_refs = pack_result.symrefs or {}
pack_buffer.flush()
pack_size = pack_buffer.tell()
pack_buffer.seek(0)
logger.debug("fetched_pack_size=%s", pack_size)
# check if repository only supports git dumb transfer protocol,
# fetched pack file will be empty in that case as dulwich do
# not support it and do not fetch any refs
self.dumb = transport_url.startswith("http") and client.dumb
return FetchPackReturn(
remote_refs=utils.filter_refs(remote_refs),
symbolic_refs=utils.filter_refs(symbolic_refs),
pack_buffer=pack_buffer,
pack_size=pack_size,
)
def prepare_origin_visit(self) -> None:
self.visit_date = datetime.datetime.now(tz=datetime.timezone.utc)
self.origin = Origin(url=self.origin_url)
def get_full_snapshot(self, origin_url) -> Optional[Snapshot]:
return snapshot_get_latest(self.storage, origin_url)
def prepare(self) -> None:
assert self.origin is not None
prev_snapshot: Optional[Snapshot] = None
if self.incremental:
prev_snapshot = self.get_full_snapshot(self.origin.url)
if self.base_url and prev_snapshot is None:
base_origin = list(self.storage.origin_get([self.base_url]))[0]
if base_origin:
prev_snapshot = self.get_full_snapshot(base_origin.url)
if prev_snapshot is not None:
self.base_snapshot = prev_snapshot
else:
self.base_snapshot = Snapshot(branches={})
def fetch_data(self) -> bool:
assert self.origin is not None
base_repo = self.repo_representation(
storage=self.storage,
base_snapshot=self.base_snapshot,
incremental=self.incremental,
)
def do_progress(msg: bytes) -> None:
sys.stderr.buffer.write(msg)
sys.stderr.flush()
try:
fetch_info = self.fetch_pack_from_origin(
self.origin.url, base_repo, do_progress
)
except NotGitRepository as e:
raise NotFound(e)
except GitProtocolError as e:
# unfortunately, that kind of error is not specific to a not found
# scenario... It depends on the value of message within the exception.
for msg in [
"Repository unavailable", # e.g DMCA takedown
"Repository not found",
"unexpected http resp 401",
]:
if msg in e.args[0]:
raise NotFound(e)
# otherwise transmit the error
raise
except (AttributeError, NotImplementedError, ValueError):
# with old dulwich versions, those exceptions types can be raised
# by the fetch_pack operation when encountering a repository with
# dumb transfer protocol so we check if the repository supports it
# here to continue the loading if it is the case
self.dumb = dumb.check_protocol(self.origin_url)
if not self.dumb:
raise
logger.debug(
"Protocol used for communication: %s", "dumb" if self.dumb else "smart"
)
if self.dumb:
self.dumb_fetcher = dumb.GitObjectsFetcher(self.origin_url, base_repo)
self.dumb_fetcher.fetch_object_ids()
self.remote_refs = utils.filter_refs(self.dumb_fetcher.refs) # type: ignore
self.symbolic_refs = self.dumb_fetcher.head
else:
self.pack_buffer = fetch_info.pack_buffer
self.pack_size = fetch_info.pack_size
self.remote_refs = fetch_info.remote_refs
self.symbolic_refs = fetch_info.symbolic_refs
self.ref_object_types = {sha1: None for sha1 in self.remote_refs.values()}
logger.info(
"Listed %d refs for repo %s",
len(self.remote_refs),
self.origin.url,
extra={
"swh_type": "git_repo_list_refs",
"swh_repo": self.origin.url,
"swh_num_refs": len(self.remote_refs),
},
)
# No more data to fetch
return False
def save_data(self) -> None:
"""Store a pack for archival"""
assert isinstance(self.visit_date, datetime.datetime)
write_size = 8192
pack_dir = self.get_save_data_path()
pack_name = "%s.pack" % self.visit_date.isoformat()
refs_name = "%s.refs" % self.visit_date.isoformat()
with open(os.path.join(pack_dir, pack_name), "xb") as f:
self.pack_buffer.seek(0)
while True:
r = self.pack_buffer.read(write_size)
if not r:
break
f.write(r)
self.pack_buffer.seek(0)
with open(os.path.join(pack_dir, refs_name), "xb") as f:
pickle.dump(self.remote_refs, f)
def iter_objects(self, object_type: bytes) -> Iterator[ShaFile]:
"""Read all the objects of type `object_type` from the packfile"""
if self.dumb:
yield from self.dumb_fetcher.iter_objects(object_type)
else:
self.pack_buffer.seek(0)
count = 0
for obj in PackInflater.for_pack_data(
PackData.from_file(self.pack_buffer, self.pack_size)
):
if obj.type_name != object_type:
continue
yield obj
count += 1
logger.debug("packfile_read_count_%s=%s", object_type.decode(), count)
def get_contents(self) -> Iterable[BaseContent]:
"""Format the blobs from the git repository as swh contents"""
for raw_obj in self.iter_objects(b"blob"):
if raw_obj.id in self.ref_object_types:
self.ref_object_types[raw_obj.id] = TargetType.CONTENT
yield converters.dulwich_blob_to_content(
raw_obj, max_content_size=self.max_content_size
)
def get_directories(self) -> Iterable[Directory]:
"""Format the trees as swh directories"""
for raw_obj in self.iter_objects(b"tree"):
if raw_obj.id in self.ref_object_types:
self.ref_object_types[raw_obj.id] = TargetType.DIRECTORY
yield converters.dulwich_tree_to_directory(raw_obj)
def get_revisions(self) -> Iterable[Revision]:
"""Format commits as swh revisions"""
for raw_obj in self.iter_objects(b"commit"):
if raw_obj.id in self.ref_object_types:
self.ref_object_types[raw_obj.id] = TargetType.REVISION
yield converters.dulwich_commit_to_revision(raw_obj)
def get_releases(self) -> Iterable[Release]:
"""Retrieve all the release objects from the git repository"""
for raw_obj in self.iter_objects(b"tag"):
if raw_obj.id in self.ref_object_types:
self.ref_object_types[raw_obj.id] = TargetType.RELEASE
yield converters.dulwich_tag_to_release(raw_obj)
def get_snapshot(self) -> Snapshot:
"""Get the snapshot for the current visit.
The main complexity of this function is mapping target objects to their
types, as the `refs` dictionaries returned by the git server only give
us the identifiers for the target objects, and not their types.
The loader itself only knows the types of the objects that it has
fetched from the server (as it has parsed them while loading them to
the archive). As we only fetched an increment between the previous
snapshot and the current state of the server, we are missing the type
information for the objects that would already have been referenced by
the previous snapshot, and that the git server didn't send us. We infer
the type of these objects from the previous snapshot.
"""
branches: Dict[bytes, Optional[SnapshotBranch]] = {}
unfetched_refs: Dict[bytes, bytes] = {}
# Retrieve types from the objects loaded by the current loader
for ref_name, ref_object in self.remote_refs.items():
if ref_name in self.symbolic_refs:
continue
target = hashutil.hash_to_bytes(ref_object.decode())
target_type = self.ref_object_types.get(ref_object)
if target_type:
branches[ref_name] = SnapshotBranch(
target=target, target_type=target_type
)
else:
# The object pointed at by this ref was not fetched, supposedly
# because it existed in the base snapshot. We record it here,
# and we can get it from the base snapshot later.
unfetched_refs[ref_name] = target
dangling_branches = {}
# Handle symbolic references as alias branches
for ref_name, target in self.symbolic_refs.items():
branches[ref_name] = SnapshotBranch(
target_type=TargetType.ALIAS, target=target,
)
if target not in branches and target not in unfetched_refs:
# This handles the case where the pointer is "dangling".
# There's a chance that a further symbolic reference
# override this default value, which is totally fine.
dangling_branches[target] = ref_name
branches[target] = None
if unfetched_refs:
# Handle inference of object types from the contents of the
# previous snapshot
unknown_objects = {}
base_snapshot_reverse_branches = {
branch.target: branch
for branch in self.base_snapshot.branches.values()
if branch and branch.target_type != TargetType.ALIAS
}
for ref_name, target in unfetched_refs.items():
branch = base_snapshot_reverse_branches.get(target)
branches[ref_name] = branch
if not branch:
unknown_objects[ref_name] = target
if unknown_objects:
# This object was referenced by the server; We did not fetch
# it, and we do not know it from the previous snapshot. This is
# likely a bug in the loader.
raise RuntimeError(
"Unknown objects referenced by remote refs: %s"
% (
", ".join(
f"{name.decode()}: {hashutil.hash_to_hex(obj)}"
for name, obj in unknown_objects.items()
)
)
)
utils.warn_dangling_branches(
branches, dangling_branches, logger, self.origin_url
)
self.snapshot = Snapshot(branches=branches)
return self.snapshot
def load_status(self) -> Dict[str, Any]:
"""The load was eventful if the current snapshot is different to
the one we retrieved at the beginning of the run"""
eventful = False
if self.base_snapshot and self.snapshot:
eventful = self.snapshot.id != self.base_snapshot.id
elif self.snapshot:
eventful = bool(self.snapshot.branches)
return {"status": ("eventful" if eventful else "uneventful")}
if __name__ == "__main__":
import click
logging.basicConfig(
level=logging.DEBUG, format="%(asctime)s %(process)d %(message)s"
)
from deprecated import deprecated
@deprecated(version="1.1", reason="Use `swh loader run git --help` instead")
@click.command()
@click.option("--origin-url", help="Origin url", required=True)
@click.option("--base-url", default=None, help="Optional Base url")
@click.option(
"--ignore-history/--no-ignore-history",
help="Ignore the repository history",
default=False,
)
def main(origin_url: str, base_url: str, incremental: bool) -> Dict[str, Any]:
from swh.storage import get_storage
storage = get_storage(cls="memory")
loader = GitLoader(
storage, origin_url, base_url=base_url, incremental=incremental,
)
return loader.load()
main()
diff --git a/swh/loader/git/tests/test_converters.py b/swh/loader/git/tests/test_converters.py
index c48cb70..c25d86d 100644
--- a/swh/loader/git/tests/test_converters.py
+++ b/swh/loader/git/tests/test_converters.py
@@ -1,637 +1,592 @@
# Copyright (C) 2015-2018 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import copy
import datetime
import os
import shutil
import subprocess
import tempfile
import dulwich.objects
import dulwich.repo
import pytest
import swh.loader.git.converters as converters
from swh.model.hashutil import bytehex_to_hash, hash_to_bytehex, hash_to_bytes
from swh.model.model import (
Content,
Directory,
DirectoryEntry,
ObjectType,
Person,
Release,
Revision,
RevisionType,
Timestamp,
TimestampWithTimezone,
)
TEST_DATA = os.path.join(os.path.dirname(__file__), "data")
GPGSIG = (
b"-----BEGIN PGP SIGNATURE-----\n"
b"\n"
b"iQJLBAABCAA1FiEEAOWDevQbOk/9ITMF6ImSleOlnUcFAl8EnS4XHGRhdmlkLmRv\n"
b"dWFyZEBzZGZhMy5vcmcACgkQ6ImSleOlnUdrqQ/8C5RO4NZ5Qr/dwAy2cPA7ktkY\n"
b"1oUjKtspQoPbC1X3MXVa1aWo9B3KuOMR2URw44RhMNFwjccLOhfss06E8p7CZr2H\n"
b"uR3CzdDw7i52jHLCL2M2ZMaPAEbQuHjXWiUWIUXz9So8YwpTyd2XQneyOC2RDDEI\n"
b"I2NVbmiMeDz33jJYPrQO0QayW+ErW+xgBF7N/qS9jFWsdV1ZNfn9NxkTH8UdGuAX\n"
b"583P+0tVC2DjXc6vORVhyFzyfn1A9wHosbtWI2Mpa+zezPjoPSkcyQAJu2GyOkMC\n"
b"YzSjJdQVqyovo+INkIf6PuUNdp41886BG/06xwT8fl4sVsyO51lNIfgH0DMwfTTB\n"
b"ZgThYnvvO7SrXDm3QzBTXkvAiHiFFl3iNyGkCyxvgVmaTntuFT+cP+HD/pCiGaC+\n"
b"jHzRwfUrmuLd/lLPyq3JXBibyjnfd3SVS+7q1NZHJ4WUmCboZ0+pfrEl65mEQ/Hz\n"
b"J1qCwQ/3SsTB77ANf6lLzGSowjjrtHcBTkTbFxR4ACUhiBbosyDKpHTM7fzGFGjo\n"
b"EIjohzrEnqR3bbyxJkK+nxoOByhIRdowgyeJ02I4neMyLJqcaup8NMWCddxqjaPt\n"
b"YobghnjaDqEd+suL/v83hbZUAZHNO3i1OZYGMqzp1WHikDPoTwGP76baqBoXi56T\n"
b"4WSpxCAJRDODHLk1HgU=\n"
b"=73wF"
b"\n"
b"-----END PGP SIGNATURE-----"
)
MERGETAG = (
b"object 9768d0b576dbaaecd80abedad6dfd0d72f1476da\n"
b"type commit\n"
b"tag v0.0.1\n"
b"tagger David Douard <david.douard@sdfa3.org> 1594138133 +0200\n"
b"\n"
b"v0.0.1\n"
b"-----BEGIN PGP SIGNATURE-----\n"
b"\n"
b"iQJLBAABCAA1FiEEAOWDevQbOk/9ITMF6ImSleOlnUcFAl8EnhkXHGRhdmlkLmRv\n"
b"dWFyZEBzZGZhMy5vcmcACgkQ6ImSleOlnUcdzg//ZW9y2xU5JFQuUsBe/LfKrs+m\n"
b"0ohVInPKXwAfpB3+gn/XtTSLe+Nnr8+QEZyVRCUz2gpGZ2tNqRjhYLIX4x5KKlaV\n"
b"rfl/6Cy7zibsxxuzA1h7HylCs3IPsueQpznVHUwD9jQ5baGJSc2Lt1LufXTueHZJ\n"
b"Oc0oLiP5xCZcPqeX8R/4zUUImJZ1QrPeKmQ/3F+Iq62iWp7nWDp8PtwpykSiYlNf\n"
b"KrJM8omGvrlrWLtfPNUaQFClXwnwK1/HyNY2kYan6K5NtsIl2UX0LZ42GkRjJIrb\n"
b"q4TFIZWZ6xndtEhHEX6B8Q5TZV6sqPgNnfGpbhj8BDoZgjD0Y43fzfDiZ0Bl2tph\n"
b"tXaLg3SX/UUjFVzC1zkoQ2MR7+j8NVKauAsBINpKF4pMGsrsVRk8764pgO49iQ+S\n"
b"8JVCVV76dNNm1gd7BbhFAdIAiegBtsEF69niJBoHKYLlrT8E8hDkF/gk4IkimPqf\n"
b"UHtw/fPhVW3B4G2skd013NJGcnRj5oKtaM99d2Roxc3vhSRiTsoaM8BM9NDvLmJg\n"
b"35rWEOnet39iJIMCHk3AYaJl8QmUhllDdr6vygaBVeVEf27m2c3NzONmIKpWqa2J\n"
b"kTpF4cmzHYro34G7WuJ1bYvmLb6qWNQt9wd8RW+J1kVm5I8dkjPzLUougBpOd0YL\n"
b"Bl5UTQILbV4Tv8ZlmJM=\n"
b"=s1lv\n"
b"-----END PGP SIGNATURE-----"
)
class SWHObjectType:
"""Dulwich lookalike ObjectType class
"""
def __init__(self, type_name):
self.type_name = type_name
-class SWHTag:
- """Dulwich lookalike tag class
-
- """
-
- def __init__(
- self,
- sha,
- name,
- type_name,
- target,
- target_type,
- tagger,
- tag_time,
- tag_timezone,
- message,
- signature,
- ):
- self._sha = sha
- self.name = name
- self.type_name = type_name
- self.object = SWHObjectType(target_type), target
- self.tagger = tagger
- self.message = message
- self.signature = signature
- self.tag_time = tag_time
- self.tag_timezone = tag_timezone
- self._tag_timezone_neg_utc = False
-
- def sha(self):
- class hasher:
- def digest():
- return self._sha
-
- return hasher
-
-
@pytest.mark.fs
class TestConverters:
@classmethod
def setup_class(cls):
cls.repo_path = tempfile.mkdtemp()
bundle = os.path.join(TEST_DATA, "git-repos", "example-submodule.bundle")
git = subprocess.Popen(
["git", "clone", "--quiet", "--bare", "--mirror", bundle, cls.repo_path],
cwd=TEST_DATA,
)
# flush stdout of xz
git.communicate()
cls.repo = dulwich.repo.Repo(cls.repo_path)
@classmethod
def tearDownClass(cls):
super().tearDownClass()
shutil.rmtree(cls.repo_path)
def test_blob_to_content(self):
content_id = b"28c6f4023d65f74e3b59a2dea3c4277ed9ee07b0"
content = converters.dulwich_blob_to_content(self.repo[content_id])
expected_content = Content(
sha1_git=bytehex_to_hash(content_id),
sha1=hash_to_bytes("4850a3420a2262ff061cb296fb915430fa92301c"),
sha256=hash_to_bytes(
"fee7c8a485a10321ad94b64135073cb5" "5f22cb9f57fa2417d2adfb09d310adef"
),
blake2s256=hash_to_bytes(
"5d71873f42a137f6d89286e43677721e574" "1fa05ce4cd5e3c7ea7c44d4c2d10b"
),
data=(
b'[submodule "example-dependency"]\n'
b"\tpath = example-dependency\n"
b"\turl = https://github.com/githubtraining/"
b"example-dependency.git\n"
),
length=124,
status="visible",
)
assert content == expected_content
def test_corrupt_blob(self, mocker):
# has a signature
sha1 = hash_to_bytes("28c6f4023d65f74e3b59a2dea3c4277ed9ee07b0")
blob = copy.deepcopy(self.repo[hash_to_bytehex(sha1)])
class hasher:
def digest():
return sha1
blob._sha = hasher
converters.dulwich_blob_to_content(blob)
converters.dulwich_blob_to_content_id(blob)
sha1 = hash_to_bytes("1234" * 10)
with pytest.raises(converters.HashMismatch):
converters.dulwich_blob_to_content(blob)
with pytest.raises(converters.HashMismatch):
converters.dulwich_blob_to_content_id(blob)
def test_convertion_wrong_input(self):
class Something:
type_name = b"something-not-the-right-type"
m = {
"blob": converters.dulwich_blob_to_content,
"tree": converters.dulwich_tree_to_directory,
"commit": converters.dulwich_tree_to_directory,
"tag": converters.dulwich_tag_to_release,
}
for _callable in m.values():
with pytest.raises(ValueError):
_callable(Something())
def test_corrupt_tree(self):
# has a signature
sha1 = b"f0695c2e2fa7ce9d574023c3413761a473e500ca"
tree = copy.deepcopy(self.repo[sha1])
converters.dulwich_tree_to_directory(tree)
del tree._entries[next(iter(tree._entries))]
with pytest.raises(converters.HashMismatch):
converters.dulwich_tree_to_directory(tree)
def test_tree_perms(self):
entries = [
(b"blob_100644", 0o100644, "file"),
(b"blob_100664", 0o100664, "file"),
(b"blob_100666", 0o100666, "file"),
(b"blob_120000", 0o120000, "file"),
(b"commit_160644", 0o160644, "rev"),
(b"commit_160664", 0o160664, "rev"),
(b"commit_160666", 0o160666, "rev"),
(b"commit_normal", 0o160000, "rev"),
(b"tree_040644", 0o040644, "dir"),
(b"tree_040664", 0o040664, "dir"),
(b"tree_040666", 0o040666, "dir"),
(b"tree_normal", 0o040000, "dir"),
]
tree = dulwich.objects.Tree()
for (name, mode, _) in entries:
tree.add(name, mode, b"00" * 20)
assert converters.dulwich_tree_to_directory(tree) == Directory(
entries=tuple(
DirectoryEntry(type=type, perms=perms, name=name, target=b"\x00" * 20)
for (name, perms, type) in entries
)
)
def test_commit_to_revision(self):
sha1 = b"9768d0b576dbaaecd80abedad6dfd0d72f1476da"
revision = converters.dulwich_commit_to_revision(self.repo[sha1])
expected_revision = Revision(
id=hash_to_bytes("9768d0b576dbaaecd80abedad6dfd0d72f1476da"),
directory=b"\xf0i\\./\xa7\xce\x9dW@#\xc3A7a\xa4s\xe5\x00\xca",
type=RevisionType.GIT,
committer=Person(
name=b"Stefano Zacchiroli",
fullname=b"Stefano Zacchiroli <zack@upsilon.cc>",
email=b"zack@upsilon.cc",
),
author=Person(
name=b"Stefano Zacchiroli",
fullname=b"Stefano Zacchiroli <zack@upsilon.cc>",
email=b"zack@upsilon.cc",
),
committer_date=TimestampWithTimezone(
timestamp=Timestamp(seconds=1443083765, microseconds=0,),
negative_utc=False,
offset=120,
),
message=b"add submodule dependency\n",
metadata=None,
extra_headers=(),
date=TimestampWithTimezone(
timestamp=Timestamp(seconds=1443083765, microseconds=0,),
negative_utc=False,
offset=120,
),
parents=(b"\xc3\xc5\x88q23`\x9f[\xbb\xb2\xd9\xe7\xf3\xfbJf\x0f?r",),
synthetic=False,
)
assert revision == expected_revision
def test_commit_to_revision_with_extra_headers(self):
sha1 = b"322f5bc915e50fc25e85226b5a182bded0e98e4b"
revision = converters.dulwich_commit_to_revision(self.repo[sha1])
expected_revision = Revision(
id=hash_to_bytes(sha1.decode()),
directory=bytes.fromhex("f8ec06e4ed7b9fff4918a0241a48023143f30000"),
type=RevisionType.GIT,
committer=Person(
name=b"David Douard",
fullname=b"David Douard <david.douard@sdfa3.org>",
email=b"david.douard@sdfa3.org",
),
author=Person(
name=b"David Douard",
fullname=b"David Douard <david.douard@sdfa3.org>",
email=b"david.douard@sdfa3.org",
),
committer_date=TimestampWithTimezone(
timestamp=Timestamp(seconds=1594137902, microseconds=0,),
negative_utc=False,
offset=120,
),
message=b"Am\xe9lioration du fichier READM\xa4\n",
metadata=None,
extra_headers=((b"encoding", b"ISO-8859-15"), (b"gpgsig", GPGSIG)),
date=TimestampWithTimezone(
timestamp=Timestamp(seconds=1594136900, microseconds=0,),
negative_utc=False,
offset=120,
),
parents=(bytes.fromhex("c730509025c6e81947102b2d77bc4dc1cade9489"),),
synthetic=False,
)
assert revision == expected_revision
@pytest.mark.parametrize(
"attribute", ["_message", "_encoding", "_author", "_gpgsig"]
)
def test_corrupt_commit(self, attribute):
# has a signature
sha1 = b"322f5bc915e50fc25e85226b5a182bded0e98e4b"
commit = copy.deepcopy(self.repo[sha1])
converters.dulwich_commit_to_revision(commit)
setattr(commit, attribute, b"abcde")
with pytest.raises(converters.HashMismatch):
converters.dulwich_commit_to_revision(commit)
if attribute == "_gpgsig":
setattr(commit, attribute, None)
with pytest.raises(converters.HashMismatch):
converters.dulwich_commit_to_revision(commit)
def test_commit_to_revision_with_extra_headers_mergetag(self):
sha1 = b"3ab3da4bf0f81407be16969df09cd1c8af9ac703"
revision = converters.dulwich_commit_to_revision(self.repo[sha1])
expected_revision = Revision(
id=hash_to_bytes(sha1.decode()),
directory=bytes.fromhex("faa4b64a841ca3e3f07d6501caebda2e3e8e544e"),
type=RevisionType.GIT,
committer=Person(
name=b"David Douard",
fullname=b"David Douard <david.douard@sdfa3.org>",
email=b"david.douard@sdfa3.org",
),
author=Person(
name=b"David Douard",
fullname=b"David Douard <david.douard@sdfa3.org>",
email=b"david.douard@sdfa3.org",
),
committer_date=TimestampWithTimezone(
timestamp=Timestamp(seconds=1594138183, microseconds=0,),
negative_utc=False,
offset=120,
),
message=b"Merge tag 'v0.0.1' into readme\n\nv0.0.1\n",
metadata=None,
extra_headers=((b"encoding", b"ISO-8859-15"), (b"mergetag", MERGETAG)),
date=TimestampWithTimezone(
timestamp=Timestamp(seconds=1594138183, microseconds=0,),
negative_utc=False,
offset=120,
),
parents=(
bytes.fromhex("322f5bc915e50fc25e85226b5a182bded0e98e4b"),
bytes.fromhex("9768d0b576dbaaecd80abedad6dfd0d72f1476da"),
),
synthetic=False,
)
assert revision == expected_revision
def test_author_line_to_author(self):
# edge case out of the way
with pytest.raises(TypeError):
converters.parse_author(None)
tests = {
b"a <b@c.com>": Person(
name=b"a", email=b"b@c.com", fullname=b"a <b@c.com>",
),
b"<foo@bar.com>": Person(
name=None, email=b"foo@bar.com", fullname=b"<foo@bar.com>",
),
b"malformed <email": Person(
name=b"malformed", email=b"email", fullname=b"malformed <email"
),
b"trailing <sp@c.e> ": Person(
name=b"trailing", email=b"sp@c.e", fullname=b"trailing <sp@c.e> ",
),
b"no<sp@c.e>": Person(name=b"no", email=b"sp@c.e", fullname=b"no<sp@c.e>",),
b" <>": Person(name=None, email=None, fullname=b" <>",),
b"something": Person(name=b"something", email=None, fullname=b"something"),
}
for author in sorted(tests):
parsed_author = tests[author]
assert parsed_author == converters.parse_author(author)
def test_dulwich_tag_to_release_no_author_no_date(self):
sha = hash_to_bytes("f6e367357b446bd1315276de5e88ba3d0d99e136")
target = b"641fb6e08ddb2e4fd096dcf18e80b894bf7e25ce"
message = b"some release message"
- tag = SWHTag(
- sha=sha,
- name=b"blah",
- type_name=b"tag",
- target=target,
- target_type=b"commit",
- message=message,
- signature=None,
- tagger=None,
- tag_time=None,
- tag_timezone=None,
- )
+ tag = dulwich.objects.Tag()
+ tag.name = b"blah"
+ tag.object = (dulwich.objects.Commit, target)
+ tag.message = message
+ tag.signature = None
+ tag.tagger = None
+ tag.tag_time = None
+ tag.tag_timezone = None
+ assert tag.sha().digest() == sha
# when
actual_release = converters.dulwich_tag_to_release(tag)
# then
expected_release = Release(
author=None,
date=None,
id=sha,
message=message,
metadata=None,
name=b"blah",
synthetic=False,
target=hash_to_bytes(target.decode()),
target_type=ObjectType.REVISION,
)
assert actual_release == expected_release
def test_dulwich_tag_to_release_author_and_date(self):
sha = hash_to_bytes("fc1e6a4f1e37e93e28e78560e73efd0b12f616ef")
tagger = b"hey dude <hello@mail.org>"
target = b"641fb6e08ddb2e4fd096dcf18e80b894bf7e25ce"
message = b"some release message"
- date = datetime.datetime(2007, 12, 5, tzinfo=datetime.timezone.utc).timestamp()
-
- tag = SWHTag(
- sha=sha,
- name=b"blah",
- type_name=b"tag",
- target=target,
- target_type=b"commit",
- message=message,
- signature=None,
- tagger=tagger,
- tag_time=date,
- tag_timezone=0,
+ date = int(
+ datetime.datetime(2007, 12, 5, tzinfo=datetime.timezone.utc).timestamp()
)
+ tag = dulwich.objects.Tag()
+ tag.name = b"blah"
+ tag.object = (dulwich.objects.Commit, target)
+ tag.message = message
+ tag.signature = None
+ tag.tagger = tagger
+ tag.tag_time = date
+ tag.tag_timezone = 0
+ assert tag.sha().digest() == sha
+
# when
actual_release = converters.dulwich_tag_to_release(tag)
# then
expected_release = Release(
author=Person(
email=b"hello@mail.org",
fullname=b"hey dude <hello@mail.org>",
name=b"hey dude",
),
date=TimestampWithTimezone(
negative_utc=False,
offset=0,
timestamp=Timestamp(seconds=1196812800, microseconds=0,),
),
id=sha,
message=message,
metadata=None,
name=b"blah",
synthetic=False,
target=hash_to_bytes(target.decode()),
target_type=ObjectType.REVISION,
)
assert actual_release == expected_release
def test_dulwich_tag_to_release_author_no_date(self):
# to reproduce bug T815 (fixed)
sha = hash_to_bytes("41076e970975122dc6b2a878aa9797960bc4781d")
tagger = b"hey dude <hello@mail.org>"
target = b"641fb6e08ddb2e4fd096dcf18e80b894bf7e25ce"
message = b"some release message"
- tag = SWHTag(
- sha=sha,
- name=b"blah",
- type_name=b"tag",
- target=target,
- target_type=b"commit",
- message=message,
- signature=None,
- tagger=tagger,
- tag_time=None,
- tag_timezone=None,
- )
+ tag = dulwich.objects.Tag()
+ tag.name = b"blah"
+ tag.object = (dulwich.objects.Commit, target)
+ tag.message = message
+ tag.signature = None
+ tag.tagger = tagger
+ tag.tag_time = None
+ tag.tag_timezone = None
+ assert tag.sha().digest() == sha
# when
actual_release = converters.dulwich_tag_to_release(tag)
# then
expected_release = Release(
author=Person(
email=b"hello@mail.org",
fullname=b"hey dude <hello@mail.org>",
name=b"hey dude",
),
date=None,
id=sha,
message=message,
metadata=None,
name=b"blah",
synthetic=False,
target=hash_to_bytes(target.decode()),
target_type=ObjectType.REVISION,
)
assert actual_release == expected_release
def test_dulwich_tag_to_release_author_zero_date(self):
# to reproduce bug T815 (fixed)
sha = hash_to_bytes("6cc1deff5cdcd853428bb63b937f43dd2566c36f")
tagger = b"hey dude <hello@mail.org>"
target = b"641fb6e08ddb2e4fd096dcf18e80b894bf7e25ce"
message = b"some release message"
- date = datetime.datetime(1970, 1, 1, tzinfo=datetime.timezone.utc).timestamp()
- tag = SWHTag(
- sha=sha,
- name=b"blah",
- type_name=b"tag",
- target=target,
- target_type=b"commit",
- message=message,
- signature=None,
- tagger=tagger,
- tag_time=date,
- tag_timezone=0,
+ date = int(
+ datetime.datetime(1970, 1, 1, tzinfo=datetime.timezone.utc).timestamp()
)
+ tag = dulwich.objects.Tag()
+ tag.name = b"blah"
+ tag.object = (dulwich.objects.Commit, target)
+ tag.message = message
+ tag.signature = None
+ tag.tagger = tagger
+ tag.tag_time = date
+ tag.tag_timezone = 0
+ assert tag.sha().digest() == sha
# when
actual_release = converters.dulwich_tag_to_release(tag)
# then
expected_release = Release(
author=Person(
email=b"hello@mail.org",
fullname=b"hey dude <hello@mail.org>",
name=b"hey dude",
),
date=TimestampWithTimezone(
negative_utc=False,
offset=0,
timestamp=Timestamp(seconds=0, microseconds=0,),
),
id=sha,
message=message,
metadata=None,
name=b"blah",
synthetic=False,
target=hash_to_bytes(target.decode()),
target_type=ObjectType.REVISION,
)
assert actual_release == expected_release
def test_dulwich_tag_to_release_signature(self):
target = b"641fb6e08ddb2e4fd096dcf18e80b894bf7e25ce"
message = b"some release message"
sha = hash_to_bytes("46fff489610ed733d2cc904e363070dadee05c71")
- tag = SWHTag(
- sha=sha,
- name=b"blah",
- type_name=b"tag",
- target=target,
- target_type=b"commit",
- message=message,
- signature=GPGSIG,
- tagger=None,
- tag_time=None,
- tag_timezone=None,
- )
+ tag = dulwich.objects.Tag()
+ tag.name = b"blah"
+ tag.object = (dulwich.objects.Commit, target)
+ tag.message = message
+ tag.signature = GPGSIG
+ tag.tagger = None
+ tag.tag_time = None
+ tag.tag_timezone = None
+ assert tag.sha().digest() == sha
# when
actual_release = converters.dulwich_tag_to_release(tag)
# then
expected_release = Release(
author=None,
date=None,
id=sha,
message=message + GPGSIG,
metadata=None,
name=b"blah",
synthetic=False,
target=hash_to_bytes(target.decode()),
target_type=ObjectType.REVISION,
)
assert actual_release == expected_release
@pytest.mark.parametrize("attribute", ["name", "message", "signature"])
def test_corrupt_tag(self, attribute):
# has a signature
sha = hash_to_bytes("46fff489610ed733d2cc904e363070dadee05c71")
target = b"641fb6e08ddb2e4fd096dcf18e80b894bf7e25ce"
message = b"some release message"
- tag = SWHTag(
- sha=sha,
- name=b"blah",
- type_name=b"tag",
- target=target,
- target_type=b"commit",
- message=message,
- signature=GPGSIG,
- tagger=None,
- tag_time=None,
- tag_timezone=None,
- )
+ tag = dulwich.objects.Tag()
+ tag.name = b"blah"
+ tag.object = (dulwich.objects.Commit, target)
+ tag.message = message
+ tag.signature = GPGSIG
+ tag.tagger = None
+ tag.tag_time = None
+ tag.tag_timezone = None
+ assert tag.sha().digest() == sha
converters.dulwich_tag_to_release(tag)
+ original_sha = tag.sha()
+
setattr(tag, attribute, b"abcde")
+ tag.sha() # reset tag._needs_serialization
+ tag._sha = original_sha # force the wrong hash
with pytest.raises(converters.HashMismatch):
converters.dulwich_tag_to_release(tag)
if attribute == "signature":
setattr(tag, attribute, None)
+ tag.sha() # reset tag._needs_serialization
+ tag._sha = original_sha # force the wrong hash
with pytest.raises(converters.HashMismatch):
converters.dulwich_tag_to_release(tag)
diff --git a/tox.ini b/tox.ini
index 8b07a2a..1d1a645 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,78 +1,78 @@
[tox]
envlist=black,flake8,mypy,py3
[testenv]
extras =
testing
deps =
# the dependency below is needed for now as a workaround for
# https://github.com/pypa/pip/issues/6239
# TODO: remove when this issue is fixed
swh.core[testing] >= 0.0.61
swh.storage[testing]
swh.scheduler[testing] >= 0.5.0
pytest-cov
commands =
pytest --cov={envsitepackagesdir}/swh/loader/git \
{envsitepackagesdir}/swh/loader/git \
--cov-branch {posargs}
[testenv:black]
skip_install = true
deps =
black==19.10b0
commands =
{envpython} -m black --check swh
[testenv:flake8]
skip_install = true
deps =
flake8
commands =
{envpython} -m flake8
[testenv:mypy]
extras =
testing
deps =
- mypy
+ mypy==0.920
commands =
mypy swh
# build documentation outside swh-environment using the current
# git HEAD of swh-docs, is executed on CI for each diff to prevent
# breaking doc build
[testenv:sphinx]
whitelist_externals = make
usedevelop = true
extras =
testing
deps =
# fetch and install swh-docs in develop mode
-e git+https://forge.softwareheritage.org/source/swh-docs#egg=swh.docs
setenv =
SWH_PACKAGE_DOC_TOX_BUILD = 1
# turn warnings into errors
SPHINXOPTS = -W
commands =
make -I ../.tox/sphinx/src/swh-docs/swh/ -C docs
# build documentation only inside swh-environment using local state
# of swh-docs package
[testenv:sphinx-dev]
whitelist_externals = make
usedevelop = true
extras =
testing
deps =
# install swh-docs in develop mode
-e ../swh-docs
setenv =
SWH_PACKAGE_DOC_TOX_BUILD = 1
# turn warnings into errors
SPHINXOPTS = -W
commands =
make -I ../.tox/sphinx-dev/src/swh-docs/swh/ -C docs

File Metadata

Mime Type
text/x-diff
Expires
Fri, Jul 4, 12:42 PM (2 w, 2 d ago)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
3441258

Event Timeline

About · Developer information · Legal