Page MenuHomeSoftware Heritage
Differential D5656

Add a naive git bare cooker
ClosedPublic
Actions

Authored by vlorentz on Apr 30 2021, 7:32 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T843: Vault: Add a "git bare" tarball cooker
Summary

It can cook directories (by adding a synthetic revision pointing to it)
and revisions.

Current limitations:

  • It does not deduplicate directories and files at all, and queries all objects one by one.
  • No support for missing/absent contents
  • No support for missing submodules

Tests reuse existing tests of the DirectoryCooker and
RevisionGitfastCooker using parametrized pytest fixtures.

Diff Detail

Event Timeline

vlorentz created this revision.Apr 30 2021, 7:32 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 30 2021, 7:32 PM
swh-public-ci added a comment.Apr 30 2021, 7:32 PM

Build has FAILED

Patch application report for D5656 (id=20205)

Rebasing onto f87dd54d5d...

Current branch diff-target is up to date.
Changes applied before test
commit caeddd4492e4888598e47e4dfa87991e4db77853
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/71/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/71/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 30 2021, 7:32 PM
Harbormaster failed remote builds in B21195: Diff 20205!
vlorentz updated this revision to Diff 20206.Apr 30 2021, 7:33 PM

oops, tempdir

swh-public-ci added a comment.Apr 30 2021, 7:34 PM

Build has FAILED

Patch application report for D5656 (id=20206)

Rebasing onto f87dd54d5d...

Current branch diff-target is up to date.
Changes applied before test
commit c3984d34c2a8cf32ff63595013b64860f1e25054
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/72/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/72/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 30 2021, 7:34 PM
Harbormaster failed remote builds in B21196: Diff 20206!
vlorentz added a child revision: D5658: git_bare: Deduplicate object downloads and writes.Apr 30 2021, 11:10 PM
vlorentz requested review of this revision.May 3 2021, 10:59 AM
douardda added a subscriber: douardda.May 4 2021, 6:17 PM
douardda added inline comments.
swh/vault/cookers/git_bare.py
87

As said on IRC, I don't think this 'directory' support should be in there. If it's only purpose is to help reusing tests, then tests should be refactored instead of adding code in the business logic.

111

Maybe we should not use an hardcoded 'mater' branch here? we should at least make it configurable (nowadays)?

134

why in "append" mode ?

179

do you mean objects like SkippedContent?

vlorentz added inline comments.May 4 2021, 6:44 PM
swh/vault/cookers/git_bare.py
87

It's also useful when testing manually with the CLI.

111

We can make it configurable when we'll need it.

179

"filtered" is the generic name for "skipped or hidden or missing"

vlorentz updated this revision to Diff 20270.May 4 2021, 6:48 PM

don't use append mode for writing objects

swh-public-ci added a comment.May 4 2021, 6:49 PM

Build has FAILED

Patch application report for D5656 (id=20270)

Rebasing onto f87dd54d5d...

Current branch diff-target is up to date.
Changes applied before test
commit 91472855a218e6ebcd81b496936f6f8b234fde38
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

Link to build: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/74/
See console output for more information: https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/74/console

Harbormaster failed remote builds in B21257: Diff 20270!May 4 2021, 6:49 PM
douardda accepted this revision.May 5 2021, 12:40 PM
douardda added inline comments.
swh/vault/cookers/git_bare.py
87

I don't really understand why (what is really the added value of this git-bare-directory' cooker), but do as you wish.

179

indeed

This revision is now accepted and ready to land.May 5 2021, 12:40 PM
vlorentz added a parent revision: D5652: identifiers: Expose git_object instead of manifest.May 6 2021, 2:46 PM
vlorentz added a task: T843: Vault: Add a "git bare" tarball cooker.May 10 2021, 9:48 AM
swh-public-ci added a comment.May 11 2021, 3:21 PM

Build is green

Patch application report for D5656 (id=20270)

Rebasing onto 35c9f519cd...

First, rewinding head to replay your work on top of it...
Applying: Add a naive git bare cooker
Changes applied before test
commit 69d84aa2240a18e702892c2fe59f731d72c5d50e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 30 19:30:34 2021 +0200

    Add a naive git bare cooker
    
    It can cook directories (by adding a synthetic revision pointing to it)
    and revisions.
    
    Current limitations:
    
    * It does not deduplicate directories and files at all, and queries
      all objects one by one.
    * No support for missing/absent contents
    * No support for missing submodules
    
    Tests reuse existing tests of the DirectoryCooker and
    RevisionGitfastCooker using parametrized pytest fixtures.

See https://jenkins.softwareheritage.org/job/DVAU/job/tests-on-diff/102/ for more details.

Harbormaster completed remote builds in B21257: Diff 20270.May 11 2021, 3:22 PM
vlorentz closed this revision.May 11 2021, 3:57 PM

4052f53698454ac47a01d26d470b8ab4b0f77a6d

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
vault/
2 lines79%-
cookers/
3 lines97%100%
6 lines94%100%
184 lines96%96%
2 lines57%
Loading...
tests/
135 lines98%
Loading...

Diff 20270

View Options

swh/vault/cli.py

Show First 20 Lines • Show All 47 Lines • ▼ Show 20 Lines@click.option(
metavar="CONFIGFILE", metavar="CONFIGFILE",
type=click.Path(exists=True, dir_okay=False,), type=click.Path(exists=True, dir_okay=False,),
help="Configuration file.", help="Configuration file.",
) )
@click.argument("swhid", type=SwhidParamType()) @click.argument("swhid", type=SwhidParamType())
@click.argument("outfile", type=click.File("wb")) @click.argument("outfile", type=click.File("wb"))
@click.option( @click.option(
"--cooker-type", "--cooker-type",
type=click.Choice(["flat", "gitfast"]), type=click.Choice(["flat", "gitfast", "git_bare"]),
help="Selects which cooker to use, when there is more than one available " help="Selects which cooker to use, when there is more than one available "
"for the given object type.", "for the given object type.",
) )
@click.pass_context @click.pass_context
def cook( def cook(
ctx, ctx,
config_file: str, config_file: str,
swhid: CoreSWHID, swhid: CoreSWHID,
▲ Show 20 Lines • Show All 101 Lines • Show Last 20 Lines
View Options

swh/vault/cookers/__init__.py

# Copyright (C) 2017-2020 The Software Heritage developers # Copyright (C) 2017-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
from __future__ import annotations from __future__ import annotations
import os import os
from typing import Any, Dict from typing import Any, Dict
from swh.core.config import load_named_config from swh.core.config import load_named_config
from swh.core.config import read as read_config from swh.core.config import read as read_config
from swh.storage import get_storage from swh.storage import get_storage
from swh.vault import get_vault from swh.vault import get_vault
from swh.vault.cookers.base import DEFAULT_CONFIG, DEFAULT_CONFIG_PATH from swh.vault.cookers.base import DEFAULT_CONFIG, DEFAULT_CONFIG_PATH
from swh.vault.cookers.directory import DirectoryCooker from swh.vault.cookers.directory import DirectoryCooker
from swh.vault.cookers.git_bare import GitBareCooker
from swh.vault.cookers.revision_flat import RevisionFlatCooker from swh.vault.cookers.revision_flat import RevisionFlatCooker
from swh.vault.cookers.revision_gitfast import RevisionGitfastCooker from swh.vault.cookers.revision_gitfast import RevisionGitfastCooker
COOKER_TYPES = { COOKER_TYPES = {
"directory": DirectoryCooker, "directory": DirectoryCooker,
"revision_flat": RevisionFlatCooker, "revision_flat": RevisionFlatCooker,
"revision_gitfast": RevisionGitfastCooker, "revision_gitfast": RevisionGitfastCooker,
"revision_git_bare": GitBareCooker,
"directory_git_bare": GitBareCooker,
} }
def get_cooker_cls(obj_type): def get_cooker_cls(obj_type):
return COOKER_TYPES[obj_type] return COOKER_TYPES[obj_type]
def check_config(cfg: Dict[str, Any]) -> Dict[str, Any]: def check_config(cfg: Dict[str, Any]) -> Dict[str, Any]:
▲ Show 20 Lines • Show All 65 Lines • Show Last 20 Lines
View Options

swh/vault/cookers/base.py

Show First 20 Lines • Show All 92 Lines • ▼ Show 20 Linesclass BaseVaultCooker(metaclass=abc.ABCMeta):
@abc.abstractmethod @abc.abstractmethod
def prepare_bundle(self): def prepare_bundle(self):
"""Implementation of the cooker. Yields chunks of the bundle bytes. """Implementation of the cooker. Yields chunks of the bundle bytes.
Override this with the cooker implementation. Override this with the cooker implementation.
""" """
raise NotImplementedError raise NotImplementedError
def cache_type_key(self) -> str:
assert self.CACHE_TYPE_KEY
return self.CACHE_TYPE_KEY
def write(self, chunk): def write(self, chunk):
self.fileobj.write(chunk) self.fileobj.write(chunk)
def cook(self): def cook(self):
"""Cook the requested object into a bundle """Cook the requested object into a bundle
""" """
self.backend.set_status(self.obj_type, self.obj_id, "pending") self.backend.set_status(self.obj_type, self.obj_id, "pending")
self.backend.set_progress(self.obj_type, self.obj_id, "Processing...") self.backend.set_progress(self.obj_type, self.obj_id, "Processing...")
self.fileobj = BytesIOBundleSizeLimit(size_limit=self.max_bundle_size) self.fileobj = BytesIOBundleSizeLimit(size_limit=self.max_bundle_size)
try: try:
try: try:
self.prepare_bundle() self.prepare_bundle()
except QueryCanceledError: except QueryCanceledError:
raise PolicyError( raise PolicyError(
"Timeout reached while assembling the requested bundle" "Timeout reached while assembling the requested bundle"
) )
bundle = self.fileobj.getvalue() bundle = self.fileobj.getvalue()
# TODO: use proper content streaming instead of put_bundle() # TODO: use proper content streaming instead of put_bundle()
self.backend.put_bundle(self.CACHE_TYPE_KEY, self.obj_id, bundle) self.backend.put_bundle(self.cache_type_key(), self.obj_id, bundle)
except PolicyError as e: except PolicyError as e:
self.backend.set_status(self.obj_type, self.obj_id, "failed") self.backend.set_status(self.obj_type, self.obj_id, "failed")
self.backend.set_progress(self.obj_type, self.obj_id, str(e)) self.backend.set_progress(self.obj_type, self.obj_id, str(e))
except Exception: except Exception:
self.backend.set_status(self.obj_type, self.obj_id, "failed") self.backend.set_status(self.obj_type, self.obj_id, "failed")
self.backend.set_progress( self.backend.set_progress(
self.obj_type, self.obj_type,
self.obj_id, self.obj_id,
"Internal Server Error. This incident will be reported.", "Internal Server Error. This incident will be reported.",
) )
logging.exception("Bundle cooking failed.") logging.exception("Bundle cooking failed.")
else: else:
self.backend.set_status(self.obj_type, self.obj_id, "done") self.backend.set_status(self.obj_type, self.obj_id, "done")
self.backend.set_progress(self.obj_type, self.obj_id, None) self.backend.set_progress(self.obj_type, self.obj_id, None)
finally: finally:
self.backend.send_notif(self.obj_type, self.obj_id) self.backend.send_notif(self.obj_type, self.obj_id)
View Options

swh/vault/cookers/git_bare.py

  • This file was added.
# Copyright (C) 2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import datetime
import os.path
import subprocess
import tarfile
import tempfile
from typing import Any, Dict
import zlib
from swh.model import identifiers
from swh.model.hashutil import hash_to_bytehex, hash_to_hex
from swh.model.model import (
Person,
Revision,
RevisionType,
Sha1Git,
TimestampWithTimezone,
)
from swh.storage.algos.revisions_walker import DFSRevisionsWalker
from swh.vault.cookers.base import BaseVaultCooker
class GitBareCooker(BaseVaultCooker):
use_fsck = True
def cache_type_key(self) -> str:
return self.obj_type
def check_exists(self):
obj_type = self.obj_type.split("_")[0]
if obj_type == "revision":
return not list(self.storage.revision_missing([self.obj_id]))
elif obj_type == "directory":
return not list(self.storage.directory_missing([self.obj_id]))
else:
raise NotImplementedError(f"GitBareCooker for {obj_type}")
def obj_swhid(self) -> identifiers.CoreSWHID:
obj_type = self.obj_type.split("_")[0]
return identifiers.CoreSWHID(
object_type=identifiers.ObjectType[obj_type.upper()], object_id=self.obj_id,
)
def prepare_bundle(self):
with tempfile.TemporaryDirectory() as workdir:
# Initialize a Git directory
self.workdir = workdir
self.gitdir = os.path.join(workdir, "clone.git")
os.mkdir(self.gitdir)
self.init_git()
# Load and write all the objects to disk
self.load_subgraph(self.obj_type.split("_")[0], self.obj_id)
# Write the root object as a ref.
# This must be done before repacking; git-repack ignores orphan objects.
self.write_refs()
self.repack()
self.write_archive()
def init_git(self) -> None:
subprocess.run(["git", "-C", self.gitdir, "init", "--bare"], check=True)
# Create all possible dirs ahead of time, so we don't have to check for
# existence every time.
for byte in range(256):
os.mkdir(os.path.join(self.gitdir, "objects", f"{byte:02x}"))
def repack(self) -> None:
if self.use_fsck:
subprocess.run(["git", "-C", self.gitdir, "fsck"], check=True)
# Add objects we wrote in a pack
subprocess.run(["git", "-C", self.gitdir, "repack"], check=True)
# Remove their non-packed originals
subprocess.run(["git", "-C", self.gitdir, "prune-packed"], check=True)
def write_refs(self):
obj_type = self.obj_type.split("_")[0]
if obj_type == "directory":
# We need a synthetic revision pointing to the directory
douarddaUnsubmitted
Not Done
Inline Actions

As said on IRC, I don't think this 'directory' support should be in there. If it's only purpose is to help reusing tests, then tests should be refactored instead of adding code in the business logic.

douardda: As said on IRC, I don't think this 'directory' support should be in there. If it's only purpose…
vlorentzAuthorUnsubmitted
Done
Inline Actions

It's also useful when testing manually with the CLI.

vlorentz: It's also useful when testing manually with the CLI.
douarddaUnsubmitted
Not Done
Inline Actions

I don't really understand why (what is really the added value of this git-bare-directory' cooker), but do as you wish.

douardda: I don't really understand why (what is really the added value of this git-bare-directory'…
author = Person.from_fullname(
b"swh-vault, git-bare cooker <robot@softwareheritage.org>"
)
dt = datetime.datetime.now(tz=datetime.timezone.utc)
dt = dt.replace(microsecond=0) # not supported by git
date = TimestampWithTimezone.from_datetime(dt)
revision = Revision(
author=author,
committer=author,
date=date,
committer_date=date,
message=b"Initial commit",
type=RevisionType.GIT,
directory=self.obj_id,
synthetic=True,
)
self.write_revision_node(revision.to_dict())
head = revision.id
elif obj_type == "revision":
head = self.obj_id
else:
assert False, obj_type
with open(os.path.join(self.gitdir, "refs", "heads", "master"), "wb") as fd:
douarddaUnsubmitted
Not Done
Inline Actions

Maybe we should not use an hardcoded 'mater' branch here? we should at least make it configurable (nowadays)?

douardda: Maybe we should not use an hardcoded 'mater' branch here? we should at least make it…
vlorentzAuthorUnsubmitted
Done
Inline Actions

We can make it configurable when we'll need it.

vlorentz: We can make it configurable when we'll need it.
fd.write(hash_to_bytehex(head))
def write_archive(self):
with tarfile.TarFile(mode="w", fileobj=self.fileobj) as tf:
tf.add(self.gitdir, arcname=f"{self.obj_swhid()}.git", recursive=True)
def write_object(self, obj_id: Sha1Git, obj: bytes) -> bool:
"""Writes a git object on disk.
Returns whether it was already written."""
obj_id_hex = hash_to_hex(obj_id)
directory = obj_id_hex[0:2]
filename = obj_id_hex[2:]
path = os.path.join(self.gitdir, "objects", directory, filename)
if os.path.exists(path):
# Already written
return False
# Git requires objects to be zlib-compressed; but repacking decompresses and
# removes them, so we don't need to compress them too much.
data = zlib.compress(obj, level=1)
with open(path, "wb") as fd:
douarddaUnsubmitted
Not Done
Inline Actions

why in "append" mode ?

douardda: why in "append" mode ?
fd.write(data)
return True
def load_subgraph(self, obj_type, obj_id) -> None:
if obj_type == "revision":
self.load_revision_subgraph(obj_id)
elif obj_type == "directory":
self.load_directory_subgraph(obj_id)
else:
raise NotImplementedError(f"GitBareCooker.load_subgraph({obj_type!r}, ...)")
def load_revision_subgraph(self, obj_id: Sha1Git) -> None:
"""Fetches a revision and all its children, and writes them to disk"""
walker = DFSRevisionsWalker(self.storage, obj_id)
for revision in walker:
self.write_revision_node(revision)
self.load_directory_subgraph(revision["directory"])
def write_revision_node(self, revision: Dict[str, Any]) -> bool:
"""Writes a revision object to disk"""
git_object = identifiers.revision_git_object(revision)
return self.write_object(revision["id"], git_object)
def load_directory_subgraph(self, obj_id: Sha1Git) -> None:
"""Fetches a directory and all its children, and writes them to disk"""
directory = self.load_directory_node(obj_id)
entry_loaders = {
"file": self.load_content,
"dir": self.load_directory_subgraph,
"rev": self.load_revision_subgraph,
}
for entry in directory["entries"]:
entry_loader = entry_loaders[entry["type"]]
entry_loader(entry["target"])
def load_directory_node(self, obj_id: Sha1Git) -> Dict[str, Any]:
"""Fetches a directory, writes it to disk (non-recursively), and returns it."""
entries = list(self.storage.directory_ls(obj_id, recursive=False))
directory = {"id": obj_id, "entries": entries}
git_object = identifiers.directory_git_object(directory)
self.write_object(obj_id, git_object)
return directory
def load_content(self, obj_id: Sha1Git) -> None:
# TODO: add support of filtered objects, somehow?
douarddaUnsubmitted
Not Done
Inline Actions

do you mean objects like SkippedContent?

douardda: do you mean objects like `SkippedContent`?
vlorentzAuthorUnsubmitted
Done
Inline Actions

"filtered" is the generic name for "skipped or hidden or missing"

vlorentz: "filtered" is the generic name for "skipped or hidden or missing"
douarddaUnsubmitted
Not Done
Inline Actions

indeed

douardda: indeed
# It's tricky, because, by definition, we can't write a git object with
# the expected hash, so git-fsck *will* choke on it.
content_sha1 = self.storage.content_find({"sha1_git": obj_id})[0].sha1
content = self.storage.content_get_data(content_sha1)
self.write_object(obj_id, f"blob {len(content)}\0".encode("ascii") + content)
View Options

swh/vault/in_memory_backend.py

Loading...
View Options

swh/vault/tests/test_cookers.py

Loading...
About · Developer information · Legal