Page MenuHomeSoftware Heritage
Differential D4197

proxmox/terraform: Align production nodes declaration with current module version
ClosedPublic
Actions

Authored by ardumont on Oct 8 2020, 11:35 AM.

Details

Reviewers
vsellier
Group Reviewers
Reviewers
Commits
rSPREe0df92bba55b: proxmox/terraform/production: Align declaration with module
Summary

Just aligning the production declaration with the current "node module" interface.

So the terraform plan actually "compiles" instead of possibly breaking [1]

[1]*possibly* because i did not check but the old declaration could not have worked given the evolution of that module

Test Plan

terraform plan

P822

Diff Detail

Repository
rSPRE sysadm-provisioning
Branch
align-production
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 16286
Build 25064: arc lint + arc unit

Event Timeline

ardumont created this revision.Oct 8 2020, 11:35 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 8 2020, 11:35 AM
Harbormaster completed remote builds in B16033: Diff 14767.Oct 8 2020, 11:35 AM
ardumont edited the test plan for this revision. (Show Details)Oct 8 2020, 5:35 PM
ardumont added inline comments.Oct 12 2020, 3:25 PM
proxmox/terraform/zookeeper.tf
82

no need, that's the default.
plus it's now a storages declaration.

vsellier added a subscriber: vsellier.Oct 13 2020, 12:14 PM
vsellier added inline comments.
proxmox/terraform/zookeeper.tf
46

The vmid for zookeeper3 is 102, I suppose it must be the same on the tf.

ardumont added inline comments.Oct 14 2020, 6:37 PM
proxmox/terraform/zookeeper.tf
46

oh yes, thanks.

ardumont updated this revision to Diff 15058.Oct 14 2020, 6:39 PM
ardumont edited the test plan for this revision. (Show Details)

Fix zookeeper3 vmid

Harbormaster completed remote builds in B16285: Diff 15058.Oct 14 2020, 6:39 PM
ardumont updated this revision to Diff 15059.Oct 14 2020, 6:42 PM

Refactor some more.

terraform is aggregating all the .tf file into one. So we can leverage this to
declare a common state to be shared in the common.tf file. This way, no need to
duplicate the local.config dictionary.

Harbormaster completed remote builds in B16286: Diff 15059.Oct 14 2020, 6:42 PM
ardumont updated this revision to Diff 15061.Oct 14 2020, 6:55 PM

Add terraform.tfstate for the production nodes as well

Harbormaster completed remote builds in B16288: Diff 15061.Oct 14 2020, 6:55 PM
ardumont edited the test plan for this revision. (Show Details)Oct 14 2020, 6:57 PM
ardumont retitled this revision from proxmox/terraform/production: Align declaration with module to proxmox/terraform: Align production nodes declaration with current module version.
ardumont edited the summary of this revision. (Show Details)
vsellier accepted this revision as: vsellier.Oct 15 2020, 9:34 AM

👍 it looks synchronized

➜  terraform git:(arcpatch-D4197) terraform  plan -no-color | grep -E -e "^(\s)+[~+-]" -e "(name|target_node)\s+="
  ~ update in-place
  ~ resource "proxmox_vm_qemu" "node" {
        name         = "kelvingrove"
        target_node  = "hypervisor3"
      + vmid         = 123
  ~ resource "proxmox_vm_qemu" "node" {
        name         = "zookeeper1"
        target_node  = "hypervisor3"
      + vmid         = 125
  ~ resource "proxmox_vm_qemu" "node" {
        name         = "zookeeper2"
        target_node  = "branly"
      + vmid         = 124
  ~ resource "proxmox_vm_qemu" "node" {
        name         = "zookeeper3"
        target_node  = "beaubourg"
      + vmid         = 102
This revision is now accepted and ready to land.Oct 15 2020, 9:34 AM
ardumont edited the summary of this revision. (Show Details)Oct 15 2020, 9:48 AM
ardumont edited the summary of this revision. (Show Details)
Closed by commit rSPREe0df92bba55b: proxmox/terraform/production: Align declaration with module (authored by ardumont). · Explain WhyOct 15 2020, 9:50 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rSPREe0df92bba55b: proxmox/terraform/production: Align declaration with module.

Revision Contents
Changeset List

PathSize
proxmox/
terraform/
11 lines
20 lines
65 lines

Diff 15059

View Options

proxmox/terraform/common.tf

# Copyright (C) 2015-2017 The Software Heritage developers # Copyright (C) 2015-2018 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
"""Module in charge of hashing function definitions. This is the base """Module in charge of hashing function definitions. This is the base
module use to compute swh's hashes. module use to compute swh's hashes.
Only a subset of hashing algorithms is supported as defined in the Only a subset of hashing algorithms is supported as defined in the
Show All 30 Lines
Subset of :const:`ALGORITHMS`. Subset of :const:`ALGORITHMS`.
""" """
HASH_BLOCK_SIZE = 32768 HASH_BLOCK_SIZE = 32768
"""Block size for streaming hash computations made in this module""" """Block size for streaming hash computations made in this module"""
_blake2_hash_cache = {} _blake2_hash_cache = {}
HASH_FORMATS = set(['bytes', 'bytehex', 'hex'])
"""Supported output hash formats
"""
class MultiHash:
"""Hashutil class to support multiple hashes computation.
Args:
hash_names (set): Set of hash algorithms (+ length) to compute
hashes (cf. DEFAULT_ALGORITHMS)
length (int): Length of the total sum of chunks to read
If the length is provided as algorithm, the length is also
computed and returned.
"""
def __init__(self, hash_names=DEFAULT_ALGORITHMS, length=None):
self.state = {}
self.track_length = False
for name in hash_names:
if name == 'length':
self.state['length'] = 0
self.track_length = True
else:
self.state[name] = _new_hash(name, length)
@classmethod
def from_state(cls, state, track_length):
ret = cls([])
ret.state = state
ret.track_length = track_length
def update(self, chunk):
for name, h in self.state.items():
if name == 'length':
continue
h.update(chunk)
if self.track_length:
self.state['length'] += len(chunk)
def digest(self):
return {
name: h.digest() if name != 'length' else h
for name, h in self.state.items()
}
def hexdigest(self):
return {
name: h.hexdigest() if name != 'length' else h
for name, h in self.state.items()
ardumontAuthorUnsubmitted
Not Done
Inline Actions

I don't think this is used much. IIRC, it's only used once.

So, i'm for removing that part later (when we will have migrated the base code).

ardumont: I don't think this is used much. IIRC, it's only used once. So, i'm for removing that part…
olasdUnsubmitted
Done
Inline Actions

In any case, we should just use the length that we've retrieved from os.path.getsize rather than computing it again.

olasd: In any case, we should just use the length that we've retrieved from os.path.getsize rather…
ardumontAuthorUnsubmitted
Not Done
Inline Actions

Right, fixing that as well.

ardumont: Right, fixing that as well.
}
def bytehexdigest(self):
return {
name: hash_to_bytehex(h.digest()) if name != 'length' else h
for name, h in self.state.items()
}
def copy(self):
copied_state = {
name: h.copy() if name != 'length' else h
for name, h in self.state.items()
}
return self.from_state(copied_state, self.track_length)
def _new_blake2_hash(algo): def _new_blake2_hash(algo):
"""Return a function that initializes a blake2 hash. """Return a function that initializes a blake2 hash.
""" """
if algo in _blake2_hash_cache: if algo in _blake2_hash_cache:
return _blake2_hash_cache[algo]() return _blake2_hash_cache[algo]()
▲ Show 20 Lines • Show All 101 Lines • ▼ Show 20 Lines if algo.endswith('_git'):
if length is None: if length is None:
raise ValueError('Missing length for git hashing algorithm') raise ValueError('Missing length for git hashing algorithm')
base_algo = algo[:-4] base_algo = algo[:-4]
return _new_git_hash(base_algo, 'blob', length) return _new_git_hash(base_algo, 'blob', length)
return _new_hashlib_hash(algo) return _new_hashlib_hash(algo)
def hash_file(fobj, length=None, algorithms=DEFAULT_ALGORITHMS, chunk_cb=None): def hash_file(fobj, length=None, algorithms=DEFAULT_ALGORITHMS,
chunk_cb=None, hash_format='bytes'):
"""Hash the contents of the given file object with the given algorithms. """Hash the contents of the given file object with the given algorithms.
Args: Args:
fobj: a file-like object fobj: a file-like object
length: the length of the contents of the file-like object (for the length: the length of the contents of the file-like object (for the
git-specific algorithms) git-specific algorithms)
algorithms: the hashing algorithms to be used, as an iterable over algorithms: the hashing algorithms to be used, as an iterable over
strings strings
hash_format (str): Format required for the output of the
computed hashes (cf. HASH_FORMATS)
Returns: a dict mapping each algorithm to a bytes digest. Returns: a dict mapping each algorithm to a digest (bytes by default).
Raises: Raises:
ValueError if algorithms contains an unknown hash algorithm. ValueError if:
algorithms contains an unknown hash algorithm.
hash_format is an unknown hash format
""" """
hashes = {algo: _new_hash(algo, length) for algo in algorithms} if hash_format not in HASH_FORMATS:
raise ValueError('Unexpected hash format %s, expected one of %s' % (
hash_format, HASH_FORMATS))
h = MultiHash(algorithms, length)
while True: while True:
chunk = fobj.read(HASH_BLOCK_SIZE) chunk = fobj.read(HASH_BLOCK_SIZE)
if not chunk: if not chunk:
break break
for hash in hashes.values(): h.update(chunk)
hash.update(chunk)
if chunk_cb: if chunk_cb:
chunk_cb(chunk) chunk_cb(chunk)
return {algo: hash.digest() for algo, hash in hashes.items()} if hash_format == 'bytes':
return h.digest()
if hash_format == 'bytehex':
return h.bytehexdigest()
return h.hexdigest()
def hash_path(path, algorithms=DEFAULT_ALGORITHMS, chunk_cb=None): def hash_path(path, algorithms=DEFAULT_ALGORITHMS, chunk_cb=None,
hash_format='bytes', track_length=True):
"""Hash the contents of the file at the given path with the given """Hash the contents of the file at the given path with the given
algorithms. algorithms.
Args: Args:
path: the path of the file to hash path (str): the path of the file to hash
algorithms: the hashing algorithms used algorithms (set): the hashing algorithms used
chunk_cb: a callback chunk_cb (def): a callback
hash_format (str): Format required for the output of the
computed hashes (cf. HASH_FORMATS)
Returns: a dict mapping each algorithm to a bytes digest. Returns: a dict mapping each algorithm to a bytes digest.
Raises: Raises:
ValueError if algorithms contains an unknown hash algorithm. ValueError if:
algorithms contains an unknown hash algorithm.
hash_format is an unknown hash format
OSError on file access error OSError on file access error
""" """
if track_length:
algorithms = set(['length']).union(algorithms)
length = os.path.getsize(path) length = os.path.getsize(path)
with open(path, 'rb') as fobj: with open(path, 'rb') as fobj:
hash = hash_file(fobj, length, algorithms, chunk_cb) return hash_file(fobj, length, algorithms, chunk_cb=chunk_cb,
hash['length'] = length hash_format=hash_format)
return hash
def hash_data(data, algorithms=DEFAULT_ALGORITHMS, with_length=False): def hash_data(data, algorithms=DEFAULT_ALGORITHMS, hash_format='bytes'):
"""Hash the given binary blob with the given algorithms. """Hash the given binary blob with the given algorithms.
Args: Args:
data (bytes): raw content to hash data (bytes): raw content to hash
algorithms (list): the hashing algorithms used algorithms (list): the hashing algorithms used
with_length (bool): add the length key in the resulting dict hash_format (str): Format required for the output of the
computed hashes (cf. HASH_FORMATS)
Returns: a dict mapping each algorithm to a bytes digest Returns: a dict mapping each algorithm to a bytes digest
Raises: Raises:
TypeError if data does not support the buffer interface. TypeError if data does not support the buffer interface.
ValueError if algorithms contains an unknown hash algorithm. ValueError if:
algorithms contains an unknown hash algorithm.
hash_format is an unknown hash format
""" """
fobj = BytesIO(data) fobj = BytesIO(data)
length = len(data) length = len(data)
data = hash_file(fobj, length, algorithms) return hash_file(fobj, length, algorithms, hash_format=hash_format)
if with_length:
data['length'] = length
return data
def hash_git_data(data, git_type, base_algo='sha1'): def hash_git_data(data, git_type, base_algo='sha1'):
"""Hash the given data as a git object of type git_type. """Hash the given data as a git object of type git_type.
Args: Args:
data: a bytes object data: a bytes object
git_type: the git object type git_type: the git object type
▲ Show 20 Lines • Show All 77 Lines • Show Last 20 Lines
View Options

proxmox/terraform/kelvingrove.tf

# Copyright (C) 2015-2017 The Software Heritage developers # Copyright (C) 2015-2018 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
"""Module in charge of hashing function definitions. This is the base """Module in charge of hashing function definitions. This is the base
module use to compute swh's hashes. module use to compute swh's hashes.
Only a subset of hashing algorithms is supported as defined in the Only a subset of hashing algorithms is supported as defined in the
Show All 30 Lines
Subset of :const:`ALGORITHMS`. Subset of :const:`ALGORITHMS`.
""" """
HASH_BLOCK_SIZE = 32768 HASH_BLOCK_SIZE = 32768
"""Block size for streaming hash computations made in this module""" """Block size for streaming hash computations made in this module"""
_blake2_hash_cache = {} _blake2_hash_cache = {}
HASH_FORMATS = set(['bytes', 'bytehex', 'hex'])
"""Supported output hash formats
"""
class MultiHash:
"""Hashutil class to support multiple hashes computation.
Args:
hash_names (set): Set of hash algorithms (+ length) to compute
hashes (cf. DEFAULT_ALGORITHMS)
length (int): Length of the total sum of chunks to read
If the length is provided as algorithm, the length is also
computed and returned.
"""
def __init__(self, hash_names=DEFAULT_ALGORITHMS, length=None):
self.state = {}
self.track_length = False
for name in hash_names:
if name == 'length':
self.state['length'] = 0
self.track_length = True
else:
self.state[name] = _new_hash(name, length)
@classmethod
def from_state(cls, state, track_length):
ret = cls([])
ret.state = state
ret.track_length = track_length
def update(self, chunk):
for name, h in self.state.items():
if name == 'length':
continue
h.update(chunk)
if self.track_length:
self.state['length'] += len(chunk)
def digest(self):
return {
name: h.digest() if name != 'length' else h
for name, h in self.state.items()
}
def hexdigest(self):
return {
name: h.hexdigest() if name != 'length' else h
for name, h in self.state.items()
ardumontAuthorUnsubmitted
Not Done
Inline Actions

I don't think this is used much. IIRC, it's only used once.

So, i'm for removing that part later (when we will have migrated the base code).

ardumont: I don't think this is used much. IIRC, it's only used once. So, i'm for removing that part…
olasdUnsubmitted
Done
Inline Actions

In any case, we should just use the length that we've retrieved from os.path.getsize rather than computing it again.

olasd: In any case, we should just use the length that we've retrieved from os.path.getsize rather…
ardumontAuthorUnsubmitted
Not Done
Inline Actions

Right, fixing that as well.

ardumont: Right, fixing that as well.
}
def bytehexdigest(self):
return {
name: hash_to_bytehex(h.digest()) if name != 'length' else h
for name, h in self.state.items()
}
def copy(self):
copied_state = {
name: h.copy() if name != 'length' else h
for name, h in self.state.items()
}
return self.from_state(copied_state, self.track_length)
def _new_blake2_hash(algo): def _new_blake2_hash(algo):
"""Return a function that initializes a blake2 hash. """Return a function that initializes a blake2 hash.
""" """
if algo in _blake2_hash_cache: if algo in _blake2_hash_cache:
return _blake2_hash_cache[algo]() return _blake2_hash_cache[algo]()
▲ Show 20 Lines • Show All 101 Lines • ▼ Show 20 Lines if algo.endswith('_git'):
if length is None: if length is None:
raise ValueError('Missing length for git hashing algorithm') raise ValueError('Missing length for git hashing algorithm')
base_algo = algo[:-4] base_algo = algo[:-4]
return _new_git_hash(base_algo, 'blob', length) return _new_git_hash(base_algo, 'blob', length)
return _new_hashlib_hash(algo) return _new_hashlib_hash(algo)
def hash_file(fobj, length=None, algorithms=DEFAULT_ALGORITHMS, chunk_cb=None): def hash_file(fobj, length=None, algorithms=DEFAULT_ALGORITHMS,
chunk_cb=None, hash_format='bytes'):
"""Hash the contents of the given file object with the given algorithms. """Hash the contents of the given file object with the given algorithms.
Args: Args:
fobj: a file-like object fobj: a file-like object
length: the length of the contents of the file-like object (for the length: the length of the contents of the file-like object (for the
git-specific algorithms) git-specific algorithms)
algorithms: the hashing algorithms to be used, as an iterable over algorithms: the hashing algorithms to be used, as an iterable over
strings strings
hash_format (str): Format required for the output of the
computed hashes (cf. HASH_FORMATS)
Returns: a dict mapping each algorithm to a bytes digest. Returns: a dict mapping each algorithm to a digest (bytes by default).
Raises: Raises:
ValueError if algorithms contains an unknown hash algorithm. ValueError if:
algorithms contains an unknown hash algorithm.
hash_format is an unknown hash format
""" """
hashes = {algo: _new_hash(algo, length) for algo in algorithms} if hash_format not in HASH_FORMATS:
raise ValueError('Unexpected hash format %s, expected one of %s' % (
hash_format, HASH_FORMATS))
h = MultiHash(algorithms, length)
while True: while True:
chunk = fobj.read(HASH_BLOCK_SIZE) chunk = fobj.read(HASH_BLOCK_SIZE)
if not chunk: if not chunk:
break break
for hash in hashes.values(): h.update(chunk)
hash.update(chunk)
if chunk_cb: if chunk_cb:
chunk_cb(chunk) chunk_cb(chunk)
return {algo: hash.digest() for algo, hash in hashes.items()} if hash_format == 'bytes':
return h.digest()
if hash_format == 'bytehex':
return h.bytehexdigest()
return h.hexdigest()
def hash_path(path, algorithms=DEFAULT_ALGORITHMS, chunk_cb=None): def hash_path(path, algorithms=DEFAULT_ALGORITHMS, chunk_cb=None,
hash_format='bytes', track_length=True):
"""Hash the contents of the file at the given path with the given """Hash the contents of the file at the given path with the given
algorithms. algorithms.
Args: Args:
path: the path of the file to hash path (str): the path of the file to hash
algorithms: the hashing algorithms used algorithms (set): the hashing algorithms used
chunk_cb: a callback chunk_cb (def): a callback
hash_format (str): Format required for the output of the
computed hashes (cf. HASH_FORMATS)
Returns: a dict mapping each algorithm to a bytes digest. Returns: a dict mapping each algorithm to a bytes digest.
Raises: Raises:
ValueError if algorithms contains an unknown hash algorithm. ValueError if:
algorithms contains an unknown hash algorithm.
hash_format is an unknown hash format
OSError on file access error OSError on file access error
""" """
if track_length:
algorithms = set(['length']).union(algorithms)
length = os.path.getsize(path) length = os.path.getsize(path)
with open(path, 'rb') as fobj: with open(path, 'rb') as fobj:
hash = hash_file(fobj, length, algorithms, chunk_cb) return hash_file(fobj, length, algorithms, chunk_cb=chunk_cb,
hash['length'] = length hash_format=hash_format)
return hash
def hash_data(data, algorithms=DEFAULT_ALGORITHMS, with_length=False): def hash_data(data, algorithms=DEFAULT_ALGORITHMS, hash_format='bytes'):
"""Hash the given binary blob with the given algorithms. """Hash the given binary blob with the given algorithms.
Args: Args:
data (bytes): raw content to hash data (bytes): raw content to hash
algorithms (list): the hashing algorithms used algorithms (list): the hashing algorithms used
with_length (bool): add the length key in the resulting dict hash_format (str): Format required for the output of the
computed hashes (cf. HASH_FORMATS)
Returns: a dict mapping each algorithm to a bytes digest Returns: a dict mapping each algorithm to a bytes digest
Raises: Raises:
TypeError if data does not support the buffer interface. TypeError if data does not support the buffer interface.
ValueError if algorithms contains an unknown hash algorithm. ValueError if:
algorithms contains an unknown hash algorithm.
hash_format is an unknown hash format
""" """
fobj = BytesIO(data) fobj = BytesIO(data)
length = len(data) length = len(data)
data = hash_file(fobj, length, algorithms) return hash_file(fobj, length, algorithms, hash_format=hash_format)
if with_length:
data['length'] = length
return data
def hash_git_data(data, git_type, base_algo='sha1'): def hash_git_data(data, git_type, base_algo='sha1'):
"""Hash the given data as a git object of type git_type. """Hash the given data as a git object of type git_type.
Args: Args:
data: a bytes object data: a bytes object
git_type: the git object type git_type: the git object type
▲ Show 20 Lines • Show All 77 Lines • Show Last 20 Lines
View Options

proxmox/terraform/zookeeper.tf

# Copyright (C) 2015-2017 The Software Heritage developers # Copyright (C) 2015-2018 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution # See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version # License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information # See top-level LICENSE file for more information
"""Module in charge of hashing function definitions. This is the base """Module in charge of hashing function definitions. This is the base
module use to compute swh's hashes. module use to compute swh's hashes.
Only a subset of hashing algorithms is supported as defined in the Only a subset of hashing algorithms is supported as defined in the
Show All 30 Lines
Subset of :const:`ALGORITHMS`. Subset of :const:`ALGORITHMS`.
""" """
HASH_BLOCK_SIZE = 32768 HASH_BLOCK_SIZE = 32768
"""Block size for streaming hash computations made in this module""" """Block size for streaming hash computations made in this module"""
_blake2_hash_cache = {} _blake2_hash_cache = {}
HASH_FORMATS = set(['bytes', 'bytehex', 'hex'])
"""Supported output hash formats
"""
class MultiHash:
"""Hashutil class to support multiple hashes computation.
Args:
hash_names (set): Set of hash algorithms (+ length) to compute
hashes (cf. DEFAULT_ALGORITHMS)
length (int): Length of the total sum of chunks to read
If the length is provided as algorithm, the length is also
computed and returned.
"""
def __init__(self, hash_names=DEFAULT_ALGORITHMS, length=None):
self.state = {}
self.track_length = False
for name in hash_names:
if name == 'length':
self.state['length'] = 0
self.track_length = True
else:
self.state[name] = _new_hash(name, length)
@classmethod
def from_state(cls, state, track_length):
ret = cls([])
ret.state = state
ret.track_length = track_length
def update(self, chunk):
for name, h in self.state.items():
if name == 'length':
continue
h.update(chunk)
if self.track_length:
self.state['length'] += len(chunk)
def digest(self):
return {
name: h.digest() if name != 'length' else h
for name, h in self.state.items()
}
def hexdigest(self):
return {
name: h.hexdigest() if name != 'length' else h
for name, h in self.state.items()
ardumontAuthorUnsubmitted
Not Done
Inline Actions

I don't think this is used much. IIRC, it's only used once.

So, i'm for removing that part later (when we will have migrated the base code).

ardumont: I don't think this is used much. IIRC, it's only used once. So, i'm for removing that part…
olasdUnsubmitted
Done
Inline Actions

In any case, we should just use the length that we've retrieved from os.path.getsize rather than computing it again.

olasd: In any case, we should just use the length that we've retrieved from os.path.getsize rather…
ardumontAuthorUnsubmitted
Not Done
Inline Actions

Right, fixing that as well.

ardumont: Right, fixing that as well.
}
def bytehexdigest(self):
return {
name: hash_to_bytehex(h.digest()) if name != 'length' else h
for name, h in self.state.items()
}
def copy(self):
copied_state = {
name: h.copy() if name != 'length' else h
for name, h in self.state.items()
}
return self.from_state(copied_state, self.track_length)
def _new_blake2_hash(algo): def _new_blake2_hash(algo):
"""Return a function that initializes a blake2 hash. """Return a function that initializes a blake2 hash.
""" """
if algo in _blake2_hash_cache: if algo in _blake2_hash_cache:
return _blake2_hash_cache[algo]() return _blake2_hash_cache[algo]()
▲ Show 20 Lines • Show All 101 Lines • ▼ Show 20 Lines if algo.endswith('_git'):
if length is None: if length is None:
raise ValueError('Missing length for git hashing algorithm') raise ValueError('Missing length for git hashing algorithm')
base_algo = algo[:-4] base_algo = algo[:-4]
return _new_git_hash(base_algo, 'blob', length) return _new_git_hash(base_algo, 'blob', length)
return _new_hashlib_hash(algo) return _new_hashlib_hash(algo)
def hash_file(fobj, length=None, algorithms=DEFAULT_ALGORITHMS, chunk_cb=None): def hash_file(fobj, length=None, algorithms=DEFAULT_ALGORITHMS,
chunk_cb=None, hash_format='bytes'):
"""Hash the contents of the given file object with the given algorithms. """Hash the contents of the given file object with the given algorithms.
Args: Args:
fobj: a file-like object fobj: a file-like object
length: the length of the contents of the file-like object (for the length: the length of the contents of the file-like object (for the
git-specific algorithms) git-specific algorithms)
algorithms: the hashing algorithms to be used, as an iterable over algorithms: the hashing algorithms to be used, as an iterable over
strings strings
hash_format (str): Format required for the output of the
computed hashes (cf. HASH_FORMATS)
Returns: a dict mapping each algorithm to a bytes digest. Returns: a dict mapping each algorithm to a digest (bytes by default).
Raises: Raises:
ValueError if algorithms contains an unknown hash algorithm. ValueError if:
algorithms contains an unknown hash algorithm.
hash_format is an unknown hash format
""" """
hashes = {algo: _new_hash(algo, length) for algo in algorithms} if hash_format not in HASH_FORMATS:
raise ValueError('Unexpected hash format %s, expected one of %s' % (
hash_format, HASH_FORMATS))
h = MultiHash(algorithms, length)
while True: while True:
chunk = fobj.read(HASH_BLOCK_SIZE) chunk = fobj.read(HASH_BLOCK_SIZE)
if not chunk: if not chunk:
break break
for hash in hashes.values(): h.update(chunk)
hash.update(chunk)
if chunk_cb: if chunk_cb:
chunk_cb(chunk) chunk_cb(chunk)
return {algo: hash.digest() for algo, hash in hashes.items()} if hash_format == 'bytes':
return h.digest()
if hash_format == 'bytehex':
return h.bytehexdigest()
return h.hexdigest()
def hash_path(path, algorithms=DEFAULT_ALGORITHMS, chunk_cb=None): def hash_path(path, algorithms=DEFAULT_ALGORITHMS, chunk_cb=None,
hash_format='bytes', track_length=True):
"""Hash the contents of the file at the given path with the given """Hash the contents of the file at the given path with the given
algorithms. algorithms.
Args: Args:
path: the path of the file to hash path (str): the path of the file to hash
algorithms: the hashing algorithms used algorithms (set): the hashing algorithms used
chunk_cb: a callback chunk_cb (def): a callback
hash_format (str): Format required for the output of the
computed hashes (cf. HASH_FORMATS)
Returns: a dict mapping each algorithm to a bytes digest. Returns: a dict mapping each algorithm to a bytes digest.
Raises: Raises:
ValueError if algorithms contains an unknown hash algorithm. ValueError if:
algorithms contains an unknown hash algorithm.
hash_format is an unknown hash format
OSError on file access error OSError on file access error
""" """
if track_length:
algorithms = set(['length']).union(algorithms)
length = os.path.getsize(path) length = os.path.getsize(path)
with open(path, 'rb') as fobj: with open(path, 'rb') as fobj:
hash = hash_file(fobj, length, algorithms, chunk_cb) return hash_file(fobj, length, algorithms, chunk_cb=chunk_cb,
hash['length'] = length hash_format=hash_format)
return hash
def hash_data(data, algorithms=DEFAULT_ALGORITHMS, with_length=False): def hash_data(data, algorithms=DEFAULT_ALGORITHMS, hash_format='bytes'):
"""Hash the given binary blob with the given algorithms. """Hash the given binary blob with the given algorithms.
Args: Args:
data (bytes): raw content to hash data (bytes): raw content to hash
algorithms (list): the hashing algorithms used algorithms (list): the hashing algorithms used
with_length (bool): add the length key in the resulting dict hash_format (str): Format required for the output of the
computed hashes (cf. HASH_FORMATS)
Returns: a dict mapping each algorithm to a bytes digest Returns: a dict mapping each algorithm to a bytes digest
Raises: Raises:
TypeError if data does not support the buffer interface. TypeError if data does not support the buffer interface.
ValueError if algorithms contains an unknown hash algorithm. ValueError if:
algorithms contains an unknown hash algorithm.
hash_format is an unknown hash format
""" """
fobj = BytesIO(data) fobj = BytesIO(data)
length = len(data) length = len(data)
data = hash_file(fobj, length, algorithms) return hash_file(fobj, length, algorithms, hash_format=hash_format)
if with_length:
data['length'] = length
return data
def hash_git_data(data, git_type, base_algo='sha1'): def hash_git_data(data, git_type, base_algo='sha1'):
"""Hash the given data as a git object of type git_type. """Hash the given data as a git object of type git_type.
Args: Args:
data: a bytes object data: a bytes object
git_type: the git object type git_type: the git object type
▲ Show 20 Lines • Show All 77 Lines • Show Last 20 Lines
About · Developer information · Legal