Paths

Table of Contentst

2/2
.pre-commit-config.yaml
2/2
tox.ini

Differential D8732

pre-commit, tox: Bump pre-commit, codespell, black and flake8
ClosedPublic
Actions

Authored by douardda on Oct 18 2022, 7:18 PM.

Details

Reviewers

None

Group Reviewers

Reviewers

Commits

rDSCRUBafacc488cd3d: pre-commit, tox: Bump pre-commit, codespell, black and flake8

Summary

pre-commit from 4.1.0 to 4.3.0,
codespell from 2.2.1 to 2.2.2,
black from 22.3.0 to 22.10.0 and
flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

Diff Detail

Repository

rDSCRUB Datastore Scrubber

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Oct 18 2022, 7:18 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptOct 18 2022, 7:18 PM

Build is green

Patch application report for D8732 (id=31456)

Rebasing onto 2822428737...

First, rewinding head to replay your work on top of it...
Applying: pre-commit, tox: Bump pre-commit, codespell, black and flake8
Using index info to reconstruct a base tree...
M	.pre-commit-config.yaml
Falling back to patching base and 3-way merge...
Auto-merging .pre-commit-config.yaml

Changes applied before test

commit d133fa74b54e80cb7d72c187981a2470a16bd3fa
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Oct 18 18:21:01 2022 +0200

    pre-commit, tox: Bump pre-commit, codespell, black and flake8
    
    - pre-commit from 4.1.0 to 4.3.0,
    - codespell from 2.2.1 to 2.2.2,
    - black from 22.3.0 to 22.10.0 and
    - flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.
    
    Also change flake8's repo config to github (the gitlab mirror
    being outdated).

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/72/ for more details.

Harbormaster completed remote builds in B32402: Diff 31456.Oct 18 2022, 7:35 PM

douardda requested review of this revision.Oct 18 2022, 7:35 PM

rebase

This revision was not accepted when it landed; it landed in state Needs Review.Oct 18 2022, 7:58 PM

This revision was landed with ongoing or failed builds.

Closed by commit rDSCRUBafacc488cd3d: pre-commit, tox: Bump pre-commit, codespell, black and flake8 (authored by douardda). · Explain Why

This revision was automatically updated to reflect the committed changes.

douardda added a commit: rDSCRUBafacc488cd3d: pre-commit, tox: Bump pre-commit, codespell, black and flake8.

Build is green

Patch application report for D8732 (id=31492)

Rebasing onto afacc488cd...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-73-D8732.

Changes applied before test

See https://jenkins.softwareheritage.org/job/DSCRUB/job/tests-on-diff/73/ for more details.

Harbormaster completed remote builds in B32437: Diff 31492.Oct 18 2022, 8:11 PM

Revision Contents
Changeset List

Path

Size

.pre-commit-config.yaml

12 lines

tox.ini

7 lines

Diff 31493

View Options

.pre-commit-config.yaml

Show All 39 Lines	def convert(self, value, param, ctx):
self.fail('%s is not a valid PID. %s.' % (value, e), param, ctx)		self.fail('%s is not a valid PID. %s.' % (value, e), param, ctx)


def pid_of_file(path):		def pid_of_file(path):
object = Content.from_file(path=path).get_data()		object = Content.from_file(path=path).get_data()
return pids.persistent_identifier(pids.CONTENT, object)		return pids.persistent_identifier(pids.CONTENT, object)


		def pid_of_file_content(data):
		object = Content.from_bytes(mode=644, data=data).get_data()
		return pids.persistent_identifier(pids.CONTENT, object)


def pid_of_dir(path):		def pid_of_dir(path):
object = Directory.from_disk(path=path).get_data()		object = Directory.from_disk(path=path).get_data()
return pids.persistent_identifier(pids.DIRECTORY, object)		return pids.persistent_identifier(pids.DIRECTORY, object)


def pid_of_origin(url):		def pid_of_origin(url):
pid = pids.PersistentId(object_type='origin',		pid = pids.PersistentId(object_type='origin',
object_id=pids.origin_identifier({'url': url}))		object_id=pids.origin_identifier({'url': url}))
Show All 24 Lines	def pid_of_git_repo(path):

pid = pids.PersistentId(object_type='snapshot',		pid = pids.PersistentId(object_type='snapshot',
object_id=pids.snapshot_identifier(snapshot))		object_id=pids.snapshot_identifier(snapshot))
return str(pid)		return str(pid)


def identify_object(obj_type, follow_symlinks, obj):		def identify_object(obj_type, follow_symlinks, obj):
if obj_type == 'auto':		if obj_type == 'auto':
if os.path.isfile(obj):		if obj == '-' or os.path.isfile(obj):
obj_type = 'content'		obj_type = 'content'
elif os.path.isdir(obj):		elif os.path.isdir(obj):
obj_type = 'directory'		obj_type = 'directory'
else:		else:
try: # URL parsing		try: # URL parsing
if urlparse(obj).scheme:		if urlparse(obj).scheme:
obj_type = 'origin'		obj_type = 'origin'
else:		else:
raise ValueError		raise ValueError
except ValueError:		except ValueError:
raise click.BadParameter('cannot detect object type for %s' %		raise click.BadParameter('cannot detect object type for %s' %
obj)		obj)

pid = None		pid = None

if obj_type in ['content', 'directory']:		if obj == '-':
		content = sys.stdin.buffer.read()
		pid = pid_of_file_content(content)
		elif obj_type in ['content', 'directory']:
path = obj.encode(sys.getfilesystemencoding())		path = obj.encode(sys.getfilesystemencoding())
if follow_symlinks and os.path.islink(obj):		if follow_symlinks and os.path.islink(obj):
path = os.path.realpath(obj)		path = os.path.realpath(obj)
if obj_type == 'content':		if obj_type == 'content':
pid = pid_of_file(path)		pid = pid_of_file(path)
elif obj_type == 'directory':		elif obj_type == 'directory':
pid = pid_of_dir(path)		pid = pid_of_dir(path)
elif obj_type == 'origin':		elif obj_type == 'origin':
Show All 16 Lines
@click.option('--filename/--no-filename', 'show_filename', default=True,		@click.option('--filename/--no-filename', 'show_filename', default=True,
help='show/hide file name (default: show)')		help='show/hide file name (default: show)')
@click.option('--type', '-t', 'obj_type', default='auto',		@click.option('--type', '-t', 'obj_type', default='auto',
type=click.Choice(['auto', 'content', 'directory', 'origin',		type=click.Choice(['auto', 'content', 'directory', 'origin',
'snapshot']),		'snapshot']),
help='type of object to identify (default: auto)')		help='type of object to identify (default: auto)')
@click.option('--verify', '-v', metavar='PID', type=PidParamType(),		@click.option('--verify', '-v', metavar='PID', type=PidParamType(),
help='reference identifier to be compared with computed one')		help='reference identifier to be compared with computed one')
@click.argument('objects', nargs=-1, required=True)		@click.argument('objects', nargs=-1)
def identify(obj_type, verify, show_filename, follow_symlinks, objects):		def identify(obj_type, verify, show_filename, follow_symlinks, objects):
"""Compute the Software Heritage persistent identifier (PID) for the given		"""Compute the Software Heritage persistent identifier (PID) for the given
source code object(s).		source code object(s).

For more details about Software Heritage PIDs see:		For more details about Software Heritage PIDs see:

\b		\b
https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html		https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html
Show All 12 Lines	\b
swh:1:dir:f9f858a48d663b3809c9e2f336412717496202ab		swh:1:dir:f9f858a48d663b3809c9e2f336412717496202ab

\b		\b
$ git clone --mirror https://forge.softwareheritage.org/source/helloworld.git		$ git clone --mirror https://forge.softwareheritage.org/source/helloworld.git
$ swh identify --type snapshot helloworld.git/		$ swh identify --type snapshot helloworld.git/
swh:1:snp:510aa88bdc517345d258c1fc2babcd0e1f905e93 helloworld.git		swh:1:snp:510aa88bdc517345d258c1fc2babcd0e1f905e93 helloworld.git

""" # NoQA # overlong lines in shell examples are fine		""" # NoQA # overlong lines in shell examples are fine
		if not objects:
		vlorentzUnsubmitted Done Inline Actions `objects = ['-']` vlorentz: `objects = ['-']`
		douarddaAuthorUnsubmitted Done Inline Actions It works also with a string but I guess you are right :-) douardda: It works also with a string but I guess you are right :-)
		objects = ['-']

if verify and len(objects) != 1:		if verify and len(objects) != 1:
raise click.BadParameter('verification requires a single object')		raise click.BadParameter('verification requires a single object')

results = map(partial(identify_object, obj_type, follow_symlinks), objects)		results = map(partial(identify_object, obj_type, follow_symlinks), objects)

if verify:		if verify:
pid = next(results)[1]		pid = next(results)[1]
if verify == pid:		if verify == pid:
Show All 15 Lines

View Options

tox.ini

Show All 39 Lines	def convert(self, value, param, ctx):
self.fail('%s is not a valid PID. %s.' % (value, e), param, ctx)		self.fail('%s is not a valid PID. %s.' % (value, e), param, ctx)


def pid_of_file(path):		def pid_of_file(path):
object = Content.from_file(path=path).get_data()		object = Content.from_file(path=path).get_data()
return pids.persistent_identifier(pids.CONTENT, object)		return pids.persistent_identifier(pids.CONTENT, object)


		def pid_of_file_content(data):
		object = Content.from_bytes(mode=644, data=data).get_data()
		return pids.persistent_identifier(pids.CONTENT, object)


def pid_of_dir(path):		def pid_of_dir(path):
object = Directory.from_disk(path=path).get_data()		object = Directory.from_disk(path=path).get_data()
return pids.persistent_identifier(pids.DIRECTORY, object)		return pids.persistent_identifier(pids.DIRECTORY, object)


def pid_of_origin(url):		def pid_of_origin(url):
pid = pids.PersistentId(object_type='origin',		pid = pids.PersistentId(object_type='origin',
object_id=pids.origin_identifier({'url': url}))		object_id=pids.origin_identifier({'url': url}))
Show All 24 Lines	def pid_of_git_repo(path):

pid = pids.PersistentId(object_type='snapshot',		pid = pids.PersistentId(object_type='snapshot',
object_id=pids.snapshot_identifier(snapshot))		object_id=pids.snapshot_identifier(snapshot))
return str(pid)		return str(pid)


def identify_object(obj_type, follow_symlinks, obj):		def identify_object(obj_type, follow_symlinks, obj):
if obj_type == 'auto':		if obj_type == 'auto':
if os.path.isfile(obj):		if obj == '-' or os.path.isfile(obj):
obj_type = 'content'		obj_type = 'content'
elif os.path.isdir(obj):		elif os.path.isdir(obj):
obj_type = 'directory'		obj_type = 'directory'
else:		else:
try: # URL parsing		try: # URL parsing
if urlparse(obj).scheme:		if urlparse(obj).scheme:
obj_type = 'origin'		obj_type = 'origin'
else:		else:
raise ValueError		raise ValueError
except ValueError:		except ValueError:
raise click.BadParameter('cannot detect object type for %s' %		raise click.BadParameter('cannot detect object type for %s' %
obj)		obj)

pid = None		pid = None

if obj_type in ['content', 'directory']:		if obj == '-':
		content = sys.stdin.buffer.read()
		pid = pid_of_file_content(content)
		elif obj_type in ['content', 'directory']:
path = obj.encode(sys.getfilesystemencoding())		path = obj.encode(sys.getfilesystemencoding())
if follow_symlinks and os.path.islink(obj):		if follow_symlinks and os.path.islink(obj):
path = os.path.realpath(obj)		path = os.path.realpath(obj)
if obj_type == 'content':		if obj_type == 'content':
pid = pid_of_file(path)		pid = pid_of_file(path)
elif obj_type == 'directory':		elif obj_type == 'directory':
pid = pid_of_dir(path)		pid = pid_of_dir(path)
elif obj_type == 'origin':		elif obj_type == 'origin':
Show All 16 Lines
@click.option('--filename/--no-filename', 'show_filename', default=True,		@click.option('--filename/--no-filename', 'show_filename', default=True,
help='show/hide file name (default: show)')		help='show/hide file name (default: show)')
@click.option('--type', '-t', 'obj_type', default='auto',		@click.option('--type', '-t', 'obj_type', default='auto',
type=click.Choice(['auto', 'content', 'directory', 'origin',		type=click.Choice(['auto', 'content', 'directory', 'origin',
'snapshot']),		'snapshot']),
help='type of object to identify (default: auto)')		help='type of object to identify (default: auto)')
@click.option('--verify', '-v', metavar='PID', type=PidParamType(),		@click.option('--verify', '-v', metavar='PID', type=PidParamType(),
help='reference identifier to be compared with computed one')		help='reference identifier to be compared with computed one')
@click.argument('objects', nargs=-1, required=True)		@click.argument('objects', nargs=-1)
def identify(obj_type, verify, show_filename, follow_symlinks, objects):		def identify(obj_type, verify, show_filename, follow_symlinks, objects):
"""Compute the Software Heritage persistent identifier (PID) for the given		"""Compute the Software Heritage persistent identifier (PID) for the given
source code object(s).		source code object(s).

For more details about Software Heritage PIDs see:		For more details about Software Heritage PIDs see:

\b		\b
https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html		https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html
Show All 12 Lines	\b
swh:1:dir:f9f858a48d663b3809c9e2f336412717496202ab		swh:1:dir:f9f858a48d663b3809c9e2f336412717496202ab

\b		\b
$ git clone --mirror https://forge.softwareheritage.org/source/helloworld.git		$ git clone --mirror https://forge.softwareheritage.org/source/helloworld.git
$ swh identify --type snapshot helloworld.git/		$ swh identify --type snapshot helloworld.git/
swh:1:snp:510aa88bdc517345d258c1fc2babcd0e1f905e93 helloworld.git		swh:1:snp:510aa88bdc517345d258c1fc2babcd0e1f905e93 helloworld.git

""" # NoQA # overlong lines in shell examples are fine		""" # NoQA # overlong lines in shell examples are fine
		if not objects:
		vlorentzUnsubmitted Done Inline Actions `objects = ['-']` vlorentz: `objects = ['-']`
		douarddaAuthorUnsubmitted Done Inline Actions It works also with a string but I guess you are right :-) douardda: It works also with a string but I guess you are right :-)
		objects = ['-']

if verify and len(objects) != 1:		if verify and len(objects) != 1:
raise click.BadParameter('verification requires a single object')		raise click.BadParameter('verification requires a single object')

results = map(partial(identify_object, obj_type, follow_symlinks), objects)		results = map(partial(identify_object, obj_type, follow_symlinks), objects)

if verify:		if verify:
pid = next(results)[1]		pid = next(results)[1]
if verify == pid:		if verify == pid:
Show All 15 Lines

pre-commit, tox: Bump pre-commit, codespell, black and flake8ClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D8732 (id=31456)

Changes applied before test

Patch application report for D8732 (id=31492)

Changes applied before test

Revision ContentsChangeset List

Diff 31493

.pre-commit-config.yaml

tox.ini

pre-commit, tox: Bump pre-commit, codespell, black and flake8
ClosedPublic
Actions

Revision Contents
Changeset List