Page MenuHomeSoftware Heritage
Differential D700

Add a Debian scripts directory as well as a job to keep them updated
ClosedPublic
Actions

Authored by olasd on Nov 22 2018, 3:53 PM.

Details

Reviewers
douardda
Commits
rCJSWH37649ff67507: Add a Debian scripts directory as well as a job to keep them updated
Summary

Depends on D699

Test Plan

tox again

Diff Detail

Repository
rCJSWH Jenkins jobs
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 2538
Build 3150: arc lint + arc unit

Event Timeline

olasd created this revision.Nov 22 2018, 3:53 PM
Harbormaster completed remote builds in B2534: Diff 2195.Nov 22 2018, 3:53 PM
olasd added a child revision: D701: Add a job to create debian chroots.Nov 22 2018, 3:54 PM
douardda added a comment.Nov 22 2018, 4:09 PM

Why a matrix here?

olasd added a comment.Nov 22 2018, 4:13 PM
In D700#14590, @douardda wrote:

Why a matrix here?

Because it needs to run on all slaves. That started off as a checkout of the swh-jenkins-jobs repo that would be available on master and jenkins-debian01, then got refined down to a debian-slave-specific job.

douardda added a comment.Nov 22 2018, 4:15 PM

might be useful to replace this .keep file by a README reminding the reader where these scripts come from

douardda accepted this revision.Nov 22 2018, 4:23 PM
This revision is now accepted and ready to land.Nov 22 2018, 4:23 PM
olasd updated this revision to Diff 2199.Nov 22 2018, 4:26 PM

Move .keep to README.md

Harbormaster completed remote builds in B2538: Diff 2199.Nov 22 2018, 4:26 PM
Closed by commit rCJSWH37649ff67507: Add a Debian scripts directory as well as a job to keep them updated (authored by olasd). · Explain WhyNov 22 2018, 4:26 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
jobs/
debian/
5 lines
18 lines
scripts/
debian/
7 lines

Diff 2199

View Options

jobs/debian/axis-slave.yml.inc

Show First 20 Lines • Show All 51 Lines • ▼ Show 20 Linesdef _write_obj_file(hex_obj_id, objstorage):
(tmp, tmp_path) = tempfile.mkstemp(suffix='.tmp', prefix='hex_obj_id.', (tmp, tmp_path) = tempfile.mkstemp(suffix='.tmp', prefix='hex_obj_id.',
dir=dir) dir=dir)
# Open the file and yield it for writing. # Open the file and yield it for writing.
tmp_f = os.fdopen(tmp, 'wb') tmp_f = os.fdopen(tmp, 'wb')
with gzip.GzipFile(filename=tmp_path, fileobj=tmp_f) as f: with gzip.GzipFile(filename=tmp_path, fileobj=tmp_f) as f:
yield f yield f
# Make sure the contents of the temporary file are written to disk
tmp_f.flush()
if objstorage.use_fdatasync:
os.fdatasync(tmp)
else:
os.fsync(tmp)
# Then close the temporary file and move it to the right directory. # Then close the temporary file and move it to the right directory.
tmp_f.close() tmp_f.close()
os.chmod(tmp_path, FILE_MODE) os.chmod(tmp_path, FILE_MODE)
os.rename(tmp_path, path) os.rename(tmp_path, path)
@contextmanager @contextmanager
def _read_obj_file(hex_obj_id, objstorage): def _read_obj_file(hex_obj_id, objstorage):
▲ Show 20 Lines • Show All 52 Lines • ▼ Show 20 Lines def __init__(self, root, slicing, **kwargs):
self.root = root self.root = root
# Make a list of tuples where each tuple contains the beginning # Make a list of tuples where each tuple contains the beginning
# and the end of each slicing. # and the end of each slicing.
self.bounds = [ self.bounds = [
slice(*map(int, sbounds.split(':'))) slice(*map(int, sbounds.split(':')))
for sbounds in slicing.split('/') for sbounds in slicing.split('/')
if sbounds if sbounds
] ]
zackUnsubmitted
Not Done
Inline Actions

Is this because some python versions don't have it, or just for OS portability?

I'm asking because we now have evidence (in T1817) that fdatasync is critical for us, in the sense that we can lose data if it's not available. So just silently ignoring the fact it's not available makes me shiver :-)

I'm tempted to suggest failing to initialize if fdatasync is not available.

Either way, we should log an error message if it's not available.

zack: Is this because some python versions don't have it, or just for OS portability? I'm asking…
olasdAuthorUnsubmitted
Done
Inline Actions

The new version falls back to fsync (available on all platforms) if fdatasync isn't available.

olasd: The new version falls back to fsync (available on all platforms) if fdatasync isn't available.
zackUnsubmitted
Not Done
Inline Actions

*thumb up*

zack: *thumb up*
self.use_fdatasync = hasattr(os, 'fdatasync')
self.check_config(check_write=False) self.check_config(check_write=False)
def check_config(self, *, check_write): def check_config(self, *, check_write):
"""Check whether this object storage is properly configured""" """Check whether this object storage is properly configured"""
root = self.root root = self.root
if not os.path.isdir(root): if not os.path.isdir(root):
▲ Show 20 Lines • Show All 238 Lines • Show Last 20 Lines
View Options

jobs/debian/update-scripts.yml

Show First 20 Lines • Show All 51 Lines • ▼ Show 20 Linesdef _write_obj_file(hex_obj_id, objstorage):
(tmp, tmp_path) = tempfile.mkstemp(suffix='.tmp', prefix='hex_obj_id.', (tmp, tmp_path) = tempfile.mkstemp(suffix='.tmp', prefix='hex_obj_id.',
dir=dir) dir=dir)
# Open the file and yield it for writing. # Open the file and yield it for writing.
tmp_f = os.fdopen(tmp, 'wb') tmp_f = os.fdopen(tmp, 'wb')
with gzip.GzipFile(filename=tmp_path, fileobj=tmp_f) as f: with gzip.GzipFile(filename=tmp_path, fileobj=tmp_f) as f:
yield f yield f
# Make sure the contents of the temporary file are written to disk
tmp_f.flush()
if objstorage.use_fdatasync:
os.fdatasync(tmp)
else:
os.fsync(tmp)
# Then close the temporary file and move it to the right directory. # Then close the temporary file and move it to the right directory.
tmp_f.close() tmp_f.close()
os.chmod(tmp_path, FILE_MODE) os.chmod(tmp_path, FILE_MODE)
os.rename(tmp_path, path) os.rename(tmp_path, path)
@contextmanager @contextmanager
def _read_obj_file(hex_obj_id, objstorage): def _read_obj_file(hex_obj_id, objstorage):
▲ Show 20 Lines • Show All 52 Lines • ▼ Show 20 Lines def __init__(self, root, slicing, **kwargs):
self.root = root self.root = root
# Make a list of tuples where each tuple contains the beginning # Make a list of tuples where each tuple contains the beginning
# and the end of each slicing. # and the end of each slicing.
self.bounds = [ self.bounds = [
slice(*map(int, sbounds.split(':'))) slice(*map(int, sbounds.split(':')))
for sbounds in slicing.split('/') for sbounds in slicing.split('/')
if sbounds if sbounds
] ]
zackUnsubmitted
Not Done
Inline Actions

Is this because some python versions don't have it, or just for OS portability?

I'm asking because we now have evidence (in T1817) that fdatasync is critical for us, in the sense that we can lose data if it's not available. So just silently ignoring the fact it's not available makes me shiver :-)

I'm tempted to suggest failing to initialize if fdatasync is not available.

Either way, we should log an error message if it's not available.

zack: Is this because some python versions don't have it, or just for OS portability? I'm asking…
olasdAuthorUnsubmitted
Done
Inline Actions

The new version falls back to fsync (available on all platforms) if fdatasync isn't available.

olasd: The new version falls back to fsync (available on all platforms) if fdatasync isn't available.
zackUnsubmitted
Not Done
Inline Actions

*thumb up*

zack: *thumb up*
self.use_fdatasync = hasattr(os, 'fdatasync')
self.check_config(check_write=False) self.check_config(check_write=False)
def check_config(self, *, check_write): def check_config(self, *, check_write):
"""Check whether this object storage is properly configured""" """Check whether this object storage is properly configured"""
root = self.root root = self.root
if not os.path.isdir(root): if not os.path.isdir(root):
▲ Show 20 Lines • Show All 238 Lines • Show Last 20 Lines
View Options

scripts/debian/README.md

Show First 20 Lines • Show All 51 Lines • ▼ Show 20 Linesdef _write_obj_file(hex_obj_id, objstorage):
(tmp, tmp_path) = tempfile.mkstemp(suffix='.tmp', prefix='hex_obj_id.', (tmp, tmp_path) = tempfile.mkstemp(suffix='.tmp', prefix='hex_obj_id.',
dir=dir) dir=dir)
# Open the file and yield it for writing. # Open the file and yield it for writing.
tmp_f = os.fdopen(tmp, 'wb') tmp_f = os.fdopen(tmp, 'wb')
with gzip.GzipFile(filename=tmp_path, fileobj=tmp_f) as f: with gzip.GzipFile(filename=tmp_path, fileobj=tmp_f) as f:
yield f yield f
# Make sure the contents of the temporary file are written to disk
tmp_f.flush()
if objstorage.use_fdatasync:
os.fdatasync(tmp)
else:
os.fsync(tmp)
# Then close the temporary file and move it to the right directory. # Then close the temporary file and move it to the right directory.
tmp_f.close() tmp_f.close()
os.chmod(tmp_path, FILE_MODE) os.chmod(tmp_path, FILE_MODE)
os.rename(tmp_path, path) os.rename(tmp_path, path)
@contextmanager @contextmanager
def _read_obj_file(hex_obj_id, objstorage): def _read_obj_file(hex_obj_id, objstorage):
▲ Show 20 Lines • Show All 52 Lines • ▼ Show 20 Lines def __init__(self, root, slicing, **kwargs):
self.root = root self.root = root
# Make a list of tuples where each tuple contains the beginning # Make a list of tuples where each tuple contains the beginning
# and the end of each slicing. # and the end of each slicing.
self.bounds = [ self.bounds = [
slice(*map(int, sbounds.split(':'))) slice(*map(int, sbounds.split(':')))
for sbounds in slicing.split('/') for sbounds in slicing.split('/')
if sbounds if sbounds
] ]
zackUnsubmitted
Not Done
Inline Actions

Is this because some python versions don't have it, or just for OS portability?

I'm asking because we now have evidence (in T1817) that fdatasync is critical for us, in the sense that we can lose data if it's not available. So just silently ignoring the fact it's not available makes me shiver :-)

I'm tempted to suggest failing to initialize if fdatasync is not available.

Either way, we should log an error message if it's not available.

zack: Is this because some python versions don't have it, or just for OS portability? I'm asking…
olasdAuthorUnsubmitted
Done
Inline Actions

The new version falls back to fsync (available on all platforms) if fdatasync isn't available.

olasd: The new version falls back to fsync (available on all platforms) if fdatasync isn't available.
zackUnsubmitted
Not Done
Inline Actions

*thumb up*

zack: *thumb up*
self.use_fdatasync = hasattr(os, 'fdatasync')
self.check_config(check_write=False) self.check_config(check_write=False)
def check_config(self, *, check_write): def check_config(self, *, check_write):
"""Check whether this object storage is properly configured""" """Check whether this object storage is properly configured"""
root = self.root root = self.root
if not os.path.isdir(root): if not os.path.isdir(root):
▲ Show 20 Lines • Show All 238 Lines • Show Last 20 Lines
About · Developer information · Legal