Page MenuHomeSoftware Heritage
Differential D79

Update the archiver for the new db shema
ClosedPublic
Actions

Authored by qcampos on Jul 21 2016, 3:36 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDSTOC5c8085d1785b: Update the archiver for the new db shema
rDSTOC9451c5f4b83f: Use postgres 9.5's functions for jsonb manipulation
R65:5c8085d1785b: Update the archiver for the new db shema
R65:9451c5f4b83f: Use postgres 9.5's functions for jsonb manipulation
rDSTO9451c5f4b83f: Use postgres 9.5's functions for jsonb manipulation
rDSTO5c8085d1785b: Update the archiver for the new db shema
Summary

Make the required changes into the db and keep up-to-date the archiver's tests.

Diff Detail

Repository
rDSTO Storage manager
Branch
archiver
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 294
Build 432: Software Heritage Python tests
Build 431: arc lint + arc unit

Event Timeline

qcampos updated this revision to Diff 257.Jul 21 2016, 3:36 PM
qcampos retitled this revision from to Update the archiver for the new db shema.
qcampos updated this object.
qcampos edited the test plan for this revision. (Show Details)
qcampos added a reviewer: ardumont.
qcampos added a subscriber: zack.
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 21 2016, 3:36 PM
qcampos added a comment.Jul 21 2016, 3:39 PM

@ardumont : Submitted a first version for review, but there is still some work to do for the Jsonb access.

swh/storage/archiver/storage.py
74–82

FIXME: check how to alter direclty the json object with postgres for better efficiency.

qcampos updated this revision to Diff 258.Jul 21 2016, 4:06 PM
qcampos edited edge metadata.

Remove string interpolation when constructing the db query

Herald edited edge metadata. · View Herald TranscriptJul 21 2016, 4:06 PM
ardumont added inline comments.Jul 21 2016, 4:10 PM
swh/storage/archiver/storage.py
74–82

directly

For jsonb manipulation btw https://www.postgresql.org/docs/9.5/static/functions-json.html.

swh/storage/archiver/worker.py
130

its mtime

swh/storage/db.py
665

Retrieve

684

'... its mtime...'

690–691

As this is a first draft, i imagine that you'll update the docstring later.
I'll say it nethertheless ^^

ardumont edited edge metadata.Jul 21 2016, 4:12 PM

This seems quite ok so far.

Submitted a first version for review, but there is still some work to do for the Jsonb access.

What do you mean?
The jsonb access for the update?

qcampos updated this revision to Diff 259.EditedJul 21 2016, 4:52 PM
qcampos marked 6 inline comments as done.
qcampos edited edge metadata.
In D79#1796, @ardumont wrote:

The jsonb access for the update?

Yes, I used the jsonb_set function to let postgres do the update.

  • Use postgres 9.5's functions for jsonb manipulation
  • Some typos corrections
Herald edited edge metadata. · View Herald TranscriptJul 21 2016, 4:52 PM
qcampos updated this revision to Diff 260.Jul 21 2016, 5:08 PM
qcampos edited edge metadata.
  • Correct the content_archive_update in order to follow properly its contract
Herald edited edge metadata. · View Herald TranscriptJul 21 2016, 5:08 PM
ardumont added inline comments.Jul 21 2016, 5:21 PM
swh/storage/db.py
695

Indeed, postgres would not let us do that.
What's strange is that it was not captured by test.

qcampos updated this revision to Diff 261.Jul 21 2016, 6:13 PM
qcampos edited edge metadata.
  • Do some corrections on the archiver director
Herald edited edge metadata. · View Herald TranscriptJul 21 2016, 6:13 PM
qcampos marked an inline comment as done.Jul 21 2016, 6:14 PM
qcampos added inline comments.
swh/storage/db.py
695

It should have been captured. If the test was launched. :)

swh/storage/tests/test_archiver.py
144

That was pretty confusing.

ardumont accepted this revision.Jul 21 2016, 6:49 PM
ardumont edited edge metadata.
ardumont added inline comments.
swh/storage/tests/test_archiver.py
144

/o\
but awesome that you saw it ^^

150

you can remove the comment then since you made it in code (which is cooler ^^)

This revision is now accepted and ready to land.Jul 21 2016, 6:49 PM
Closed by commit rDSTO5c8085d1785b: Update the archiver for the new db shema (authored by qcampos). · Explain WhyJul 21 2016, 11:21 PM
This revision was automatically updated to reflect the committed changes.
qcampos marked an inline comment as done.
qcampos marked 3 inline comments as done.Jul 21 2016, 11:21 PM

Revision Contents
Changeset List

PathSize
swh/
storage/
archiver/
64 lines
49 lines
15 lines
66 lines
tests/
30 lines

Diff 258

View Options

swh/storage/archiver/director.py

Loading...
View Options

swh/storage/archiver/storage.py

Loading...
View Options

swh/storage/archiver/worker.py

Loading...
View Options

swh/storage/db.py

Loading...
View Options

swh/storage/tests/test_archiver.py

Loading...
About · Developer information · Legal