Page MenuHomeSoftware Heritage
Differential D181

Add journal client implementation which updates content archiver db with new contents
AbandonedPublic
Actions

Authored by ardumont on Feb 25 2017, 1:30 AM.

Details

Reviewers
olasd
zack
Group Reviewers
Reviewers
Summary

This leverages the base class client (D180) to do thy bidding.

Note that this is a draft and some implementation needs improvment.

I mainly target the archiver storage functions (existing one + that
new one in this revision) which should be improved/refactored/merged +
one of the archiver director which is already able to cope with
missing entries... (T569)

Related T494

Diff Detail

Repository
rDSTO Storage manager
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 708
Build 951: Software Heritage Python tests
Build 950: arc lint + arc unit

Event Timeline

ardumont created this revision.Feb 25 2017, 1:30 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 25 2017, 1:30 AM
ardumont updated this revision to Diff 558.Mar 1 2017, 11:04 AM

Sum up

  • Refactor: Merge common behavior in director and content updater client
  • test: Remove impossible and commented test
  • content_archive_add: Use the right 'missing' status
  • Refactor: Reuse swh.scheduler.get_task function
  • archiver-storage: Improve content_archive_add function
  • Refactor: Unify the content_archive_add with swh.storage.content_add
  • swh.storage.archiver.updater: Add logging level to INFO
  • swh.storage.archiver.updater: Override configuration filename
ardumont mentioned this in D182: Deploy journal_publisher and archiver_content_updater manifest.Mar 1 2017, 11:15 AM
ardumont mentioned this in D183: Add journal_publisher and archiver_content_updater's configuration.
olasd added inline comments.Mar 2 2017, 3:37 PM
swh/storage/archiver/director.py
206

# yesqa :)

That line should not be needed anyway.

swh/storage/archiver/storage.py
180

If possible this mtime should be pulled from the database (so you'll pass a list of content_id/mtime data), even though that's okay for a first pass (this can land as is).

In any case it doesn't really need to be coerced to an int, a float will do.

182–186

This is not useful as the status for random archives will always default to "missing". Cleans up some configuration as well which is always nice.

ardumont added inline comments.Mar 2 2017, 4:16 PM
swh/storage/archiver/storage.py
180

Right.

182–186

Indeed.

ardumont updated this revision to Diff 567.Mar 2 2017, 4:17 PM

Clean up revision after code separation between director/archiver.storage

  • Add journal client to update content archiver with new content
  • d/control: Add swh-journal dependency
  • Refactor: Merge common behavior in director and content updater client
  • Refactor: Unify the content_archive_add with swh.storage.content_add
  • swh.storage.archiver.updater: Add logging level to INFO
  • swh.storage.archiver.updater: Override configuration filename
ardumont updated this revision to Diff 569.Mar 2 2017, 5:05 PM

Rebased on latest master + remove sources_missing option

  • Add journal client to update content archiver with new content
  • d/control: Add swh-journal dependency
  • Refactor: Merge common behavior in director and content updater client
  • Refactor: Unify the content_archive_add with swh.storage.content_add
  • swh.storage.archiver.updater: Add logging level to INFO
  • swh.storage.archiver.updater: Override configuration filename
  • swh.storage.archiver.updater: Remove sources_missing
ardumont updated this revision to Diff 576.Mar 3 2017, 10:49 AM

Stabilize the git commit messages to what they actually do

Some commit remanipulation were done yesterday to untangle perimeters
(archiver + content updater). To be fair, they were related.

The code did not change though.

  • Add journal client to update content archiver with new content
  • d/control: Add swh-journal dependency to swh.storage.archiver
  • swh.storage.archiver.updater: Call directly content_archive_add
  • swh.storage.archiver.updater: Add logging level to INFO
  • swh.storage.archiver.updater: Add specific configuration filename
ardumont abandoned this revision.EditedMar 13 2017, 2:33 PM

Merged through 4f1d48ce6a0a281f2553284b7078735ace740755..d2d45b28ed8214f4a4f5b0a3b566ce05a788b335

Revision Contents
Changeset List

PathSize
debian/
7 lines
2 lines
sql/
archiver/
35 lines
2 lines
upgrades/
42 lines
swh/
storage/
archiver/
29 lines
32 lines
44 lines
49 lines
6 lines
6 lines
tests/
23 lines

Diff 558

View Options

debian/control

Loading...
View Options

requirements-swh.txt

Loading...
View Options

sql/archiver/swh-archiver-func.sql

Loading...
View Options

sql/archiver/swh-archiver-schema.sql

Loading...
View Options

sql/archiver/upgrades/009.sql

Loading...
View Options

swh/storage/archiver/db.py

Loading...
View Options

swh/storage/archiver/director.py

Loading...
View Options

swh/storage/archiver/storage.py

Loading...
View Options

swh/storage/archiver/updater.py

Loading...
View Options

swh/storage/archiver/worker.py

Loading...
View Options

swh/storage/db.py

Loading...
View Options

swh/storage/tests/test_archiver.py

Loading...
About · Developer information · Legal