Depends on D3010.
Details
Details
- Reviewers
ardumont - Group Reviewers
Reviewers - Maniphest Tasks
- T2355: Make swh-journal independent from swh-storage or swh-objstorage
- Commits
- rDSTO038c30b40838: Adapt journal client loading to swh.journal 0.0.31
Diff Detail
Diff Detail
- Repository
- rDSTO Storage manager
- Lint
Automatic diff as part of commit; lint not applicable. - Unit
Automatic diff as part of commit; unit tests not applicable.
Event Timeline
Comment Actions
Build is green
Patch application report for D3058 (id=10875)
Could not rebase; Attempt merge onto fe56005555...
Updating fe56005..bff1125 Fast-forward mypy.ini | 9 + requirements-swh-journal.txt | 2 +- swh/storage/backfill.py | 490 +++++++++++++++++++++++++++++++++ swh/storage/cli.py | 147 +++++++++- swh/storage/fixer.py | 293 ++++++++++++++++++++ swh/storage/replay.py | 128 +++++++++ swh/storage/tests/conftest.py | 2 - swh/storage/tests/test_backfill.py | 160 +++++++++++ swh/storage/tests/test_cli.py | 193 +++++++++++++ swh/storage/tests/test_replay.py | 388 ++++++++++++++++++++++++++ swh/storage/tests/test_write_replay.py | 112 ++++++++ 11 files changed, 1916 insertions(+), 8 deletions(-) create mode 100644 swh/storage/backfill.py create mode 100644 swh/storage/fixer.py create mode 100644 swh/storage/replay.py create mode 100644 swh/storage/tests/test_backfill.py create mode 100644 swh/storage/tests/test_cli.py create mode 100644 swh/storage/tests/test_replay.py create mode 100644 swh/storage/tests/test_write_replay.py
Changes applied before test
commit bff11258065a7bcf7f871599fb402ce1304ec9fd
Author: David Douard <david.douard@sdfa3.org>
Date: Wed Apr 22 17:00:45 2020 +0200
Adapt journal client loading to swh.journal 0.0.31
commit bd1ac8c758217eb734f33e1f67e24606ab307006
Author: David Douard <david.douard@sdfa3.org>
Date: Thu Apr 9 17:45:55 2020 +0200
Copy the graph replayer component from swh-journal
The CLI command is included as well as `swh storage replay`.
Copied test test_replay, in test_cli.py, should be identical.
commit e245a465589fa24288bb62bd57aaf925d665c353
Author: David Douard <david.douard@sdfa3.org>
Date: Thu Apr 23 15:27:28 2020 +0200
Deprecate the `config-path` argument of the `swh storage rpc-serve` command
in favor of the standard `--config-file` option of `swh storage`.
Attempt to write a couple of tests for the rpc-serve command.
commit cd32cf42538ff256bed718fd747e289c9a311ebf
Author: David Douard <david.douard@sdfa3.org>
Date: Wed Apr 22 16:57:07 2020 +0200
Normalize the configuration file handling in the `swh storage` CLI command
Almost every other swh package handles the loading of the config file
from the main click command group of the package.
So we make storage behaves the same.
commit 698af8ca0028c5847326cf57f867608e533af9ed
Author: David Douard <david.douard@sdfa3.org>
Date: Fri Apr 10 11:03:21 2020 +0200
cli: rename the command 'backfiller' as 'backfill'
for the sake of consistency.
commit b2bba450aaa808406b024800dcd50b4778313903
Author: David Douard <david.douard@sdfa3.org>
Date: Thu Apr 9 16:56:44 2020 +0200
Copy the backfiller component from swh-journal
This componant makes more sense in the swh-storage package.See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/123/ for more details.
Comment Actions
Build is green
Patch application report for D3058 (id=10878)
Could not rebase; Attempt merge onto fe56005555...
Updating fe56005..fb20c7d Fast-forward mypy.ini | 9 + requirements-swh-journal.txt | 2 +- swh/storage/backfill.py | 490 +++++++++++++++++++++++++++++++++ swh/storage/cli.py | 147 +++++++++- swh/storage/fixer.py | 293 ++++++++++++++++++++ swh/storage/replay.py | 128 +++++++++ swh/storage/tests/conftest.py | 2 - swh/storage/tests/test_backfill.py | 160 +++++++++++ swh/storage/tests/test_cli.py | 194 +++++++++++++ swh/storage/tests/test_kafka_writer.py | 60 ++++ swh/storage/tests/test_replay.py | 388 ++++++++++++++++++++++++++ swh/storage/tests/test_write_replay.py | 112 ++++++++ 12 files changed, 1977 insertions(+), 8 deletions(-) create mode 100644 swh/storage/backfill.py create mode 100644 swh/storage/fixer.py create mode 100644 swh/storage/replay.py create mode 100644 swh/storage/tests/test_backfill.py create mode 100644 swh/storage/tests/test_cli.py create mode 100644 swh/storage/tests/test_kafka_writer.py create mode 100644 swh/storage/tests/test_replay.py create mode 100644 swh/storage/tests/test_write_replay.py
Changes applied before test
commit fb20c7d433acb81bec10f6a7d3f466029c8dfd10
Author: David Douard <david.douard@sdfa3.org>
Date: Wed Apr 22 17:00:45 2020 +0200
Adapt journal client loading to swh.journal 0.0.31
commit 726dd9a6f5fd25bb43b395bc0c162c4068b7c9c8
Author: David Douard <david.douard@sdfa3.org>
Date: Thu Apr 9 17:45:55 2020 +0200
Copy the graph replayer component from swh-journal
The CLI command is included as well as `swh storage replay`.
Copied test test_replay, in test_cli.py, should be identical.
commit 5fd9b563298f1068636ba57be10d2f1975f09781
Author: David Douard <david.douard@sdfa3.org>
Date: Thu Apr 23 15:27:28 2020 +0200
Deprecate the `config-path` argument of the `swh storage rpc-serve` command
in favor of the standard `--config-file` option of `swh storage`.
Attempt to write a couple of tests for the rpc-serve command.
commit cd32cf42538ff256bed718fd747e289c9a311ebf
Author: David Douard <david.douard@sdfa3.org>
Date: Wed Apr 22 16:57:07 2020 +0200
Normalize the configuration file handling in the `swh storage` CLI command
Almost every other swh package handles the loading of the config file
from the main click command group of the package.
So we make storage behaves the same.
commit 698af8ca0028c5847326cf57f867608e533af9ed
Author: David Douard <david.douard@sdfa3.org>
Date: Fri Apr 10 11:03:21 2020 +0200
cli: rename the command 'backfiller' as 'backfill'
for the sake of consistency.
commit b2bba450aaa808406b024800dcd50b4778313903
Author: David Douard <david.douard@sdfa3.org>
Date: Thu Apr 9 16:56:44 2020 +0200
Copy the backfiller component from swh-journal
This componant makes more sense in the swh-storage package.See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/126/ for more details.
Comment Actions
Build is green
Patch application report for D3058 (id=10884)
Could not rebase; Attempt merge onto 5fd9b56329...
Updating 5fd9b56..038c30b Fast-forward mypy.ini | 6 + requirements-swh-journal.txt | 2 +- swh/storage/cli.py | 55 +++++ swh/storage/fixer.py | 293 +++++++++++++++++++++++++ swh/storage/replay.py | 128 +++++++++++ swh/storage/tests/conftest.py | 2 - swh/storage/tests/test_cli.py | 81 ++++++- swh/storage/tests/test_kafka_writer.py | 60 +++++ swh/storage/tests/test_replay.py | 388 +++++++++++++++++++++++++++++++++ swh/storage/tests/test_write_replay.py | 112 ++++++++++ 10 files changed, 1120 insertions(+), 7 deletions(-) create mode 100644 swh/storage/fixer.py create mode 100644 swh/storage/replay.py create mode 100644 swh/storage/tests/test_kafka_writer.py create mode 100644 swh/storage/tests/test_replay.py create mode 100644 swh/storage/tests/test_write_replay.py
Changes applied before test
commit 038c30b40838defb27a7686e456047488be9a29d
Author: David Douard <david.douard@sdfa3.org>
Date: Wed Apr 22 17:00:45 2020 +0200
Adapt journal client loading to swh.journal 0.0.31
commit 4b7ba1fc6bccd18c391a841da636a950bc66111d
Author: David Douard <david.douard@sdfa3.org>
Date: Thu Apr 9 17:45:55 2020 +0200
Copy the graph replayer component from swh-journal
The CLI command is included as well as `swh storage replay`.
Copied test test_replay, in test_cli.py, should be identical.See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/128/ for more details.