Page MenuHomeSoftware Heritage

Copy the backfiller component from swh-journal
ClosedPublic

Authored by douardda on Apr 10 2020, 3:33 PM.

Diff Detail

Repository
rDSTO Storage manager
Branch
skipped_content
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 11938
Build 18094: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 18093: arc lint + arc unit

Event Timeline

add the commit "cli: rename the command 'backfiller' as 'backfill'" in the diff

Build is green

Patch application report for D3008 (id=10676)

Could not rebase; Attempt merge onto ddac3d27e3...

Updating ddac3d2..93ca058
Fast-forward
 mypy.ini                                   |   3 +
 swh/storage/backfill.py                    | 490 +++++++++++++++++++++++++++++
 swh/storage/cli.py                         |  50 +++
 swh/storage/tests/conftest.py              |  45 ++-
 swh/storage/tests/test_api_client.py       |   7 +-
 swh/storage/tests/test_api_client_dicts.py |  28 +-
 swh/storage/tests/test_backfill.py         | 160 ++++++++++
 swh/storage/tests/test_storage.py          | 239 ++++++++------
 8 files changed, 900 insertions(+), 122 deletions(-)
 create mode 100644 swh/storage/backfill.py
 create mode 100644 swh/storage/tests/test_backfill.py
Changes applied before test
commit 93ca058fc487aa13f6c3e4895be1ea82aea2408a
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 9 16:56:44 2020 +0200

    Copy the backfiller component from swh-journal
    
    This componant makes more sense in the swh-storage package.

commit fd5ed546581506906dad640ba8c686a8ae754bbb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Apr 7 16:17:39 2020 +0200

    test: update storage tests to (future) swh.journal 0.0.30
    
    which will handle swh.model objects everywhere instead of dicts.
    
    Also add a BW compat version os the InMemoryJournalWriter so tests
    will pass with current version of swh.journal (0.0.29).

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/86/ for more details.

Build is green

Patch application report for D3008 (id=10678)

Could not rebase; Attempt merge onto ddac3d27e3...

Updating ddac3d2..687fa2b
Fast-forward
 mypy.ini                                   |   3 +
 swh/storage/backfill.py                    | 490 +++++++++++++++++++++++++++++
 swh/storage/cli.py                         |  50 +++
 swh/storage/tests/conftest.py              |  45 ++-
 swh/storage/tests/test_api_client.py       |   7 +-
 swh/storage/tests/test_api_client_dicts.py |  28 +-
 swh/storage/tests/test_backfill.py         | 160 ++++++++++
 swh/storage/tests/test_storage.py          | 239 ++++++++------
 8 files changed, 900 insertions(+), 122 deletions(-)
 create mode 100644 swh/storage/backfill.py
 create mode 100644 swh/storage/tests/test_backfill.py
Changes applied before test
commit 687fa2b3a4ba40ae58746b1504721c766dba5de9
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 10 11:03:21 2020 +0200

    cli: rename the command 'backfiller' as 'backfill'
    
    for the sake of consistency.

commit 93ca058fc487aa13f6c3e4895be1ea82aea2408a
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 9 16:56:44 2020 +0200

    Copy the backfiller component from swh-journal
    
    This componant makes more sense in the swh-storage package.

commit fd5ed546581506906dad640ba8c686a8ae754bbb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Apr 7 16:17:39 2020 +0200

    test: update storage tests to (future) swh.journal 0.0.30
    
    which will handle swh.model objects everywhere instead of dicts.
    
    Also add a BW compat version os the InMemoryJournalWriter so tests
    will pass with current version of swh.journal (0.0.29).

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/87/ for more details.

ardumont edited the summary of this revision. (Show Details)
This revision is now accepted and ready to land.Apr 14 2020, 10:14 AM

rebase *and* pick all the revisions involved

Build is green

Patch application report for D3008 (id=10789)

Rebasing onto bca643acab...

Current branch diff-target is up to date.
Changes applied before test
commit b4784245fb4b0e1683b398dc0d7987a136b10763
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 9 16:56:44 2020 +0200

    Copy the backfiller component from swh-journal
    
    This componant makes more sense in the swh-storage package.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/101/ for more details.

Build is green

Patch application report for D3008 (id=10790)

Rebasing onto bca643acab...

Current branch diff-target is up to date.
Changes applied before test
commit 02f2ef2a00dd02156a8282aeedcff10a3b209e23
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 10 11:03:21 2020 +0200

    cli: rename the command 'backfiller' as 'backfill'
    
    for the sake of consistency.

commit b4784245fb4b0e1683b398dc0d7987a136b10763
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 9 16:56:44 2020 +0200

    Copy the backfiller component from swh-journal
    
    This componant makes more sense in the swh-storage package.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/102/ for more details.

Build is green

Patch application report for D3008 (id=10846)

Rebasing onto fe56005555...

Current branch diff-target is up to date.
Changes applied before test
commit 698af8ca0028c5847326cf57f867608e533af9ed
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 10 11:03:21 2020 +0200

    cli: rename the command 'backfiller' as 'backfill'
    
    for the sake of consistency.

commit b2bba450aaa808406b024800dcd50b4778313903
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 9 16:56:44 2020 +0200

    Copy the backfiller component from swh-journal
    
    This componant makes more sense in the swh-storage package.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/115/ for more details.

swh/storage/backfill.py
22

Won't that pose a problem for module depending on the storage?

it might be a transitional state though, i'm not up-to-date on where you are at with the transition.

olasd added inline comments.
swh/storage/backfill.py
22

Not anymore than the other submodules of swh.storage already depending on swh.journal?

swh/storage/backfill.py
22

Yes the idea is we already have the storage -> journal dependency but we should soon (!) have no mode journal -> storage dependency.

I've let the swh.journal dependency in requirements-swh-journal.txt because technically, the journal is not mandatory for a simple storage. Not sure if we want to go more complex than that...

swh/storage/backfill.py
22

I was thinking on those depending on the storage without the journal.
I think your answer and the completion on david's part answered my concerns.

Thanks.

swh/storage/backfill.py
22

I'm waiting for the the current stack of diff to land in the storage as well as the to-be-sent swh-objstorage-replayer before submitting my pile of diffs that removes the backfiller and replayers from the journal.