Page MenuHomeSoftware Heritage

writer: make the writer use swh.model objects instead of (possibly) dicts
ClosedPublic

Authored by douardda on Apr 3 2020, 3:14 PM.

Details

Summary

And annotate the code as we go.

Depends on D2955.

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build has FAILED

Patch application report for D2956 (id=10516)

Could not rebase; Attempt merge onto ca7e9d20c5...

Updating ca7e9d2..17d8c0f
Fast-forward
 .pre-commit-config.yaml                |  15 +++-
 setup.cfg                              |   6 ++
 swh/journal/serializers.py             |  10 ++-
 swh/journal/tests/conftest.py          |   8 +-
 swh/journal/tests/test_kafka_writer.py |  27 +++---
 swh/journal/writer/inmemory.py         |  11 ++-
 swh/journal/writer/kafka.py            | 150 +++++++++++++++++++++------------
 7 files changed, 147 insertions(+), 80 deletions(-)
 create mode 100644 setup.cfg
Changes applied before test
commit 17d8c0fd81c008416a612fdc50fdef2ec01399a1
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 3 15:03:32 2020 +0200

    writer: make the writer use swh.model objects instead of (possibly) dicts
    
    And annotate the code as we go.

commit 7e51665eae837164342a5d49bf4cf11d861d16a7
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 3 14:54:25 2020 +0200

    Update pre-commit config to use a 'local' configuration
    
    This is needed to be able to use flake8 with pyflakes > 2.1.1
    to gain support for @overload on methods.

commit eb6ff79a76a470482624886b50f1b25fd8c03ad6
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 3 13:32:44 2020 +0200

    Add a setup.cfg to make flake8 compatible with blackified files

commit 9c697f6812899d002ee411807fe85611b163a8b7
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Apr 2 15:18:00 2020 +0200

    test: use Multihash to create a valid dict for a Content object

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/8/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/8/console

ardumont added inline comments.
swh/journal/tests/test_kafka_writer.py
24

MODEL_OBJECTS: Dict[str, Type[BaseModel]]

?

swh/journal/writer/kafka.py
92

What is the @overload annotation?

swh/journal/writer/kafka.py
92
15:31 <+douardda> ardumont: declare multiple signatures for a function / method

see https://docs.python.org/3/library/typing.html#typing.overload

132–143

Shoudn't there be an implementation in each of the previous overloaded methods then?
And remove some if/else here though?

swh/journal/writer/kafka.py
132–143

nvm, i read again the documentation and realized that it's only to improve on the type definition combination.
Not to define specific implementation per type signature.

swh/journal/tests/test_kafka_writer.py
24

not needed, mypy seems to infer this from OBJECT_TYPES type declaration.

This revision is now accepted and ready to land.Apr 3 2020, 3:47 PM

rebase + add comment on @overload usage

Build is green

Patch application report for D2956 (id=10524)

Could not rebase; Attempt merge onto f6a398f99a...

Updating f6a398f..7d6ead5
Fast-forward
 .pre-commit-config.yaml                |  15 +++-
 setup.cfg                              |   6 ++
 swh/journal/serializers.py             |  10 ++-
 swh/journal/tests/test_kafka_writer.py |  27 +++---
 swh/journal/writer/inmemory.py         |  11 ++-
 swh/journal/writer/kafka.py            | 152 ++++++++++++++++++++++-----------
 tox.ini                                |   1 +
 7 files changed, 148 insertions(+), 74 deletions(-)
 create mode 100644 setup.cfg
Changes applied before test
commit 7d6ead58a4300b2cefc3df939a5ae1845d6f5f71
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 3 15:03:32 2020 +0200

    writer: make the writer use swh.model objects instead of (possibly) dicts
    
    And annotate the code as we go.

commit f3112dbf7cceed2520a1cd44d0b6701bee271dd8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 3 14:54:25 2020 +0200

    Enforce current dev version of pyflakes in tox and pre-commit
    
    - Enforce installation of pyflakes from upstream git repo.
    - Update pre-commit config to use a 'local' configuration.
      This is needed to be able to use flake8 with pyflakes > 2.1.1
      to gain support for @overload on methods.

commit a25e1222aed63fa0a83df959161f98698ac15484
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Apr 3 13:32:44 2020 +0200

    Add a setup.cfg to make flake8 compatible with blackified files

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/13/ for more details.