Page MenuHomeSoftware Heritage

Remove conflict_update option from _add() endpoints.
ClosedPublic

Authored by vlorentz on Nov 3 2020, 2:24 PM.

Details

Summary

By removing the False behavior, which we didn't use in practice,
and was removed from the indexers in the previous commit.

The main motivation is to make _add endpoints write to Kafka in a
future commit, as Kafka's semantics are closer to conflict_update=True
than conflict_update=False.

Diff Detail

Event Timeline

Build is green

Patch application report for D4400 (id=15587)

Could not rebase; Attempt merge onto 94c8259193...

Updating 94c8259..e0b2ec8
Fast-forward
 sql/upgrades/133.sql                      | 184 +++++++++++++++++++++++++++
 swh/indexer/cli.py                        |   2 +-
 swh/indexer/ctags.py                      |   8 +-
 swh/indexer/fossology_license.py          |   9 +-
 swh/indexer/indexer.py                    |  57 +++++----
 swh/indexer/journal_client.py             |   1 -
 swh/indexer/metadata.py                   |  31 ++---
 swh/indexer/mimetype.py                   |   9 +-
 swh/indexer/origin_head.py                |   4 +-
 swh/indexer/sql/30-schema.sql             |   2 +-
 swh/indexer/sql/50-func.sql               | 198 +++++++++++-------------------
 swh/indexer/storage/__init__.py           |  56 +++------
 swh/indexer/storage/db.py                 |  34 +++--
 swh/indexer/storage/in_memory.py          |  40 +++---
 swh/indexer/storage/interface.py          |  35 ++----
 swh/indexer/tests/storage/test_storage.py | 171 +++++---------------------
 swh/indexer/tests/test_cli.py             |   2 +-
 swh/indexer/tests/test_indexer.py         |  20 ++-
 swh/indexer/tests/test_journal_client.py  |  16 +--
 swh/indexer/tests/test_metadata.py        |   6 +-
 swh/indexer/tests/test_origin_head.py     |   2 +-
 swh/indexer/tests/utils.py                |   7 +-
 22 files changed, 407 insertions(+), 487 deletions(-)
 create mode 100644 sql/upgrades/133.sql
Changes applied before test
commit e0b2ec8bf38dc195c80000dc01040158a0b69d47
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 14:21:41 2020 +0100

    Remove conflict_update option from _add() endpoints.
    
    By removing the False behavior, which we didn't use in practice,
    and was removed from the indexers in the previous commit.
    
    The main motivation is to make _add endpoints write to Kafka in a
    future commit, as Kafka's semantics are closer to conflict_update=True
    than conflict_update=False.

commit 72528376f5dbd22fa915c0fac7c0853906f1d38b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 14:19:23 2020 +0100

    Remove conflict_update/policy_update option from BaseIndexer.run()
    
    By removing the False/ignore-dups behavior, which we didn't use.
    
    The main motivation is to make _add endpoints write to Kafka in a
    future commit, as Kafka's semantics are closer to True/update-dups
    than False/ignore-dups.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/107/ for more details.

Build is green

Patch application report for D4400 (id=15590)

Could not rebase; Attempt merge onto 94c8259193...

Updating 94c8259..1127804
Fast-forward
 sql/upgrades/133.sql                      | 184 +++++++++++++++++++++++++++
 swh/indexer/cli.py                        |   2 +-
 swh/indexer/ctags.py                      |   8 +-
 swh/indexer/fossology_license.py          |   9 +-
 swh/indexer/indexer.py                    |  57 +++++----
 swh/indexer/journal_client.py             |   1 -
 swh/indexer/metadata.py                   |  31 ++---
 swh/indexer/mimetype.py                   |   9 +-
 swh/indexer/origin_head.py                |   4 +-
 swh/indexer/sql/30-schema.sql             |   2 +-
 swh/indexer/sql/50-func.sql               | 198 +++++++++++-------------------
 swh/indexer/storage/__init__.py           |  56 +++------
 swh/indexer/storage/db.py                 |  34 +++--
 swh/indexer/storage/in_memory.py          |  40 +++---
 swh/indexer/storage/interface.py          |  35 ++----
 swh/indexer/tests/storage/test_storage.py | 171 +++++---------------------
 swh/indexer/tests/test_cli.py             |   2 +-
 swh/indexer/tests/test_indexer.py         |  20 ++-
 swh/indexer/tests/test_journal_client.py  |  16 +--
 swh/indexer/tests/test_metadata.py        |   6 +-
 swh/indexer/tests/test_origin_head.py     |   2 +-
 swh/indexer/tests/utils.py                |   7 +-
 22 files changed, 407 insertions(+), 487 deletions(-)
 create mode 100644 sql/upgrades/133.sql
Changes applied before test
commit 1127804dec29b1217c518ec5e3c4e64aeeb41d1b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 14:21:41 2020 +0100

    Remove conflict_update option from _add() endpoints.
    
    By removing the False behavior, which we didn't use in practice,
    and was removed from the indexers in the previous commit.
    
    The main motivation is to make _add endpoints write to Kafka in a
    future commit, as Kafka's semantics are closer to conflict_update=True
    than conflict_update=False.

commit 8e69633f639e31cf5efa3edd784397b351d3b2db
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 14:19:23 2020 +0100

    Remove conflict_update/policy_update option from BaseIndexer.run()
    
    By removing the False/ignore-dups behavior, which we didn't use.
    
    The main motivation is to make _add endpoints write to Kafka in a
    future commit, as Kafka's semantics are closer to True/update-dups
    than False/ignore-dups.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/110/ for more details.

This revision is now accepted and ready to land.Nov 3 2020, 2:33 PM