Page MenuHomeSoftware Heritage
Differential D4399

Un-skip some ctags and license storage tests.
ClosedPublic
Actions

Authored by vlorentz on Nov 3 2020, 1:12 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDCIDXe2835bfff6ba: Un-skip some ctags and license storage tests.
Summary

Commits in the last month made these endpoints more consistent with
the other ones (though not completely), so these tests don't need to
be skipped anymore.

Depends on D4397.

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
remove-conflict-update
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 16805
Build 25915: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 25914: arc lint + arc unit

Event Timeline

vlorentz created this revision.Nov 3 2020, 1:12 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 3 2020, 1:12 PM
swh-public-ci added a comment.Nov 3 2020, 1:13 PM

Build is green

Patch application report for D4399 (id=15582)

Could not rebase; Attempt merge onto 94c8259193...

Updating 94c8259..ffd1588
Fast-forward
 sql/upgrades/133.sql                      | 177 ++++++++++++++++++++++++++
 swh/indexer/cli.py                        |   2 +-
 swh/indexer/ctags.py                      |   8 +-
 swh/indexer/fossology_license.py          |   9 +-
 swh/indexer/indexer.py                    |  57 +++++----
 swh/indexer/journal_client.py             |   1 -
 swh/indexer/metadata.py                   |  31 ++---
 swh/indexer/mimetype.py                   |   9 +-
 swh/indexer/origin_head.py                |   4 +-
 swh/indexer/sql/30-schema.sql             |   2 +-
 swh/indexer/sql/50-func.sql               | 197 +++++++++++------------------
 swh/indexer/storage/__init__.py           |  56 +++------
 swh/indexer/storage/db.py                 |  34 +++--
 swh/indexer/storage/in_memory.py          |  40 +++---
 swh/indexer/storage/interface.py          |  35 ++----
 swh/indexer/tests/storage/test_storage.py | 199 ++++++++----------------------
 swh/indexer/tests/test_cli.py             |   2 +-
 swh/indexer/tests/test_indexer.py         |  20 ++-
 swh/indexer/tests/test_journal_client.py  |  16 +--
 swh/indexer/tests/test_metadata.py        |   6 +-
 swh/indexer/tests/test_origin_head.py     |   2 +-
 swh/indexer/tests/utils.py                |   7 +-
 22 files changed, 419 insertions(+), 495 deletions(-)
 create mode 100644 sql/upgrades/133.sql
Changes applied before test
commit ffd1588d767b93efdbc5a18011bf35f9daeef51e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 13:11:48 2020 +0100

    Un-skip some ctags and license storage tests.
    
    Commits in the last month made these endpoints more consistent with
    the other ones (though not completely), so these tests don't need to
    be skipped anymore.

commit 6e8a901b95f3852a1b90611b1e1068d79ae6063d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 12:50:31 2020 +0100

    Remove conflict_update/policy_update option.
    
    By removing the False/ignore-dups behavior, which we didn't use.
    
    The main motivation is to make `_add` endpoints write to Kafka in a
    future commit, as Kafka's semantics are closer to True/update-dups
    than False/ignore-dups.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/104/ for more details.

Harbormaster completed remote builds in B16805: Diff 15582.Nov 3 2020, 1:13 PM
vlorentz updated this revision to Diff 15584.Nov 3 2020, 1:15 PM

rebase

swh-public-ci added a comment.Nov 3 2020, 1:18 PM

Build is green

Patch application report for D4399 (id=15584)

Rebasing onto 94c8259193...

Current branch diff-target is up to date.
Changes applied before test
commit 2e044e3b0fd373ab94b35aca6ed7df0e6bbe040e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 13:11:48 2020 +0100

    Un-skip some ctags and license storage tests.
    
    Commits in the last month made these endpoints more consistent with
    the other ones (though not completely), so these tests don't need to
    be skipped anymore.

commit 6022e53417f104f7ddd642d3c20ce7f8a98b6e3e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 12:50:31 2020 +0100

    Remove conflict_update/policy_update option.
    
    By removing the False/ignore-dups behavior, which we didn't use.
    
    The main motivation is to make `_add` endpoints write to Kafka in a
    future commit, as Kafka's semantics are closer to True/update-dups
    than False/ignore-dups.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/106/ for more details.

Harbormaster completed remote builds in B16807: Diff 15584.Nov 3 2020, 1:18 PM
ardumont requested changes to this revision.Nov 3 2020, 2:07 PM
ardumont added a subscriber: ardumont.

This diff contains one commit too much (the one from the previous diff which is already too long).
Please, update it so the review is simpler.

Cheers,

This revision now requires changes to proceed.Nov 3 2020, 2:07 PM
vlorentz updated this revision to Diff 15588.Nov 3 2020, 2:24 PM

rebase

vlorentz updated this revision to Diff 15591.Nov 3 2020, 2:26 PM

rebase

swh-public-ci added a comment.Nov 3 2020, 2:27 PM

Build is green

Patch application report for D4399 (id=15588)

Could not rebase; Attempt merge onto 94c8259193...

Updating 94c8259..bb47843
Fast-forward
 sql/upgrades/133.sql                      | 184 +++++++++++++++++++++++++++
 swh/indexer/cli.py                        |   2 +-
 swh/indexer/ctags.py                      |   8 +-
 swh/indexer/fossology_license.py          |   9 +-
 swh/indexer/indexer.py                    |  57 +++++----
 swh/indexer/journal_client.py             |   1 -
 swh/indexer/metadata.py                   |  31 ++---
 swh/indexer/mimetype.py                   |   9 +-
 swh/indexer/origin_head.py                |   4 +-
 swh/indexer/sql/30-schema.sql             |   2 +-
 swh/indexer/sql/50-func.sql               | 198 +++++++++++------------------
 swh/indexer/storage/__init__.py           |  56 +++------
 swh/indexer/storage/db.py                 |  34 +++--
 swh/indexer/storage/in_memory.py          |  40 +++---
 swh/indexer/storage/interface.py          |  35 ++----
 swh/indexer/tests/storage/test_storage.py | 199 ++++++++----------------------
 swh/indexer/tests/test_cli.py             |   2 +-
 swh/indexer/tests/test_indexer.py         |  20 ++-
 swh/indexer/tests/test_journal_client.py  |  16 +--
 swh/indexer/tests/test_metadata.py        |   6 +-
 swh/indexer/tests/test_origin_head.py     |   2 +-
 swh/indexer/tests/utils.py                |   7 +-
 22 files changed, 426 insertions(+), 496 deletions(-)
 create mode 100644 sql/upgrades/133.sql
Changes applied before test
commit bb47843eded6bbc0cbb94cd86f41c612bd20f8f5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 13:11:48 2020 +0100

    Un-skip some ctags and license storage tests.
    
    Commits in the last month made these endpoints more consistent with
    the other ones (though not completely), so these tests don't need to
    be skipped anymore.

commit e0b2ec8bf38dc195c80000dc01040158a0b69d47
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 14:21:41 2020 +0100

    Remove conflict_update option from _add() endpoints.
    
    By removing the False behavior, which we didn't use in practice,
    and was removed from the indexers in the previous commit.
    
    The main motivation is to make _add endpoints write to Kafka in a
    future commit, as Kafka's semantics are closer to conflict_update=True
    than conflict_update=False.

commit 72528376f5dbd22fa915c0fac7c0853906f1d38b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 14:19:23 2020 +0100

    Remove conflict_update/policy_update option from BaseIndexer.run()
    
    By removing the False/ignore-dups behavior, which we didn't use.
    
    The main motivation is to make _add endpoints write to Kafka in a
    future commit, as Kafka's semantics are closer to True/update-dups
    than False/ignore-dups.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/108/ for more details.

Harbormaster completed remote builds in B16811: Diff 15588.Nov 3 2020, 2:27 PM
vlorentz added a parent revision: D4400: Remove conflict_update option from _add() endpoints..Nov 3 2020, 2:27 PM
ardumont added a comment.Nov 3 2020, 2:27 PM

thanks.

Now though you allow tests to run on ctags as explained in the description.

But you inhibit from for the license indexer... The description should mention this as well.

vlorentz removed a parent revision: D4397: Remove conflict_update/policy_update option..Nov 3 2020, 2:28 PM
In D4399#109653, @ardumont wrote:

But you inhibit from for the license indexer... The description should mention this as well.

No, I don't. I made it inherit from StorageETypeTester which *adds* tests, then mark some of them as skipped.

ardumont accepted this revision.Nov 3 2020, 2:31 PM

No, I don't. I made it inherit from StorageETypeTester which *adds* tests, then mark some of them as skipped.

ok, i misread.
cool

This revision is now accepted and ready to land.Nov 3 2020, 2:31 PM
swh-public-ci added a comment.Nov 3 2020, 2:32 PM

Build is green

Patch application report for D4399 (id=15591)

Could not rebase; Attempt merge onto 94c8259193...

Updating 94c8259..e2835bf
Fast-forward
 sql/upgrades/133.sql                      | 184 +++++++++++++++++++++++++++
 swh/indexer/cli.py                        |   2 +-
 swh/indexer/ctags.py                      |   8 +-
 swh/indexer/fossology_license.py          |   9 +-
 swh/indexer/indexer.py                    |  57 +++++----
 swh/indexer/journal_client.py             |   1 -
 swh/indexer/metadata.py                   |  31 ++---
 swh/indexer/mimetype.py                   |   9 +-
 swh/indexer/origin_head.py                |   4 +-
 swh/indexer/sql/30-schema.sql             |   2 +-
 swh/indexer/sql/50-func.sql               | 198 +++++++++++------------------
 swh/indexer/storage/__init__.py           |  56 +++------
 swh/indexer/storage/db.py                 |  34 +++--
 swh/indexer/storage/in_memory.py          |  40 +++---
 swh/indexer/storage/interface.py          |  35 ++----
 swh/indexer/tests/storage/test_storage.py | 199 ++++++++----------------------
 swh/indexer/tests/test_cli.py             |   2 +-
 swh/indexer/tests/test_indexer.py         |  20 ++-
 swh/indexer/tests/test_journal_client.py  |  16 +--
 swh/indexer/tests/test_metadata.py        |   6 +-
 swh/indexer/tests/test_origin_head.py     |   2 +-
 swh/indexer/tests/utils.py                |   7 +-
 22 files changed, 426 insertions(+), 496 deletions(-)
 create mode 100644 sql/upgrades/133.sql
Changes applied before test
commit e2835bfff6baffd9a0a998ac4a1dc6d67326d950
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 13:11:48 2020 +0100

    Un-skip some ctags and license storage tests.
    
    Commits in the last month made these endpoints more consistent with
    the other ones (though not completely), so these tests don't need to
    be skipped anymore.

commit 1127804dec29b1217c518ec5e3c4e64aeeb41d1b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 14:21:41 2020 +0100

    Remove conflict_update option from _add() endpoints.
    
    By removing the False behavior, which we didn't use in practice,
    and was removed from the indexers in the previous commit.
    
    The main motivation is to make _add endpoints write to Kafka in a
    future commit, as Kafka's semantics are closer to conflict_update=True
    than conflict_update=False.

commit 8e69633f639e31cf5efa3edd784397b351d3b2db
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Nov 3 14:19:23 2020 +0100

    Remove conflict_update/policy_update option from BaseIndexer.run()
    
    By removing the False/ignore-dups behavior, which we didn't use.
    
    The main motivation is to make _add endpoints write to Kafka in a
    future commit, as Kafka's semantics are closer to True/update-dups
    than False/ignore-dups.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/111/ for more details.

Harbormaster completed remote builds in B16814: Diff 15591.Nov 3 2020, 2:32 PM
Closed by commit rDCIDXe2835bfff6ba: Un-skip some ctags and license storage tests. (authored by vlorentz). · Explain WhyNov 3 2020, 5:03 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDCIDXe2835bfff6ba: Un-skip some ctags and license storage tests..

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
indexer/
tests/
storage/
28 lines99%
Loading...

Diff 15582

View Options

swh/indexer/tests/storage/test_storage.py

Loading...
About · Developer information · Legal