Page MenuHomeSoftware Heritage
Differential D5670

Stop storing authority/fetcher metadata.
ClosedPublic
Actions

Authored by vlorentz on May 4 2021, 4:06 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T3303: Replaying REMD topics is broken
Commits
rDSTOe170fb29fde7: Stop storing authority/fetcher metadata.
Summary

We still don't have a use for them, and they are causing issues;
such as being unable to add an authority/fetcher based only
on a REMD object, which is needed by the replayer.

Resolves T3303.

Diff Detail

Repository
rDSTO Storage manager
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 21243
Build 32984: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 32983: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
16,813 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_backfill::test_backfiller
swh_storage_backend_config = {'check_config': {'check_write': True}, 'cls': 'local', 'db': "dbname=storage user=postgres host=127.0.0.1 port=31785 ...riter': {'brokers': ['127.0.0.1:47457'], 'client_id': 'kafka_writer-1', 'cls': 'kafka', 'prefix': 'ndzoegbslf-1'}, ...} kafka_prefix = 'ndzoegbslf', kafka_consumer_group = 'test-consumer-ndzoegbslf' kafka_server = '127.0.0.1:47457'
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin_visit
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_raw_extrinsic_metadata
View Full Test Results (1 Failed · 819 Passed · 29 Skipped)

Event Timeline

vlorentz created this revision.May 4 2021, 4:06 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMay 4 2021, 4:07 PM
swh-public-ci added a comment.May 4 2021, 4:16 PM

Build has FAILED

Patch application report for D5670 (id=20256)

Rebasing onto 051b771523...

Current branch diff-target is up to date.
Changes applied before test
commit e2afe93b756bd073ff855bfc45d0b1baa197f22e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 4 16:04:38 2021 +0200

    Stop storing authority/fetcher metadata.
    
    We still don't have a use for them, and they are causing issues;
    such as being unable to add an authority/fetcher based only
    on a REMD object, which is needed by the replayer.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1301/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1301/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 4 2021, 4:16 PM
Harbormaster failed remote builds in B21243: Diff 20256!
douardda added a subscriber: douardda.May 4 2021, 4:39 PM

maybe add a "fixer" also (in fixers.py) for the replayer to not break because of a metadata field in a kafka message?

vlorentz updated this revision to Diff 20263.May 4 2021, 5:04 PM

Update backfiller too

vlorentz added a comment.May 4 2021, 5:08 PM

@douardda It won't break, because the metadata field will simply be ignored.

But I'll add a fixer in a future diff, so I can remove the field from the model.

swh-public-ci added a comment.May 4 2021, 5:25 PM

Build was aborted

Patch application report for D5670 (id=20263)

Rebasing onto 051b771523...

Current branch diff-target is up to date.
Changes applied before test
commit f9ca2fa32a122c8e839b7ef252a6b2813b971328
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 4 16:04:38 2021 +0200

    Stop storing authority/fetcher metadata.
    
    We still don't have a use for them, and they are causing issues;
    such as being unable to add an authority/fetcher based only
    on a REMD object, which is needed by the replayer.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1303/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1303/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 4 2021, 5:25 PM
Harbormaster failed remote builds in B21250: Diff 20263!
swh-public-ci added a comment.May 4 2021, 5:33 PM

Build is green

Patch application report for D5670 (id=20263)

Rebasing onto 051b771523...

Current branch diff-target is up to date.
Changes applied before test
commit f9ca2fa32a122c8e839b7ef252a6b2813b971328
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 4 16:04:38 2021 +0200

    Stop storing authority/fetcher metadata.
    
    We still don't have a use for them, and they are causing issues;
    such as being unable to add an authority/fetcher based only
    on a REMD object, which is needed by the replayer.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1304/ for more details.

Harbormaster completed remote builds in B21250: Diff 20263.May 4 2021, 5:33 PM
vlorentz requested review of this revision.May 4 2021, 5:33 PM
vlorentz added a comment.May 5 2021, 11:33 AM

But I'll add a fixer in a future diff, so I can remove the field from the model.

Actually, that's not even needed because that field was not written to Kafka.

douardda accepted this revision.May 5 2021, 12:34 PM
This revision is now accepted and ready to land.May 5 2021, 12:34 PM
vlorentz updated this revision to Diff 20291.May 5 2021, 12:54 PM

rebase

swh-public-ci added a comment.May 5 2021, 1:01 PM

Build is green

Patch application report for D5670 (id=20291)

Rebasing onto 77ef651d95...

Current branch diff-target is up to date.
Changes applied before test
commit e170fb29fde7ddc896b28f60505f64adf3631410
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 4 16:04:38 2021 +0200

    Stop storing authority/fetcher metadata.
    
    We still don't have a use for them, and they are causing issues;
    such as being unable to add an authority/fetcher based only
    on a REMD object, which is needed by the replayer.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1308/ for more details.

Harbormaster completed remote builds in B21277: Diff 20291.May 5 2021, 1:01 PM
Closed by commit rDSTOe170fb29fde7: Stop storing authority/fetcher metadata. (authored by vlorentz). · Explain WhyMay 5 2021, 1:06 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSTOe170fb29fde7: Stop storing authority/fetcher metadata..

Revision Contents
Changeset List

PathSize
sql/
upgrades/
12 lines
swh/
storage/
cassandra/
2 lines
2 lines
23 lines
postgresql/
26 lines
17 lines
sql/
10 lines
tests/
24 lines

Diff 20256

View Options

sql/upgrades/174.sql

Loading...
View Options

swh/storage/cassandra/model.py

Loading...
View Options

swh/storage/cassandra/schema.py

Loading...
View Options

swh/storage/cassandra/storage.py

Loading...
View Options

swh/storage/postgresql/db.py

Loading...
View Options

swh/storage/postgresql/storage.py

Loading...
View Options

swh/storage/sql/30-schema.sql

Loading...
View Options

swh/storage/tests/storage_data.py

Loading...
About · Developer information · Legal