Page MenuHomeSoftware Heritage

common/archive: Ensure backward compatibility with swh-indexer 1.x
ClosedPublic

Authored by anlambert on Jun 8 2022, 1:44 PM.

Details

Summary

idx_storage.origin_intrinsic_metadata_get does not return the same
dict fields according to the swh-indexer version used.

As swh-indexer 2.0.0 is not yet deployed in production, we must ensure
backward compatibility with swh-indexer 1.x or the Web API endpoint
/origin/metadata-search/ will break.

Diff Detail

Repository
rDWAPPS Web applications
Branch
swh-indexer-1.x-backward-compatibility
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 29804
Build 46580: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 46579: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D7971 (id=28719)

Rebasing onto 66ded032dc...

Current branch diff-target is up to date.
Changes applied before test
commit caedd83a5998b288a08bd18df47ba9a8ee00bcc5
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Jun 8 13:39:48 2022 +0200

    common/archive: Ensure backward compatibility with swh-indexer 1.x
    
    idx_storage.origin_intrinsic_metadata_get does not return the same
    dict fields according to the swh-indexer version used.
    
    As swh-indexer 2.0.0 is not yet deployed in production, we must ensure
    backward compatibility with swh-indexer 1.x or the Web API endpoint
    /origin/metadata-search/ will break.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1888/ for more details.

Why bother with swh-indexer 1.x? updating prod should be good enough

and I don't think you meant to stage debian/

Why bother with swh-indexer 1.x? updating prod should be good enough

and I don't think you meant to stage debian/

Because if we deploy swh-web before swh-indexer (which is likely to happen), this would break the origin metadata search endpoint.

and I don't think you meant to stage debian/

Oops, classic mistake from myself.

Remove debian folder staged by mistake.

updating prod should be good enough

@vlorentz I'm lost as to when deploying the new indexer v2.0 should happen by the way, any clues?

Build is green

Patch application report for D7971 (id=28723)

Rebasing onto 66ded032dc...

Current branch diff-target is up to date.
Changes applied before test
commit 051d59325b99931b67c8dfe52eb01766f24e3973
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Jun 8 13:39:48 2022 +0200

    common/archive: Ensure backward compatibility with swh-indexer 1.x
    
    idx_storage.origin_intrinsic_metadata_get does not return the same
    dict fields according to the swh-indexer version used.
    
    As swh-indexer 2.0.0 is not yet deployed in production, we must ensure
    backward compatibility with swh-indexer 1.x or the Web API endpoint
    /origin/metadata-search/ will break.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1890/ for more details.

Sorry for the late reply. I thought swh-indexer v2 was already deployed; but @ardumont is working on it now

Sorry for the late reply. I thought swh-indexer v2 was already deployed; but @ardumont is working on it now

Well not now now ¯\_(ツ)_/¯ but i've opened [1] to address it soonishly.

[1] T4319

This revision is now accepted and ready to land.Jun 10 2022, 9:10 AM
This revision was landed with ongoing or failed builds.Jun 13 2022, 11:42 AM
This revision was automatically updated to reflect the committed changes.

Build has FAILED

Patch application report for D7971 (id=28751)

Rebasing onto de50f0b449...

Current branch diff-target is up to date.
Changes applied before test
commit 284e0d9857b26674b258be17b08352ed7c68d0ad
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Jun 8 13:39:48 2022 +0200

    common/archive: Ensure backward compatibility with swh-indexer 1.x
    
    idx_storage.origin_intrinsic_metadata_get does not return the same
    dict fields according to the swh-indexer version used.
    
    As swh-indexer 2.0.0 is not yet deployed in production, we must ensure
    backward compatibility with swh-indexer 1.x or the Web API endpoint
    /origin/metadata-search/ will break.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1898/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1898/console