Paths

Table of Contentst

Differential D7160

Update storage for swh.core 2.0.0
ClosedPublic
Actions

Authored by douardda on Feb 11 2022, 4:20 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T3894: Add a 'swh db upgrade' cli

Commits

rDSTO215162b2c129: Update for swh.core 2.0.0
rDSTO386fb4d61f19: Add types-toml to requirements-test.txt

Summary

Add expected entry points for swh.core 2 db handling new features:
- add a `swh.storge.get_datastore()` function
- add `swh.storage.postgreql.storage.Storage.get_current_version()` method
- move sql migration scripts in `swh/storage/sql/upgrades`
- modify sql initialization scripts to match swh.core 2 (remove dbversion management code).

Update tests to use the new template-based database handling; this should have only minimal impact on test execution performances.

Depends on D7155
Related to T3894

Diff Detail

Repository

rDSTO Storage manager

Branch

master

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 27104
Build 42390: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 42389: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	2,030 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_replay::test_storage_replayer_with_validation_nok kafka_prefix = 'begirrfpde', kafka_consumer_group = 'test-consumer-begirrfpde' kafka_server = '127.0.0.1:53617'
	4 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
	2 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_raw_extrinsic_metadata
	361 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_diff::test__get_rev
	4 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_diff.TestDiffRevisions::test_insert_delete
		View Full Test Results (1 Failed · 1,170 Passed · 40 Skipped)

Event Timeline

douardda created this revision.Feb 11 2022, 4:20 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 11 2022, 4:20 PM

Build has FAILED

Patch application report for D7160 (id=25954)

Rebasing onto f578377db5...

Current branch diff-target is up to date.

Changes applied before test

commit 035c34911170ae9766287f4e40c9815134cc0a54
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 2 15:56:18 2022 +0100

    Update for swh.core 2.0.0
    
    - Add expected entry points for swh.core 2 db handling new features:
    
      - add a ``swh.storge.get_datastore()`` function
      - add ``swh.storage.postgreql.storage.Storage.get_current_version()`` method
      - move sql migration scripts in ``swh/storage/sql/upgrades``
      - modify sql initialization scripts to match swh.core 2 (remove
        dbversion management code).
    
    - Update tests to use the new template-based database handling; this
      should have only minimal impact on test execution performances.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1559/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1559/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 11 2022, 4:20 PM

Harbormaster failed remote builds in B26830: Diff 25954!

douardda retitled this revision from Update for swh.core 2.0.0 to Update storage for swh.core 2.0.0.Feb 11 2022, 5:13 PM

Build is green

Patch application report for D7160 (id=25954)

Rebasing onto f578377db5...

Current branch diff-target is up to date.

Changes applied before test

commit 035c34911170ae9766287f4e40c9815134cc0a54
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 2 15:56:18 2022 +0100

    Update for swh.core 2.0.0
    
    - Add expected entry points for swh.core 2 db handling new features:
    
      - add a ``swh.storge.get_datastore()`` function
      - add ``swh.storage.postgreql.storage.Storage.get_current_version()`` method
      - move sql migration scripts in ``swh/storage/sql/upgrades``
      - modify sql initialization scripts to match swh.core 2 (remove
        dbversion management code).
    
    - Update tests to use the new template-based database handling; this
      should have only minimal impact on test execution performances.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1560/ for more details.

Harbormaster completed remote builds in B26830: Diff 25954.Feb 23 2022, 11:38 AM

douardda requested review of this revision.Feb 23 2022, 11:38 AM

ardumont accepted this revision.Feb 23 2022, 6:18 PM

This revision is now accepted and ready to land.Feb 23 2022, 6:18 PM

Add types-toml to requirements-test.txt (in a dedicated revision)

This revision was landed with ongoing or failed builds.Feb 24 2022, 11:14 AM

Closed by commit rDSTO386fb4d61f19: Add types-toml to requirements-test.txt (authored by douardda). · Explain Why

This revision was automatically updated to reflect the committed changes.

douardda added a commit: rDSTO386fb4d61f19: Add types-toml to requirements-test.txt.

douardda added a commit: rDSTO215162b2c129: Update for swh.core 2.0.0.

Build has FAILED

Patch application report for D7160 (id=26232)

Rebasing onto f578377db5...

Current branch diff-target is up to date.

Changes applied before test

commit 215162b2c12962803188220f4bef8bcfd710905a
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 2 15:56:18 2022 +0100

    Update for swh.core 2.0.0
    
    - Add expected entry points for swh.core 2 db handling new features:
    
      - add a ``swh.storge.get_datastore()`` function
      - add ``swh.storage.postgreql.storage.Storage.get_current_version()`` method
      - move sql migration scripts in ``swh/storage/sql/upgrades``
      - modify sql initialization scripts to match swh.core 2 (remove
        dbversion management code).
    
    - Update tests to use the new template-based database handling; this
      should have only minimal impact on test execution performances.

commit 386fb4d61f190dfd90d0af30735f060c6dc4732e
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Feb 24 11:00:49 2022 +0100

    Add types-toml to requirements-test.txt

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1561/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1561/console

Harbormaster failed remote builds in B27104: Diff 26232!Feb 24 2022, 11:16 AM