Page MenuHomeSoftware Heritage
Differential D816

swh-db-init: Make the db initialization idempotent
ClosedPublic
Actions

Authored by ardumont on Dec 14 2018, 12:03 PM.

Details

Reviewers
douardda
vlorentz
Group Reviewers
Reviewers
Commits
rDCORE829b92cc9ad3: db_testing: Simplify pg_createdb
rDCORE590aeefbc832: core.cli: Reference the possibility to migrate to latest version
rDCORE6aa0f32547ee: swh.core.cli: Remove no longer used --create/--no-create flags
rDCOREdb7c1d0b322b: swh-db-init: Make the db initialization idempotent
rDCORE6eea4a05588c: tests.db_testing: Add a function to read swh's db version
Summary

The first time, the db will be initialized.
The next time this is called, this will do nothing.

Related D815
Related T1444

Diff Detail

Repository
rDCORE Foundations and core functionalities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 3026
Build 3864: tox-on-jenkinsJenkins
Build 3863: arc lint + arc unit

Event Timeline

ardumont created this revision.Dec 14 2018, 12:03 PM
swh-public-ci added a subscriber: swh-public-ci.Dec 14 2018, 12:05 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCORE/job/tox/102/ for more details.

Harbormaster completed remote builds in B3025: Diff 2602.Dec 14 2018, 12:05 PM
ardumont updated this revision to Diff 2603.Dec 14 2018, 12:07 PM

Fix print statements

swh-public-ci added a comment.Dec 14 2018, 12:09 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCORE/job/tox/107/ for more details.

Harbormaster completed remote builds in B3026: Diff 2603.Dec 14 2018, 12:09 PM
ardumont added a child revision: D817: storage/entrypoint.sh: Make the storage initialization transparent.Dec 14 2018, 12:18 PM
ardumont marked 2 inline comments as done.Dec 14 2018, 12:21 PM
ardumont added inline comments.
swh/core/cli.py
28

Oh yes, this can be removed, it tries to create the db each time without breaking if it cannot now.

80

A priori the directive to include the sql upgrade scripts are there so the swh.* pypi service artifact should contain the necessary sql scripts for it.

ardumont updated this revision to Diff 2606.Dec 14 2018, 1:25 PM
  • swh.core.cli: Remove no longer used --create/--no-create flags
ardumont marked an inline comment as done.Dec 14 2018, 1:26 PM
ardumont added inline comments.
swh/core/cli.py
28

done

swh-public-ci added a comment.Dec 14 2018, 1:27 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCORE/job/tox/109/ for more details.

Harbormaster completed remote builds in B3029: Diff 2606.Dec 14 2018, 1:27 PM
vlorentz accepted this revision.Dec 14 2018, 1:33 PM
vlorentz added inline comments.
swh/core/cli.py
76

redundant test

swh/core/tests/db_testing.py
16–40

Can't we use psycopg2 for that?

77–78

Starting with Python 3.5, you can do this instead:

subprocess.run(['createdb', dbname], check=check)
This revision is now accepted and ready to land.Dec 14 2018, 1:33 PM
ardumont marked 3 inline comments as done.Dec 14 2018, 1:36 PM
ardumont added inline comments.
swh/core/cli.py
76

It's to have the db version when we are done migrating the schema.
The log at the end of the file will display at what version we are.

swh/core/tests/db_testing.py
16–40

I tried but then that would mean i'd need to change the input.
As dbname is not enough.

At the moment, we can both use dbname and a string like 'service=swh' (providing you have the right .pgass and .pg_service.conf installed).

77–78

So true!

ardumont marked an inline comment as done.Dec 14 2018, 1:37 PM
ardumont added inline comments.
swh/core/cli.py
76

s/migrating/initializing/

ardumont updated this revision to Diff 2607.Dec 14 2018, 1:40 PM
  • db_testing: Simplify pg_createdb
swh-public-ci added a comment.Dec 14 2018, 1:42 PM

Build is green
See https://jenkins.softwareheritage.org/job/DCORE/job/tox/110/ for more details.

Harbormaster completed remote builds in B3030: Diff 2607.Dec 14 2018, 1:42 PM
Closed by commit rDCORE6eea4a05588c: tests.db_testing: Add a function to read swh's db version (authored by ardumont). · Explain WhyDec 14 2018, 1:43 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added inline comments.Dec 14 2018, 1:43 PM
swh/core/cli.py
76

I know. But since not db_version is always True at this point, you can replace this:

if not db_version:
    db_version = swh_db_version(db_name)

with this:

db_version = swh_db_version(db_name)
ardumont marked an inline comment as done.Dec 14 2018, 1:44 PM
ardumont added inline comments.
swh/core/cli.py
76

Right!

vlorentz added a comment.EditedDec 14 2018, 1:44 PM

What happened to this diff? changes to swh/core/cli.py disappeared

ardumont added a comment.Dec 14 2018, 1:45 PM
In D816#17415, @vlorentz wrote:

What happened to this diff? changes to swh/core/cli.py disappeared?

I pushed the commits.
So that closes it (good).
and then somehow that screws everything in the diff view (bad).

ardumont mentioned this in rDCORE36b28c10cedc: swh.core.cli: Remove redundant check.Dec 14 2018, 1:48 PM
ardumont mentioned this in D818: Add scheduler services.Dec 14 2018, 2:57 PM
ardumont marked an inline comment as done.Dec 17 2018, 10:05 AM
ardumont added inline comments.
swh/core/tests/db_testing.py
16–40

What i meant was, in the current context it's used (swh.core.cli), this is fine.
If we want to actually implement using psycopg2, we will have to draw a not so simple refactoring (including the swh.core.cli which is not tested yet).

Revision Contents
Changeset List

PathSize
swh/
core/
34 lines
tests/
40 lines

Diff 2603

View Options

swh/core/cli.py

Loading...
View Options

swh/core/tests/db_testing.py

Loading...
About · Developer information · Legal