Page MenuHomeSoftware Heritage

Make swh db init and swh db-init behavior much closer to one another
ClosedPublic

Authored by olasd on Sep 16 2020, 7:21 PM.

Details

Summary

While named similarly, these two CLI utilities used to take quite a different
approach to doing the same thing.

They now both call out the same initialization function for a given module. The
main difference is now the source of information for what databases to
initialize: swh db init uses the SWH_CONFIG_FILENAME configuration file, while
swh db-init only uses its own command line arguments.

(also sneak in a cosmetic change in a separate commit to db_testing.swh_db_version)

Test Plan
  • reviewed the use of swh db init and swh db-init in all modules and their

documentation; All documented usage is still valid.

  • ran the docker initialization with the new swh-core which worked fine

Diff Detail

Repository
rDCORE Foundations and core functionalities
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build has FAILED

Patch application report for D3970 (id=13992)

Rebasing onto 22e77212f1...

Current branch diff-target is up to date.
Changes applied before test
commit 2c086ddee9f249a0ba2ba8d47c7eff62faf24b98
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Sep 16 19:13:51 2020 +0200

    Make swh db init and swh db-init behavior much closer to one another
    
    While named similarly, these two CLI utilities used to take quite a different
    approach to doing the same thing.
    
    They now both call out the same initialization function for a given module. The
    main difference is now the source of information for what databases to
    initialize: `swh db init` uses the SWH_CONFIG_FILENAME configuration file, while
    `swh db-init` only uses its own command line arguments.

commit 327f98d1860280c9cbde86af14addf183eff112a
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Sep 16 18:57:11 2020 +0200

    Ignore stderr in db_testing.swh_db_version instead of printing it out

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/56/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/56/console

Fix flake8 issue that snuck past pre-commit thanks to rebase shenanigans

Build is green

Patch application report for D3970 (id=13993)

Rebasing onto 22e77212f1...

Current branch diff-target is up to date.
Changes applied before test
commit 6b983c28a3bea0987281cbeb48004c164f55d07c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Sep 16 19:13:51 2020 +0200

    Make swh db init and swh db-init behavior much closer to one another
    
    While named similarly, these two CLI utilities used to take quite a different
    approach to doing the same thing.
    
    They now both call out the same initialization function for a given module. The
    main difference is now the source of information for what databases to
    initialize: `swh db init` uses the SWH_CONFIG_FILENAME configuration file, while
    `swh db-init` only uses its own command line arguments.

commit 327f98d1860280c9cbde86af14addf183eff112a
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Sep 16 18:57:11 2020 +0200

    Ignore stderr in db_testing.swh_db_version instead of printing it out

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/57/ for more details.

ardumont added a subscriber: ardumont.

lgtm

one remark/question above.

Thanks.

swh/core/cli/db.py
78 ↗(On Diff #13993)

i think we could also have the case of cfg.get("db") return something directly.

Configuration can be either:

storage:
    cls: local
    args: 
      db: ...

covered here.

or with the "new" abilities from get_storage (iirc):

storage:
  cls: local
  db: ...
88 ↗(On Diff #13993)
This revision is now accepted and ready to land.Sep 17 2020, 9:15 AM
swh/core/cli/db.py
78 ↗(On Diff #13993)

Sure, this should be fixed in a followup commit.

I don't think anything actually uses swh db init anyway.