Page MenuHomeSoftware Heritage

Make postgresql's Storage client options configurable from config
ClosedPublic

Authored by douardda on Mar 30 2022, 2:50 PM.

Details

Summary

Adding a query_options member to postgresql's Storage, in conjunction
with swh.core >= 2.5, allows to set/overwrite SQL client options from
the storage configuration file.

Default values are set, as they used to be, from the decorator
arguments. But in addition to this, one can overload these value at run
time from the storage configuration file. For example:

.. code-block:: yaml

storage:
  cls: postgresql
  db: testdb
  objstorage:
    cls: memory
  query_options:
    directory_ls:
      statement_timeout: 180000

will provide a Storage instance for which the timeout value for the
directory_ls endpoint is 3mn (instead of the default 20s).

See D7472 for the swh.core counter part needed for this feature.

Diff Detail

Event Timeline

Build is green

Patch application report for D7473 (id=27110)

Rebasing onto 835feb6842...

Current branch diff-target is up to date.
Changes applied before test
commit 35f036c6b2dbc9f5a4c4e236b763ed6374613d76
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 30 14:39:28 2022 +0200

    Make postgresql's Storage client options configurable from config
    
    Adding a `query_options` member to postgresql's Storage, in conjunction
    with swh.core >= 2.5, allows to set/overwrite SQL client options from
    the storage configuration file.
    
    Default values are set, as they used to be, from the decorator
    arguments. But in addition to this, one can overload these value at run
    time from the storage configuration file. For example:
    
    .. code-block:: yaml
    
       storage:
         cls: postgresql
         db: testdb
         objstorage:
           cls: memory
         query_options:
           directory_ls:
             statement_timeout: 180000
    
    will provide a Storage instance for which the timeout value for the
    `directory_ls` endpoint is 3mn (instead of the default 20s).

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1585/ for more details.

Please add a docstring; it's unclear this is used by swh-core otherwise

This revision is now accepted and ready to land.Mar 30 2022, 3:12 PM

Please add a docstring; it's unclear this is used by swh-core otherwise

Oh I missed this comment, sorry... so I guess, now I need to actually write a docstring for this class, instead of the current mostly unrelated one... thx :-)

Build has FAILED

Patch application report for D7473 (id=27188)

Rebasing onto 835feb6842...

Current branch diff-target is up to date.
Changes applied before test
commit ad9e2a70bc65a3f129dd8735dc489bc73fac12ce
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 30 14:39:28 2022 +0200

    Make postgresql's Storage client options configurable from config
    
    Adding a `query_options` member to postgresql's Storage, in conjunction
    with swh.core >= 2.5, allows to set/overwrite SQL client options from
    the storage configuration file.
    
    Default values are set, as they used to be, from the decorator
    arguments. But in addition to this, one can overload these value at run
    time from the storage configuration file. For example:
    
    .. code-block:: yaml
    
       storage:
         cls: postgresql
         db: testdb
         objstorage:
           cls: memory
         query_options:
           directory_ls:
             statement_timeout: 180000
    
    will provide a Storage instance for which the timeout value for the
    `directory_ls` endpoint is 3mn (instead of the default 20s).

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1586/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1586/console

olasd added inline comments.
swh/storage/postgresql/storage.py
120–121

After playing with tox -e sphinx-dev I came up with this...

better docstring (thx olasd)

Build is green

Patch application report for D7473 (id=27195)

Rebasing onto 835feb6842...

Current branch diff-target is up to date.
Changes applied before test
commit c6dc5cd3b58a59b58ed136f6708dc7df461f7d14
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 30 14:39:28 2022 +0200

    Make postgresql's Storage client options configurable from config
    
    Adding a `query_options` member to postgresql's Storage, in conjunction
    with swh.core >= 2.5, allows to set/overwrite SQL client options from
    the storage configuration file.
    
    Default values are set, as they used to be, from the decorator
    arguments. But in addition to this, one can overload these value at run
    time from the storage configuration file. For example:
    
    .. code-block:: yaml
    
       storage:
         cls: postgresql
         db: testdb
         objstorage:
           cls: memory
         query_options:
           directory_ls:
             statement_timeout: 180000
    
    will provide a Storage instance for which the timeout value for the
    `directory_ls` endpoint is 3mn (instead of the default 20s).

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1587/ for more details.