Page MenuHomeSoftware Heritage

scrubber does not comply to what's expected by the swh db tooling
Closed, MigratedEdits Locked

Description

After crafting of a "proper" config.yml [1] because otherwise it complains [2].

It still complains about missing implementations [3] [4].

This currently prevents migration with the proper tooling as per documentation [5].

Current version of the used scrubber is the latest v0.0.3 [6]

[1] config.yml:

swhworker@scrubber0:~$ cat config.yml
---
scrubber:
  cls: postgresql
  db: host=<host> port=<port> dbname=swh-scrubber user=swh-scrubber password=<password>

[2]

swhworker@scrubber0:~$ swh db --config-file /etc/softwareheritage/scrubber/postgres.yml upgrade scrubber --to-version=2
Usage: swh db upgrade [OPTIONS] MODULE

Error: Invalid value: Configuration cls must be set to 'postgresql' for this command.

[3]

swhworker@scrubber0:~$ swh db --config-file config.yml version scrubber
module: scrubber
Traceback (most recent call last):
  File "/usr/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core==2.8.0', 'console_scripts', 'swh')())
  File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 184, in main
    return swh(auto_envvar_prefix="SWH")
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/swh/core/cli/db.py", line 302, in db_version
    code_version = datastore.get_current_version()
AttributeError: 'ScrubberDb' object has no attribute 'get_current_version'

[4]

swhworker@scrubber0:~$ swh db --config-file config.yml upgrade scrubber --to-version=2
Traceback (most recent call last):
  File "/usr/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core==2.8.0', 'console_scripts', 'swh')())
  File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 184, in main
    return swh(auto_envvar_prefix="SWH")
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/swh/core/cli/db.py", line 390, in db_upgrade
    ds_version = datastore.get_current_version()
AttributeError: 'ScrubberDb' object has no attribute 'get_current_version'

[5] https://docs.softwareheritage.org/devel/swh-core/db.html

[6]

swhworker@scrubber0:~$ dpkg -l python3-swh.scrubber | grep ii
ii  python3-swh.scrubber 0.0.3-1~swh1~bpo10+1 all          Software Heritage Datastore Scrubber
ii  python3-swh.core 2.8.0-1~swh1~bpo10+1 all          Software Heritage core utilities

Event Timeline

ardumont triaged this task as Normal priority.May 30 2022, 4:03 PM
ardumont created this task.
ardumont updated the task description. (Show Details)

How great, now, we got the following with D7910 applied:

swhworker@scrubber0:~$ swh db --config-file /etc/softwareheritage/scrubber/postgres.yml upgrade scrubber --to-version=2 --module-config-key=scrubber_db
Traceback (most recent call last):
  File "/usr/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core==2.8.0', 'console_scripts', 'swh')())
  File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 184, in main
    return swh(auto_envvar_prefix="SWH")
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/swh/core/cli/db.py", line 412, in db_upgrade
    new_db_version = swh_db_upgrade(dbname, module, to_version)
  File "/usr/lib/python3/dist-packages/swh/core/db/db_utils.py", line 181, in swh_db_upgrade
    for fname in get_sql_for_package(modname, upgrade=True)
  File "/usr/lib/python3/dist-packages/swh/core/db/db_utils.py", line 508, in get_sql_for_package
    "Module {} does not provide a db schema (no sql/ dir)".format(modname)
ValueError: Module scrubber does not provide a db schema (no sql/ dir)

Scrubber got finally adapted so migration script is now ok [1]
Closing this.

[1]

swhworker@scrubber0:~$ swh db --config-file /etc/softwareheritage/scrubber/postgres.yml upgrade scrubber --to-version=2 --module-config-key=scrubber_db
INFO:swh.core.db.db_utils:Executing migration script '/usr/lib/python3/dist-packages/swh/scrubber/sql/upgrades/2.sql'

swhworker@scrubber0:~$ swh db --config-file /etc/softwareheritage/scrubber/postgres.yml version scrubber --module-config-key=scrubber_db
module: scrubber
current code version: 2
version: 2
ardumont claimed this task.