While I was locally playing to change swh-web database backend from SQLite to PostgreSQL,
I stumbled across that error while trying to load a JSON dump of the webapp database into
the new Postgres one.
17:25 $ django-admin loaddata datadump.json --settings=swh.web.settings.development
[08/Jan/2021 16:26:07] [DEBUG] urllib3.util.retry.from_int:332 - Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
[08/Jan/2021 16:26:07] [DEBUG] urllib3.util.retry.from_int:332 - Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
Traceback (most recent call last):
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/serializers/json.py", line 69, in Deserializer
yield from PythonDeserializer(objects, **options)
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/serializers/python.py", line 92, in Deserializer
Model = _get_model(d["model"])
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/serializers/python.py", line 154, in _get_model
return apps.get_model(model_identifier)
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/apps/registry.py", line 206, in get_model
app_label, model_name = app_label.split(".")
ValueError: too many values to unpack (expected 2)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/anlambert/.virtualenvs/swh/bin/django-admin", line 8, in <module>
sys.exit(execute_from_command_line())
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
self.loaddata(fixture_labels)
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
self.load_label(fixture_label)
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 172, in load_label
for obj in objects:
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/core/serializers/json.py", line 73, in Deserializer
raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/home/anlambert/swh/swh-environment/swh-web/datadump.json':This is due to the fact that swh-web declares the following labels for
some custom django applications: swh.web.common, swh.web.auth.
But Django application label must be a valid Python identifier and thus must
not contain dot characters.
As a result, having invalid identifiers will make database data importing from JSON
fails.
This was not clearly indicated in Django documentation and application label validity
check has been added a couple of weeks ago in upstream Django source code.
https://code.djangoproject.com/ticket/32285
That diff renames the invalid django application labels and ensure migrations can
still be applied.
In order to apply the renaming changes to the swh-web instance in production,
the migration command will have to be the following:
django-admin migrate --settings=swh.web.settings.production --fake
This will simulate the migrations but without any effects apart filling the django migrations
table with the new application labels.
Calling migrate without the fake option will raise errors otherwise as the changes
have already been applied to database, see below.
17:38 $ django-admin migrate --settings=swh.web.settings.development
[08/Jan/2021 16:38:12] [DEBUG] urllib3.util.retry.from_int:332 - Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
[08/Jan/2021 16:38:12] [DEBUG] urllib3.util.retry.from_int:332 - Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
Operations to perform:
Apply all migrations: admin, auth, contenttypes, sessions, swh_web_auth, swh_web_common
Running migrations:
Applying swh_web_auth.0001_initial...Traceback (most recent call last):
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/db/backends/utils.py", line 82, in _execute
return self.cursor.execute(sql)
File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 381, in execute
return Database.Cursor.execute(self, query)
sqlite3.OperationalError: table "oidc_user_offline_tokens" already existsThis could be implemented the following way in debian/postinst:
#!/bin/bash set -e if [ -f /var/lib/swh/web.sqlite3 ]; then echo "Software Heritage production db detected; running migrations" if [ -d /usr/lib/python3/dist-packages/swh.web-0.0.279.egg-info/ ] then # use fake migrations after django app labels renaming in v0.0.279 django-admin migrate --settings=swh.web.settings.production --fake else django-admin migrate --settings=swh.web.settings.production fi fi
Related to T2945