Change Details

Now that the "Save code now" service gets more and more used, we started to encounter some SQLite concurrency issue, see T2891 and [[ https://sentry.softwareheritage.org/organizations/swh/issues/4464/?project=13&query=is%3Aunresolved+OperationalError&statsPeriod=90d | sentry related issue ]]. This calls for the use of a better Django database backend in production, PostgreSQL being the obvious choice. This task will track the actions that need to be done to perform that migration process, notably: - the creation of a `swh-web` database in a PostgreSQL server somewhere in swh staging / production environment - the adaptation of Django production settings and `swh-web` configuration to use that database - the dump of current `swh-web` SQLite database content to JSON - the load from those JSON data into the new `swh-web` PostgreSQL database Plan or action so far: - [X] D5392: Migrate swh.web settings to declare the use of postgres - [X] D5391: Ensure migration tools and setup works in docker - [X] D5393: staging: Prepare db (puppet) - [ ] production: Prepare new swh-web db (manual creation) - [ ] Prepare swh-site setup change to webapps - [ ] D5402: staging - [ ] production - [ ] production: Deactivate puppet on moma - [ ] Land, tag package swh-web (so the migration tools is available on nodes to migrate) - [ ] production: Run swh-web upgrade and migrate previous data dump on new db - [ ] production: Run upgrade on moma (reactivate puppe)

Now that the "Save code now" service gets more and more used, we started to encounter some SQLite concurrency issue, see T2891 and [[ https://sentry.softwareheritage.org/organizations/swh/issues/4464/?project=13&query=is%3Aunresolved+OperationalError&statsPeriod=90d | sentry related issue ]]. This calls for the use of a better Django database backend in production, PostgreSQL being the obvious choice. This task will track the actions that need to be done to perform that migration process, notably: - the creation of a `swh-web` database in a PostgreSQL server somewhere in swh staging / production environment - the adaptation of Django production settings and `swh-web` configuration to use that database - the dump of current `swh-web` SQLite database content to JSON - the load from those JSON data into the new `swh-web` PostgreSQL database Plan or action so far: - [X] D5392: Migrate swh.web settings to declare the use of postgres - [X] D5391: Ensure migration tools and setup works in docker - [X] D5393: staging: Prepare db (puppet) - [X] production: Deactivate puppet on moma, webapp1 - [ ] D5402: staging: Prepare swh-site setup change to webapps - [ ] Land, tag package swh-web (so the migration tools is available on nodes to migrate) - [ ] Load the dump on new db - [ ] Check everything is fine in webapp.staging.swh.network ---- - [ ] production: Prepare swh-site setup change to webapps - [ ] production: Prepare new swh-web db (manual creation) - [ ] production: Run swh-web upgrade and migrate previous data dump on new db - [ ] production: Run upgrade on moma (reactivate puppe)

Now that the "Save code now" service gets more and more used, we started to encounter some SQLite concurrency issue, see T2891 and [[ https://sentry.softwareheritage.org/organizations/swh/issues/4464/?project=13&query=is%3Aunresolved+OperationalError&statsPeriod=90d | sentry related issue ]]. This calls for the use of a better Django database backend in production, PostgreSQL being the obvious choice. This task will track the actions that need to be done to perform that migration process, notably: - the creation of a `swh-web` database in a PostgreSQL server somewhere in swh staging / production environment - the adaptation of Django production settings and `swh-web` configuration to use that database - the dump of current `swh-web` SQLite database content to JSON - the load from those JSON data into the new `swh-web` PostgreSQL database Plan or action so far: - [X] D5392: Migrate swh.web settings to declare the use of postgres - [X] D5391: Ensure migration tools and setup works in docker - [X] D5393: staging: Prepare db (puppet) - [ - [X] production: Prepare new swh-web db (manual creation)Deactivate puppet on moma, webapp1 - [ ] D5402: staging: Prepare swh-site setup change to webapps - [ ] D5402: staging- [ ] Land, tag package swh-web (so the migration tools is available on nodes to migrate) - [ ] production- [ ] Load the dump on new db - [ ] Check everything is fine in webapp.staging.swh.network ---- - [ ] production: Deactivate puppet on momaPrepare swh-site setup change to webapps - [ ] Land, tag packageproduction: Prepare new swh-web (so the migration tools is available on nodes to migratedb (manual creation) - [ ] production: Run swh-web upgrade and migrate previous data dump on new db - [ ] production: Run upgrade on moma (reactivate puppe)