Page MenuHomeSoftware Heritage
Create Task
Maniphest T486

DB storage cleanup
Closed, MigratedEdits Locked
Actions

Description

Our current mount point for our main db is saturated (the one for the worker and currently for the archiver in its main design).

But, as it turns out, we have softwareheritage-log db which is around 1.1Tb (this is one quarter of the available space for the main cluster).

So we can:

  • clean up a bit (by removing logs with log level > info older than 1 month).
  • dump the db softwareheritage-log
  • restore that dump in the other cluster (the other mount point which is standard hdd with much space).

-> The db will be compressed to what's needed only so less than 1.1Tb this time.

  • drop softwareheritage-log in the main cluster
  • Update /etc/pbbouncer/pgbouncer.ini with the new db port to keep serving softwareheritage-log

Note:

  • As we use pgbouncer, this should not change anything downstream.

Revisions and Commits

rDWAPPS Web applications
ClosedD76 Remove dead code related to file upload (T468)

Related Objects
Search...

Event Timeline

ardumont created this task.Jul 12 2016, 6:42 PM
ardumont mentioned this in T412: Bootstrap archiver's database.Jul 12 2016, 7:00 PM
zack added a subscriber: olasd.Jul 13 2016, 9:01 AM

Thanks for opening this task, I agree this is the way to go. Once done we can easily deploy the archiver as it is supposed to work in the long-term, reducing the need of further migrations down the road.

There are several sub-tasks here though, that I think should be untangled:

  1. moving the log DB from the SSD-based DB to the spinning-drive one
  2. removing old verbose log entries (level > info)
  3. reducing the current log configuration of our various services so that log size becomes more manageable in the medium term

What is blocking for T240 is probably only (1). But if you start looking into this (which I recommend you do), it'd be nice to fix all of it, while the context is clear.

ardumont mentioned this in T487: Moving the log DB from the SSD-based DB to the spinning-drive one.Jul 13 2016, 10:52 AM
ardumont created subtask T487: Moving the log DB from the SSD-based DB to the spinning-drive one.
ardumont mentioned this in T488: Removing old verbose log entries (level > info).
ardumont created subtask T488: Removing old verbose log entries (level > info).
ardumont created subtask T489: Reduce current log configuration of various services.
ardumont renamed this task from db storage cleanup to DB storage cleanup.Jul 13 2016, 10:55 AM
ardumont removed a parent task: T240: content archiver.
ardumont updated the task description. (Show Details)Jul 13 2016, 11:23 AM
ardumont changed the status of subtask T487: Moving the log DB from the SSD-based DB to the spinning-drive one from Open to Work in Progress.Jul 13 2016, 11:36 AM
ardumont closed subtask T488: Removing old verbose log entries (level > info) as Resolved.
ardumont added a comment.Jul 13 2016, 11:39 AM

But if you start looking into this (which I recommend you do), it'd be nice to fix all of it, while the context is clear.

Agreed and on it ^^

Note for readers:

  • The bullet points do not represent order, it's merely references to get back to ^^
ardumont updated the task description. (Show Details)Jul 13 2016, 2:02 PM
jbertran added a revision: D76: Remove dead code related to file upload (T468).Jul 13 2016, 4:17 PM
ardumont changed the status of subtask T489: Reduce current log configuration of various services from Open to Work in Progress.Jul 14 2016, 12:33 PM
ardumont closed subtask T487: Moving the log DB from the SSD-based DB to the spinning-drive one as Resolved.Jul 16 2016, 10:09 AM
ardumont closed this task as Resolved.Jul 21 2016, 11:19 AM
ardumont closed subtask T489: Reduce current log configuration of various services as Resolved.
jbertran mentioned this in rDWAPPS6007da59720f: Remove dead file upload code missed in D76.Aug 16 2016, 10:42 AM
jbertran mentioned this in R65:6007da59720f: Remove dead file upload code missed in D76.Feb 23 2019, 1:55 AM
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:19 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T487: Moving the log DB from the SSD-based DB to the spinning-drive one from Resolved to Migrated.Jan 8 2023, 4:19 PM
gitlab-migration changed the status of subtask T488: Removing old verbose log entries (level > info) from Resolved to Migrated.
gitlab-migration changed the status of subtask T489: Reduce current log configuration of various services from Resolved to Migrated.
About · Developer information · Legal