Page MenuHomeSoftware Heritage
Create Task
Maniphest T2865

Prepare an environment to test the ClearlyDefined integration
Closed, MigratedEdits Locked
Actions

Description

ClearlyDefined[1] project could help to generate extrinsic metadata of the archive content.
The service can be accessed via a rest api[2] but there is a rate limiting in place.
To get rid of this limit, a miror can be installed on our infrastructure and keep in sync with a ingestion proxy [3]

An intern will work on this subject. We should provide a VM with enough disk space and memory to create it.
a preconfigured postgresql could be useful too,

The disk space needed is estimated to 2To (cf homepage of the github project[3])

Steps:

  • T2890: Onboard tg19999
  • D4745: Create db instance on staging for now [4]
  • D4753, D4754 : Create vm instance with db access

[1] https://clearlydefined.io/
[2] https://api.clearlydefined.io/api-docs/#/definitions/get_definitions
[3] https://github.com/nexB/clearcode-toolkit
[4] uffizi has some disk limitations in the end

Revisions and Commits

rSPRE sysadm-provisioning
D4754rSPRE0cc4a1d72f75 staging: Add clearly-defined node
rSPSITE puppet-swh-site
D4753rSPSITE2fbf746bab01 Add clearly-defined vm role to access the staging clearly defined db instance
D4745rSPSITE89d439f119be staging: Add clearly-defined postgresql instance

Related Objects
Search...

Event Timeline

vsellier triaged this task as Normal priority.Dec 8 2020, 5:57 PM
vsellier created this task.
vsellier mentioned this in T2868: Integrate former ceph-mon1 server to the proxmox cluster.Dec 8 2020, 6:09 PM
ardumont mentioned this in rSPPRIVC2fd73771ad81: Add clearly defined db instance password.Dec 15 2020, 4:02 PM
ardumont added a revision: D4745: staging: Add clearly-defined postgresql instance.Dec 15 2020, 4:06 PM
ardumont updated the task description. (Show Details)Dec 15 2020, 4:11 PM
ardumont added a commit: rSPSITE89d439f119be: staging: Add clearly-defined postgresql instance.Dec 15 2020, 5:19 PM
ardumont updated the task description. (Show Details)Dec 15 2020, 5:45 PM
ardumont added a revision: D4753: Add clearly-defined vm role to access the staging clearly defined db instance.Dec 16 2020, 12:52 PM
ardumont updated the task description. (Show Details)Dec 16 2020, 1:37 PM
ardumont added a revision: D4754: staging: Add clearly-defined node.Dec 16 2020, 1:47 PM
ardumont updated the task description. (Show Details)Dec 16 2020, 1:48 PM
ardumont added a commit: rSPSITE2fbf746bab01: Add clearly-defined vm role to access the staging clearly defined db instance.Dec 16 2020, 3:10 PM
ardumont mentioned this in rSPSITEdee3f241eefc: common: Replace old public key with the new one.Dec 16 2020, 3:27 PM
ardumont added a commit: rSPRE0cc4a1d72f75: staging: Add clearly-defined node.Dec 16 2020, 3:51 PM
ardumont changed the task status from Open to Work in Progress.Dec 16 2020, 6:38 PM
ardumont updated the task description. (Show Details)
ardumont mentioned this in rSPSITE9b324ac18cc6: clearly-defined: Fix dbname typo (dash not underscore).Dec 16 2020, 7:00 PM
ardumont added a subscriber: ardumont.Dec 16 2020, 7:03 PM
  • vm access with db access:
root@clearly-defined:~# su - tg1999
tg1999@clearly-defined:~$ psql service=clearly-defined
psql (13.1 (Debian 13.1-1.pgdg100+1), server 12.4 (Debian 12.4-1.pgdg100+1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

clearly-defined=> \conninfo
You are connected to database "clearly-defined" as user "guest" on host "db1.internal.staging.swh.network" (address "192.168.130.11") at port "5432".
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
clearly-defined=> \q
tg1999@clearly-defined:~$ psql service=admin-clearly-defined
psql (13.1 (Debian 13.1-1.pgdg100+1), server 12.4 (Debian 12.4-1.pgdg100+1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

clearly-defined=> \conninfo
You are connected to database "clearly-defined" as user "guest" on host "db1.internal.staging.swh.network" (address "192.168.130.11") at port "5432".
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
ardumont added a comment.Dec 17 2020, 3:05 PM

Following the onboarding task of tg1999, we checked he could connect to the node clearly-defined and access the db.
So it's all good now.

ardumont updated the task description. (Show Details)Dec 17 2020, 3:05 PM
ardumont removed a subscriber: ardumont.
ardumont added a comment.Dec 17 2020, 3:34 PM

The main part of this task is done.

Note that:

  • the disk on the vm clearly-defined [1] may need disk increase [2]
  • there is no current limitation for the db instance regarding the size though (we have more than 2To of space there, it's also shared with other staging db instances)

[1] clearly-defined.internal.staging.swh.network

[2] looking at the clearsync tool, there is a way to sync directly to db
afawui so it should be fine

clearsync --save-to-db  --verbose -n3
ardumont closed this task as Resolved.Dec 17 2020, 3:34 PM
ardumont claimed this task.
ardumont mentioned this in rSENVaddf98947be1: Add clearly-defined node.Dec 18 2020, 12:02 PM
ardumont mentioned this in rSPSITE3f807415c0f6: clearly-defined: Define the main admin user connection.Dec 18 2020, 8:57 PM
ardumont changed the status of subtask T2866: Integrate former Uffizi server to the proxmox cluster from Open to Work in Progress.Jan 6 2021, 11:05 AM
ardumont moved this task from Backlog to deployed/landed/monitoring on the System administration board.Jan 6 2021, 3:45 PM
vsellier closed subtask T2866: Integrate former Uffizi server to the proxmox cluster as Resolved.Jan 19 2021, 7:51 PM
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.Apr 21 2021, 6:57 PM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:00 PM
gitlab-migration claimed this task.
gitlab-migration added subscribers: ardumont, gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T2866: Integrate former Uffizi server to the proxmox cluster from Resolved to Migrated.Oct 19 2022, 6:00 PM
gitlab-migration changed the status of subtask T2868: Integrate former ceph-mon1 server to the proxmox cluster from Duplicate to Migrated.
About · Developer information · Legal