Page MenuHomeSoftware Heritage

Storage managerFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Details

Recent Activity

Today

vlorentz reopened Restricted Maniphest Task, a subtask of T3841: regularly scrub all the data stores of swh, as Open.
Mon, Aug 8, 2:23 PM · meta-task, Roadmap 2022, Storage manager

Fri, Aug 5

ardumont added a subtask for T3841: regularly scrub all the data stores of swh: T4371: Deploy swh-scrubber on all storage instances.
Fri, Aug 5, 4:31 PM · meta-task, Roadmap 2022, Storage manager
ardumont added a parent task for T4371: Deploy swh-scrubber on all storage instances: T3841: regularly scrub all the data stores of swh.
Fri, Aug 5, 4:31 PM · System administration, Archive integrity, Storage manager
vlorentz closed Restricted Maniphest Task, a subtask of T3841: regularly scrub all the data stores of swh, as Resolved.
Fri, Aug 5, 3:07 PM · meta-task, Roadmap 2022, Storage manager

Thu, Aug 4

ardumont closed T4371: Deploy swh-scrubber on all storage instances as Resolved.
Thu, Aug 4, 3:57 PM · System administration, Archive integrity, Storage manager
ardumont moved T4371: Deploy swh-scrubber on all storage instances from in-progress to deployed/landed/monitoring on the System administration board.
Thu, Aug 4, 3:49 PM · System administration, Archive integrity, Storage manager
ardumont added a comment to T4371: Deploy swh-scrubber on all storage instances.

Deployed both in staging and production [1]:

Thu, Aug 4, 3:49 PM · System administration, Archive integrity, Storage manager
ardumont changed the status of T4371: Deploy swh-scrubber on all storage instances from Open to Work in Progress.
Thu, Aug 4, 3:03 PM · System administration, Archive integrity, Storage manager
ardumont added a revision to T4371: Deploy swh-scrubber on all storage instances: D8181: scrubber: Make service parametric on the db instance to scrub.
Thu, Aug 4, 3:03 PM · System administration, Archive integrity, Storage manager
ardumont moved T4371: Deploy swh-scrubber on all storage instances from Backlog to Weekly backlog on the System administration board.
Thu, Aug 4, 11:34 AM · System administration, Archive integrity, Storage manager

Wed, Jul 13

vsellier added a comment to T4373: [cassandra] Test the new hardware.

Unfortunately, the operator test is a failure due to the lake of configuration possibility

  • non blocker, the init containers are OOMkilled during the start, it can be solved by editing the cassandra statefulset created by the operator to extends the limits
  • blocker, it's not possible to configure the commitlog_directory explicitely. it's by default on /var/lib/cassandra/commitlog
    • it's not easy to propagate the host mount to use 2 mountpoints /srv/cassadra and /srv/cassandra/commitlog without tweking the kernel / rancher configuration
    • it's not possible to add a second volume on the pod description created by the operator
Wed, Jul 13, 10:25 AM · Storage manager, System administration
vlorentz added a subtask for T3841: regularly scrub all the data stores of swh: Restricted Maniphest Task.
Wed, Jul 13, 9:40 AM · meta-task, Roadmap 2022, Storage manager

Tue, Jul 12

vsellier added a revision to T4373: [cassandra] Test the new hardware: D8116: Deploy the cassandra operator on the production cassandra cluster.
Tue, Jul 12, 3:01 PM · Storage manager, System administration
vsellier changed the status of T4374: [cassandra] Test basic topology, a subtask of T4379: [cassandra] create etcd / controlplane servers, from Open to Work in Progress.
Tue, Jul 12, 12:10 PM · Storage manager, System administration
vsellier changed the status of T4374: [cassandra] Test basic topology from Open to Work in Progress.
Tue, Jul 12, 12:10 PM · Storage manager, System administration
vsellier closed T4389: [cassandra] Configure the monitoring of the cluster, a subtask of T4373: [cassandra] Test the new hardware, as Resolved.
Tue, Jul 12, 12:09 PM · Storage manager, System administration

Mon, Jul 11

vsellier closed T4379: [cassandra] create etcd / controlplane servers as Resolved.

Finally, the cluster is up.
I'm not sure what unstuck the node registration, but I suspect a node with all the roles is needed to bootstrap the cluster.
I tried this initially, it didn't worked, but I'm not sure in which status the cluster was.

Mon, Jul 11, 4:33 PM · Storage manager, System administration
vsellier closed T4379: [cassandra] create etcd / controlplane servers, a subtask of T4373: [cassandra] Test the new hardware, as Resolved.
Mon, Jul 11, 4:33 PM · Storage manager, System administration
vsellier added a revision to T4373: [cassandra] Test the new hardware: D8105: Install zfs and docker on the cassandra node to prepare the cass operator tests.
Mon, Jul 11, 9:33 AM · Storage manager, System administration

Jul 7 2022

vsellier added a comment to T4379: [cassandra] create etcd / controlplane servers.

The management nodes were correctly created but it seems rancher is having some issuer to register them in the cluster.

Jul 7 2022, 6:52 PM · Storage manager, System administration
vsellier added a revision to T4379: [cassandra] create etcd / controlplane servers: D8094: Declare the kubernetes cluster and management nodes for cassandra.
Jul 7 2022, 12:11 PM · Storage manager, System administration
vsellier changed the status of T4379: [cassandra] create etcd / controlplane servers, a subtask of T4373: [cassandra] Test the new hardware, from Open to Work in Progress.
Jul 7 2022, 11:56 AM · Storage manager, System administration
vsellier changed the status of T4379: [cassandra] create etcd / controlplane servers from Open to Work in Progress.
Jul 7 2022, 11:56 AM · Storage manager, System administration

Jul 5 2022

vsellier removed a parent task for T4374: [cassandra] Test basic topology: T4373: [cassandra] Test the new hardware.
Jul 5 2022, 5:50 PM · Storage manager, System administration
vsellier removed a subtask for T4373: [cassandra] Test the new hardware: T4374: [cassandra] Test basic topology.
Jul 5 2022, 5:50 PM · Storage manager, System administration
vsellier removed a subtask for T4373: [cassandra] Test the new hardware: T4375: [cassandra] One cassandra per data disk.
Jul 5 2022, 5:50 PM · Storage manager, System administration
vsellier removed a parent task for T4375: [cassandra] One cassandra per data disk: T4373: [cassandra] Test the new hardware.
Jul 5 2022, 5:50 PM · Storage manager, System administration
vsellier added a parent task for T4374: [cassandra] Test basic topology: T4379: [cassandra] create etcd / controlplane servers.
Jul 5 2022, 5:49 PM · Storage manager, System administration
vsellier added a subtask for T4379: [cassandra] create etcd / controlplane servers: T4374: [cassandra] Test basic topology.
Jul 5 2022, 5:49 PM · Storage manager, System administration
vsellier added a parent task for T4375: [cassandra] One cassandra per data disk: T4379: [cassandra] create etcd / controlplane servers.
Jul 5 2022, 5:49 PM · Storage manager, System administration
vsellier added a subtask for T4379: [cassandra] create etcd / controlplane servers: T4375: [cassandra] One cassandra per data disk.
Jul 5 2022, 5:49 PM · Storage manager, System administration
vsellier triaged T4379: [cassandra] create etcd / controlplane servers as Normal priority.
Jul 5 2022, 5:47 PM · Storage manager, System administration
vsellier changed the status of T4373: [cassandra] Test the new hardware from Open to Work in Progress.
Jul 5 2022, 5:41 PM · Storage manager, System administration
ardumont updated the task description for T4373: [cassandra] Test the new hardware.
Jul 5 2022, 10:04 AM · Storage manager, System administration
vsellier triaged T4375: [cassandra] One cassandra per data disk as Normal priority.
Jul 5 2022, 9:52 AM · Storage manager, System administration
vsellier triaged T4374: [cassandra] Test basic topology as Normal priority.
Jul 5 2022, 9:43 AM · Storage manager, System administration
vsellier triaged T4373: [cassandra] Test the new hardware as Normal priority.
Jul 5 2022, 9:36 AM · Storage manager, System administration

Jul 4 2022

ardumont added a project to T4371: Deploy swh-scrubber on all storage instances: System administration.
Jul 4 2022, 10:47 AM · System administration, Archive integrity, Storage manager
ardumont added projects to T4371: Deploy swh-scrubber on all storage instances: Storage manager, Archive integrity.
Jul 4 2022, 10:47 AM · System administration, Archive integrity, Storage manager

Jul 1 2022

douardda triaged T4370: Refactor the origin visit data model (aka get rid of the OriginVisit model object) as High priority.
Jul 1 2022, 4:35 PM · Storage manager, Data Model
douardda triaged T4368: Loosen "foreign key" validation in storages used as mirror ingestion endpoint as High priority.
Jul 1 2022, 11:13 AM · Storage manager
douardda created T4368: Loosen "foreign key" validation in storages used as mirror ingestion endpoint.
Jul 1 2022, 11:13 AM · Storage manager

Jun 23 2022

vlorentz added a comment to T4185: Loader profiling : Add Measure of ignored objects .

the Git loader now exports a swh_loader_filtered_objects_total metric. We should generalize this to other loaders eventually; using one of the options above

Jun 23 2022, 10:40 AM · Storage manager

Jun 10 2022

douardda triaged T4325: Remove (useless) metadata_authority and metadata_fetcher from the journal as Normal priority.
Jun 10 2022, 2:51 PM · Storage manager

Jun 2 2022

vlorentz moved T3841: regularly scrub all the data stores of swh from Backlog to Work in progress on the Roadmap 2022 board.
Jun 2 2022, 9:58 AM · meta-task, Roadmap 2022, Storage manager
vlorentz moved T2214: Scale-out graph and database storage in production from Backlog to Work in progress on the Roadmap 2022 board.
Jun 2 2022, 9:57 AM · meta-task, Roadmap 2022, Roadmap 2021, Storage manager

May 31 2022

douardda closed T4286: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Resolved.
May 31 2022, 7:28 PM · Storage manager
douardda added a revision to T4286: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function : D7918: pytest_plugin: use the stock pytest_postgresql postgresql factory.
May 31 2022, 4:43 PM · Storage manager
douardda triaged T4286: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Normal priority.
May 31 2022, 4:23 PM · Storage manager

May 4 2022

olasd added a project to T4185: Loader profiling : Add Measure of ignored objects : Storage manager.
May 4 2022, 10:54 AM · Storage manager