Page MenuHomeSoftware Heritage
Feed Advanced Search

Jul 6 2021

vsellier closed T3395: cassandra - Timeouts during revision import as Resolved.

The problem was solved with an increase of the default timeout in the cassandra configuration and by reducing the journal_client batch size.

Jul 6 2021, 12:44 PM · System administration, Storage manager
vsellier committed rDSNIP68cf74d2fc57: grid5000/cassandra: open unauthenticated access to cassandra (authored by vsellier).
grid5000/cassandra: open unauthenticated access to cassandra
Jul 6 2021, 12:38 PM
vsellier committed rDSNIPae799c543610: grid5000/cassandra: restore the initial number of replayers per object type (authored by vsellier).
grid5000/cassandra: restore the initial number of replayers per object type
Jul 6 2021, 12:17 PM
vsellier committed rDSNIP63cb43673e29: grid5000/cassandra: Add a statistics dashboards (authored by vsellier).
grid5000/cassandra: Add a statistics dashboards
Jul 6 2021, 12:17 PM
vsellier committed rDSNIP8b617533498b: grid5000/cassandra: increase commit log size (authored by vsellier).
grid5000/cassandra: increase commit log size
Jul 6 2021, 12:17 PM
vsellier changed the status of T3396: cassandra - allow to configure the consistency level used by the queries from Open to Work in Progress.
Jul 6 2021, 12:12 PM · System administration, Storage manager
vsellier changed the status of T3396: cassandra - allow to configure the consistency level used by the queries, a subtask of T3357: Perform some tests of the cassandra storage on Grid5000, from Open to Work in Progress.
Jul 6 2021, 12:12 PM · System administration, Storage manager
vsellier added a comment to T3357: Perform some tests of the cassandra storage on Grid5000.

A replay was run during during 13 hours the previous night with the current default consistency=ONE. It can be used as the reference for the next test with the LOCAL_QUORUM consistency.
After trying several options to render the result, the simpler was to export the content of a spreadsheet in a hedgedoc document [1] .
The data are stored in a prometheus instance on a proxmox vms so it will always possible to improve the statistics later[2].

Jul 6 2021, 11:55 AM · System administration, Storage manager

Jul 5 2021

vsellier added inline comments to D5964: origin_search: Filters and sorting for date_{created,modified,published}.
Jul 5 2021, 7:51 PM
vsellier updated the task description for T3408: Provide read-only access to production servers.
Jul 5 2021, 6:20 PM · System administration
vsellier closed D5965: swh-search: bind the service to the internal network address.
Jul 5 2021, 5:39 PM
vsellier committed rSPSITE415363937661: swh-search: bind the service to the internal network address (authored by vsellier).
swh-search: bind the service to the internal network address
Jul 5 2021, 5:39 PM
vsellier updated the test plan for D5965: swh-search: bind the service to the internal network address.
Jul 5 2021, 4:36 PM
vsellier updated the test plan for D5965: swh-search: bind the service to the internal network address.
Jul 5 2021, 4:35 PM
vsellier updated the diff for D5965: swh-search: bind the service to the internal network address.

Good catch. A variable renaming was missing.

Jul 5 2021, 4:33 PM
vsellier added a comment to P1088 swh-search 494 dates.

one result:

{
   "_index" : "origin-production",
   "_type" : "_doc",
   "_id" : "4a7ff7d5b3827d34f81c7112835928bfd2e701a1",
   "_score" : 1.0,
   "_source" : {
     "intrinsic_metadata" : [
       {
         "http://schema.org/dateModified" : [
           {
             "@value" : "2018-01-29"
           }
         ],
         "http://schema.org/datePublished" : [
           {
             "@value" : "2018-01-29"
           }
         ],
         "http://schema.org/dateCreated" : [
           {
             "@value" : "2018-01-29"
           }
         ]
       }
     ],
     "url" : "https://github.com/XinlongSBU/Pynucastro_weakrates"
   }
 },
Jul 5 2021, 3:54 PM
vsellier created P1088 swh-search 494 dates.
Jul 5 2021, 3:53 PM
vsellier committed rDSNIPd74a14befd7c: grid5000/cassandra: add annotations on graphs (authored by vsellier).
grid5000/cassandra: add annotations on graphs
Jul 5 2021, 3:26 PM
vsellier committed rDSNIP80d3f8366372: grid5000/cassandra: add read/write statistics on the system dashboard (authored by vsellier).
grid5000/cassandra: add read/write statistics on the system dashboard
Jul 5 2021, 2:56 PM
vsellier committed rDSNIP045b0a5a087a: grid5000/cassandra: automatically start the replayers on best effort nodes (authored by vsellier).
grid5000/cassandra: automatically start the replayers on best effort nodes
Jul 5 2021, 12:13 PM
vsellier committed rDSNIPa775273bb3bf: grid5000/cassandra: grafana dashboards (authored by vsellier).
grid5000/cassandra: grafana dashboards
Jul 5 2021, 12:13 PM
vsellier committed rDSNIPe32234d9a97c: grid5000/cassandra: improve perfs and resiliency (authored by vsellier).
grid5000/cassandra: improve perfs and resiliency
Jul 5 2021, 12:13 PM
vsellier committed rDSNIPdcc7cfcae65f: grid5000/cassandra: fix best effort nodes computation (authored by vsellier).
grid5000/cassandra: fix best effort nodes computation
Jul 5 2021, 12:13 PM
vsellier updated the task description for T3408: Provide read-only access to production servers.
Jul 5 2021, 12:02 PM · System administration
vsellier added a comment to T3408: Provide read-only access to production servers.

Sure, there is actually only a read-write access, but I will reopen it.

Jul 5 2021, 11:59 AM · System administration

Jul 2 2021

vsellier requested review of D5965: swh-search: bind the service to the internal network address.
Jul 2 2021, 7:35 PM
vsellier added a revision to T3408: Provide read-only access to production servers: D5965: swh-search: bind the service to the internal network address.
Jul 2 2021, 7:35 PM · System administration
vsellier updated the task description for T3408: Provide read-only access to production servers.
Jul 2 2021, 6:59 PM · System administration
vsellier committed rSPSITE5488e3ef78b0: monitoring: allow to monitor rpc service bound on a specific address (authored by vsellier).
monitoring: allow to monitor rpc service bound on a specific address
Jul 2 2021, 6:30 PM
vsellier added a comment to P1086 100 origins with programming languages.

content copied from

Jul 2 2021, 5:41 PM
vsellier created P1086 100 origins with programming languages.
Jul 2 2021, 5:40 PM
vsellier closed D5960: webapp: bind the local storage to the internal network address.
Jul 2 2021, 4:40 PM
vsellier committed rSPSITE30a27bc6af13: webapp: bind the local storage to the internal network address (authored by vsellier).
webapp: bind the local storage to the internal network address
Jul 2 2021, 4:40 PM
vsellier added inline comments to D5960: webapp: bind the local storage to the internal network address.
Jul 2 2021, 4:37 PM

Jul 1 2021

vsellier requested review of D5960: webapp: bind the local storage to the internal network address.
Jul 1 2021, 7:31 PM
vsellier added a revision to T3408: Provide read-only access to production servers: D5960: webapp: bind the local storage to the internal network address.
Jul 1 2021, 7:31 PM · System administration
vsellier committed rSENV5b7cb7238096: vagrant: fix moma installation (authored by vsellier).
vagrant: fix moma installation
Jul 1 2021, 7:23 PM
vsellier updated the task description for T3357: Perform some tests of the cassandra storage on Grid5000.
Jul 1 2021, 6:33 PM · System administration, Storage manager
vsellier committed rSENV9fd498f4fcd0: vagrant: fix moma hostname (authored by vsellier).
vagrant: fix moma hostname
Jul 1 2021, 5:21 PM
vsellier added a comment to T3357: Perform some tests of the cassandra storage on Grid5000.

A new run was launched with 4x10 replayers per object type.
A limit is still reached for the revision replayers which ends with read timeout.
A test with only the revision replayers shows the problems start to append when the number of // replayers is greater than ~20

Jul 1 2021, 1:07 PM · System administration, Storage manager
vsellier committed rDSNIP961c843d400d: grid5000/cassandra: Add new and fix existent grafana dashboards (authored by vsellier).
grid5000/cassandra: Add new and fix existent grafana dashboards
Jul 1 2021, 11:35 AM
vsellier committed rDSNIPc810356b5a45: grid5000/cassandra: ensure the data of the monitoring is up-to-date (authored by vsellier).
grid5000/cassandra: ensure the data of the monitoring is up-to-date
Jul 1 2021, 11:35 AM
vsellier added a comment to T3396: cassandra - allow to configure the consistency level used by the queries.

One example of the wrong behavior of the read consistencty level ONE for read requests:
One of the probe of the monitoring query is based on the objectcount table content.
The servers were hard stopped after the end of the grid5000 resrvation and it seems some replication messages were lost. After the restart the content of the table in not in sync on all the servers.

Jul 1 2021, 11:12 AM · System administration, Storage manager

Jun 30 2021

vsellier committed rDSNIPd41ce9665136: grid5000/cassandra: reduce write pressure for 3 nodes cluster (authored by vsellier).
grid5000/cassandra: reduce write pressure for 3 nodes cluster
Jun 30 2021, 3:48 PM
vsellier committed rDSNIP9c2f1d5f10af: grid5000/cassandra: fix start/stop scripts (authored by vsellier).
grid5000/cassandra: fix start/stop scripts
Jun 30 2021, 3:48 PM
vsellier added a comment to T3357: Perform some tests of the cassandra storage on Grid5000.

A first run was launched with 4x20 replayered per object type.
It seems it's too much for 3 cassandra nodes. The default 1s timeout for cassandra reads is often reached.
It's seems the batch size of 1000 is also too much for some object types like snapshots
A new test will be launched this night with some changes:

  • reduce the number of replayers processes to 4x10 per object type
  • reduce the journal client batch size to 500
Jun 30 2021, 3:07 PM · System administration, Storage manager

Jun 29 2021

vsellier added a comment to T1526: Install a new VPN endpoint at Rocquencourt.

Regarding the recurrent disconnection of the azure VPN, it seems the only difference is on the reauth=no activated on louvre but not on opnsense.
The option was activated on opnsense too, the connection should not killed anymore on a key renegociation (if I have well undertood ;))

Jun 29 2021, 7:26 PM · System administration
vsellier committed rSPSITE144a796abf87: network: clean old gateways and routes (authored by vsellier).
network: clean old gateways and routes
Jun 29 2021, 4:31 PM
vsellier closed D5929: network: clean old gateways and routes.
Jun 29 2021, 4:31 PM
vsellier updated the diff for D5929: network: clean old gateways and routes.

rebase

Jun 29 2021, 4:30 PM
vsellier added a comment to P1083 ES documents with metadata.
{
  "_index": "origin-v0.9.0",
  "_type": "_doc",
  "_id": "6925508eeef1c21e32f41040849c5851ac02920e",
  "_version": 6,
  "_seq_no": 16879311,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "intrinsic_metadata": [
      {
        "http://schema.org/author": [
          {
            "@list": [
              {
                "http://schema.org/name": [
                  {
                    "@value": "Evgeniy Malyarov"
                  }
                ],
                "@type": [
                  "http://schema.org/Person"
                ],
                "http://schema.org/url": [
                  {
                    "@id": "http://www.oknosoft.ru"
                  }
                ],
                "http://schema.org/email": [
                  {
                    "@value": "info@oknosoft.ru"
                  }
                ]
              }
            ]
          }
        ],
        "http://schema.org/description": [
          {
            "@value": "Library for building offline-first browser-based business applications"
          }
        ],
        "http://schema.org/name": [
          {
            "@value": "metadata-js"
          }
        ],
        "https://codemeta.github.io/terms/issueTracker": [
          {
            "@id": "https://github.com/oknosoft/metadata.js/issues"
          }
        ],
        "http://schema.org/license": [
          {
            "@id": "https://spdx.org/licenses/MIT"
          }
        ],
        "http://schema.org/codeRepository": [
          {
            "@id": "git+https://github.com/oknosoft/metadata.js.git"
          }
        ],
        "@type": [
          "http://schema.org/SoftwareSourceCode"
        ],
        "http://schema.org/version": [
          {
            "@value": "0.11.223"
          }
        ],
        "http://schema.org/keywords": [
          {
            "@value": "metadata"
          },
          {
            "@value": "browser data engine"
          },
          {
            "@value": "spa offline"
          },
          {
            "@value": "rest"
          },
          {
            "@value": "odata"
          },
          {
            "@value": "1c"
          },
          {
            "@value": "1с"
          },
          {
            "@value": "web сервис"
          },
          {
            "@value": "клиент 1с"
          },
          {
            "@value": "ui framework"
          },
          {
            "@value": "offline framework"
          },
          {
            "@value": "offline data engine"
          },
          {
            "@value": "rest client"
          },
          {
            "@value": "CRDT"
          },
          {
            "@value": "offline-first"
          },
          {
            "@value": "replication"
          }
        ],
        "http://schema.org/url": [
          {
            "@id": "http://www.oknosoft.ru/metadata/"
          }
        ]
      }
    ],
    "sha1": "6925508eeef1c21e32f41040849c5851ac02920e",
    "url": "https://github.com/SMAlik93/metadata.js",
    "visit_types": [
      "git"
    ],
    "has_visits": true,
    "last_visit_date": "2021-01-13T13:12:40.822142Z",
    "nb_visits": 2
  }
}
Jun 29 2021, 2:25 PM
vsellier added a comment to P1083 ES documents with metadata.
{
  "_index": "origin-v0.9.0",
  "_type": "_doc",
  "_id": "014cea90c907f8c4af2d8e88d9c0b328388f766f",
  "_version": 17,
  "_seq_no": 19142389,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "sha1": "014cea90c907f8c4af2d8e88d9c0b328388f766f",
    "url": "https://github.com/mvolz/html-metadata",
    "intrinsic_metadata": [
      {
        "http://schema.org/author": [
          {
            "@list": [
              {
                "http://schema.org/name": [
                  {
                    "@value": "Marielle Volz"
                  }
                ],
                "@type": [
                  "http://schema.org/Person"
                ],
                "http://schema.org/email": [
                  {
                    "@value": "marielle.volz@gmail.com"
                  }
                ]
              }
            ]
          }
        ],
        "http://schema.org/description": [
          {
            "@value": "Scrapes metadata of several different standards"
          }
        ],
        "http://schema.org/name": [
          {
            "@value": "html-metadata"
          }
        ],
        "https://codemeta.github.io/terms/issueTracker": [
          {
            "@id": "https://github.com/wikimedia/html-metadata/issues"
          }
        ],
        "http://schema.org/license": [
          {
            "@id": "https://spdx.org/licenses/MIT"
          }
        ],
        "http://schema.org/codeRepository": [
          {
            "@id": "git+https://github.com/wikimedia/html-metadata.git"
          }
        ],
        "@type": [
          "http://schema.org/SoftwareSourceCode"
        ],
        "http://schema.org/version": [
          {
            "@value": "1.7.0"
          }
        ],
        "http://schema.org/keywords": [
          {
            "@value": "bepress"
          },
          {
            "@value": "coins"
          },
          {
            "@value": "dublin core"
          },
          {
            "@value": "eprints"
          },
          {
            "@value": "highwire press"
          },
          {
            "@value": "json-ld"
          },
          {
            "@value": "open graph"
          },
          {
            "@value": "metadata"
          },
          {
            "@value": "microdata"
          },
          {
            "@value": "prism"
          },
          {
            "@value": "twitter cards"
          },
          {
            "@value": "web scraper"
          }
        ],
        "http://schema.org/url": [
          {
            "@id": "https://github.com/wikimedia/html-metadata"
          }
        ]
      }
    ],
    "visit_types": [
      "git"
    ],
    "nb_visits": 8,
    "has_visits": true,
    "last_visit_date": "2020-04-07T05:42:09.217233+00:00"
  }
}
Jun 29 2021, 2:24 PM
vsellier edited P1083 ES documents with metadata.
Jun 29 2021, 2:23 PM
vsellier added a comment to P1083 ES documents with metadata.
{
  "_index": "origin-v0.9.0",
  "_type": "_doc",
  "_id": "cca7bef1de938d94d11d0a8b67b4b2e3e9791a70",
  "_version": 1331,
  "_seq_no": 25110599,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "intrinsic_metadata": [
      {
        "http://schema.org/identifier": [
          {
            "@id": "com.blazegraph"
          }
        ],
        "http://schema.org/description": [
          {
            "@value": "Blazegraph™ DB is our ultra high-performance graph database supporting Blueprints and RDF/SPARQL APIs. It supports up to 50 Billion edges on a single machine and has a High Availability and Scale-out architecture. It is in production use for customers such as EMC, Syapse, Wikidata Query Service, the British Museum, and many others.  GPU acceleration and High Availability (HA) are available in the Enterprise edition.  It contains war, jar, deb, rpm, and tar.gz deployment artifacts."
          }
        ],
        "http://schema.org/name": [
          {
            "@value": "Blazegraph Database Platform"
          }
        ],
        "http://schema.org/license": [
          {
            "@id": "http://www.gnu.org/licenses/gpl-2.0.html"
          }
        ],
        "http://schema.org/codeRepository": [
          {
            "@id": "https://repo.maven.apache.org/maven2/com/blazegraph/blazegraph-parent"
          }
        ],
        "@type": [
          "http://schema.org/SoftwareSourceCode"
        ],
        "http://schema.org/version": [
          {
            "@value": "2.1.6-wmf.2-SNAPSHOT"
          }
        ]
      }
    ],
    "sha1": "cca7bef1de938d94d11d0a8b67b4b2e3e9791a70",
    "url": "https://phabricator.wikimedia.org/diffusion/WDQB/wikidata-query-blazegraph.git",
    "visit_types": [
      "git"
    ],
    "has_visits": true,
    "last_visit_date": "2020-10-02T11:36:24.415385Z",
    "nb_visits": 665
  }
}
Jun 29 2021, 2:23 PM
vsellier edited P1083 ES documents with metadata.
Jun 29 2021, 2:23 PM
vsellier added a comment to P1083 ES documents with metadata.
{
  "_index": "origin-v0.9.0",
  "_type": "_doc",
  "_id": "94247b453f4290fb234e8f05be470673c044a7d1",
  "_version": 1940,
  "_seq_no": 25440556,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "intrinsic_metadata": [
      {
        "http://schema.org/author": [
          {
            "@list": [
              {
                "http://schema.org/name": [
                  {
                    "@value": "FaBo"
                  }
                ],
                "@type": [
                  "http://schema.org/Person"
                ],
                "http://schema.org/email": [
                  {
                    "@value": "info@fabo.io"
                  }
                ]
              }
            ]
          }
        ],
        "http://schema.org/description": [
          {
            "@value": "FaBoTemperature-ADT7410-Python\n==============================\n\nHow to install.\n---------------\n\n::\n\n    pip install FaBoTemperature_ADT7410\n\nFaBo Temperature I2C Brick\n--------------------------\n\n `#207 Temperature I2C Brick <http://fabo.io/207.html>`__\n\nADT7410\n-------\n\n ADT7410 is 16-Bit Digital I2C Temperature Sensor.\n\nADT7410 Datasheet\n~~~~~~~~~~~~~~~~~\n\n `ADT7410\nDatasheet <http://www.analog.com/media/en/technical-documentation/data-sheets/ADT7410.pdf>`__\n\nReleases\n--------\n\n-  1.0.0 Initial release.\n"
          },
          {
            "@value": "This is a library for the FaBo Temperature I2C Brick."
          }
        ],
        "http://schema.org/name": [
          {
            "@value": "FaBoTemperature_ADT7410"
          }
        ],
        "http://schema.org/license": [
          {
            "@id": "Apache License 2.0"
          }
        ],
        "@type": [
          "http://schema.org/SoftwareSourceCode"
        ],
        "http://schema.org/version": [
          {
            "@value": "1.0.0"
          }
        ],
        "http://schema.org/url": [
          {
            "@id": "https://github.com/FaBoPlatform/FaBoTemperature-ADT7410-Python"
          }
        ]
      }
    ],
    "sha1": "94247b453f4290fb234e8f05be470673c044a7d1",
    "url": "https://pypi.org/project/FaBoTemperature_ADT7410/",
    "visit_types": [
      "pypi"
    ],
    "has_visits": true,
    "last_visit_date": "2021-06-29T11:23:44.141572+00:00",
    "nb_visits": 969
  }
}
Jun 29 2021, 2:23 PM
vsellier edited P1083 ES documents with metadata.
Jun 29 2021, 2:23 PM
vsellier created P1083 ES documents with metadata.
Jun 29 2021, 2:23 PM
vsellier committed rDSNIP184ee8c0a595: grid5000/cassandra: prepare the run environments (authored by vsellier).
grid5000/cassandra: prepare the run environments
Jun 29 2021, 1:09 PM
vsellier committed rDSNIPa8367fe02594: grid5000/cassandra: Start the replayer when the environment is up (authored by vsellier).
grid5000/cassandra: Start the replayer when the environment is up
Jun 29 2021, 1:09 PM
vsellier committed rDSNIPf76ff24e6fca: grid5000/cassandra: Monitor the object_count table content (authored by vsellier).
grid5000/cassandra: Monitor the object_count table content
Jun 29 2021, 1:09 PM
vsellier committed rDSNIP77acf7325374: grid5000/cassandra: fix hosts configuration in ansible (authored by vsellier).
grid5000/cassandra: fix hosts configuration in ansible
Jun 29 2021, 1:09 PM
vsellier added a comment to T3357: Perform some tests of the cassandra storage on Grid5000.

These are the first tests that will be executed:

  1. baseline configuration : 3 cassandra nodes (parasilo[1]), commitlog on a dedicated SSD, data on 4 HDD. Goal: testing minimal configuration (a night or a complete weekend if possible)
  2. baseline configuration + 1 cassandra nodes. Goal: Testing the performance impact of having 1 more server (duration enough to have tendencies)
  3. baseline configuration + 3 cassandra nodes: Goal: Testing the performance impact of having the cluster size x 2 (duration enough to have tendencies)
  4. baseline configuration but with the commitlog on the data partition. Goal check the impact of data/commitlog mutualization (duration enough to have tendencies)
  5. baseline configuration but with 2 HDD. Goal check the impact of the number of disks + have a reference for the next run (a night)
  6. baseline configuration but with 2 HDD + commitlog on a dedicated HDD. Goal check the impact of having the commitlog on a slower disk (duration enough to have tendencies)
  7. baseline configuration but with 2x the default heap allocated to cassandra. Goal check the impact of the memory configuration ((!) check the gc profile)
Jun 29 2021, 12:59 PM · System administration, Storage manager
vsellier updated the task description for T3417: Cleanup the old counters environment.
Jun 29 2021, 11:25 AM · System administration, Monitoring
vsellier triaged T3417: Cleanup the old counters environment as Normal priority.
Jun 29 2021, 11:20 AM · System administration, Monitoring
vsellier added a comment to T3394: cassandra - origin url hashing encoding issue.

Thanks, it was tested this night on grid5000, all the origins were correctly replayed without issues.

Jun 29 2021, 10:36 AM · System administration, Storage manager

Jun 28 2021

vsellier updated the task description for T3357: Perform some tests of the cassandra storage on Grid5000.
Jun 28 2021, 8:49 PM · System administration, Storage manager
vsellier committed rDSNIP51adac9ac3dc: grid5000/cassandra: 2021-06-28 run (authored by vsellier).
grid5000/cassandra: 2021-06-28 run
Jun 28 2021, 7:05 PM
vsellier added a comment to T3396: cassandra - allow to configure the consistency level used by the queries.

only one confirmation of the write is needed

It's not perfect though. If the server that confirmed the writes breaks before it replicates the write, then the write is lost.

Jun 28 2021, 5:54 PM · System administration, Storage manager
vsellier added a comment to T3396: cassandra - allow to configure the consistency level used by the queries.

IMO, we should first try to have a global configuration for all the read/write queries, and improve that later if needed for performance or if it creates some problems. At worst, it will be possible to use the default ONE values by configuration.

Jun 28 2021, 4:48 PM · System administration, Storage manager
vsellier updated the task description for T3357: Perform some tests of the cassandra storage on Grid5000.
Jun 28 2021, 4:09 PM · System administration, Storage manager
vsellier updated the task description for T3357: Perform some tests of the cassandra storage on Grid5000.
Jun 28 2021, 4:09 PM · System administration, Storage manager
vsellier updated the task description for T3357: Perform some tests of the cassandra storage on Grid5000.
Jun 28 2021, 3:39 PM · System administration, Storage manager
vsellier added a comment to T3394: cassandra - origin url hashing encoding issue.

released in swh-storage:v0.32.0

Jun 28 2021, 3:37 PM · System administration, Storage manager
vsellier committed rDSNIPc80554db8de0: grid5000/cassandra: deploy some grafana dashboards (authored by vsellier).
grid5000/cassandra: deploy some grafana dashboards
Jun 28 2021, 12:27 PM
vsellier committed rDSNIPd940be8d5575: grid5000/cassandra: improve replayer tracability (authored by vsellier).
grid5000/cassandra: improve replayer tracability
Jun 28 2021, 12:27 PM
vsellier committed rDSNIPdabe01fc0194: grid5000/cassandra: Remove possible lvm volumes (authored by vsellier).
grid5000/cassandra: Remove possible lvm volumes
Jun 28 2021, 12:27 PM
vsellier committed rDSNIP75624f36dd16: grid5000/cassandra fix a type on the cassandra directories (authored by vsellier).
grid5000/cassandra fix a type on the cassandra directories
Jun 28 2021, 12:27 PM
vsellier committed rDSNIP7039064fcb13: grid5000/cassandra: don't fail the reservation if something goes wrong (authored by vsellier).
grid5000/cassandra: don't fail the reservation if something goes wrong
Jun 28 2021, 12:27 PM
vsellier committed rDSNIPc1eac7dd278a: grid5000/cassandra: Add a script to clean the cassandra data (authored by vsellier).
grid5000/cassandra: Add a script to clean the cassandra data
Jun 28 2021, 12:27 PM
vsellier accepted D5932: cassandra: Add support for non-ASCII origin 'URLs'..

LGTM, thank you for fixing that.

Jun 28 2021, 12:12 PM
vsellier updated the task description for T3398: [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation.
Jun 28 2021, 9:58 AM · System administration, Archive search
vsellier closed T3398: [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation, a subtask of T3373: Metadata search is failing due to a boolean field in the mapping of the metadata fields, as Resolved.
Jun 28 2021, 9:57 AM · System administration, Archive search
vsellier closed T3398: [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation as Resolved.

The lag on the topics has recovered.
The configuration update of moma will be followed in T3373

Jun 28 2021, 9:57 AM · System administration, Archive search
vsellier added a comment to T3373: Metadata search is failing due to a boolean field in the mapping of the metadata fields.

The lag has recovered, the search on webapp1[1] is now fully up-to-date and can be tested before changing the configuration on the main webapp.

Jun 28 2021, 9:50 AM · System administration, Archive search

Jun 25 2021

vsellier changed the status of T3408: Provide read-only access to production servers, a subtask of T1526: Install a new VPN endpoint at Rocquencourt, from Open to Work in Progress.
Jun 25 2021, 6:03 PM · System administration
vsellier changed the status of T3408: Provide read-only access to production servers from Open to Work in Progress.
Jun 25 2021, 6:03 PM · System administration
vsellier triaged T3408: Provide read-only access to production servers as High priority.
Jun 25 2021, 5:50 PM · System administration
vsellier updated the diff for D5929: network: clean old gateways and routes.

improve commit message

Jun 25 2021, 5:09 PM
vsellier requested review of D5929: network: clean old gateways and routes.
Jun 25 2021, 5:08 PM
vsellier added a revision to T1526: Install a new VPN endpoint at Rocquencourt: D5929: network: clean old gateways and routes.
Jun 25 2021, 5:08 PM · System administration
vsellier triaged T3407: Upgrade sphinx docker image to use a more recent version of plantuml as Normal priority.
Jun 25 2021, 3:44 PM · System administration, Documentation
vsellier committed rDDOC2c97beb78a57: Remove the unsupported 'description' group option on old versions of nwdiag (authored by vsellier).
Remove the unsupported 'description' group option on old versions of nwdiag
Jun 25 2021, 3:13 PM
vsellier accepted D5920: Drop search instance from swh_storage_cloud role.

lgtm

Jun 25 2021, 10:46 AM

Jun 23 2021

vsellier added a comment to T3373: Metadata search is failing due to a boolean field in the mapping of the metadata fields.

The metadata indexation is finished, https://webapp1.internal.softwareheritage.org can now search on them via elasticsearch without any issue.
Let's now wait for the lag on origin* topics to recover: https://grafana.softwareheritage.org/goto/iOvBK6gnk?orgId=1

Jun 23 2021, 4:12 PM · System administration, Archive search
vsellier closed D5911: swh-search: configure webapp1 to use the v0.9.0 index.
Jun 23 2021, 3:38 PM
vsellier committed rSPSITEf5d049c76829: swh-search: configure webapp1 to use the v0.9.0 index (authored by vsellier).
swh-search: configure webapp1 to use the v0.9.0 index
Jun 23 2021, 3:38 PM
vsellier updated the test plan for D5911: swh-search: configure webapp1 to use the v0.9.0 index.
Jun 23 2021, 1:01 PM
vsellier updated the diff for D5911: swh-search: configure webapp1 to use the v0.9.0 index.

Also update the index name. It's not necessary as only the aliases are used, for the searchbut it's cleaner

Jun 23 2021, 1:00 PM
vsellier requested review of D5911: swh-search: configure webapp1 to use the v0.9.0 index.
Jun 23 2021, 12:56 PM