Page MenuHomeSoftware Heritage

Storage managerFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Details

Recent Activity

Thu, Dec 2

vlorentz updated the task description for T3752: Store/represent time offsets as strings.
Thu, Dec 2, 4:04 PM · Data Model, Storage manager
vlorentz closed T3586: Figure out what to do with 'misordered' directories in Cassandra, a subtask of T3585: Fix inconsistencies of the Cassandra backend with postgres, as Resolved.
Thu, Dec 2, 3:14 PM · meta-task, Storage manager
vlorentz closed T3586: Figure out what to do with 'misordered' directories in Cassandra as Resolved.

We don't care anymore, this will be handled by T3753.

Thu, Dec 2, 3:14 PM · Data Model, Storage manager
vlorentz removed a parent task for T3752: Store/represent time offsets as strings: T3753: Store original git manifests.
Thu, Dec 2, 3:01 PM · Data Model, Storage manager
vlorentz removed a subtask for T3753: Store original git manifests: T3752: Store/represent time offsets as strings.
Thu, Dec 2, 3:01 PM · Data Model, Storage manager
vlorentz added a parent task for T3752: Store/represent time offsets as strings: T3753: Store original git manifests.
Thu, Dec 2, 3:00 PM · Data Model, Storage manager
vlorentz added a subtask for T3753: Store original git manifests: T3752: Store/represent time offsets as strings.
Thu, Dec 2, 3:00 PM · Data Model, Storage manager
vlorentz updated the task description for T3752: Store/represent time offsets as strings.
Thu, Dec 2, 2:59 PM · Data Model, Storage manager
vlorentz updated the task description for T3752: Store/represent time offsets as strings.
Thu, Dec 2, 2:55 PM · Data Model, Storage manager
vlorentz updated the task description for T3752: Store/represent time offsets as strings.
Thu, Dec 2, 2:52 PM · Data Model, Storage manager
vlorentz updated the task description for T3753: Store original git manifests.
Thu, Dec 2, 2:48 PM · Data Model, Storage manager
vlorentz updated the task description for T3752: Store/represent time offsets as strings.
Thu, Dec 2, 2:22 PM · Data Model, Storage manager
vsellier closed T3357: Perform some tests of the cassandra storage on Grid5000, a subtask of T1892: Cassandra as a storage backend, as Resolved.
Thu, Dec 2, 10:10 AM · meta-task, Storage manager
vsellier closed T3357: Perform some tests of the cassandra storage on Grid5000 as Resolved.

The slide of the restrospective of the experiment are available at : https://hedgedoc.softwareheritage.org/VOP9qh1MTqm4DjPQfFgNbQ

Thu, Dec 2, 10:10 AM · System administration, Storage manager
vsellier closed T3573: [cassandra] directory and content read benchmarks, a subtask of T3357: Perform some tests of the cassandra storage on Grid5000, as Resolved.
Thu, Dec 2, 10:08 AM · System administration, Storage manager
vsellier closed T3573: [cassandra] directory and content read benchmarks as Resolved.

It was not easy to know if it's a lot of call or long running calls because it's regular sample and we don't have this granularity.

Thu, Dec 2, 10:08 AM · System administration, Storage manager

Wed, Dec 1

zack moved T2053: support graph export for the cassandra backend from Backlog to Deployed on the Graph service board.
Wed, Dec 1, 4:37 PM · Graph service, Storage manager
zack moved T2045: add support for reverse lookup from swh:1:ori:... PIDs to origin URLs from Backlog to Deployed on the Graph service board.
Wed, Dec 1, 4:35 PM · Graph service, Storage manager

Fri, Nov 26

vlorentz removed a project from T3752: Store/represent time offsets as strings: meta-task.
Fri, Nov 26, 5:19 PM · Data Model, Storage manager
vlorentz removed a project from T3753: Store original git manifests: meta-task.
Fri, Nov 26, 5:19 PM · Data Model, Storage manager
vlorentz claimed T3594: Faithfully store weird git objects.
Fri, Nov 26, 4:43 PM · meta-task, Data Model, Storage manager
vlorentz claimed T3753: Store original git manifests.
Fri, Nov 26, 4:43 PM · Data Model, Storage manager
vlorentz triaged T3753: Store original git manifests as Normal priority.
Fri, Nov 26, 4:43 PM · Data Model, Storage manager
vlorentz triaged T3752: Store/represent time offsets as strings as Normal priority.
Fri, Nov 26, 4:42 PM · Data Model, Storage manager
vlorentz closed T3598: Support revisions with "extra headers" not at the end, a subtask of T3594: Faithfully store weird git objects, as Wontfix.
Fri, Nov 26, 4:41 PM · meta-task, Data Model, Storage manager
vlorentz closed T3598: Support revisions with "extra headers" not at the end as Wontfix.

We decided to store manifests instead. T3594#74385

Fri, Nov 26, 4:41 PM · Data Model, Storage manager
vlorentz closed T3596: Support "weird" permissions in directories as Wontfix.

We decided to store manifests instead. T3594#74385

Fri, Nov 26, 4:41 PM · meta-task, Data Model, Storage manager
vlorentz closed T3596: Support "weird" permissions in directories, a subtask of T3594: Faithfully store weird git objects, as Wontfix.
Fri, Nov 26, 4:41 PM · meta-task, Data Model, Storage manager
vlorentz closed T3595: Support disordered directory entries in git, a subtask of T3594: Faithfully store weird git objects, as Wontfix.
Fri, Nov 26, 4:41 PM · meta-task, Data Model, Storage manager
vlorentz closed T3595: Support disordered directory entries in git as Wontfix.

We decided to store manifests instead. T3594#74385

Fri, Nov 26, 4:41 PM · meta-task, Data Model, Storage manager
vlorentz added a comment to T3594: Faithfully store weird git objects.

Copy of an email I sent today:

Fri, Nov 26, 4:40 PM · meta-task, Data Model, Storage manager
vlorentz added a revision to T399: (Re-)Compute data checksums before insertion: D6281: converters: Recompute hashes and check they match the originals.
Fri, Nov 26, 3:52 PM · Storage manager
douardda added a revision to T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror): D6693: Add support for a redis-based reporter for failed replayed objects.
Fri, Nov 26, 1:33 PM · Storage manager

Mon, Nov 15

vsellier updated the task description for T3357: Perform some tests of the cassandra storage on Grid5000.
Mon, Nov 15, 9:46 AM · System administration, Storage manager

Mon, Nov 8

vsellier closed T3683: cassandra - benchmark the vault, a subtask of T3357: Perform some tests of the cassandra storage on Grid5000, as Resolved.
Mon, Nov 8, 9:55 AM · System administration, Storage manager

Oct 27 2021

douardda added a revision to T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror): D6571: Add support for a redis-based reporting for invalid mirrorred objects.
Oct 27 2021, 6:24 PM · Storage manager
douardda added a revision to T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror): D6565: Pass the object_type to JournalClient.value_serializer().
Oct 27 2021, 4:19 PM · Storage manager
vlorentz updated the task description for T3594: Faithfully store weird git objects.
Oct 27 2021, 2:08 PM · meta-task, Data Model, Storage manager
vlorentz updated the task description for T3594: Faithfully store weird git objects.
Oct 27 2021, 2:03 PM · meta-task, Data Model, Storage manager
vlorentz updated the task description for T3594: Faithfully store weird git objects.
Oct 27 2021, 2:03 PM · meta-task, Data Model, Storage manager

Oct 26 2021

douardda added a revision to T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror): D6554: [WIP] Add a (redis-based) validation error reporting facility.
Oct 26 2021, 5:48 PM · Storage manager
douardda triaged T3693: Provide a mecanism to report (with persistence) objects that fails to get replayed (mirror) as High priority.
Oct 26 2021, 5:41 PM · Storage manager

Oct 22 2021

ardumont added a comment to T3595: Support disordered directory entries in git.

I came across a rather small repository [1] which i believe raise the same issue.
So it may help to keep its reference to ease the testing of the improvment discussed here.
Feel free to dismiss if not that useful.

Oct 22 2021, 1:54 PM · meta-task, Data Model, Storage manager
vsellier reopened T3683: cassandra - benchmark the vault, a subtask of T3357: Perform some tests of the cassandra storage on Grid5000, as Work in Progress.
Oct 22 2021, 11:49 AM · System administration, Storage manager
vsellier closed T3683: cassandra - benchmark the vault, a subtask of T3357: Perform some tests of the cassandra storage on Grid5000, as Resolved.
Oct 22 2021, 11:48 AM · System administration, Storage manager

Oct 21 2021

vsellier added revisions to T3577: Parallel loaders performances : D6423: cassandra: Add alternative algorithms to list missing objects, D6494: cassandra: Fix incomplete check of content existence in object_find_by_sha1_git, D6495: cassandra: Rewrite content_missing to run queries concurrently..
Oct 21 2021, 2:58 PM · System administration, Storage manager
vsellier closed T3577: Parallel loaders performances , a subtask of T3357: Perform some tests of the cassandra storage on Grid5000, as Resolved.
Oct 21 2021, 2:56 PM · System administration, Storage manager
vsellier closed T3577: Parallel loaders performances as Resolved.

Stopping here the investigations as the limit is now the hardware running cassandra (HDD). Great improvements were made compared to the first tries

Oct 21 2021, 2:56 PM · System administration, Storage manager
vsellier added a comment to T3577: Parallel loaders performances .

Changing the cache size doesn't seem very effective for the particular workload of the loaders:
Exception some contextual differences due probably due to the different origin loaded, the performances are quite similar.
The hit ratios between the different configurations are also very close

Oct 21 2021, 2:36 PM · System administration, Storage manager
vlorentz added a revision to T3135: Improve integrity of ingested content: D6504: converters: Fix detection of tree entries with non-standard commit/tree mode..
Oct 21 2021, 10:57 AM · Storage manager, Roadmap 2021, meta-task