Page MenuHomeSoftware Heritage

Storage managerFolder
ActivePublic

Members

  • This project does not have any members.

Watchers

  • This project does not have any watchers.

Details

Recent Activity

Today

ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Fri, Feb 22, 11:22 AM · Web app, SWORD deposit, Object storage, Storage manager, Vault

Yesterday

ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Thu, Feb 21, 8:34 PM · Web app, SWORD deposit, Object storage, Storage manager, Vault
ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Thu, Feb 21, 2:11 PM · Web app, SWORD deposit, Object storage, Storage manager, Vault

Wed, Feb 20

ardumont updated the task description for T1533: Make sure api server uses explicit configurations.
Wed, Feb 20, 7:11 PM · Web app, SWORD deposit, Object storage, Storage manager, Vault
vlorentz closed T1495: Move swh.storage.db.BaseDb class in swh.core as Resolved.
Wed, Feb 20, 11:45 AM · Storage manager, Core & foundations

Tue, Feb 19

vlorentz closed T1507: Investigate CPU usage of swh-storage on storage0.euwest.azure as Resolved.
Tue, Feb 19, 2:54 PM · Storage manager
ardumont renamed T1533: Make sure api server uses explicit configurations from Use explicit configuration for all modules to Make sure api server uses explicit configurations.
Tue, Feb 19, 11:36 AM · Web app, SWORD deposit, Object storage, Storage manager, Vault

Thu, Feb 14

vlorentz triaged T1527: Have comments on all columns of all databases as Normal priority.
Thu, Feb 14, 11:08 AM · Storage manager, Scheduling utilities, Indexer

Tue, Feb 12

ardumont updated the task description for T1495: Move swh.storage.db.BaseDb class in swh.core.
Tue, Feb 12, 1:18 PM · Storage manager, Core & foundations

Mon, Feb 11

vlorentz updated the task description for T1495: Move swh.storage.db.BaseDb class in swh.core.
Mon, Feb 11, 11:38 AM · Storage manager, Core & foundations

Fri, Feb 1

vlorentz updated the task description for T1495: Move swh.storage.db.BaseDb class in swh.core.
Fri, Feb 1, 4:24 PM · Storage manager, Core & foundations
ardumont added a comment to T1507: Investigate CPU usage of swh-storage on storage0.euwest.azure.

@ardumont These timeouts are either caused by the high CPU usage of swh-storage, or an unrelated issue.

Fri, Feb 1, 9:03 AM · Storage manager

Thu, Jan 31

vlorentz added a comment to T1507: Investigate CPU usage of swh-storage on storage0.euwest.azure.

@ardumont These timeouts are either a cause of the high CPU usage of swh-storage, or an unrelated issue.

Thu, Jan 31, 5:59 PM · Storage manager
vlorentz added a revision to T1507: Investigate CPU usage of swh-storage on storage0.euwest.azure: D1053: Remove entry_to_bytes and other *_to_bytes functions.
Thu, Jan 31, 5:50 PM · Storage manager
ardumont added a comment to T1507: Investigate CPU usage of swh-storage on storage0.euwest.azure.

There now remains errors :

  • directly related to indexers (bad input) so indexer fails
  • Sometimes it's the indexer's api call to the indexer storage that fails. I've created a dashboard to see those errors [1]
Thu, Jan 31, 4:36 PM · Storage manager
ardumont added a comment to T1507: Investigate CPU usage of swh-storage on storage0.euwest.azure.

To exercise my hypothesis, i've increased that timeout to 1second for storage0.euwest.azure.
So far so good.

Thu, Jan 31, 3:15 PM · Storage manager
ardumont added a comment to T1507: Investigate CPU usage of swh-storage on storage0.euwest.azure.

oh, i think i know... [1] We have a timeout of some endpoint...

Thu, Jan 31, 3:04 PM · Storage manager
ardumont added a comment to T1507: Investigate CPU usage of swh-storage on storage0.euwest.azure.

Once in a while, an indexer worker crashes with the following error:

Thu, Jan 31, 2:30 PM · Storage manager
vlorentz added a project to T1487: Add a public API endpoint to retrieve a set of files with a given name: Easy hack.
Thu, Jan 31, 1:29 PM · Easy hack, Storage manager, Object storage
vlorentz updated the task description for T1507: Investigate CPU usage of swh-storage on storage0.euwest.azure.
Thu, Jan 31, 1:16 PM · Storage manager
vlorentz claimed T1507: Investigate CPU usage of swh-storage on storage0.euwest.azure.
Thu, Jan 31, 1:14 PM · Storage manager
vlorentz triaged T1507: Investigate CPU usage of swh-storage on storage0.euwest.azure as Normal priority.
Thu, Jan 31, 1:13 PM · Storage manager

Tue, Jan 29

douardda updated the task description for T1495: Move swh.storage.db.BaseDb class in swh.core.
Tue, Jan 29, 10:48 AM · Storage manager, Core & foundations
douardda merged T904: Factor-out common DB wrappers in swh.core into T1495: Move swh.storage.db.BaseDb class in swh.core.
Tue, Jan 29, 10:22 AM · Storage manager, Core & foundations
douardda merged task T904: Factor-out common DB wrappers in swh.core into T1495: Move swh.storage.db.BaseDb class in swh.core.
Tue, Jan 29, 10:22 AM · Core & foundations, Storage manager, Scheduling utilities, Vault
douardda added a comment to T904: Factor-out common DB wrappers in swh.core.

I agree we must be careful with not bloating swh.core, but the current subject, it really makes sense to me to put this basic db access wrapper as a core functionality.

Tue, Jan 29, 10:14 AM · Core & foundations, Storage manager, Scheduling utilities, Vault

Thu, Jan 24

douardda added projects to T1495: Move swh.storage.db.BaseDb class in swh.core: Core & foundations, Storage manager.
Thu, Jan 24, 3:38 PM · Storage manager, Core & foundations

Jan 21 2019

vlorentz added a project to T1487: Add a public API endpoint to retrieve a set of files with a given name: Storage manager.

The object storage doesn't have content names, so it cannot address this feature as stated.

Jan 21 2019, 4:55 PM · Easy hack, Storage manager, Object storage

Jan 13 2019

ardumont closed T494: swh-journal: archiver-client: Keep archiver table in sync with new contents, a subtask of T240: content archiver, as Invalid.
Jan 13 2019, 12:33 PM · Storage manager
ardumont closed T494: swh-journal: archiver-client: Keep archiver table in sync with new contents as Invalid.
Jan 13 2019, 12:33 PM · Journal, Restricted Project, Storage manager
ardumont closed T1275: swh-journal: Complete missing snapshot insertion event from storage to journal as Resolved.
Jan 13 2019, 12:27 PM · Storage manager, Journal

Dec 18 2018

vlorentz added a parent task for T1446: Add support for slices in Storage.content_get: T803: Indexer - Retrieval error when contents is too big.
Dec 18 2018, 4:35 PM · Easy hack, Storage manager
vlorentz triaged T1446: Add support for slices in Storage.content_get as Low priority.
Dec 18 2018, 4:34 PM · Easy hack, Storage manager
vlorentz created T1446: Add support for slices in Storage.content_get.
Dec 18 2018, 4:34 PM · Easy hack, Storage manager

Dec 13 2018

vlorentz edited projects for T1377: in-memory storage: compute all counters., added: Easy hack; removed Good first contribution.
Dec 13 2018, 3:07 PM · Easy hack, Storage manager
vlorentz edited projects for T1377: in-memory storage: compute all counters., added: Good first contribution; removed Easy hack.
Dec 13 2018, 12:33 PM · Easy hack, Storage manager
vlorentz updated the task description for T1377: in-memory storage: compute all counters..
Dec 13 2018, 12:19 PM · Easy hack, Storage manager
vlorentz added a project to T1377: in-memory storage: compute all counters.: Easy hack.
Dec 13 2018, 12:19 PM · Easy hack, Storage manager

Dec 11 2018

vlorentz updated the task description for T1307: Remove mock storages used in tests..
Dec 11 2018, 4:18 PM · Storage manager

Dec 7 2018

vlorentz updated the task description for T1307: Remove mock storages used in tests..
Dec 7 2018, 6:21 PM · Storage manager
vlorentz closed T1432: Remove mock storages from the indexers, a subtask of T1307: Remove mock storages used in tests., as Resolved.
Dec 7 2018, 6:20 PM · Storage manager
vlorentz closed T1432: Remove mock storages from the indexers as Resolved.
Dec 7 2018, 6:20 PM · Storage manager
vlorentz added revisions to T1307: Remove mock storages used in tests.: D792: Kill mocks., D791: Remove mock storages from test_origin_metadata.py., D790: Add in-mem storage for intrinsic metadata., D789: Remove languages mock., D788: Remove mocks from range tests of mimetype and license indexers., D786: Add content_fossology_license_get_range + Remove mock from non-range tests., D784: Add content_fossology_license_{get,add} to the mem idx storage., D785: Start removing mocks for the mimetype indexer., D787: Remove mocks from range tests of mimetype and license indexers., D783: rm ctags mocks + add ctags to idx db + fix doc..
Dec 7 2018, 5:13 PM · Storage manager
anlambert changed the status of T1271: Use the in-memory storage instead of mocks to test the webapp, a subtask of T1307: Remove mock storages used in tests., from Open to Work in Progress.
Dec 7 2018, 10:49 AM · Storage manager

Dec 5 2018

vlorentz added a revision to T1432: Remove mock storages from the indexers: D783: rm ctags mocks + add ctags to idx db + fix doc..
Dec 5 2018, 5:07 PM · Storage manager
anlambert updated the task description for T1307: Remove mock storages used in tests..
Dec 5 2018, 3:38 PM · Storage manager
vlorentz placed T1307: Remove mock storages used in tests. up for grabs.
Dec 5 2018, 3:31 PM · Storage manager
vlorentz triaged T1432: Remove mock storages from the indexers as Normal priority.
Dec 5 2018, 3:31 PM · Storage manager
anlambert updated the task description for T1307: Remove mock storages used in tests..
Dec 5 2018, 2:07 PM · Storage manager
anlambert updated the task description for T1307: Remove mock storages used in tests..
Dec 5 2018, 1:57 PM · Storage manager