Page MenuHomeSoftware Heritage

Storage managerFolder
ActivePublic

Members

  • This project does not have any members.

Watchers

  • This project does not have any watchers.

Details

Recent Activity

Today

vlorentz added a project to T2215: Streaming support everywhere: meta-task.
Tue, Feb 18, 4:52 PM · meta-task, Web app, Object storage, Storage manager, Restricted Project
vlorentz updated the task description for T2290: Implement origin_metadata endpoints in swh/storage/cassandra/.
Tue, Feb 18, 4:50 PM · Easy hack, Storage manager
vlorentz renamed T2291: Implement metadata_provider endpoints in swh/storage/cassandra/ from T2290: Implement metadata_provider endpoints in swh/storage/cassandra/ to Implement metadata_provider endpoints in swh/storage/cassandra/.
Tue, Feb 18, 4:50 PM · Easy hack, Storage manager
vlorentz triaged T2291: Implement metadata_provider endpoints in swh/storage/cassandra/ as Normal priority.
Tue, Feb 18, 4:49 PM · Easy hack, Storage manager
vlorentz added projects to T2290: Implement origin_metadata endpoints in swh/storage/cassandra/: Storage manager, Easy hack.
Tue, Feb 18, 4:48 PM · Easy hack, Storage manager

Yesterday

vlorentz added a comment to T2239: storage: kafka issue: Can't pickle <class 'cimpl.KafkaException'>: import of module 'cimpl' failed.

python-cassandra too has some un-unpicklable errors:

Mon, Feb 17, 12:14 PM · Storage manager

Fri, Feb 14

vlorentz triaged T2287: Improve code in BufferingProxyStorage as Normal priority.
Fri, Feb 14, 6:02 PM · Easy hack, Storage manager

Thu, Feb 6

ardumont closed T2185: Make webapp0 use Cassandra as storage backend., a subtask of T1892: Cassandra as a storage backend, as Resolved.
Thu, Feb 6, 12:31 PM · meta-task, Storage manager
ardumont closed T2185: Make webapp0 use Cassandra as storage backend. as Resolved.
Thu, Feb 6, 12:31 PM · Storage manager

Wed, Feb 5

vlorentz added a revision to T2239: storage: kafka issue: Can't pickle <class 'cimpl.KafkaException'>: import of module 'cimpl' failed: D2627: In case of errors, return a simple dictionary instead of pickled exception..
Wed, Feb 5, 5:12 PM · Storage manager
vlorentz claimed T2239: storage: kafka issue: Can't pickle <class 'cimpl.KafkaException'>: import of module 'cimpl' failed.
Wed, Feb 5, 4:18 PM · Storage manager
vlorentz added a comment to T2239: storage: kafka issue: Can't pickle <class 'cimpl.KafkaException'>: import of module 'cimpl' failed.

Relatedly, errors raised by tenacity cannot be pickled because they contain a Lock: https://github.com/jd/tenacity/issues/147

Wed, Feb 5, 4:17 PM · Storage manager
ardumont changed the status of T2185: Make webapp0 use Cassandra as storage backend., a subtask of T1892: Cassandra as a storage backend, from Open to Work in Progress.
Wed, Feb 5, 4:05 PM · meta-task, Storage manager
ardumont changed the status of T2185: Make webapp0 use Cassandra as storage backend. from Open to Work in Progress.
Wed, Feb 5, 4:05 PM · Storage manager
ardumont added a comment to T2185: Make webapp0 use Cassandra as storage backend..

storage02.euwest.azure exposes a rpc server using cassandra as storage backend.
webapp0 has been updated to use it.

Wed, Feb 5, 4:04 PM · Storage manager
ardumont closed T2183: Switch webapp0 to use swh-search instead of postgresql search. as Resolved.
Wed, Feb 5, 3:02 PM · Archive search, Storage manager
ardumont closed T2183: Switch webapp0 to use swh-search instead of postgresql search., a subtask of T2185: Make webapp0 use Cassandra as storage backend., as Resolved.
Wed, Feb 5, 3:02 PM · Storage manager
ardumont closed T2183: Switch webapp0 to use swh-search instead of postgresql search., a subtask of T2182: Switch production swh-web to use swh-search instead of postgresql search., as Resolved.
Wed, Feb 5, 3:02 PM · Archive search, Storage manager
ardumont added a project to T2266: Use python-cassandra 3.21: Storage manager.
Wed, Feb 5, 1:50 PM · Storage manager

Mon, Feb 3

vlorentz added a comment to T2186: Merge swh-storage-cassandra in swh-storage master.

thx

Mon, Feb 3, 10:38 PM · Storage manager
ardumont added a comment to T2186: Merge swh-storage-cassandra in swh-storage master.

and swh-storage debian package built (passing the cassandra tests ;)

Mon, Feb 3, 5:45 PM · Storage manager
vlorentz closed T2186: Merge swh-storage-cassandra in swh-storage master, a subtask of T2185: Make webapp0 use Cassandra as storage backend., as Resolved.
Mon, Feb 3, 1:33 PM · Storage manager
vlorentz closed T2186: Merge swh-storage-cassandra in swh-storage master as Resolved.
Mon, Feb 3, 1:33 PM · Storage manager

Thu, Jan 30

zack added a comment to T2262: Dealing with IRIs.

I'm fine with switching to IRIs in the doc, just please expand what it means on first use (with a mention like "they are like URIs but"), as I don't think the acronym is that well-known yet, especially in the US.

Thu, Jan 30, 5:47 PM · Storage manager, Data Model
vlorentz triaged T2262: Dealing with IRIs as Normal priority.
Thu, Jan 30, 4:26 PM · Storage manager, Data Model

Wed, Jan 29

ardumont renamed T2211: Go beyond git expressivity from Go beyound git expressivity to Go beyond git expressivity.
Wed, Jan 29, 6:43 PM · Mercurial loader, Storage manager, Data Model, Restricted Project
ardumont closed T2243: Add Debian package python3-cassandra as Resolved.

done indeed.

Wed, Jan 29, 3:49 PM · Storage manager
vlorentz closed T2184: Replay origins to ElasticSearch's "origin" index as Resolved.
Wed, Jan 29, 1:39 PM · Archive search, Storage manager
vlorentz closed T2184: Replay origins to ElasticSearch's "origin" index, a subtask of T2183: Switch webapp0 to use swh-search instead of postgresql search., as Resolved.
Wed, Jan 29, 1:39 PM · Archive search, Storage manager

Mon, Jan 27

ardumont added a comment to T2183: Switch webapp0 to use swh-search instead of postgresql search..

It's done and deployed btw.

Mon, Jan 27, 12:01 PM · Archive search, Storage manager

Fri, Jan 24

ardumont closed T2167: Deploy swh-search, a subtask of T1910: Redesign origin search using a dedicated component (swh-search), as Resolved.
Fri, Jan 24, 1:46 PM · Archive search, Storage manager
ardumont closed T2167: Deploy swh-search, a subtask of T2183: Switch webapp0 to use swh-search instead of postgresql search., as Resolved.
Fri, Jan 24, 1:46 PM · Archive search, Storage manager
ardumont changed the status of T2167: Deploy swh-search, a subtask of T1910: Redesign origin search using a dedicated component (swh-search), from Open to Work in Progress.
Fri, Jan 24, 9:38 AM · Archive search, Storage manager
ardumont changed the status of T2167: Deploy swh-search, a subtask of T2183: Switch webapp0 to use swh-search instead of postgresql search., from Open to Work in Progress.
Fri, Jan 24, 9:38 AM · Archive search, Storage manager
ardumont added a comment to T2185: Make webapp0 use Cassandra as storage backend..

Cool, looks like this is all ready within our code base:

Fri, Jan 24, 9:38 AM · Storage manager
ardumont renamed T1910: Redesign origin search using a dedicated component (swh-search) from Redesign origin search using a dedicated component to Redesign origin search using a dedicated component (swh-search).
Fri, Jan 24, 9:28 AM · Archive search, Storage manager

Thu, Jan 23

olasd closed T546: Update debian loader to register origin_visit's state, a subtask of T534: Add completion information to softwareheritage.origin_visit table, as Resolved.
Thu, Jan 23, 2:11 PM · Storage manager
olasd closed T757: Memory leak in swh.storage.api.server as Wontfix.

Considering the age of the bug report and how many underlying libraries have been upgraded, we can reopen this when we notice it again.

Thu, Jan 23, 2:10 PM · Storage manager
douardda added a comment to T757: Memory leak in swh.storage.api.server.

Is this still "a thing"?

Thu, Jan 23, 1:58 PM · Storage manager

Wed, Jan 22

vlorentz claimed T2214: Scale-out storage in production.
Wed, Jan 22, 4:46 PM · Storage manager, Restricted Project
vlorentz changed the status of T2214: Scale-out storage in production from Open to Work in Progress.
Wed, Jan 22, 4:46 PM · Storage manager, Restricted Project
vlorentz added projects to T2211: Go beyond git expressivity: Data Model, Storage manager, Mercurial loader.
Wed, Jan 22, 4:38 PM · Mercurial loader, Storage manager, Data Model, Restricted Project
vlorentz added a project to T2214: Scale-out storage in production: Storage manager.
Wed, Jan 22, 4:24 PM · Storage manager, Restricted Project
vlorentz added projects to T2215: Streaming support everywhere: Storage manager, Object storage, Web app.
Wed, Jan 22, 4:24 PM · meta-task, Web app, Object storage, Storage manager, Restricted Project
vlorentz renamed T1892: Cassandra as a storage backend from Cassandra as a storage backend (meta-task) to Cassandra as a storage backend.
Wed, Jan 22, 4:23 PM · meta-task, Storage manager
vlorentz updated the task description for T2243: Add Debian package python3-cassandra.
Wed, Jan 22, 3:22 PM · Storage manager
vlorentz triaged T2243: Add Debian package python3-cassandra as Normal priority.
Wed, Jan 22, 3:16 PM · Storage manager
vlorentz closed T1731: Intrinsic identifiers for origins as Resolved.

in addition we will also need to modify storage to store and allow retrieval of hashed origin URLs.

Wed, Jan 22, 2:18 PM · Storage manager, Data Model
vlorentz closed T1731: Intrinsic identifiers for origins, a subtask of T1892: Cassandra as a storage backend, as Resolved.
Wed, Jan 22, 2:18 PM · meta-task, Storage manager

Tue, Jan 21

ardumont updated the task description for T2239: storage: kafka issue: Can't pickle <class 'cimpl.KafkaException'>: import of module 'cimpl' failed.
Tue, Jan 21, 11:37 AM · Storage manager