Page MenuHomeSoftware Heritage
Create Task
Maniphest T2213

Storage
Open, NormalPublic
Actions

Related Objects
Search...

StatusAssignedTask
 OpenNoneT2213 Storage
 Work in ProgressvlorentzT2214 Scale-out graph and database storage in production
 OpenvlorentzT1892 Cassandra as a storage backend
 ResolvedvlorentzT1816 Make all clients of swh-storage use origin URL as identifier and use visit types instead of origin types
 ResolvedvlorentzT1731 Intrinsic identifiers for origins
 ResolvedvlorentzT1910 Redesign origin search using a dedicated component (swh-search)
 ResolvedolasdT2052 Publish swh-search on PyPI
 ResolvedvsellierT2182 Switch production swh-web to use swh-search instead of postgresql search.
 ResolvedvsellierT2497 Create an ElasticSearch cluster tuned for origin/metadata search
 ResolvedvlorentzT2590 Finish the indexer -> swh-search pipeline
 ResolvedvlorentzT2651 Make the indexer-storage publish its rows to Kafka
 ResolvedvlorentzT2652 Make the indexer-storage interface use attr classes instead of dicts
 ResolvedvsellierT2816 Enable the journal-writer for the swh-idx-storage in staging
 ResolvedvsellierT2817 Enable the swh-search environment in staging
 ResolvedvlorentzT2876 metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types
 ResolvedvsellierT2904 Create a new production webapp using the frozen index on the staging ES
 ResolvedvsellierT2905 Deploy swh-search for production
 ResolvedvlorentzT2936 Update the swh-search journal client to only set "has_visit" on "full" status of the visit
 ResolvedvsellierT2944 Deploy swh-search v0.4.1
 ResolvedardumontT3037 Reschedule origin-intrinsic-metadata tasks for all origins
 ResolvedardumontT2780 Enable the journal-writer for the swh-idx-storage in production
 ResolvedvsellierT3040 [production] Enable swh-search's journal-client for indexed objects
 ResolvedvsellierT3041 [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata
 ResolvedvsellierT3373 Metadata search is failing due to a boolean field in the mapping of the metadata fields
 ResolvedvsellierT3391 [swh-search] Deploy v0.9.0 on staging and execute a full origin and metadata reindexation
 ResolvedvsellierT3392 [staging] Properly recreate the origin_intrinsic_metadata topic
 ResolvedvsellierT3398 [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation
 ResolvedanlambertT3047 Enable to search in origin metadata with swh-search in webapp
 ResolvedvlorentzT3058 Metadata search is failing with "failed to parse date field"
 ResolvedvsellierT3060 Deploy swh-search v0.6.0 in **staging**
 ResolvedvsellierT3076 [swh-search] Improve the index/mapping migration process
 ResolvedvsellierT3083 Deploy swh-search v0.7.0/v0.7.1
 ResolvedvlorentzT1912 Support origin pagination without origin ids
 Work in ProgressvlorentzT2033 Run Cassandra storage backend with production data
 OpenNoneT2498 Re-create the Cassandra cluster using on-premise servers
 ResolvedvlorentzT2185 Make webapp0 use Cassandra as storage backend.
 ResolvedvlorentzT2186 Merge swh-storage-cassandra in swh-storage master
 ResolvedvlorentzT2183 Switch webapp0 to use swh-search instead of postgresql search.
 ResolvedardumontT2167 Deploy swh-search
 DuplicateolasdT2174 Add debian package for swh-search
 ResolvedvlorentzT2184 Replay origins to ElasticSearch's "origin" index
 ResolvedvlorentzT2602 Investigate how to upgrade the schema of the Cassandra storage
 ResolvedvlorentzT3314 Test swh.storage.cassandra with ScyllaDB
 Work in ProgressvsellierT3357 Perform some tests of the cassandra storage on Grid5000
 ResolvedvlorentzT3394 cassandra - origin url hashing encoding issue
 ResolvedvsellierT3395 cassandra - Timeouts during revision import
 ResolvedvsellierT3396 cassandra - allow to configure the consistency level used by the queries
 ResolvedvsellierT3464 Prepare a quote for the cassandra servers
 ResolvedvsellierT3465 Test multidatacenter replication
 DuplicatevsellierT3491 Origin visit ids restart from 1 even if there is previous visits
 ResolvedvsellierT3493 [cassandra] Git loader performance are very bad
 ResolvedvsellierT3517 [cassandra] decorate the method calls to have statsd metrics
Work in ProgressvsellierT3573 [cassandra] directory and content read benchmarks
 Work in ProgressvsellierT3577 Parallel loaders performances
OpenvlorentzT3585 Fix inconsistencies of the Cassandra backend with postgres
 WontfixvlorentzT3582 cassandra: Use 'git ordering' for directory entries
 OpenNoneT3586 Figure out what to do with 'misordered' directories in Cassandra
 OpenNoneT2215 Streaming support everywhere
 OpenNoneT2216 Packing object storage

Event Timeline

douardda created this task.Jan 20 2020, 2:23 PM
vlorentz triaged this task as Normal priority.Jan 22 2020, 4:19 PM
vlorentz renamed this task from Storage to [meta-task] Storage.Jan 22 2020, 4:21 PM
vlorentz renamed this task from [meta-task] Storage to Storage.
vlorentz added a project: meta-task.
vlorentz changed the status of subtask T2214: Scale-out graph and database storage in production from Open to Work in Progress.Jan 22 2020, 4:46 PM
rdicosmo mentioned this in T3116: Roll out first operational mirror.Mar 11 2021, 7:59 PM
About · Developer information · Legal