Page MenuHomeSoftware Heritage

Storage managerFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Recent Activity

Yesterday

tenma closed T2287: Improve code in BufferingProxyStorage as Resolved.
Fri, Sep 25, 4:47 PM · Easy hack, Storage manager
zack added a revision to T1926: FUSE filesystem to navigate the archive: D4042: docs: add design notes.
Fri, Sep 25, 4:03 PM · Storage manager, Graph service

Thu, Sep 24

vlorentz added a parent task for T1910: Redesign origin search using a dedicated component (swh-search): T1117: Origin search is *slow* when you look for very common words.
Thu, Sep 24, 11:05 AM · Archive search, Storage manager
vlorentz added a subtask for T1117: Origin search is *slow* when you look for very common words: T1910: Redesign origin search using a dedicated component (swh-search).
Thu, Sep 24, 11:05 AM · Web app, Storage manager
vlorentz claimed T1117: Origin search is *slow* when you look for very common words.
Thu, Sep 24, 11:05 AM · Web app, Storage manager
vlorentz added a comment to T1117: Origin search is *slow* when you look for very common words.

aka T1910

Thu, Sep 24, 11:05 AM · Web app, Storage manager

Wed, Sep 23

ardumont closed D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.
Wed, Sep 23, 1:58 PM · Storage manager
douardda accepted D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.
Wed, Sep 23, 1:11 PM · Storage manager
tenma added a revision to T2287: Improve code in BufferingProxyStorage: D4017: Improve code quality and doc in BufferedProxyStorage.
Wed, Sep 23, 1:08 PM · Easy hack, Storage manager
swh-public-ci added a comment to D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.

Build is green

Wed, Sep 23, 12:35 PM · Storage manager
ardumont updated the diff for D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.

Rename tests db to "storage"

Wed, Sep 23, 12:28 PM · Storage manager
ardumont added inline comments to D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.
Wed, Sep 23, 12:06 PM · Storage manager
ardumont added inline comments to D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.
Wed, Sep 23, 12:02 PM · Storage manager
ardumont added inline comments to D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.
Wed, Sep 23, 12:00 PM · Storage manager
ardumont added a comment to D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.

it's unclear to me if the hunk in swh_storage_backend_config is related with the fix or not.

Wed, Sep 23, 12:00 PM · Storage manager
swh-public-ci added a comment to D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.

Build is green

Wed, Sep 23, 11:58 AM · Storage manager
douardda added a comment to D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.

it's unclear to me if the hunk in swh_storage_backend_config is related with the fix or not.
isn't the very last diff hunk enough for the job?

Wed, Sep 23, 11:56 AM · Storage manager
ardumont updated the diff for D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.
  • Drop unneeded postgresql_proc
Wed, Sep 23, 11:54 AM · Storage manager
ardumont updated the summary of D4014: pytest_plugin: Change dbname to storage to avoid clash in tests.
Wed, Sep 23, 11:52 AM · Storage manager
ardumont added a comment to P773 storage check at startup in deposit tests: dbversion discrepancy make the tests refuse to start....

Related to D4013

Wed, Sep 23, 10:55 AM · Storage manager
ardumont updated subscribers of P773 storage check at startup in deposit tests: dbversion discrepancy make the tests refuse to start....

More information:

(Pdb) cfg['storage']
{'check_config': {'check_write': True}, 'cls': 'local', 'db': 'postgresql://postgres@127.0.0.1:16607/tests', 'objstorage': {'args': {}, 'cls': 'memory'}}
(Pdb) cfg['scheduler']
{'args': {'db': "user=postgres password=xxx dbname=tests host=127.0.0.1 port=16607 options=''"}, 'cls': 'local'}
Wed, Sep 23, 10:55 AM · Storage manager
ardumont created P773 storage check at startup in deposit tests: dbversion discrepancy make the tests refuse to start....
Wed, Sep 23, 10:08 AM · Storage manager

Tue, Sep 22

olasd placed T1117: Origin search is *slow* when you look for very common words up for grabs.

This is very probably superseded by @vlorentz 's work on swh.search.

Tue, Sep 22, 4:47 PM · Web app, Storage manager
ardumont triaged T2622: Proxy storages: Split storage.'*_missing' calls in chunks as Normal priority.
Tue, Sep 22, 9:45 AM · Storage manager

Mon, Sep 21

vlorentz closed T2053: support graph export for the cassandra backend as Resolved.
Mon, Sep 21, 3:39 PM · Graph service, Storage manager

Fri, Sep 18

tenma claimed T2287: Improve code in BufferingProxyStorage.
Fri, Sep 18, 11:59 AM · Easy hack, Storage manager

Thu, Sep 17

olasd added a comment to T2604: Handle multiple "database profiles" in the swh-storage (/...) SQL configurations.

Btw, sqitch uses the native scripting for the database engine (in case of postgres, psql scripts), so if we ever end up using it, we can keep this approach.

Thu, Sep 17, 8:34 PM · Storage manager
olasd added a comment to T2604: Handle multiple "database profiles" in the swh-storage (/...) SQL configurations.

So, I've first attempted something along the lines of https://www.depesz.com/2008/06/18/conditional-ddl/, which uses a function to execute DDL commands stored in a string.

Thu, Sep 17, 8:26 PM · Storage manager
olasd added a revision to T2604: Handle multiple "database profiles" in the swh-storage (/...) SQL configurations: D3981: Support different database flavors in the SQL scripts.
Thu, Sep 17, 8:05 PM · Storage manager

Wed, Sep 16

olasd triaged T2604: Handle multiple "database profiles" in the swh-storage (/...) SQL configurations as Normal priority.
Wed, Sep 16, 5:32 PM · Storage manager

Tue, Sep 15

vlorentz triaged T2602: Investigate how to upgrade the schema of the Cassandra storage as Normal priority.
Tue, Sep 15, 1:56 PM · Storage manager, Restricted Project
seirl added a comment to T2600: SQL storage: experiment with flattened layouts for directory nodes.

We considered three possibilities for the schema (assuming that we want to get rid of the three separate tables for dir_entries, rev_entries and file_entries -- otherwise, there's 6 possibilities).

Tue, Sep 15, 1:40 PM · Storage manager
zack triaged T2600: SQL storage: experiment with flattened layouts for directory nodes as Normal priority.
Tue, Sep 15, 12:53 PM · Storage manager

Mon, Sep 14

vlorentz triaged T2590: Finish the indexer -> swh-search pipeline as Normal priority.
Mon, Sep 14, 5:39 PM · Journal, Archive search
ardumont closed T2547: pgstorage: Migrate db to storage 0.13.2 (db versions 160, 161) as Resolved.
Mon, Sep 14, 1:09 PM · System administration, Storage manager
ardumont added a comment to T2547: pgstorage: Migrate db to storage 0.13.2 (db versions 160, 161).

everything ran now including and up to D3936.

Mon, Sep 14, 12:24 PM · System administration, Storage manager
ardumont added a revision to T2547: pgstorage: Migrate db to storage 0.13.2 (db versions 160, 161): D3936: sql: Make the extra_headers not null a constraint.
Mon, Sep 14, 11:00 AM · System administration, Storage manager
zack assigned T1926: FUSE filesystem to navigate the archive to haltode.
Mon, Sep 14, 9:59 AM · Storage manager, Graph service

Thu, Sep 10

ardumont changed the status of T2547: pgstorage: Migrate db to storage 0.13.2 (db versions 160, 161) from Open to Work in Progress.

started back the migration, this time running from a shared tmux session on belvedere (before it was with my user):

Thu, Sep 10, 4:58 PM · System administration, Storage manager

Wed, Sep 9

ardumont added a comment to T2547: pgstorage: Migrate db to storage 0.13.2 (db versions 160, 161).

revisions done so far (as of yesterday, when it got stopped):

Wed, Sep 9, 5:39 PM · System administration, Storage manager
ardumont changed the status of T2547: pgstorage: Migrate db to storage 0.13.2 (db versions 160, 161) from Work in Progress to Open.
Wed, Sep 9, 11:20 AM · System administration, Storage manager
ardumont added a project to T2547: pgstorage: Migrate db to storage 0.13.2 (db versions 160, 161): System administration.
Wed, Sep 9, 11:18 AM · System administration, Storage manager
ardumont updated the task description for T2547: pgstorage: Migrate db to storage 0.13.2 (db versions 160, 161).
Wed, Sep 9, 11:17 AM · System administration, Storage manager
ardumont added a comment to T2547: pgstorage: Migrate db to storage 0.13.2 (db versions 160, 161).

stand-by related to T2561

Wed, Sep 9, 11:13 AM · System administration, Storage manager

Tue, Sep 8

vlorentz closed T2556: using InMemoryStorage.directory_ls may return `status: None` as Resolved.
Tue, Sep 8, 2:14 PM · Storage manager

Mon, Sep 7

zack raised the priority of T1926: FUSE filesystem to navigate the archive from Wishlist to Normal.
Mon, Sep 7, 10:59 AM · Storage manager, Graph service

Sun, Sep 6

zack updated subscribers of T1926: FUSE filesystem to navigate the archive.

Noting down that I had a tentative very preliminary implementation in the feature/fuse branch of swh-graph.
It's probably no worth picking up and we should restart from scratch at this point, but might still contain useful material.
(The webclient in there has since become a proper thing, see T2279. So that part is definitely obsolete.)

Sun, Sep 6, 4:48 PM · Storage manager, Graph service

Fri, Sep 4

ardumont added a revision to T645: Type swh-storage endpoints with swh.model objects: D3883: algos.diff: Add missed revision_get conversion.
Fri, Sep 4, 3:37 PM · Data Model, Storage manager
vlorentz removed a project from T2564: migrate existing revisions metadata extra_headers to actual extra_headers field: Datasets.
Fri, Sep 4, 11:34 AM · Storage manager
ardumont added projects to T2564: migrate existing revisions metadata extra_headers to actual extra_headers field: Storage manager, Datasets.
Fri, Sep 4, 11:30 AM · Storage manager