Page MenuHomeSoftware Heritage
Differential D637 Diff 2013 swh/storage/storage.py

Changeset View

Standalone View

View Options

swh/storage/storage.py

Show First 20 Lines • Show All 1,283 Lines • ▼ Show 20 Lines def tool_get(self, tool, db=None, cur=None):
tool_conf) tool_conf)
if not idx: if not idx:
return None return None
return dict(zip(db.tool_cols, idx)) return dict(zip(db.tool_cols, idx))
@db_transaction() @db_transaction()
def metadata_provider_add(self, provider_name, provider_type, provider_url, def metadata_provider_add(self, provider_name, provider_type, provider_url,
metadata, db=None, cur=None): metadata, db=None, cur=None):
"""Add a metadata provider.
Args:
provider_name (str): Its name
moraneggUnsubmitted
Done
Inline Actions

It might be pertinent to write a type example or the potential types list.

moranegg: It might be pertinent to write a type example or the potential types list.
douarddaUnsubmitted
Not Done
Inline Actions

I'll go one step further:

  • what are the possible values for this provider_type (I've not checked the code for this) ? If its a limited list of possible values, give them, otherwise tell how valid values can be guessed.
  • add a complete and valid example of arguments (or give a clue on where such a usage example can be found in the code)

Also, there should somewhere a description of what this metadata object stands for. I guess it is the metadata describing the metadata provider (which might be source of confusion; which metadata are we talking about in there?)

douardda: I'll go one step further: - what are the possible values for this provider_type (I've not…
vlorentzAuthorUnsubmitted
Done
Inline Actions

tbh, I don't know the answers to these questions

vlorentz: tbh, I don't know the answers to these questions
moraneggUnsubmitted
Not Done
Inline Actions

what are the possible values for this provider_type (I've not checked the code for this) ? If its a limited list of possible values, give them, otherwise tell how valid values can be guessed.

It isn't a limited list for now, we have at the moment only one provider_type deposit_client, but the metadata provider entity should be used in the following future cases:

  • when listing: lister type
  • when loading: loader type
  • when fetching metadata from registries: registry type

Because it is not implemented, I'm not sure if we should list the types above in the docs

add a complete and valid example of arguments (or give a clue on where such a usage example can be found in the code)

Here is the actual metadata_provider in the storage today

idprovider_nameprovider_typeprovider_urlmetadata
3haldeposit_clienthttps://hal.archives-ouvertes.fr/{}
4swhdeposit_clienthttps://www.softwareheritage.org{}

with the use cases I mentioned we could add:

idprovider_nameprovider_typeprovider_urlmetadata
xgitlab_listerlisterhttps://gitlab.com/{}
ygitlab_loaderloaderhttps://gitlab.com/{}
zwikidataregistryhttps://www.wikidata.org{}

Also, there should somewhere a description of what this metadata object stands for. I guess it is the metadata describing the metadata provider (which might be source of confusion; which metadata are we talking about in there?)

You are right ! we do not use at the moment the metadata property for a metadata_provider, this property was added for registries where we want to keep more information about the registry itself.

@vlorentz you can add the existing hal deposit_client example as is
The rest isn't documented, we had a couple of emails on swh-devel, but we might need specs on this subject.

For a larger discussion:
@douardda where would you put specs on future features?
For the deposit, we used the \docs\specs for it, maybe wiki pages are more appropriate.

moranegg: > what are the possible values for this provider_type (I've not checked the code for this) ? If…
provider_type (str): Its type (eg. `'deposit-client'`)
provider_url (str): Its URL
metadata: JSON-encodable object
Returns:
dict: same as args, plus an 'id' key.
"""
return db.metadata_provider_add(provider_name, provider_type, return db.metadata_provider_add(provider_name, provider_type,
provider_url, metadata, cur) provider_url, metadata, cur)
@db_transaction() @db_transaction()
def metadata_provider_get(self, provider_id, db=None, cur=None): def metadata_provider_get(self, provider_id, db=None, cur=None):
"""Get a metadata provider
Args:
provider_id: Its identifier, as given by `metadata_provider_add`.
Returns:
dict: same as `metadata_provider_add`.
"""
result = db.metadata_provider_get(provider_id) result = db.metadata_provider_get(provider_id)
if not result: if not result:
return None return None
return dict(zip(db.metadata_provider_cols, result)) return dict(zip(db.metadata_provider_cols, result))
@db_transaction() @db_transaction()
def metadata_provider_get_by(self, provider, db=None, cur=None): def metadata_provider_get_by(self, provider, db=None, cur=None):
"""Get a metadata provider
Args:
provider (dict): A dictionary with keys:
* provider_name: Its name
* provider_url: Its URL
Returns:
dict: same as `metadata_provider_add`.
"""
result = db.metadata_provider_get_by(provider['provider_name'], result = db.metadata_provider_get_by(provider['provider_name'],
provider['provider_url']) provider['provider_url'])
if not result: if not result:
return None return None
return dict(zip(db.metadata_provider_cols, result)) return dict(zip(db.metadata_provider_cols, result))
def diff_directories(self, from_dir, to_dir, track_renaming=False): def diff_directories(self, from_dir, to_dir, track_renaming=False):
"""Compute the list of file changes introduced between two arbitrary """Compute the list of file changes introduced between two arbitrary
▲ Show 20 Lines • Show All 46 Lines • Show Last 20 Lines
About · Developer information · Legal