Software Heritage storage server

diff --git a/PKG-INFO b/PKG-INFO index 0c4fb9bb..8eac0506 100644 --- a/PKG-INFO +++ b/PKG-INFO @@ -1,202 +1,202 @@ Metadata-Version: 2.1 Name: swh.storage -Version: 0.0.157 +Version: 0.0.158 Summary: Software Heritage storage manager Home-page: https://forge.softwareheritage.org/diffusion/DSTO/ Author: Software Heritage developers Author-email: swh-devel@inria.fr License: UNKNOWN Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Project-URL: Funding, https://www.softwareheritage.org/donate Project-URL: Source, https://forge.softwareheritage.org/source/swh-storage Description: swh-storage =========== Abstraction layer over the archive, allowing to access all stored source code artifacts as well as their metadata. See the [documentation](https://docs.softwareheritage.org/devel/swh-storage/index.html) for more details. ## Quick start ### Dependencies Python tests for this module include tests that cannot be run without a local Postgresql database, so you need the Postgresql server executable on your machine (no need to have a running Postgresql server). On a Debian-like host: ``` $ sudo apt install libpq-dev postgresql ``` ### Installation It is strongly recommended to use a virtualenv. In the following, we consider you work in a virtualenv named `swh`. See the [developer setup guide](https://docs.softwareheritage.org/devel/developer-setup.html#developer-setup) for a more details on how to setup a working environment. You can install the package directly from [pypi](https://pypi.org/p/swh.storage): ``` (swh) :~$ pip install swh.storage [...] ``` Or from sources: ``` (swh) :~$ git clone https://forge.softwareheritage.org/source/swh-storage.git [...] (swh) :~$ cd swh-storage (swh) :~/swh-storage$ pip install . [...] ``` Then you can check it's properly installed: ``` (swh) :~$ swh storage --help Usage: swh storage [OPTIONS] COMMAND [ARGS]... Software Heritage Storage tools. Options: -h, --help Show this message and exit. Commands: rpc-serve Software Heritage Storage RPC server. ``` ## Tests The best way of running Python tests for this module is to use [tox](https://tox.readthedocs.io/). ``` (swh) :~$ pip install tox ``` ### tox From the sources directory, simply use tox: ``` (swh) :~/swh-storage$ tox [...] ========= 315 passed, 6 skipped, 15 warnings in 40.86 seconds ========== _______________________________ summary ________________________________ flake8: commands succeeded py3: commands succeeded congratulations :) ``` ## Development The storage server can be locally started. It requires a configuration file and a running Postgresql database. ### Sample configuration A typical configuration `storage.yml` file is: ``` storage: cls: local args: db: "dbname=softwareheritage-dev user= password=" objstorage: cls: pathslicing args: root: /tmp/swh-storage/ slicing: 0:2/2:4/4:6 ``` which means, this uses: - a local storage instance whose db connection is to `softwareheritage-dev` local instance, - the objstorage uses a local objstorage instance whose: - `root` path is /tmp/swh-storage, - slicing scheme is `0:2/2:4/4:6`. This means that the identifier of the content (sha1) which will be stored on disk at first level with the first 2 hex characters, the second level with the next 2 hex characters and the third level with the next 2 hex characters. And finally the complete hash file holding the raw content. For example: 00062f8bd330715c4f819373653d97b3cd34394c will be stored at 00/06/2f/00062f8bd330715c4f819373653d97b3cd34394c Note that the `root` path should exist on disk before starting the server. ### Starting the storage server If the python package has been properly installed (e.g. in a virtual env), you should be able to use the command: ``` (swh) :~/swh-storage$ swh storage rpc-serve storage.yml ``` This runs a local swh-storage api at 5002 port. ``` (swh) :~/swh-storage$ curl http://127.0.0.1:5002 Software Heritage storage server

You have reached the Software Heritage storage server.
See its documentation and API for more information

``` ### And then what? In your upper layer ([loader-git](https://forge.softwareheritage.org/source/swh-loader-git/), [loader-svn](https://forge.softwareheritage.org/source/swh-loader-svn/), etc...), you can define a remote storage with this snippet of yaml configuration. ``` storage: cls: remote args: url: http://localhost:5002/ ``` You could directly define a local storage with the following snippet: ``` storage: cls: local args: db: service=swh-dev objstorage: cls: pathslicing args: root: /home/storage/swh-storage/ slicing: 0:2/2:4/4:6 ``` Platform: UNKNOWN Classifier: Programming Language :: Python :: 3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Development Status :: 5 - Production/Stable Description-Content-Type: text/markdown Provides-Extra: testing Provides-Extra: schemata Provides-Extra: journal diff --git a/swh.storage.egg-info/SOURCES.txt b/swh.storage.egg-info/SOURCES.txt index f5a6cc87..70788d23 100644 --- a/swh.storage.egg-info/SOURCES.txt +++ b/swh.storage.egg-info/SOURCES.txt @@ -1,225 +1,220 @@ MANIFEST.in Makefile Makefile.local README.md pytest.ini setup.py tox.ini version.txt ./requirements-swh-journal.txt ./requirements-swh.txt ./requirements-test.txt ./requirements.txt bin/swh-storage-add-dir sql/.gitignore sql/Makefile sql/TODO sql/clusters.dot sql/bin/db-upgrade sql/bin/dot_add_content sql/doc/json/.gitignore sql/doc/json/Makefile sql/doc/json/entity.lister_metadata.schema.json sql/doc/json/entity.metadata.schema.json sql/doc/json/entity_history.lister_metadata.schema.json sql/doc/json/entity_history.metadata.schema.json sql/doc/json/fetch_history.result.schema.json sql/doc/json/list_history.result.schema.json sql/doc/json/listable_entity.list_params.schema.json sql/doc/json/origin_visit.metadata.json sql/doc/json/tool.tool_configuration.schema.json sql/json/.gitignore sql/json/Makefile sql/json/entity.lister_metadata.schema.json sql/json/entity.metadata.schema.json sql/json/entity_history.lister_metadata.schema.json sql/json/entity_history.metadata.schema.json sql/json/fetch_history.result.schema.json sql/json/list_history.result.schema.json sql/json/listable_entity.list_params.schema.json sql/json/origin_visit.metadata.json sql/json/tool.tool_configuration.schema.json sql/upgrades/015.sql sql/upgrades/016.sql sql/upgrades/017.sql sql/upgrades/018.sql sql/upgrades/019.sql sql/upgrades/020.sql sql/upgrades/021.sql sql/upgrades/022.sql sql/upgrades/023.sql sql/upgrades/024.sql sql/upgrades/025.sql sql/upgrades/026.sql sql/upgrades/027.sql sql/upgrades/028.sql sql/upgrades/029.sql sql/upgrades/030.sql sql/upgrades/032.sql sql/upgrades/033.sql sql/upgrades/034.sql sql/upgrades/035.sql sql/upgrades/036.sql sql/upgrades/037.sql sql/upgrades/038.sql sql/upgrades/039.sql sql/upgrades/040.sql sql/upgrades/041.sql sql/upgrades/042.sql sql/upgrades/043.sql sql/upgrades/044.sql sql/upgrades/045.sql sql/upgrades/046.sql sql/upgrades/047.sql sql/upgrades/048.sql sql/upgrades/049.sql sql/upgrades/050.sql sql/upgrades/051.sql sql/upgrades/052.sql sql/upgrades/053.sql sql/upgrades/054.sql sql/upgrades/055.sql sql/upgrades/056.sql sql/upgrades/057.sql sql/upgrades/058.sql sql/upgrades/059.sql sql/upgrades/060.sql sql/upgrades/061.sql sql/upgrades/062.sql sql/upgrades/063.sql sql/upgrades/064.sql sql/upgrades/065.sql sql/upgrades/066.sql sql/upgrades/067.sql sql/upgrades/068.sql sql/upgrades/069.sql sql/upgrades/070.sql sql/upgrades/071.sql sql/upgrades/072.sql sql/upgrades/073.sql sql/upgrades/074.sql sql/upgrades/075.sql sql/upgrades/076.sql sql/upgrades/077.sql sql/upgrades/078.sql sql/upgrades/079.sql sql/upgrades/080.sql sql/upgrades/081.sql sql/upgrades/082.sql sql/upgrades/083.sql sql/upgrades/084.sql sql/upgrades/085.sql sql/upgrades/086.sql sql/upgrades/087.sql sql/upgrades/088.sql sql/upgrades/089.sql sql/upgrades/090.sql sql/upgrades/091.sql sql/upgrades/092.sql sql/upgrades/093.sql sql/upgrades/094.sql sql/upgrades/095.sql sql/upgrades/096.sql sql/upgrades/097.sql sql/upgrades/098.sql sql/upgrades/099.sql sql/upgrades/100.sql sql/upgrades/101.sql sql/upgrades/102.sql sql/upgrades/103.sql sql/upgrades/104.sql sql/upgrades/105.sql sql/upgrades/106.sql sql/upgrades/107.sql sql/upgrades/108.sql sql/upgrades/109.sql sql/upgrades/110.sql sql/upgrades/111.sql sql/upgrades/112.sql sql/upgrades/113.sql sql/upgrades/114.sql sql/upgrades/115.sql sql/upgrades/116.sql sql/upgrades/117.sql sql/upgrades/118.sql sql/upgrades/119.sql sql/upgrades/120.sql sql/upgrades/121.sql sql/upgrades/122.sql sql/upgrades/123.sql sql/upgrades/124.sql sql/upgrades/125.sql sql/upgrades/126.sql sql/upgrades/127.sql sql/upgrades/128.sql sql/upgrades/129.sql sql/upgrades/130.sql sql/upgrades/131.sql sql/upgrades/132.sql sql/upgrades/133.sql sql/upgrades/134.sql sql/upgrades/135.sql sql/upgrades/136.sql sql/upgrades/137.sql sql/upgrades/138.sql sql/upgrades/139.sql sql/upgrades/140.sql sql/upgrades/141.sql sql/upgrades/142.sql sql/upgrades/143.sql swh/__init__.py swh.storage.egg-info/PKG-INFO swh.storage.egg-info/SOURCES.txt swh.storage.egg-info/dependency_links.txt swh.storage.egg-info/entry_points.txt swh.storage.egg-info/requires.txt swh.storage.egg-info/top_level.txt swh/storage/__init__.py swh/storage/buffer.py swh/storage/cli.py swh/storage/common.py swh/storage/converters.py swh/storage/db.py swh/storage/exc.py swh/storage/filter.py swh/storage/in_memory.py swh/storage/py.typed swh/storage/storage.py swh/storage/algos/__init__.py swh/storage/algos/diff.py swh/storage/algos/dir_iterators.py swh/storage/algos/origin.py swh/storage/algos/revisions_walker.py swh/storage/algos/snapshot.py swh/storage/api/__init__.py swh/storage/api/client.py swh/storage/api/server.py -swh/storage/schemata/__init__.py -swh/storage/schemata/distribution.py swh/storage/sql/10-swh-init.sql swh/storage/sql/20-swh-enums.sql swh/storage/sql/30-swh-schema.sql swh/storage/sql/40-swh-func.sql swh/storage/sql/60-swh-indexes.sql swh/storage/tests/__init__.py swh/storage/tests/conftest.py swh/storage/tests/generate_data_test.py swh/storage/tests/storage_data.py swh/storage/tests/test_api_client.py swh/storage/tests/test_buffer.py swh/storage/tests/test_converters.py swh/storage/tests/test_db.py swh/storage/tests/test_filter.py swh/storage/tests/test_in_memory.py swh/storage/tests/test_init.py swh/storage/tests/test_server.py swh/storage/tests/test_storage.py swh/storage/tests/algos/__init__.py swh/storage/tests/algos/test_diff.py swh/storage/tests/algos/test_dir_iterator.py swh/storage/tests/algos/test_origin.py swh/storage/tests/algos/test_revisions_walker.py -swh/storage/tests/algos/test_snapshot.py -swh/storage/tests/schemata/__init__.py -swh/storage/tests/schemata/conftest.py -swh/storage/tests/schemata/test_distribution.py \ No newline at end of file +swh/storage/tests/algos/test_snapshot.py \ No newline at end of file diff --git a/swh/storage/schemata/__init__.py b/swh/storage/schemata/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/swh/storage/schemata/distribution.py b/swh/storage/schemata/distribution.py deleted file mode 100644 index b67e1151..00000000 --- a/swh/storage/schemata/distribution.py +++ /dev/null @@ -1,251 +0,0 @@ -# Copyright (C) 2017 the Software Heritage developers -# License: GNU General Public License version 3, or any later version -# See top-level LICENSE file for more information - -import binascii -from collections import defaultdict -import datetime - -from sqlalchemy import ( - Boolean, - Column, - DateTime, - Enum, - ForeignKey, - Integer, - LargeBinary, - String, - Table, - UniqueConstraint, -) - -try: - from sqlalchemy import JSON -except ImportError: - # SQLAlchemy < 1.1 - from sqlalchemy.dialects.postgresql import JSONB as JSON - -from sqlalchemy.ext.declarative import declarative_base -from sqlalchemy.orm import relationship - -SQLBase = declarative_base() - - -class Distribution(SQLBase): - """A distribution (e.g. Debian, Ubuntu, Fedora, ...)""" - __tablename__ = 'distribution' - - id = Column(Integer, primary_key=True) - name = Column(String, unique=True, nullable=False) - type = Column(Enum('deb', 'rpm', name='distribution_types'), - nullable=False) - mirror_uri = Column(String, nullable=False) - - areas = relationship('Area', back_populates='distribution') - - def origin_for_package(self, package_name, package_versions): - """Return the origin dictionary for the given package""" - return { - 'type': self.type, - 'url': '%s://%s/packages/%s' % ( - self.type, self.name, package_name - ), - } - - def __repr__(self): - return 'Distribution(%s (%s) on %s)' % ( - self.name, - self.type, - self.mirror_uri, - ) - - -class Area(SQLBase): - __tablename__ = 'area' - __table_args__ = ( - UniqueConstraint('distribution_id', 'name'), - ) - - id = Column(Integer, primary_key=True) - distribution_id = Column(Integer, ForeignKey('distribution.id'), - nullable=False) - name = Column(String, nullable=False) - active = Column(Boolean, nullable=False, default=True) - - distribution = relationship('Distribution', back_populates='areas') - - def index_uris(self): - """Get possible URIs for this component's package index""" - if self.distribution.type == 'deb': - compression_exts = ('xz', 'bz2', 'gz', None) - base_uri = '%s/dists/%s/source/Sources' % ( - self.distribution.mirror_uri, - self.name, - ) - for ext in compression_exts: - if ext: - yield (base_uri + '.' + ext, ext) - else: - yield (base_uri, None) - - else: - raise NotImplementedError( - 'Do not know how to build index URI for Distribution type %s' % - self.distribution.type - ) - - def __repr__(self): - return 'Area(%s of %s)' % ( - self.name, - self.distribution.name, - ) - - -class Package(SQLBase): - __tablename__ = 'package' - __table_args__ = ( - UniqueConstraint('area_id', 'name', 'version'), - ) - - id = Column(Integer, primary_key=True) - area_id = Column(Integer, ForeignKey('area.id'), nullable=False) - name = Column(String, nullable=False) - version = Column(String, nullable=False) - directory = Column(String, nullable=False) - files = Column(JSON, nullable=False) - - origin_id = Column(Integer) - task_id = Column(Integer) - - revision_id = Column(LargeBinary(20)) - - area = relationship('Area') - - @property - def distribution(self): - return self.area.distribution - - def fetch_uri(self, filename): - """Get the URI to fetch the `filename` file associated with the - package""" - if self.distribution.type == 'deb': - return '%s/%s/%s' % ( - self.distribution.mirror_uri, - self.directory, - filename, - ) - else: - raise NotImplementedError( - 'Do not know how to build fetch URI for Distribution type %s' % - self.distribution.type - ) - - def loader_dict(self): - ret = { - 'id': self.id, - 'name': self.name, - 'version': self.version, - } - if self.revision_id: - ret['revision_id'] = binascii.hexlify(self.revision_id).decode() - else: - files = { - name: checksums.copy() - for name, checksums in self.files.items() - } - for name in files: - files[name]['uri'] = self.fetch_uri(name) - - ret.update({ - 'revision_id': None, - 'files': files, - }) - return ret - - def __repr__(self): - return 'Package(%s_%s of %s %s)' % ( - self.name, - self.version, - self.distribution.name, - self.area.name, - ) - - -class DistributionSnapshot(SQLBase): - __tablename__ = 'distribution_snapshot' - - id = Column(Integer, primary_key=True) - date = Column(DateTime, nullable=False, index=True) - distribution_id = Column(Integer, - ForeignKey('distribution.id'), - nullable=False) - - distribution = relationship('Distribution') - areas = relationship('AreaSnapshot', back_populates='snapshot') - - def task_for_package(self, package_name, package_versions): - """Return the task dictionary for the given list of package versions""" - origin = self.distribution.origin_for_package( - package_name, package_versions, - ) - - return { - 'policy': 'oneshot', - 'type': 'load-%s-package' % self.distribution.type, - 'next_run': datetime.datetime.now(tz=datetime.timezone.utc), - 'arguments': { - 'args': [], - 'kwargs': { - 'origin': origin['url'], - 'date': self.date.isoformat(), - 'packages': package_versions, - }, - } - } - - def get_packages(self): - packages = defaultdict(dict) - for area_snapshot in self.areas: - area_name = area_snapshot.area.name - for package in area_snapshot.packages: - ref_name = '%s/%s' % (area_name, package.version) - packages[package.name][ref_name] = package.loader_dict() - - return packages - - -area_snapshot_package_assoc = Table( - 'area_snapshot_package', SQLBase.metadata, - Column('area_snapshot_id', Integer, ForeignKey('area_snapshot.id'), - nullable=False), - Column('package_id', Integer, ForeignKey('package.id'), - nullable=False), -) - - -class AreaSnapshot(SQLBase): - __tablename__ = 'area_snapshot' - - id = Column(Integer, primary_key=True) - snapshot_id = Column(Integer, - ForeignKey('distribution_snapshot.id'), - nullable=False) - area_id = Column(Integer, - ForeignKey('area.id'), - nullable=False) - - snapshot = relationship('DistributionSnapshot', back_populates='areas') - area = relationship('Area') - packages = relationship('Package', secondary=area_snapshot_package_assoc) - - -class TempPackage(SQLBase): - __tablename__ = 'temp_package' - __table_args__ = { - 'prefixes': ['TEMPORARY'], - } - - id = Column(Integer, primary_key=True) - area_id = Column(Integer) - name = Column(String) - version = Column(String) diff --git a/swh/storage/tests/schemata/__init__.py b/swh/storage/tests/schemata/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/swh/storage/tests/schemata/conftest.py b/swh/storage/tests/schemata/conftest.py deleted file mode 100644 index 5cfa8ae7..00000000 --- a/swh/storage/tests/schemata/conftest.py +++ /dev/null @@ -1,25 +0,0 @@ -# Copyright (C) 2019 The Software Heritage developers -# See the AUTHORS file at the top-level directory of this distribution -# License: GNU General Public License version 3, or any later version -# See top-level LICENSE file for more information - -import pytest -from pytest_postgresql.janitor import DatabaseJanitor -from sqlalchemy import create_engine - - -@pytest.fixture -def sqlalchemy_engine(postgresql_proc): - pg_host = postgresql_proc.host - pg_port = postgresql_proc.port - pg_user = postgresql_proc.user - - pg_db = 'sqlalchemy-tests' - - url = f'postgresql://{pg_user}@{pg_host}:{pg_port}/{pg_db}' - with DatabaseJanitor( - pg_user, pg_host, pg_port, pg_db, postgresql_proc.version - ): - engine = create_engine(url) - yield engine - engine.dispose() diff --git a/swh/storage/tests/schemata/test_distribution.py b/swh/storage/tests/schemata/test_distribution.py deleted file mode 100644 index 8c90c3e9..00000000 --- a/swh/storage/tests/schemata/test_distribution.py +++ /dev/null @@ -1,51 +0,0 @@ -# Copyright (C) 2019 The Software Heritage developers -# See the AUTHORS file at the top-level directory of this distribution -# License: GNU General Public License version 3, or any later version -# See top-level LICENSE file for more information - -import pytest -from swh.storage.schemata.distribution import SQLBase, Distribution, Area - -from sqlalchemy.orm import sessionmaker - - -@pytest.fixture -def session(sqlalchemy_engine): - SQLBase.metadata.create_all(sqlalchemy_engine) - Session = sessionmaker(bind=sqlalchemy_engine) - session = Session() - yield session - session.close() - - -def test_area_index_uris_deb(session): - d = Distribution( - name='Debian', type='deb', mirror_uri='http://deb.debian.org/debian' - ) - a = Area( - distribution=d, - name='unstable/main', - active=True, - ) - session.add_all([d, a]) - session.commit() - - uris = list(a.index_uris()) - assert uris - - -def test_area_index_uris_rpm(session): - d = Distribution( - name='CentOS', type='rpm', - mirror_uri='http://centos.mirrors.proxad.net/' - ) - a = Area( - distribution=d, - name='8', - active=True, - ) - session.add_all([d, a]) - session.commit() - - with pytest.raises(NotImplementedError): - list(a.index_uris()) diff --git a/tox.ini b/tox.ini index d719433b..4cca042f 100644 --- a/tox.ini +++ b/tox.ini @@ -1,41 +1,39 @@ [tox] envlist=flake8,mypy,py3 [testenv:py3] deps = .[testing] .[listener] - .[schemata] pytest-cov commands = pytest --hypothesis-profile=fast \ --cov={envsitepackagesdir}/swh/storage \ {envsitepackagesdir}/swh/storage \ --cov-branch {posargs} [testenv:py3-slow] deps = .[testing] .[listener] - .[schemata] pytest-cov commands = pytest --hypothesis-profile=slow \ --cov={envsitepackagesdir}/swh/storage \ {envsitepackagesdir}/swh/storage \ --cov-branch {posargs} [testenv:flake8] skip_install = true deps = flake8 commands = {envpython} -m flake8 [testenv:mypy] skip_install = true deps = .[testing] mypy commands = mypy swh diff --git a/version.txt b/version.txt index a5236fd6..057bf524 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -v0.0.157-0-gbb5d405 \ No newline at end of file +v0.0.158-0-ge296dfb \ No newline at end of file