diff --git a/Makefile b/Makefile index 524175c2b..5db623f03 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,5 @@ -# Makefile driver for SWH Python modules. DO NOT CHANGE. +# Makefile driver for swh Python modules. DO NOT CHANGE. # You can add custom Makefile rules to Makefile.local include ../Makefile.python -include Makefile.local diff --git a/README.md b/README.md index b203ec393..42acbe3a6 100644 --- a/README.md +++ b/README.md @@ -1,106 +1,106 @@ # swh-web This repository holds the development of Software Heritage web applications: -* SWH Web API (https://archive.softwareheritage.org/api): enables to query the content of the SWH archive through HTTP requests and get responses in JSON or YAML. +* swh-web API (https://archive.softwareheritage.org/api): enables to query the content of the archive through HTTP requests and get responses in JSON or YAML. -* SWH Web browse (https://archive.softwareheritage.org/browse): graphical interface that eases the navigation in the SWH archive. +* swh-web browse (https://archive.softwareheritage.org/browse): graphical interface that eases the navigation in the archive. Documentation about how to use these components but also the details of their URI schemes can be found in the docs folder. The produced HTML documentation can be read and browsed at https://docs.softwareheritage.org/devel/swh-web/index.html. ## Technical details Those applications are powered by: * [Django Web Framework](https://www.djangoproject.com/) on the backend side with the following extensions enabled: * [django-rest-framework](http://www.django-rest-framework.org/) * [django-webpack-loader](https://github.com/owais/django-webpack-loader) * [django-js-reverse](http://django-js-reverse.readthedocs.io/en/latest/) * [webpack](https://webpack.js.org/) on the frontend side for better static assets management, including: * assets dependencies management and retrieval through [npm](https://www.npmjs.com/) * linting of custom javascript code (through [eslint](https://eslint.org/)) and stylesheets (through [stylelint](https://stylelint.io/)) * use of [es6](http://es6-features.org) syntax and advanced javascript feature like [async/await](https://javascript.info/async-await) or [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) thanks to [babel](https://babeljs.io/) (es6 to es5 transpiler and polyfills provider) * assets minification (using [UglifyJS](https://github.com/mishoo/UglifyJS2) and [cssnano](http://cssnano.co/)) but also dead code elimination for production use ## How to build and run ### Requirements First you will need [Python 3](https://www.python.org) and a complete [swh development environment](https://forge.softwareheritage.org/source/swh-environment/) installed. To run the backend, you need to have the following Python 3 modules installed: * beautifulsoup4 * django >= 1.10.7 * djangorestframework >= 3.4.0 * django_webpack_loader * django_js_reverse * docutils * file_magic >= 0.3.0 * htmlmin * lxml * pygments * pypandoc * python-dateutil * pyyaml * requests To compile the frontend assets, you need to have [nodejs](https://nodejs.org/en/) >= 8.x (preferably version 8.x LTS) and [npm](https://www.npmjs.com/) (or [yarn](https://yarnpkg.com/en/)) installed. If you are on Debian stretch, you can easily install an up to date nodejs/npm from the stretch-backports repository or by following the instructions located at https://github.com/nodesource/distributions. Once you have installed nodejs, issue the following command in the root directory of swh-web in order to retrieve all the frontend dependencies: ``` $ npm install ``` or if you prefer to use yarn (faster than npm): ``` $ yarn install ``` Please note that the static assets bundles generated by webpack are not stored in the git repository. Follow the instructions below in order to generate them in order to be able to run the frontend part of the web applications. ### Make targets Below is the list of available make targets that can be executed from the root directory of swh-web in order to build and/or execute the web applications under various configurations: * **run-django-webpack-devserver**: Compile and serve not optimized (without mignification and dead code elimination) frontend static assets using [webpack-dev-server](https://github.com/webpack/webpack-dev-server) and run django server with development settings. This is the recommended target to use when developing swh-web as it enables automatic reloading of backend and frontend part of the applications when modifying source files (*.py, *.js, *.css, *.html). * **run-django-webpack-dev**: Compile not optimized (no minification, no dead code elimination) frontend static assets using webpack and run django server with development settings. This is the recommended target when one only wants to develop the backend side of the application. * **run-django-webpack-prod**: Compile optimized (with minification and dead code elimination) frontend static assets using webpack and run django server with production settings. This is useful to test the applications in production mode (with the difference that static assets are served by django). Production settings notably enable advanced django caching and you will need to have [memcached](https://memcached.org/) installed for that feature to work. * **run-django-server-dev**: Run the django server with development settings but without compiling frontend static assets through webpack. * **run-django-server-prod**: Run the django server with production settings but without compiling frontend static assets through webpack. * **run-gunicorn-server**: Run the web applications with production settings in a [gunicorn](http://gunicorn.org/) worker as they will be in real production environment. Once one of these targets executed, the web applications can be executed by pointing your browser to http://localhost:5004. ### Npm/Yarn targets Below is a list of available npm/yarn targets in order to only execute the frontend static assets compilation (no web server will be executed): * **build-dev**: compile not optimized (without mignification and dead code elimination) frontend static assets and store the results in the `swh/web/static` folder. * **build**: compile optimized (with mignification and dead code elimination) frontend static assets and store the results in the `swh/web/static` folder. **The build target must be executed prior performing the Debian packaging of swh-web** in order for the package to contain the optimized assets dedicated to production environment. To execute these targets, issue the following commmand: ``` $ npm run ``` or if you prefer using yarn: ``` $ yarn ``` diff --git a/docs/developers-info.rst b/docs/developers-info.rst index 51f75de89..72f25993a 100644 --- a/docs/developers-info.rst +++ b/docs/developers-info.rst @@ -1,135 +1,131 @@ Developers Information ====================== Sample configuration -------------------- -The configuration will be taken from the default configuration file: *~/.config/swh/web/web.yml*. +The configuration will be taken from the default configuration file: ``~/.config/swh/web/web.yml``. The following introduces a default configuration file: .. sourcecode:: yaml storage: cls: remote args: url: http://localhost:5002 debug: false throttling: cache_uri: None scopes: swh_api: limiter_rate: default: 120/h exempted_networks: - 127.0.0.0/8 Run server ---------- Either use the django manage script directly (useful in development mode as it offers various commands): .. sourcecode:: shell $ python3 -m swh.web.manage runserver or use the following shortcut: .. sourcecode:: shell $ make run Modules description ------------------- Common to all web applications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Configuration and settings """""""""""""""""""""""""" - * :mod:`swh.web.config`: holds the SWH configuration for the web applications. + * :mod:`swh.web.config`: holds the configuration for the web applications. * :mod:`swh.web.doc_config`: utility module used to extend the sphinx configuration when building the documentation. * :mod:`swh.web.manage`: Django management module for developers. * :mod:`swh.web.urls`: module that holds the whole URI scheme of all the web applications. * :mod:`swh.web.wsgi`: WSGI module to use when deploying the web applications in production. * :mod:`swh.web.settings.common`: Common Django settings * :mod:`swh.web.settings.development`: Django settings for development * :mod:`swh.web.settings.production`: Django settings for production * :mod:`swh.web.settings.tests`: Django settings for tests Common utilities """""""""""""""" - * :mod:`swh.web.common.converters`: conversion module used to transform SWH raw data + * :mod:`swh.web.common.converters`: conversion module used to transform raw data to serializable ones. It is used by :mod:`swh.web.common.service`: to convert data before transmitting then to Django views. * :mod:`swh.web.common.exc`: module defining exceptions used in the web applications. * :mod:`swh.web.common.highlightjs`: utility module to ease the use of the highlightjs_ library in produced Django views. * :mod:`swh.web.common.query`: Utilities to parse data from HTTP endpoints. It is used by :mod:`swh.web.common.service`. * :mod:`swh.web.common.service`: Orchestration layer used by views module in charge of communication with :mod:`swh.storage` to retrieve information and perform conversion for the upper layer. * :mod:`swh.web.common.swh_templatetags`: Custom Django template tags library for swh. * :mod:`swh.web.common.throttling`: Custom request rate limiter to use with the `Django REST Framework `_ * :mod:`swh.web.common.urlsindex`: Utilities to help the registering of endpoints for the web applications * :mod:`swh.web.common.utils`: Utility functions used in the web applications implementation -SWH Web API application +swh-web API application ^^^^^^^^^^^^^^^^^^^^^^^ * :mod:`swh.web.api.apidoc`: Utilities to document the web api for its html browsable rendering. * :mod:`swh.web.api.apiresponse`: Utility module to ease the generation of web api responses. - * :mod:`swh.web.api.apiurls`: Utilities to facilitate the registration of SWH - web api endpoints. + * :mod:`swh.web.api.apiurls`: Utilities to facilitate the registration of web api endpoints. * :mod:`swh.web.api.urls`: Module that defines the whole URI scheme for the api endpoints - * :mod:`swh.web.api.utils`: Utility functions used in the SWH web api implementation. + * :mod:`swh.web.api.utils`: Utility functions used in the web api implementation. * :mod:`swh.web.api.views.content`: Implementation of API endpoints for getting information - about SWH contents. + about contents. * :mod:`swh.web.api.views.directory`: Implementation of API endpoints for getting information - about SWH directories. + about directories. * :mod:`swh.web.api.views.entity`: Implementation of API endpoints for getting information - about SWH entities. + about entities. * :mod:`swh.web.api.views.origin`: Implementation of API endpoints for getting information - about SWH origins. + about origins. * :mod:`swh.web.api.views.person`: Implementation of API endpoints for getting information - about SWH persons. + about persons. * :mod:`swh.web.api.views.release`: Implementation of API endpoints for getting information - about SWH releases. + about releases. * :mod:`swh.web.api.views.revision`: Implementation of API endpoints for getting information - about SWH revisions. + about revisions. * :mod:`swh.web.api.views.snapshot`: Implementation of API endpoints for getting information - about SWH snapshots. + about snapshots. * :mod:`swh.web.api.views.stat`: Implementation of API endpoints for getting information - about SWH archive statistics. + about archive statistics. * :mod:`swh.web.api.views.utils`: Utilities used in the web api endpoints implementation. -SWH Web browse application +swh-web browse application ^^^^^^^^^^^^^^^^^^^^^^^^^^ - * :mod:`swh.web.browse.browseurls`: Utilities to facilitate the registration of SWH web - browse endpoints. - * :mod:`swh.web.browse.urls`: Module that defines the whole URI scheme for the SWH web - browse endpoints. - * :mod:`swh.web.browse.utils`: Utilities functions used troughout the SWH web browse - endpoints implementation. - * :mod:`swh.web.browse.views.content`: Implementation of endpoints for browsing SWH contents. - * :mod:`swh.web.browse.views.directory`: Implementation of endpoints for browsing SWH directories. - * :mod:`swh.web.browse.views.identifiers`: Implementation of endpoints for browsing SWH objects + * :mod:`swh.web.browse.browseurls`: Utilities to facilitate the registration of browse endpoints. + * :mod:`swh.web.browse.urls`: Module that defines the whole URI scheme for the browse endpoints. + * :mod:`swh.web.browse.utils`: Utilities functions used throughout the browse endpoints implementation. + * :mod:`swh.web.browse.views.content`: Implementation of endpoints for browsing contents. + * :mod:`swh.web.browse.views.directory`: Implementation of endpoints for browsing directories. + * :mod:`swh.web.browse.views.identifiers`: Implementation of endpoints for browsing objects through persistent identifiers. - * :mod:`swh.web.browse.views.origin`: Implementation of endpoints for browsing SWH origins. - * :mod:`swh.web.browse.views.person`: Implementation of endpoints for browsing SWH persons. - * :mod:`swh.web.browse.views.release`: Implementation of endpoints for browsing SWH releases. - * :mod:`swh.web.browse.views.revision`: Implementation of endpoints for browsing SWH revisions. - * :mod:`swh.web.browse.views.snapshot`: Implementation of endpoints for browsing SWH snapshots. + * :mod:`swh.web.browse.views.origin`: Implementation of endpoints for browsing origins. + * :mod:`swh.web.browse.views.person`: Implementation of endpoints for browsing persons. + * :mod:`swh.web.browse.views.release`: Implementation of endpoints for browsing releases. + * :mod:`swh.web.browse.views.revision`: Implementation of endpoints for browsing revisions. + * :mod:`swh.web.browse.views.snapshot`: Implementation of endpoints for browsing snapshots. .. _highlightjs: https://highlightjs.org/ diff --git a/docs/uri-scheme-api.rst b/docs/uri-scheme-api.rst index b059c15e4..d451f8799 100644 --- a/docs/uri-scheme-api.rst +++ b/docs/uri-scheme-api.rst @@ -1,24 +1,24 @@ .. _swh-web-api-urls: -SWH Web API URLs +swh-web API URLs ================ .. include:: uri-scheme-api-content.rst .. include:: uri-scheme-api-directory.rst .. include:: uri-scheme-api-identifiers.rst .. include:: uri-scheme-api-origin.rst .. include:: uri-scheme-api-person.rst .. include:: uri-scheme-api-release.rst .. include:: uri-scheme-api-revision.rst .. include:: uri-scheme-api-snapshot.rst .. include:: uri-scheme-api-stat.rst .. include:: uri-scheme-api-vault.rst diff --git a/docs/uri-scheme-browse-content.rst b/docs/uri-scheme-browse-content.rst index d977fac19..73e42a06c 100644 --- a/docs/uri-scheme-browse-content.rst +++ b/docs/uri-scheme-browse-content.rst @@ -1,92 +1,92 @@ Content ^^^^^^^ .. http:get:: /browse/content/[(algo_hash):](hash)/ - HTML view that displays a SWH content identified by its hash value. + HTML view that displays a content identified by its hash value. If the content to display is textual, it will be highlighted client-side if possible using highlightjs_. In order for that operation to be performed, a programming language must first be associated to the content. The following procedure is used in order to find the language: 1) First try to find a language from the content filename (provided as query parameter when navigating from a directory view). 2) If no language has been found from the filename, try to find one from the content mime type. The mime type is retrieved from the content metadata stored - in the SWH archive or is computed server-side using Python + in the archive or is computed server-side using Python magic module. It is also possible to highlight specific lines of a textual content (not in terms of syntax highlighting but to emphasize some relevant content part) by either: * clicking on line numbers (holding shift to highlight a lines range) * using an url fragment in the form '#Ln' or '#Lm-Ln' When that view is called in the context of a navigation coming from a directory view, a breadcrumb will be displayed on top of the rendered content in order to easily navigate up to the associated root directory. In that case, the path query parameter will be used and filled with the path of the file relative to the root directory. :param string algo_hash: optional parameter to indicate the algorithm used - to compute the content checksum (can be either *sha1*, - *sha1_git*, *sha256* or *blake2s256*, default to *sha1*) + to compute the content checksum (can be either ``sha1``, + ``sha1_git``, ``sha256`` or ``blake2s256``, default to ``sha1``) :param string hash: hexadecimal representation for the checksum from which - to retrieve the associated content in the SWH archive + to retrieve the associated content in the archive :query string path: describe the path of the content relative to a root directory (used to add context aware navigation links when navigating from a directory view) :statuscode 200: no error :statuscode 400: an invalid query string has been provided - :statuscode 404: requested content can not be found in the SWH archive + :statuscode 404: requested content can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`content/sha1_git:f5d0b39a0cdddb91a31a537052b7d8d31a4aa79f/` :swh_web_browse:`content/sha1_git:f5d0b39a0cdddb91a31a537052b7d8d31a4aa79f/#L23-L41` :swh_web_browse:`content/blake2s256:1cc1e3124957c9be8a454c58e92eb925cf4aa9823984bd01451c5b7e0fee99d1/` :swh_web_browse:`content/sha1:1cb1447c1c7ddc1b03eac88398e40bd914d46b62/` :swh_web_browse:`content/sha256:8ceb4b9ee5adedde47b31e975c1d90c73ad27b6b165a1dcd80c7c545eb65b903/` .. http:get:: /browse/content/[(algo_hash):](hash)/raw/ - HTML view that produces a raw display of a SWH content identified by its hash value. + HTML view that produces a raw display of a content identified by its hash value. The behaviour of that view depends on the mime type of the requested content. If the mime type is from the text family, the view will return a response whose content type is 'text/plain' that will be rendered by the browser. Otherwise, the view will return a response whose content type is 'application/octet-stream' and the browser will then offer to download the file. In the context of a navigation coming from a directory view, the filename query parameter will be used in order to provide the real name of the file when one wants to save it locally. :param string algo_hash: optional parameter to indicate the algorithm used - to compute the content checksum (can be either *sha1*, - *sha1_git*, *sha256* or *blake2s256*, default to *sha1*) + to compute the content checksum (can be either ``sha1``, + ``sha1_git``, ``sha256`` or ``blake2s256``, default to ``sha1``) :param string hash: hexadecimal representation for the checksum from which - to retrieve the associated content in the SWH archive + to retrieve the associated content in the archive :query string filename: indicate the name of the file holding the requested content (used when one wants to save the content to a local file) :statuscode 200: no error :statuscode 400: an invalid query string has been provided - :statuscode 404: requested content can not be found in the SWH archive + :statuscode 404: requested content can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`content/sha1_git:f5d0b39a0cdddb91a31a537052b7d8d31a4aa79f/raw/?filename=LICENSE` :swh_web_browse:`content/blake2s256:1cc1e3124957c9be8a454c58e92eb925cf4aa9823984bd01451c5b7e0fee99d1/raw/?filename=MAINTAINERS` :swh_web_browse:`content/sha1:1cb1447c1c7ddc1b03eac88398e40bd914d46b62/raw/` :swh_web_browse:`content/sha256:8ceb4b9ee5adedde47b31e975c1d90c73ad27b6b165a1dcd80c7c545eb65b903/raw/?filename=COPYING` .. _highlightjs: https://highlightjs.org/ \ No newline at end of file diff --git a/docs/uri-scheme-browse-directory.rst b/docs/uri-scheme-browse-directory.rst index 4c8b9e5c5..f8d63b533 100644 --- a/docs/uri-scheme-browse-directory.rst +++ b/docs/uri-scheme-browse-directory.rst @@ -1,31 +1,31 @@ Directory ^^^^^^^^^ .. http:get:: /browse/directory/(sha1_git)/[(path)/] - HTML view for browsing the content of a SWH directory reachable from - the provided root one (including itself) identified by its *sha1_git* value. + HTML view for browsing the content of a directory reachable from + the provided root one (including itself) identified by its **sha1_git** value. The content of the directory is first sorted in lexicographical order and the sub-directories are displayed before the regular files. The view enables to navigate from the requested directory to directories reachable from it in a recursive way but also up to the root directory. A breadcrumb located in the top part of the view allows to keep track of the paths navigated so far. - :param string sha1_git: hexadecimal representation for the *sha1_git* identifier + :param string sha1_git: hexadecimal representation for the **sha1_git** identifier of the directory to browse :param string path: optional parameter used to specify the path of a directory reachable from the provided root one :statuscode 200: no error - :statuscode 400: an invalid *sha1_git* value has been provided - :statuscode 404: requested directory can not be found in the SWH archive + :statuscode 400: an invalid **sha1_git** value has been provided + :statuscode 404: requested directory can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`directory/977fc4b98c0e85816348cebd3b12026407c368b6/` :swh_web_browse:`directory/9650ed370c0330d2cd2b6fd1e9febf649ffe538d/kernel/sched/` diff --git a/docs/uri-scheme-browse-origin.rst b/docs/uri-scheme-browse-origin.rst index 6001405b0..67b0dcef4 100644 --- a/docs/uri-scheme-browse-origin.rst +++ b/docs/uri-scheme-browse-origin.rst @@ -1,489 +1,500 @@ Origin ^^^^^^ This describes the URI scheme when one wants to browse the Software Heritage archive in the context of an origin (for instance, a repository crawled from GitHub or a Debian source package). All the views pointed by that scheme offer quick links to browse objects as found during the associated crawls performed by Software Heritage: * the root directory of the origin * the list of branches of the origin * the list of releases of the origin Origin visits """"""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visits/ - HTML view that displays a visits reporting for a SWH origin identified by + HTML view that displays a visits reporting for a software origin identified by its type and url. - :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) + :param string origin_type: the type of software origin (possible values are ``git``, ``svn``, + ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive + :statuscode 404: requested origin can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/visits/` :swh_web_browse:`origin/git/url/https://github.com/python/cpython/visits/` :swh_web_browse:`origin/deb://Debian-Security/packages/mediawiki/visits/` :swh_web_browse:`origin/https://gitorious.org/qt/qtbase.git/visits/` Origin directory """""""""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/directory/[(path)/] HTML view for browsing the content of a directory reachable from the root directory - (including itself) associated to the latest full visit of a SWH origin. + (including itself) associated to the latest full visit of a software origin. The content of the directory is first sorted in lexicographical order and the sub-directories are displayed before the regular files. The view enables to navigate from the requested directory to directories reachable from it in a recursive way but also up to the origin root directory. A breadcrumb located in the top part of the view allows to keep track of the paths navigated so far. The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the directory content can also be specified by using the branch query parameter. - :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) + :param string origin_type: the type of software origin (possible values are ``git``, + ``svn``, ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string path: optional parameter used to specify the path of a directory reachable from the origin root one :query string branch: specify the origin branch name from which to retrieve the root directory :query string release: specify the origin release name from which to retrieve the root directory :query string revision: specify the origin revision, identified by the hexadecimal - representation of its *sha1_git* value, from which to retrieve the root directory - :query int visit_id: specify a SWH visit id to retrieve the directory from instead + representation of its **sha1_git** value, from which to retrieve the root directory + :query int visit_id: specify a visit id to retrieve the directory from instead of using the latest full visit by default :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive + :statuscode 404: requested origin can not be found in the archive or the provided path does not exist from the origin root directory **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/directory/` :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/directory/net/ethernet/` :swh_web_browse:`origin/https://github.com/python/cpython/directory/` :swh_web_browse:`origin/https://github.com/python/cpython/directory/Python/` :swh_web_browse:`origin/https://github.com/python/cpython/directory/?branch=refs/heads/2.7` .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/directory/[(path)/] HTML view for browsing the content of a directory reachable from - the root directory (including itself) associated to a visit of a SWH + the root directory (including itself) associated to a visit of a software origin closest to a provided timestamp. The content of the directory is first sorted in lexicographical order and the sub-directories are displayed before the regular files. The view enables to navigate from the requested directory to directories reachable from it in a recursive way but also up to the origin root directory. A breadcrumb located in the top part of the view allows to keep track of the paths navigated so far. The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the directory content can also be specified by using the branch query parameter. - :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) + :param string origin_type: the type of software origin (possible values are ``git``, + ``svn``, ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string timestamp: a date string (any format parsable by `dateutil.parser.parse`_) - or Unix timestamp to parse in order to find the closest SWH visit. + or Unix timestamp to parse in order to find the closest visit. :param path: optional parameter used to specify the path of a directory reachable from the origin root one :type path: string :query string branch: specify the origin branch name from which to retrieve the root directory :query string release: specify the origin release name from which to retrieve the root directory :query string revision: specify the origin revision, identified by the hexadecimal - representation of its *sha1_git* value, from which to retrieve the directory - :query int visit_id: specify a SWH visit id to retrieve the directory from instead + representation of its **sha1_git** value, from which to retrieve the directory + :query int visit_id: specify a visit id to retrieve the directory from instead of using the provided timestamp :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive, + :statuscode 404: requested origin can not be found in the archive, requested visit timestamp does not exist or the provided path does not exist from the origin root directory **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/visit/1493926809/directory/` :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/visit/2016-09-14T10:36:21/directory/net/ethernet/` :swh_web_browse:`origin/git/url/https://github.com/python/cpython/visit/1474620651/directory/` :swh_web_browse:`origin/git/url/https://github.com/python/cpython/visit/2017-05-05/directory/Python/` :swh_web_browse:`origin/git/url/https://github.com/python/cpython/visit/2015-08/directory/?branch=refs/heads/2.7` Origin content """""""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/content/(path)/ - HTML view that produces a display of a SWH content - associated to the latest full visit of a SWH origin. + HTML view that produces a display of a content + associated to the latest full visit of a software origin. If the content to display is textual, it will be highlighted client-side if possible using highlightjs_. The procedure to perform that task is described in :http:get:`/browse/content/[(algo_hash):](hash)/`. It is also possible to highlight specific lines of a textual content (not in terms of syntax highlighting but to emphasize some relevant content part) by either: * clicking on line numbers (holding shift to highlight a lines range) * using an url fragment in the form '#Ln' or '#Lm-Ln' The view displays a breadcrumb on top of the rendered content in order to easily navigate up to the origin root directory. The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the content can also be specified by using the branch query parameter. - :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) + :param string origin_type: the type of software origin (possible values are ``git``, + ``svn``, ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string path: path of a content reachable from the origin root directory :query string branch: specify the origin branch name from which to retrieve the content :query string release: specify the origin release name from which to retrieve the content :query string revision: specify the origin revision, identified by the hexadecimal - representation of its *sha1_git* value, from which to retrieve the content - :query int visit_id: specify a SWH visit id to retrieve the content from instead + representation of its **sha1_git** value, from which to retrieve the content + :query int visit_id: specify a visit id to retrieve the content from instead of using the latest full visit by default :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive, + :statuscode 404: requested origin can not be found in the archive, or the provided content path does not exist from the origin root directory **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/git/git/content/git.c/` :swh_web_browse:`origin/git/url/https://github.com/git/git/content/git.c/` :swh_web_browse:`origin/https://github.com/mozilla/gecko-dev/content/js/src/json.cpp/` :swh_web_browse:`origin/https://github.com/git/git/content/git.c/?branch=refs/heads/next` .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/content/(path)/ - HTML view that produces a display of a SWH content associated to a - visit of a SWH origin closest to a provided timestamp. + HTML view that produces a display of a content associated to a + visit of a software origin closest to a provided timestamp. If the content to display is textual, it will be highlighted client-side if possible using highlightjs_. The procedure to perform that task is described in :http:get:`/browse/content/[(algo_hash):](hash)/`. It is also possible to highlight specific lines of a textual content (not in terms of syntax highlighting but to emphasize some relevant content part) by either: * clicking on line numbers (holding shift to highlight a lines range) * using an url fragment in the form '#Ln' or '#Lm-Ln' The view displays a breadcrumb on top of the rendered content in order to easily navigate up to the origin root directory. The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the content can also be specified by using the branch query parameter. - :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) + :param string origin_type: the type of software origin (possible values are ``git``, + ``svn``, ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string timestamp: a date string (any format parsable by `dateutil.parser.parse`_) - or Unix timestamp to parse in order to find the closest SWH visit. + or Unix timestamp to parse in order to find the closest visit. :param string path: path of a content reachable from the origin root directory :query string branch: specify the origin branch name from which to retrieve the content :query string release: specify the origin release name from which to retrieve the content :query string revision: specify the origin revision, identified by the hexadecimal - representation of its *sha1_git* value, from which to retrieve the content - :query int visit_id: specify a SWH visit id to retrieve the content from instead + representation of its **sha1_git** value, from which to retrieve the content + :query int visit_id: specify a visit id to retrieve the content from instead of using the provided timestamp :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive, + :statuscode 404: requested origin can not be found in the archive, requested visit timestamp does not exist or the provided content path does not exist from the origin root directory **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/git/git/visit/1473933564/content/git.c/` :swh_web_browse:`origin/git/url/https://github.com/git/git/visit/2016-05-05T00:0:00+00:00/content/git.c/` :swh_web_browse:`origin/https://github.com/mozilla/gecko-dev/visit/1490126182/content/js/src/json.cpp/` :swh_web_browse:`origin/https://github.com/mozilla/gecko-dev/visit/2017-03-21/content/js/src/json.cpp/#L904-L931` :swh_web_browse:`origin/https://github.com/git/git/visit/2017-09-15/content/git.c/?branch=refs/heads/next` Origin history """""""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/log/ HTML view that produces a display of revisions history heading - to the last revision found during the latest visit of a SWH origin. + to the last revision found during the latest visit of a software origin. In other words, it shows the commit log associated to the latest - full visit of a SWH origin. + full visit of a software origin. The following data are displayed for each log entry: * link to browse the associated revision in the origin context * author of the revision * date of the revision * message associated the revision * commit date of the revision By default, the revisions are ordered in reverse chronological order of their commit date. N log entries are displayed per page (default is 100). In order to navigate in a large history, two buttons are present at the bottom of the view: - * *Newer*: fetch and display if available the N more recent log entries + * **Newer**: fetch and display if available the N more recent log entries than the ones currently displayed - * *Older*: fetch and display if available the N older log entries + * **Older**: fetch and display if available the N older log entries than the ones currently displayed The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the content can also be specified by using the branch query parameter. - :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) + :param string origin_type: the type of software origin (possible values are ``git``, + ``svn``, ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :query int per_page: the number of log entries to display per page :query int offset: the number of revisions to skip before returning those to display - :query str revs_ordering: specify the revisions ordering, possible values are *committer_date*, - *dfs*, *dfs_post* and *bfs* + :query str revs_ordering: specify the revisions ordering, possible values are ``committer_date``, + ``dfs``, ``dfs_post`` and ``bfs`` :query string branch: specify the origin branch name from which to retrieve the commit log :query string release: specify the origin release name from which to retrieve the commit log :query string revision: specify the origin revision, identified by the hexadecimal - representation of its *sha1_git* value, from which to retrieve the commit log - :query int visit_id: specify a SWH visit id to retrieve the history log from instead + representation of its **sha1_git** value, from which to retrieve the commit log + :query int visit_id: specify a visit id to retrieve the history log from instead of using the latest visit by default :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive + :statuscode 404: requested origin can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/videolan/vlc/log/` :swh_web_browse:`origin/https://github.com/Kitware/CMake/log/` :swh_web_browse:`origin/https://github.com/Kitware/CMake/log/?branch=refs/heads/release` .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/log/ HTML view that produces a display of revisions history heading - to the last revision found during a visit of a SWH origin closest + to the last revision found during a visit of a software origin closest to the provided timestamp. In other words, it shows the commit log associated to a visit of - a SWH origin closest to a provided timestamp. + a software origin closest to a provided timestamp. The following data are displayed for each log entry: * author of the revision * link to the revision metadata * message associated the revision * date of the revision * link to browse the associated source tree in the origin context N log entries are displayed per page (default is 20). In order to navigate in a large history, two buttons are present at the bottom of the view: - * *Newer*: fetch and display if available the N more recent log entries + * **Newer**: fetch and display if available the N more recent log entries than the ones currently displayed - * *Older*: fetch and display if available the N older log entries + * **Older**: fetch and display if available the N older log entries than the ones currently displayed The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the content can also be specified by using the branch query parameter. - :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) + :param string origin_type: the type of software origin (possible values are ``git``, + ``svn``, ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string timestamp: a date string (any format parsable by `dateutil.parser.parse`_) - or Unix timestamp to parse in order to find the closest SWH visit. + or Unix timestamp to parse in order to find the closest visit. :query string revs_breadcrumb: used internally to store the navigation breadcrumbs (i.e. the list of descendant revisions visited so far). It must be a string in the form "(rev_1)[/(rev_2)/.../(rev_n)]" where rev_i corresponds to a - revision *sha1_git*. + revision **sha1_git**. :query int per_page: the number of log entries to display per page (default is 20, max is 50) :query string branch: specify the origin branch name from which to retrieve the commit log :query string release: specify the origin release name from which to retrieve the commit log :query string revision: specify the origin revision, identified by the hexadecimal - representation of its *sha1_git* value, from which to retrieve the commit log - :query int visit_id: specify a SWH visit id to retrieve the history log from instead + representation of its **sha1_git** value, from which to retrieve the commit log + :query int visit_id: specify a visit id to retrieve the history log from instead of using the provided timestamp :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive + :statuscode 404: requested origin can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/videolan/vlc/visit/1459651262/log/` :swh_web_browse:`origin/git/url/https://github.com/Kitware/CMake/visit/2016-04-01/log/` :swh_web_browse:`origin/https://github.com/Kitware/CMake/visit/1438116814/log/?branch=refs/heads/release` :swh_web_browse:`origin/https://github.com/Kitware/CMake/visit/2017-05-05T03:14:23/log/?branch=refs/heads/release` Origin branches """"""""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/branches/ HTML view that produces a display of the list of branches - found during the latest full visit of a SWH origin. + found during the latest full visit of a software origin. The following data are displayed for each branch: * its name * a link to browse the associated directory * a link to browse the associated revision * last commit message * last commit date That list of branches is paginated, each page displaying a maximum of 100 branches. - :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) + :param string origin_type: the type of software origin (possible values are ``git``, + ``svn``, ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive + :statuscode 404: requested origin can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/deb/url/deb://Debian/packages/linux/branches/` :swh_web_browse:`origin/https://github.com/webpack/webpack/branches/` .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/branches/ HTML view that produces a display of the list of branches - found during a visit of a SWH origin closest to the provided timestamp. + found during a visit of a software origin closest to the provided timestamp. The following data are displayed for each branch: * its name * a link to browse the associated directory * a link to browse the associated revision * last commit message * last commit date That list of branches is paginated, each page displaying a maximum of 100 branches. - :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) + :param string origin_type: the type of software origin (possible values are ``git``, + ``svn``, ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string timestamp: a date string (any format parsable by `dateutil.parser.parse`_) - or Unix timestamp to parse in order to find the closest SWH visit. + or Unix timestamp to parse in order to find the closest visit. :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive + :statuscode 404: requested origin can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/kripken/emscripten/visit/2017-05-05T12:02:03/branches/` :swh_web_browse:`origin/deb://Debian/packages/apache2-mod-xforward/visit/2017-11-15T05:15:09/branches/` Origin releases """"""""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/releases/ HTML view that produces a display of the list of releases - found during the latest full visit of a SWH origin. + found during the latest full visit of a software origin. The following data are displayed for each release: * its name * a link to browse the release details * its target type (revision, directory, content or release) * its associated message * its date That list of releases is paginated, each page displaying a maximum of 100 releases. - :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) + :param string origin_type: the type of software origin (possible values are ``git``, + ``svn``, ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive + :statuscode 404: requested origin can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/git/git/releases/` :swh_web_browse:`origin/https://github.com/webpack/webpack/releases/` .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/releases/ HTML view that produces a display of the list of releases - found during a visit of a SWH origin closest to the provided timestamp. + found during a visit of a software origin closest to the provided timestamp. The following data are displayed for each release: * its name * a link to browse the release details * its target type (revision, directory, content or release) * its associated message * its date That list of releases is paginated, each page displaying a maximum of 100 releases. - :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) + :param string origin_type: the type of software origin (possible values are ``git``, + ``svn``, ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string timestamp: a date string (any format parsable by `dateutil.parser.parse`_) - or Unix timestamp to parse in order to find the closest SWH visit. + or Unix timestamp to parse in order to find the closest visit. :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive + :statuscode 404: requested origin can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/visit/2017-11-21T19:37:42/releases/` :swh_web_browse:`origin/https://github.com/Kitware/CMake/visit/2016-09-23T14:06:35/releases/` .. _highlightjs: https://highlightjs.org/ .. _dateutil.parser.parse: http://dateutil.readthedocs.io/en/stable/parser.html diff --git a/docs/uri-scheme-browse-person.rst b/docs/uri-scheme-browse-person.rst index a64a25e6e..950ad2ccc 100644 --- a/docs/uri-scheme-browse-person.rst +++ b/docs/uri-scheme-browse-person.rst @@ -1,10 +1,10 @@ Person ^^^^^^ .. http:get:: /browse/person/(person_id)/ - HTML view that displays information regarding a SWH person. + HTML view that displays information regarding a person. - :param int person_id: the id of a SWH person + :param int person_id: the id of a person :statuscode 200: no error - :statuscode 404: requested person can not be found in the SWH archive + :statuscode 404: requested person can not be found in the archive diff --git a/docs/uri-scheme-browse-release.rst b/docs/uri-scheme-browse-release.rst index ba895ff96..862e0953b 100644 --- a/docs/uri-scheme-browse-release.rst +++ b/docs/uri-scheme-browse-release.rst @@ -1,25 +1,25 @@ Release ^^^^^^^ .. http:get:: /browse/release/(sha1_git)/ - HTML view that displays metadata associated to a SWH release: + HTML view that displays metadata associated to a release: * the author * the release date * the release name * the associated message * the type of target the release points to (revision, directory, content or release) * the link to browse the release target - :param string sha1_git: hexadecimal representation for the *sha1_git* - identifier of a SWH release + :param string sha1_git: hexadecimal representation for the **sha1_git** + identifier of a release :statuscode 200: no error - :statuscode 404: requested release can not be found in the SWH archive + :statuscode 404: requested release can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`release/208f61cc7a5dbc9879ae6e5c2f95891e270f09ef/` :swh_web_browse:`release/f883596e997fe5bcbc5e89bee01b869721326109/` diff --git a/docs/uri-scheme-browse-revision.rst b/docs/uri-scheme-browse-revision.rst index 88c3215e9..e9e55ea64 100644 --- a/docs/uri-scheme-browse-revision.rst +++ b/docs/uri-scheme-browse-revision.rst @@ -1,81 +1,82 @@ Revision ^^^^^^^^ .. http:get:: /browse/revision/(sha1_git)/ - HTML view to browse a SWH revision. It notably shows the revision date + HTML view to browse a revision. It notably shows the revision date and message but also offers links to get more details on: * its author * its parent revisions * the history log reachable from it The view also enables to navigate in the source tree associated to the revision and browse its content. Last but not least, the view displays the list of file changes introduced in the revision but also the diffs of each changed files. - :param string sha1_git: hexadecimal representation for the *sha1_git* - identifier of a SWH revision - :query string origin_type: used internally to associate a SWH origin type - (*git*, *svn*, *deb* ...) to the revision + :param string sha1_git: hexadecimal representation for the **sha1_git** + identifier of a revision + :query string origin_type: used internally to associate a software origin type + (possible values are ``git``, ``svn``, ``hg``, ``deb``, ``pypi``, ``ftp`` + or ``deposit``) to the revision :query string origin_url: used internally to associate an origin url (e.g. https://github.com/user/repo) to the revision :query string timestamp: used internally to associate an origin visit to the revision, must be a date string (any format parsable by `dateutil.parser.parse`_) - or Unix timestamp to parse in order to find the closest SWH visit. - :query int visit_id: used internally to specify a SWH visit id instead of + or Unix timestamp to parse in order to find the closest visit. + :query int visit_id: used internally to specify a visit id instead of using the provided timestamp :query string path: used internally when navigating in the source tree associated to the revision :statuscode 200: no error - :statuscode 404: requested revision can not be found in the SWH archive + :statuscode 404: requested revision can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`revision/f1b94134a4b879bc55c3dacdb496690c8ebdc03f/` :swh_web_browse:`revision/d1aa2b3f607b35dc5dbf613b2334b6d243ec2bda/` .. _dateutil.parser.parse: http://dateutil.readthedocs.io/en/stable/parser.html .. http:get:: /browse/revision/(sha1_git)/log/ HTML view that displays the list of revisions heading to a given one. In other words, it shows a commit log. The following data are displayed for each log entry: * link to browse the revision * author of the revision * date of the revision * message associated to the revision * commit date of the revision By default, the revisions are ordered in reverse chronological order of their commit date. N log entries are displayed per page (default is 100). In order to navigate in a large history, two buttons are present at the bottom of the view: - * *Newer*: fetch and display if available the N more recent log entries + * **Newer**: fetch and display if available the N more recent log entries than the ones currently displayed - * *Older*: fetch and display if available the N older log entries + * **Older**: fetch and display if available the N older log entries than the ones currently displayed - :param string sha1_git: hexadecimal representation for the *sha1_git* - identifier of a SWH revision + :param string sha1_git: hexadecimal representation for the **sha1_git** + identifier of a revision :query int per_page: the number of log entries to display per page :query int offset: the number of revisions to skip before returning those to display - :query str revs_ordering: specify the revisions ordering, possible values are *committer_date*, - *dfs*, *dfs_post* and *bfs* + :query str revs_ordering: specify the revisions ordering, possible values are ``committer_date``, + ``dfs``, ``dfs_post`` and ``bfs`` :statuscode 200: no error - :statuscode 404: requested revision can not be found in the SWH archive + :statuscode 404: requested revision can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`revision/f1b94134a4b879bc55c3dacdb496690c8ebdc03f/log/` :swh_web_browse:`revision/d1aa2b3f607b35dc5dbf613b2334b6d243ec2bda/log/` diff --git a/docs/uri-scheme-browse-snapshot.rst b/docs/uri-scheme-browse-snapshot.rst index 012630157..8f5f2fd72 100644 --- a/docs/uri-scheme-browse-snapshot.rst +++ b/docs/uri-scheme-browse-snapshot.rst @@ -1,187 +1,187 @@ Snapshot ^^^^^^^^ .. http:get:: /browse/snapshot/(snapshot_id)/ - HTML view that displays the content of a SWH snapshot from its identifier + HTML view that displays the content of a snapshot from its identifier (see :func:`swh.model.identifiers.snapshot_identifier` in our data model module for details about how they are computed). A snapshot is a set of named branches, which are pointers to objects at any level of the Software Heritage DAG. It represents a full picture of an origin at a given time. Thus, multiple visits of different origins can point to the same snapshot (for instance, when several projects are forks of a common one). Currently, that endpoint simply performs a redirection to :http:get:`/browse/snapshot/(snapshot_id)/directory/[(path)/]` in order to display the root directory associated to the default snapshot branch (usually master). - :param string snapshot_id: hexadecimal representation of the snapshot *sha1* identifier + :param string snapshot_id: hexadecimal representation of the snapshot **sha1** identifier :statuscode 200: no error :statuscode 400: an invalid snapshot identifier has been provided - :statuscode 404: requested snapshot can not be found in the SWH archive + :statuscode 404: requested snapshot can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`snapshot/baebc2109e4a2ec22a1129a3859647e191d04df4/` :swh_web_browse:`snapshot/673156c31a876c5b99b2fe3e89615529de9a3c44/` Snapshot directory """""""""""""""""" .. http:get:: /browse/snapshot/(snapshot_id)/directory/[(path)/] HTML view that displays the content of a directory reachable from - a SWH snapshot. + a snapshot. The features offered by the view are similar to the one for browsing a directory in an origin context (see :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/directory/[(path)/]`). - :param string snapshot_id: hexadecimal representation of the snapshot *sha1* identifier + :param string snapshot_id: hexadecimal representation of the snapshot **sha1** identifier :param string path: optional parameter used to specify the path of a directory reachable from the snapshot root one :query string branch: specify the snapshot branch name from which to retrieve the root directory :query string release: specify the snapshot release name from which to retrieve the root directory :query string revision: specify the snapshot revision, identified by the hexadecimal - representation of its *sha1_git* value, from which to retrieve the root directory + representation of its **sha1_git** value, from which to retrieve the root directory :statuscode 200: no error :statuscode 400: an invalid snapshot identifier has been provided - :statuscode 404: requested snapshot can not be found in the SWH archive + :statuscode 404: requested snapshot can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`snapshot/baebc2109e4a2ec22a1129a3859647e191d04df4/directory/drivers/gpu/` :swh_web_browse:`snapshot/673156c31a876c5b99b2fe3e89615529de9a3c44/directory/src/opengl/` :swh_web_browse:`snapshot/673156c31a876c5b99b2fe3e89615529de9a3c44/log/?release=v5.7.0` Snapshot content """""""""""""""" .. http:get:: /browse/snapshot/(snapshot_id)/content/(path)/ - HTML view that produces a display of a SWH content reachable from - a SWH snapshot. + HTML view that produces a display of a content reachable from + a snapshot. The features offered by the view are similar to the one for browsing a content in an origin context (see :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/content/(path)/`). - :param string snapshot_id: hexadecimal representation of the snapshot *sha1* identifier + :param string snapshot_id: hexadecimal representation of the snapshot **sha1** identifier :param string path: path of a content reachable from the snapshot root directory :query string branch: specify the snapshot branch name from which to retrieve the content :query string release: specify the snapshot release name from which to retrieve the content :query string revision: specify the snapshot revision, identified by the hexadecimal - representation of its *sha1_git* value, from which to retrieve the content + representation of its **sha1_git** value, from which to retrieve the content :statuscode 200: no error :statuscode 400: an invalid snapshot identifier has been provided - :statuscode 404: requested snapshot can not be found in the SWH archive, + :statuscode 404: requested snapshot can not be found in the archive, or the provided content path does not exist from the origin root directory **Examples:** .. parsed-literal:: :swh_web_browse:`snapshot/baebc2109e4a2ec22a1129a3859647e191d04df4/content/init/initramfs.c` :swh_web_browse:`snapshot/673156c31a876c5b99b2fe3e89615529de9a3c44/content/src/opengl/qglbuffer.h/` :swh_web_browse:`snapshot/673156c31a876c5b99b2fe3e89615529de9a3c44/content/src/opengl/qglbuffer.h/?release=v5.0.0` Snapshot history """""""""""""""" .. http:get:: /browse/snapshot/(snapshot_id)/log/ HTML view that produces a display of revisions history (aka the commit log) - heading to the last revision collected in a SWH snasphot. + heading to the last revision collected in a snapshot. The features offered by the view are similar to the one for browsing the history in an origin context (see :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/log/`). - :param string snapshot_id: hexadecimal representation of the snapshot *sha1* identifier + :param string snapshot_id: hexadecimal representation of the snapshot **sha1** identifier :query string revs_breadcrumb: used internally to store the navigation breadcrumbs (i.e. the list of descendant revisions visited so far). It must be a string in the form "(rev_1)[/(rev_2)/.../(rev_n)]" where rev_i corresponds to a - revision *sha1_git*. + revision **sha1_git**. :query int per_page: the number of log entries to display per page (default is 20, max is 50) :query string branch: specify the snapshot branch name from which to retrieve the commit log - :query string release: specify the snasphot release name from which + :query string release: specify the snapshot release name from which to retrieve the commit log :query string revision: specify the snapshot revision, identified by the hexadecimal - representation of its *sha1_git* value, from which to retrieve the commit log + representation of its **sha1_git** value, from which to retrieve the commit log :statuscode 200: no error :statuscode 400: an invalid snapshot identifier has been provided - :statuscode 404: requested snapshot can not be found in the SWH archive + :statuscode 404: requested snapshot can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`snapshot/a274b44111f777209556e94920b7e71cf5c305cd/log/` :swh_web_browse:`snapshot/9ca9e75279df5f4e3fee19bf5190ed672dcdfb33/log/?branch=refs/heads/emacs-unicode` Snapshot branches """"""""""""""""" .. http:get:: /browse/snapshot/(snapshot_id)/branches/ HTML view that produces a display of the list of branches - collected in a SWH snapshot. + collected in a snapshot. The features offered by the view are similar to the one for browsing the list of branches in an origin context (see :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/branches/`). - :param string snapshot_id: hexadecimal representation of the snapshot *sha1* identifier + :param string snapshot_id: hexadecimal representation of the snapshot **sha1** identifier :statuscode 200: no error :statuscode 400: an invalid snapshot identifier has been provided - :statuscode 404: requested snapshot can not be found in the SWH archive + :statuscode 404: requested snapshot can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`snapshot/03d7897352541e78ee7b13a580dc836778e8126a/branches/` :swh_web_browse:`snapshot/f37563b953327f8fd83e39af6ebb929ef85103d5/branches/` Snapshot releases """"""""""""""""" .. http:get:: /browse/snapshot/(snapshot_id)/releases/ HTML view that produces a display of the list of releases - collected in a SWH snapshot. + collected in a snapshot. The features offered by the view are similar to the one for browsing the list of releases in an origin context (see :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/releases/`). - :param string snapshot_id: hexadecimal representation of the snapshot *sha1* identifier + :param string snapshot_id: hexadecimal representation of the snapshot **sha1** identifier :statuscode 200: no error :statuscode 400: an invalid snapshot identifier has been provided - :statuscode 404: requested snapshot can not be found in the SWH archive + :statuscode 404: requested snapshot can not be found in the archive **Examples:** .. parsed-literal:: :swh_web_browse:`snapshot/673156c31a876c5b99b2fe3e89615529de9a3c44/releases/` :swh_web_browse:`snapshot/23e6fb084a60cc909b9e222d80d89fdb98756dee/releases/` diff --git a/docs/uri-scheme-browse.rst b/docs/uri-scheme-browse.rst index b18c6b564..c04cc241a 100644 --- a/docs/uri-scheme-browse.rst +++ b/docs/uri-scheme-browse.rst @@ -1,99 +1,99 @@ -URI scheme for SWH Web Browse application +URI scheme for swh-web Browse application ========================================= -This web application aims to provide HTML views to easily navigate in the SWH archive, +This web application aims to provide HTML views to easily navigate in the archive, thus it needs to be reached from a web browser. -If you intend to query the SWH archive programmatically through any HTTP client, +If you intend to query the archive programmatically through any HTTP client, please refer to the :ref:`swh-web-api-urls` section instead. Context-independent browsing ---------------------------- -Context-independent URLs provide information about SWH objects (e.g., +Context-independent URLs provide information about objects (e.g., revisions, directories, contents, person, ...), independently of the contexts where they have been found (e.g., specific repositories, branches, commits, ...). The following endpoints are the same of the API case (see below), and just render the corresponding information for user consumption. Where hyperlinks are created, they always point to other context-independent user URLs: - * :http:get:`/browse/content/[(algo_hash):](hash)/`: Display a SWH content - * :http:get:`/browse/content/[(algo_hash):](hash)/raw/`: Get / Download SWH content raw data - * :http:get:`/browse/directory/(sha1_git)/[(path)/]`: Browse the content of a SWH directory - * :http:get:`/browse/person/(person_id)/`: Information on a SWH person - * :http:get:`/browse/revision/(sha1_git)/`: Browse a SWH revision - * :http:get:`/browse/revision/(sha1_git)/log/`: Browse history log heading to a SWH revision + * :http:get:`/browse/content/[(algo_hash):](hash)/`: Display a content + * :http:get:`/browse/content/[(algo_hash):](hash)/raw/`: Get / Download content raw data + * :http:get:`/browse/directory/(sha1_git)/[(path)/]`: Browse the content of a directory + * :http:get:`/browse/person/(person_id)/`: Information on a person + * :http:get:`/browse/revision/(sha1_git)/`: Browse a revision + * :http:get:`/browse/revision/(sha1_git)/log/`: Browse history log heading to a revision Context-dependent browsing -------------------------- -Context-dependent URLs provide information about SWH objects, limited to +Context-dependent URLs provide information about objects, limited to specific contexts where the objects have been found. -For instance, instead of having to specify a (root) revision by *sha1_git*, users might want to -specify a place and a time. In SWH a "place" is an origin, with an optional +For instance, instead of having to specify a (root) revision by **sha1_git**, users might want to +specify a place and a time. In Software Heritage a "place" is an origin, with an optional branch name; a "time" is a timestamp at which some place has been observed by -SWH crawlers. +Software Heritage crawlers. Wherever a revision context is expected in a path (i.e., a **/browse/revision/(sha1_git)/** path fragment) we can put in its stead a path fragment of the form **/origin/(origin_type)/url/(origin_url)/[/visit/(timestamp)/][?branch=(branch)]**. -Such a fragment is resolved, internally by the SWH archive, to a revision *sha1_git* as follows: +Such a fragment is resolved, internally by the archive, to a revision **sha1_git** as follows: -- if *timestamp* is absent: look for the most recent crawl of origin - identified by *origin_type* and *origin_url* -- if *timestamp* is given: look for the closest crawl of origin identified - by *origin_type* and *origin_url* from timestamp *timestamp* -- if *branch* is given as a query parameter: look for the branch *branch* -- if *branch* is absent: look for branch "HEAD" or "master" -- return the revision *sha1_git* pointed by the chosen branch +- if **timestamp** is absent: look for the most recent crawl of origin + identified by **origin_type** and **origin_url** +- if **timestamp** is given: look for the closest crawl of origin identified + by **origin_type** and **origin_url** from timestamp **timestamp** +- if **branch** is given as a query parameter: look for the branch **branch** +- if **branch** is absent: look for branch "HEAD" or "master" +- return the revision **sha1_git** pointed by the chosen branch The already mentioned URLs for revision contexts can therefore be alternatively specified by users as: * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/directory/[(path)/]` * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/directory/[(path)/]` * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/content/(path)/` * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/content/(path)/` * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/log/` * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/log/` Typing: -- *origin_type* corresponds to the type of the archived origin: - *git*, *svn*, *hg*, *deposit*, *deb*, ... +- **origin_type** corresponds to the type of the archived origin: + ``git``, ``svn``,``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit`` -- *origin_url* corresponds to the URL the origin was crawled from, +- **origin_url** corresponds to the URL the origin was crawled from, for instance https://github.com/(user)/(repo)/ -- *branch* name is given as per the corresponding VCS (e.g., Git) as - a query parameter to the requestes URL. +- **branch** name is given as per the corresponding VCS (e.g., Git) as + a query parameter to the requested URL. -- *timestamp* is given in a format as liberal as possible, to uphold the +- **timestamp** is given in a format as liberal as possible, to uphold the principle of least surprise. At the very minimum it is possible to enter timestamps as: - Unix epoch timestamp (see for instance the output of `date +%s`) - ISO 8601 timestamps (see for instance the output of `date -I`, `date -Is`) - YYYY[MM[DD[HH[MM[SS]]]]] ad-hoc format - YYYY[-MM[-DD[ HH:[MM:[SS:]]]]] ad-hoc format -SWH Browse Urls ---------------- +swh-web Browse Urls +------------------- .. include:: uri-scheme-browse-content.rst .. include:: uri-scheme-browse-directory.rst .. include:: uri-scheme-browse-origin.rst .. include:: uri-scheme-browse-person.rst .. include:: uri-scheme-browse-release.rst .. include:: uri-scheme-browse-revision.rst .. include:: uri-scheme-browse-snapshot.rst diff --git a/docs/uri-scheme-identifiers.rst b/docs/uri-scheme-identifiers.rst index 871d5c4f5..064e15690 100644 --- a/docs/uri-scheme-identifiers.rst +++ b/docs/uri-scheme-identifiers.rst @@ -1,30 +1,30 @@ -URI scheme for SWH identifiers -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +URI scheme for Software Heritage identifiers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A subset of Software Heritage objects (contents, directories, releases and revisions) can be browsed using :ref:`persistent-identifiers`. Those identifiers are guaranteed to remain stable (persistent) over time. .. http:get:: /(swh_id)/ - End point to browse SWH objects using their persistent identifiers. + End point to browse Software Heritage objects using their persistent identifiers. A redirection to the adequate HTML view will be performed when reaching it. - :param string swh_id: a persistent identifier for a SWH object + :param string swh_id: a persistent identifier for a Software Heritage object (see the :ref:`persistent-identifiers` section to learn more about its syntax) - :resheader Location: the redirection URL for browsing the SWH object + :resheader Location: the redirection URL for browsing the Software Heritage object associated to the provided identifier :statuscode 302: no error :statuscode 400: the provided identifier is malformed **Examples:** .. parsed-literal:: :swh_web:`swh:1:cnt:0ffd12d85cdec70c88e852fc3f5ea9fd342213cd` :swh_web:`swh:1:dir:db990da9af15427455ce7836ce2b8a34b9bf67f5` :swh_web:`swh:1:rel:a9b7e3f1eada90250a6b2ab2ef3e0a846cb16831` :swh_web:`swh:1:rev:f1b94134a4b879bc55c3dacdb496690c8ebdc03f` :swh_web:`swh:1:snp:673156c31a876c5b99b2fe3e89615529de9a3c44` diff --git a/swh/web/admin/adminurls.py b/swh/web/admin/adminurls.py index e44f33aad..ba2b82c2f 100644 --- a/swh/web/admin/adminurls.py +++ b/swh/web/admin/adminurls.py @@ -1,38 +1,38 @@ # Copyright (C) 2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from swh.web.common.urlsindex import UrlsIndex class AdminUrls(UrlsIndex): """ - Class to manage SWH web admin urls. + Class to manage swh-web admin urls. """ scope = 'admin' class admin_route(object): # noqa: N801 """ - Decorator to ease the registration of a SWH web admin endpoint + Decorator to ease the registration of a swh-web admin endpoint Args: url_patterns: list of url patterns used by Django to identify the admin routes view_name: the name of the Django view associated to the routes used to reverse the url """ # noqa def __init__(self, *url_patterns, view_name=None): super().__init__() self.url_patterns = [] for url_pattern in url_patterns: self.url_patterns.append('^' + url_pattern + '$') self.view_name = view_name def __call__(self, f): # register the route and its view in the browse endpoints index for url_pattern in self.url_patterns: AdminUrls.add_url_pattern(url_pattern, f, self.view_name) return f diff --git a/swh/web/api/apidoc.py b/swh/web/api/apidoc.py index d0968891e..f39cc42ec 100644 --- a/swh/web/api/apidoc.py +++ b/swh/web/api/apidoc.py @@ -1,356 +1,358 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import docutils.nodes import docutils.parsers.rst import docutils.utils import functools import os import re from functools import wraps from rest_framework.decorators import api_view from swh.web.common.utils import parse_rst from swh.web.api.apiurls import APIUrls from swh.web.api.apiresponse import make_api_response, error_response class _HTTPDomainDocVisitor(docutils.nodes.NodeVisitor): """ docutils visitor for walking on a parsed rst document containing sphinx httpdomain roles. Its purpose is to extract relevant info regarding swh api endpoints (for instance url arguments) from their docstring written using sphinx httpdomain. """ # httpdomain roles we want to parse (based on sphinxcontrib.httpdomain 1.6) parameter_roles = ('param', 'parameter', 'arg', 'argument') response_json_object_roles = ('resjsonobj', 'resjson', '>jsonobj', '>json') response_json_array_roles = ('resjsonarr', '>jsonarr') query_parameter_roles = ('queryparameter', 'queryparam', 'qparam', 'query') request_header_roles = ('header', 'resheader', 'responseheader') status_code_roles = ('statuscode', 'status', 'code') def __init__(self, document, urls, data): super().__init__(document) self.urls = urls self.url_idx = 0 self.data = data self.args_set = set() self.params_set = set() self.returns_set = set() self.status_codes_set = set() self.reqheaders_set = set() self.resheaders_set = set() self.field_list_visited = False def process_paragraph(self, par): """ Process extracted paragraph text before display. Cleanup document model markups and transform the paragraph into a valid raw rst string (as the apidoc documentation transform rst to html when rendering). """ par = par.replace('\n', ' ') - # keep emphasized and strong text + # keep emphasized, strong and literal text par = par.replace('', '*') par = par.replace('', '*') par = par.replace('', '**') par = par.replace('', '**') + par = par.replace('', '``') + par = par.replace('', '``') # remove parsed document markups par = re.sub('<[^<]+?>', '', par) # api urls cleanup to generate valid links afterwards par = re.sub('\(\w+\)', '', par) # noqa par = re.sub('\[.*\]', '', par) # noqa par = par.replace('//', '/') # transform references to api endpoints into valid rst links par = re.sub(':http:get:`(.*)`', r'`<\1>`_', par) # transform references to some elements into bold text par = re.sub(':http:header:`(.*)`', r'**\1**', par) par = re.sub(':func:`(.*)`', r'**\1**', par) return par def visit_field_list(self, node): """ Visit parsed rst field lists to extract relevant info regarding api endpoint. """ self.field_list_visited = True for child in node.traverse(): # get the parsed field name if isinstance(child, docutils.nodes.field_name): field_name = child.astext() # parse field text elif isinstance(child, docutils.nodes.paragraph): text = self.process_paragraph(str(child)) field_data = field_name.split(' ') # Parameters if field_data[0] in self.parameter_roles: if field_data[2] not in self.args_set: self.data['args'].append({'name': field_data[2], 'type': field_data[1], 'doc': text}) self.args_set.add(field_data[2]) # Query Parameters if field_data[0] in self.query_parameter_roles: if field_data[2] not in self.params_set: self.data['params'].append({'name': field_data[2], 'type': field_data[1], 'doc': text}) self.params_set.add(field_data[2]) # Response type if field_data[0] in self.response_json_array_roles or \ field_data[0] in self.response_json_object_roles: # array if field_data[0] in self.response_json_array_roles: self.data['return_type'] = 'array' # object else: self.data['return_type'] = 'object' # returned object field if field_data[2] not in self.returns_set: self.data['returns'].append({'name': field_data[2], 'type': field_data[1], 'doc': text}) self.returns_set.add(field_data[2]) # Status Codes if field_data[0] in self.status_code_roles: if field_data[1] not in self.status_codes_set: self.data['status_codes'].append({'code': field_data[1], # noqa 'doc': text}) self.status_codes_set.add(field_data[1]) # Request Headers if field_data[0] in self.request_header_roles: if field_data[1] not in self.reqheaders_set: self.data['reqheaders'].append({'name': field_data[1], 'doc': text}) self.reqheaders_set.add(field_data[1]) # Response Headers if field_data[0] in self.response_header_roles: if field_data[1] not in self.resheaders_set: resheader = {'name': field_data[1], 'doc': text} self.data['resheaders'].append(resheader) self.resheaders_set.add(field_data[1]) if resheader['name'] == 'Content-Type' and \ resheader['doc'] == 'application/octet-stream': self.data['return_type'] = 'octet stream' def visit_paragraph(self, node): """ Visit relevant paragraphs to parse """ # only parsed top level paragraphs if isinstance(node.parent, docutils.nodes.block_quote): text = self.process_paragraph(str(node)) # endpoint description if not text.startswith('**') and self.data['description'] != text: self.data['description'] += '\n\n' if self.data['description'] else '' # noqa self.data['description'] += text # http methods elif text.startswith('**Allowed HTTP Methods:**'): text = text.replace('**Allowed HTTP Methods:**', '') http_methods = text.strip().split(',') http_methods = [m[m.find('`')+1:-1].upper() for m in http_methods] self.data['urls'].append({'rule': self.urls[self.url_idx], 'methods': http_methods}) self.url_idx += 1 def visit_literal_block(self, node): """ Visit literal blocks """ text = node.astext() # literal block in endpoint description if not self.field_list_visited: self.data['description'] += ':\n\n\t%s' % text # extract example url if ':swh_web_api:' in text: self.data['examples'].append( '/api/1/' + re.sub('.*`(.*)`.*', r'\1', text)) def visit_bullet_list(self, node): # bullet list in endpoint description if not self.field_list_visited: self.data['description'] += '\n\n' for child in node.traverse(): # process list item if isinstance(child, docutils.nodes.paragraph): line_text = self.process_paragraph(str(child)) self.data['description'] += '\t* %s\n' % line_text def unknown_visit(self, node): pass def depart_document(self, node): """ End of parsing extra processing """ default_methods = ['GET', 'HEAD', 'OPTIONS'] # ensure urls info is present and set default http methods if not self.data['urls']: for url in self.urls: self.data['urls'].append({'rule': url, 'methods': default_methods}) def unknown_departure(self, node): pass def _parse_httpdomain_doc(doc, data): doc_lines = doc.split('\n') doc_lines_filtered = [] urls = [] # httpdomain is a sphinx extension that is unknown to docutils but # fortunately we can still parse its directives' content, # so remove lines with httpdomain directives before executing the # rst parser from docutils for doc_line in doc_lines: if '.. http' not in doc_line: doc_lines_filtered.append(doc_line) else: url = doc_line[doc_line.find('/'):] # emphasize url arguments for html rendering url = re.sub(r'\((\w+)\)', r' **\(\1\)** ', url) urls.append(url) # parse the rst docstring and do not print system messages about # unknown httpdomain roles document = parse_rst('\n'.join(doc_lines_filtered), report_level=5) # remove the system_message nodes from the parsed document for node in document.traverse(docutils.nodes.system_message): node.parent.remove(node) # visit the document nodes to extract relevant endpoint info visitor = _HTTPDomainDocVisitor(document, urls, data) document.walkabout(visitor) class APIDocException(Exception): """ Custom exception to signal errors in the use of the APIDoc decorators """ class api_doc(object): # noqa: N801 """ Decorate an API function to register it in the API doc route index and create the corresponding DRF route. Args: route (str): documentation page's route noargs (boolean): set to True if the route has no arguments, and its result should be displayed anytime its documentation is requested. Default to False tags (list): Further information on api endpoints. Two values are possibly expected: * hidden: remove the entry points from the listing * upcoming: display the entry point but it is not followable handle_response (boolean): indicate if the decorated function takes care of creating the HTTP response or delegates that task to the apiresponse module api_version (str): api version string """ def __init__(self, route, noargs=False, tags=[], handle_response=False, api_version='1'): super().__init__() self.route = route self.urlpattern = '^' + api_version + route + '$' self.noargs = noargs self.tags = set(tags) self.handle_response = handle_response # @api_doc() Decorator call def __call__(self, f): # If the route is not hidden, add it to the index if 'hidden' not in self.tags: doc_data = self.get_doc_data(f) doc_desc = doc_data['description'] first_dot_pos = doc_desc.find('.') APIUrls.add_route(self.route, doc_desc[:first_dot_pos+1], tags=self.tags) # If the decorated route has arguments, we create a specific # documentation view if not self.noargs: @api_view(['GET', 'HEAD']) def doc_view(request): doc_data = self.get_doc_data(f) return make_api_response(request, None, doc_data) view_name = 'api-%s' % self.route[1:-1].replace('/', '-') APIUrls.add_url_pattern(self.urlpattern, doc_view, view_name) @wraps(f) def documented_view(request, **kwargs): doc_data = self.get_doc_data(f) try: response = f(request, **kwargs) except Exception as exc: return error_response(request, exc, doc_data) if self.handle_response: return response else: return make_api_response(request, response, doc_data) return documented_view @functools.lru_cache(maxsize=32) def get_doc_data(self, f): """ Build documentation data for the decorated api endpoint function """ data = { 'description': '', 'response_data': None, 'urls': [], 'args': [], 'params': [], 'resheaders': [], 'reqheaders': [], 'return_type': '', 'returns': [], 'status_codes': [], 'examples': [], 'route': self.route, 'noargs': self.noargs } if not f.__doc__: raise APIDocException('apidoc %s: expected a docstring' ' for function %s' % (self.__class__.__name__, f.__name__)) # use raw docstring as endpoint documentation if sphinx # httpdomain is not used if '.. http' not in f.__doc__: data['description'] = f.__doc__ # else parse the sphinx httpdomain docstring with docutils # (except when building the swh-web documentation through autodoc # sphinx extension, not needed and raise errors with sphinx >= 1.7) elif 'SWH_WEB_DOC_BUILD' not in os.environ: _parse_httpdomain_doc(f.__doc__, data) # process returned object info for nicer html display returns_list = '' for ret in data['returns']: returns_list += '\t* **%s (%s)**: %s\n' %\ (ret['name'], ret['type'], ret['doc']) data['returns_list'] = returns_list return data diff --git a/swh/web/api/views/content.py b/swh/web/api/views/content.py index 9859367ec..751508611 100644 --- a/swh/web/api/views/content.py +++ b/swh/web/api/views/content.py @@ -1,404 +1,404 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import functools from django.http import HttpResponse from swh.web.common import service from swh.web.common.utils import reverse from swh.web.common.exc import NotFoundExc from swh.web.api.apidoc import api_doc from swh.web.api import utils from swh.web.api.apiurls import api_route from swh.web.api.views.utils import api_lookup @api_route(r'/content/(?P.+)/provenance/', 'api-content-provenance') @api_doc('/content/provenance/', tags=['hidden']) def api_content_provenance(request, q): """Return content's provenance information if any. """ def _enrich_revision(provenance): p = provenance.copy() p['revision_url'] = \ reverse('api-revision', url_args={'sha1_git': provenance['revision']}) p['content_url'] = \ reverse('api-content', url_args={'q': 'sha1_git:%s' % provenance['content']}) p['origin_url'] = \ reverse('api-origin', url_args={'origin_id': provenance['origin']}) p['origin_visits_url'] = \ reverse('api-origin-visits', url_args={'origin_id': provenance['origin']}) p['origin_visit_url'] = \ reverse('api-origin-visit', url_args={'origin_id': provenance['origin'], 'visit_id': provenance['visit']}) return p return api_lookup( service.lookup_content_provenance, q, notfound_msg='Content with {} not found.'.format(q), enrich_fn=_enrich_revision) @api_route(r'/content/(?P.+)/filetype/', 'api-content-filetype') @api_doc('/content/filetype/') def api_content_filetype(request, q): """ .. http:get:: /api/1/content/[(hash_type):](hash)/filetype/ Get information about the detected MIME type of a content object. :param string hash_type: optional parameter specifying which hashing algorithm has been used - to compute the content checksum. It can be either *sha1*, *sha1_git*, *sha256* - or *blake2s256*. If that parameter is not provided, it is assumed that the - hashing algorithm used is *sha1*. + to compute the content checksum. It can be either ``sha1``, ``sha1_git``, ``sha256`` + or ``blake2s256``. If that parameter is not provided, it is assumed that the + hashing algorithm used is `sha1`. :param string hash: hexadecimal representation of the checksum value computed with the specified hashing algorithm. :>json object content_url: link to :http:get:`/api/1/content/[(hash_type):](hash)/` for getting information about the content :>json string encoding: the detected content encoding - :>json string id: the *sha1* identifier of the content + :>json string id: the **sha1** identifier of the content :>json string mimetype: the detected MIME type of the content :>json object tool: information about the tool used to detect the content filetype :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *hash_type* or *hash* has been provided - :statuscode 404: requested content can not be found in the SWH archive + :statuscode 400: an invalid **hash_type** or **hash** has been provided + :statuscode 404: requested content can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`content/sha1:dc2830a9e72f23c1dfebef4413003221baa5fb62/filetype/` """ # noqa return api_lookup( service.lookup_content_filetype, q, notfound_msg='No filetype information found for content {}.'.format(q), enrich_fn=utils.enrich_metadata_endpoint) @api_route(r'/content/(?P.+)/language/', 'api-content-language') @api_doc('/content/language/') def api_content_language(request, q): """ .. http:get:: /api/1/content/[(hash_type):](hash)/language/ Get information about the programming language used in a content object. :param string hash_type: optional parameter specifying which hashing algorithm has been used - to compute the content checksum. It can be either *sha1*, *sha1_git*, *sha256* - or *blake2s256*. If that parameter is not provided, it is assumed that the - hashing algorithm used is *sha1*. + to compute the content checksum. It can be either ``sha1``, ``sha1_git``, ``sha256`` + or ``blake2s256``. If that parameter is not provided, it is assumed that the + hashing algorithm used is ``sha1``. :param string hash: hexadecimal representation of the checksum value computed with the specified hashing algorithm. :>json object content_url: link to :http:get:`/api/1/content/[(hash_type):](hash)/` for getting information about the content - :>json string id: the *sha1* identifier of the content + :>json string id: the **sha1** identifier of the content :>json string lang: the detected programming language if any :>json object tool: information about the tool used to detect the programming language :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *hash_type* or *hash* has been provided - :statuscode 404: requested content can not be found in the SWH archive + :statuscode 400: an invalid **hash_type** or **hash** has been provided + :statuscode 404: requested content can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`content/sha1:dc2830a9e72f23c1dfebef4413003221baa5fb62/language/` """ # noqa return api_lookup( service.lookup_content_language, q, notfound_msg='No language information found for content {}.'.format(q), enrich_fn=utils.enrich_metadata_endpoint) @api_route(r'/content/(?P.+)/license/', 'api-content-license') @api_doc('/content/license/') def api_content_license(request, q): """ .. http:get:: /api/1/content/[(hash_type):](hash)/license/ Get information about the license of a content object. :param string hash_type: optional parameter specifying which hashing algorithm has been used - to compute the content checksum. It can be either *sha1*, *sha1_git*, *sha256* - or *blake2s256*. If that parameter is not provided, it is assumed that the - hashing algorithm used is *sha1*. + to compute the content checksum. It can be either ``sha1``, ``sha1_git``, ``sha256`` + or ``blake2s256``. If that parameter is not provided, it is assumed that the + hashing algorithm used is ``sha1``. :param string hash: hexadecimal representation of the checksum value computed with the specified hashing algorithm. :>json object content_url: link to :http:get:`/api/1/content/[(hash_type):](hash)/` for getting information about the content - :>json string id: the *sha1* identifier of the content + :>json string id: the **sha1** identifier of the content :>json array licenses: array of strings containing the detected license names if any :>json object tool: information about the tool used to detect the license :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *hash_type* or *hash* has been provided - :statuscode 404: requested content can not be found in the SWH archive + :statuscode 400: an invalid **hash_type** or **hash** has been provided + :statuscode 404: requested content can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`content/sha1:dc2830a9e72f23c1dfebef4413003221baa5fb62/license/` """ # noqa return api_lookup( service.lookup_content_license, q, notfound_msg='No license information found for content {}.'.format(q), enrich_fn=utils.enrich_metadata_endpoint) @api_route(r'/content/(?P.+)/ctags/', 'api-content-ctags') @api_doc('/content/ctags/', tags=['hidden']) def api_content_ctags(request, q): """ Get information about all `Ctags `_-style symbols defined in a content object. """ return api_lookup( service.lookup_content_ctags, q, notfound_msg='No ctags symbol found for content {}.'.format(q), enrich_fn=utils.enrich_metadata_endpoint) @api_route(r'/content/(?P.+)/raw/', 'api-content-raw') @api_doc('/content/raw/', handle_response=True) def api_content_raw(request, q): """ .. http:get:: /api/1/content/[(hash_type):](hash)/raw/ Get the raw content of a content object (aka a "blob"), as a byte sequence. :param string hash_type: optional parameter specifying which hashing algorithm has been used - to compute the content checksum. It can be either *sha1*, *sha1_git*, *sha256* - or *blake2s256*. If that parameter is not provided, it is assumed that the - hashing algorithm used is *sha1*. + to compute the content checksum. It can be either ``sha1``, ``sha1_git``, ``sha256`` + or ``blake2s256``. If that parameter is not provided, it is assumed that the + hashing algorithm used is ``sha1``. :param string hash: hexadecimal representation of the checksum value computed with the specified hashing algorithm. :query string filename: if provided, the downloaded content will get that filename :resheader Content-Type: application/octet-stream **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *hash_type* or *hash* has been provided - :statuscode 404: requested content can not be found in the SWH archive + :statuscode 400: an invalid **hash_type** or **hash** has been provided + :statuscode 404: requested content can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`content/sha1:dc2830a9e72f23c1dfebef4413003221baa5fb62/raw/` """ # noqa def generate(content): yield content['data'] content_raw = service.lookup_content_raw(q) if not content_raw: raise NotFoundExc('Content %s is not found.' % q) filename = request.query_params.get('filename') if not filename: filename = 'content_%s_raw' % q.replace(':', '_') response = HttpResponse(generate(content_raw), content_type='application/octet-stream') response['Content-disposition'] = 'attachment; filename=%s' % filename return response @api_route(r'/content/symbol/(?P.+)/', 'api-content-symbol') @api_doc('/content/symbol/', tags=['hidden']) def api_content_symbol(request, q=None): """Search content objects by `Ctags `_-style symbol (e.g., function name, data type, method, ...). """ result = {} last_sha1 = request.query_params.get('last_sha1', None) per_page = int(request.query_params.get('per_page', '10')) def lookup_exp(exp, last_sha1=last_sha1, per_page=per_page): return service.lookup_expression(exp, last_sha1, per_page) symbols = api_lookup( lookup_exp, q, notfound_msg="No indexed raw content match expression '{}'.".format(q), enrich_fn=functools.partial(utils.enrich_content, top_url=True)) if symbols: nb_symbols = len(symbols) if nb_symbols == per_page: query_params = {} new_last_sha1 = symbols[-1]['sha1'] query_params['last_sha1'] = new_last_sha1 if request.query_params.get('per_page'): query_params['per_page'] = per_page result['headers'] = { 'link-next': reverse('api-content-symbol', url_args={'q': q}, query_params=query_params) } result.update({ 'results': symbols }) return result @api_route(r'/content/known/search/', 'api-content-known', methods=['POST']) @api_route(r'/content/known/(?P(?!search).*)/', 'api-content-known') @api_doc('/content/known/', tags=['hidden']) def api_check_content_known(request, q=None): """ .. http:get:: /api/1/content/known/(sha1)[,(sha1), ...,(sha1)]/ - Check whether some content(s) (aka "blob(s)") is present in the SWH archive - based on its *sha1* checksum. + Check whether some content(s) (aka "blob(s)") is present in the archive + based on its **sha1** checksum. - :param string sha1: hexadecimal representation of the *sha1* checksum value + :param string sha1: hexadecimal representation of the **sha1** checksum value for the content to check existence. Multiple values can be provided separated by ','. :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request - :>json array search_res: array holding the search result for each provided *sha1* - :>json object search_stats: some statistics regarding the number of *sha1* provided - and the percentage of those found in the SWH archive + :>json array search_res: array holding the search result for each provided **sha1** + :>json object search_stats: some statistics regarding the number of **sha1** provided + and the percentage of those found in the archive **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *sha1* has been provided + :statuscode 400: an invalid **sha1** has been provided **Example:** .. parsed-literal:: :swh_web_api:`content/known/dc2830a9e72f23c1dfebef4413003221baa5fb62,0c3f19cb47ebfbe643fb19fa94c874d18fa62d12/` """ # noqa response = {'search_res': None, 'search_stats': None} search_stats = {'nbfiles': 0, 'pct': 0} search_res = None queries = [] # GET: Many hash separated values request if q: hashes = q.split(',') for v in hashes: queries.append({'filename': None, 'sha1': v}) # POST: Many hash requests in post form submission elif request.method == 'POST': data = request.data # Remove potential inputs with no associated value for k, v in data.items(): if v is not None: if k == 'q' and len(v) > 0: queries.append({'filename': None, 'sha1': v}) elif v != '': queries.append({'filename': k, 'sha1': v}) if queries: lookup = service.lookup_multiple_hashes(queries) result = [] nb_queries = len(queries) for el in lookup: res_d = {'sha1': el['sha1'], 'found': el['found']} if 'filename' in el and el['filename']: res_d['filename'] = el['filename'] result.append(res_d) search_res = result nbfound = len([x for x in lookup if x['found']]) search_stats['nbfiles'] = nb_queries search_stats['pct'] = (nbfound / nb_queries) * 100 response['search_res'] = search_res response['search_stats'] = search_stats return response @api_route(r'/content/(?P.+)/', 'api-content') @api_doc('/content/') def api_content_metadata(request, q): """ .. http:get:: /api/1/content/[(hash_type):](hash)/ Get information about a content (aka a "blob") object. - In the SWH archive, a content object is identified based on checksum + In the archive, a content object is identified based on checksum values computed using various hashing algorithms. :param string hash_type: optional parameter specifying which hashing algorithm has been used - to compute the content checksum. It can be either *sha1*, *sha1_git*, *sha256* - or *blake2s256*. If that parameter is not provided, it is assumed that the - hashing algorithm used is *sha1*. + to compute the content checksum. It can be either ``sha1``, ``sha1_git``, ``sha256`` + or ``blake2s256``. If that parameter is not provided, it is assumed that the + hashing algorithm used is ``sha1``. :param string hash: hexadecimal representation of the checksum value computed with the specified hashing algorithm. :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>json object checksums: object holding the computed checksum values for the requested content :>json string data_url: link to :http:get:`/api/1/content/[(hash_type):](hash)/raw/` for downloading the content raw bytes :>json string filetype_url: link to :http:get:`/api/1/content/[(hash_type):](hash)/filetype/` for getting information about the content MIME type :>json string language_url: link to :http:get:`/api/1/content/[(hash_type):](hash)/language/` for getting information about the programming language used in the content :>json number length: length of the content in bytes :>json string license_url: link to :http:get:`/api/1/content/[(hash_type):](hash)/license/` for getting information about the license of the content **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *hash_type* or *hash* has been provided - :statuscode 404: requested content can not be found in the SWH archive + :statuscode 400: an invalid **hash_type** or **hash** has been provided + :statuscode 404: requested content can not be found in the archive **Example:** .. parsed-literal:: curl -i :swh_web_api:`content/sha1_git:fe95a46679d128ff167b7c55df5d02356c5a1ae1/` """ # noqa return api_lookup( service.lookup_content, q, notfound_msg='Content with {} not found.'.format(q), enrich_fn=functools.partial(utils.enrich_content, query_string=q)) diff --git a/swh/web/api/views/directory.py b/swh/web/api/views/directory.py index a6dbe2b9e..c17902eec 100644 --- a/swh/web/api/views/directory.py +++ b/swh/web/api/views/directory.py @@ -1,75 +1,75 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from swh.web.common import service from swh.web.api import utils from swh.web.api.apidoc import api_doc from swh.web.api.apiurls import api_route from swh.web.api.views.utils import api_lookup @api_route(r'/directory/(?P[0-9a-f]+)/', 'api-directory') @api_route(r'/directory/(?P[0-9a-f]+)/(?P.+)/', 'api-directory') @api_doc('/directory/') def api_directory(request, sha1_git, path=None): """ .. http:get:: /api/1/directory/(sha1_git)/[(path)/] Get information about directory objects. - Directories are identified by *sha1* checksums, compatible with Git directory identifiers. + Directories are identified by **sha1** checksums, compatible with Git directory identifiers. See :func:`swh.model.identifiers.directory_identifier` in our data model module for details about how they are computed. When given only a directory identifier, this endpoint returns information about the directory itself, returning its content (usually a list of directory entries). When given a directory identifier and a path, this endpoint returns information about the directory entry pointed by the relative path, starting path resolution from the given directory. - :param string sha1_git: hexadecimal representation of the directory *sha1_git* identifier + :param string sha1_git: hexadecimal representation of the directory **sha1_git** identifier :param string path: optional parameter to get information about the directory entry pointed by that relative path :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>jsonarr object checksums: object holding the computed checksum values for a directory entry (only for file entries) - :>jsonarr string dir_id: *sha1_git* identifier of the requested directory + :>jsonarr string dir_id: **sha1_git** identifier of the requested directory :>jsonarr number length: length of a directory entry in bytes (only for file entries) for getting information about the content MIME type :>jsonarr string name: the directory entry name :>jsonarr number perms: permissions for the directory entry - :>jsonarr string target: *sha1_git* identifier of the directory entry + :>jsonarr string target: **sha1_git** identifier of the directory entry :>jsonarr string target_url: link to :http:get:`/api/1/content/[(hash_type):](hash)/` or :http:get:`/api/1/directory/(sha1_git)/[(path)/]` depending on the directory entry type - :>jsonarr string type: the type of the directory entry, can be either *dir*, *file* or *rev* + :>jsonarr string type: the type of the directory entry, can be either ``dir``, ``file`` or ``rev`` **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *hash_type* or *hash* has been provided - :statuscode 404: requested directory can not be found in the SWH archive + :statuscode 400: an invalid **hash_type** or **hash** has been provided + :statuscode 404: requested directory can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`directory/977fc4b98c0e85816348cebd3b12026407c368b6/` """ # noqa if path: error_msg_path = ('Entry with path %s relative to directory ' 'with sha1_git %s not found.') % (path, sha1_git) return api_lookup( service.lookup_directory_with_path, sha1_git, path, notfound_msg=error_msg_path, enrich_fn=utils.enrich_directory) else: error_msg_nopath = 'Directory with sha1_git %s not found.' % sha1_git return api_lookup( service.lookup_directory, sha1_git, notfound_msg=error_msg_nopath, enrich_fn=utils.enrich_directory) diff --git a/swh/web/api/views/identifiers.py b/swh/web/api/views/identifiers.py index a1499b957..068968279 100644 --- a/swh/web/api/views/identifiers.py +++ b/swh/web/api/views/identifiers.py @@ -1,77 +1,77 @@ # Copyright (C) 2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from swh.model.identifiers import ( CONTENT, DIRECTORY, RELEASE, REVISION, SNAPSHOT ) from swh.web.common import service from swh.web.common.utils import resolve_swh_persistent_id from swh.web.api.apidoc import api_doc from swh.web.api.apiurls import api_route @api_route(r'/resolve/(?P.*)/', 'api-resolve-swh-pid') @api_doc('/resolve/') def api_resolve_swh_pid(request, swh_id): """ .. http:get:: /api/1/resolve/(swh_id)/ Resolve a Software Heritage persistent identifier. Try to resolve a provided `persistent identifier `_ into an url for browsing the pointed archive object. If the provided identifier is valid, the existence of the object in the archive will also be checked. - :param string swh_id: a SWH presistent identifier + :param string swh_id: a Software Heritage presistent identifier :>json string browse_url: the url for browsing the pointed object :>json object metadata: object holding optional parts of the persistent identifier :>json string namespace: the persistent identifier namespace :>json string object_id: the hash identifier of the pointed object :>json string object_type: the type of the pointed object :>json number scheme_version: the scheme version of the persistent identifier :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error :statuscode 400: an invalid persistent identifier has been provided :statuscode 404: the pointed object does not exist in the archive **Example:** .. parsed-literal:: :swh_web_api:`resolve/swh:1:rev:96db9023b881d7cd9f379b0c154650d6c108e9a3;origin=https://github.com/openssl/openssl/` """ # noqa # try to resolve the provided pid swh_id_resolved = resolve_swh_persistent_id(swh_id) # id is well-formed, now check that the pointed # object is present in the archive, NotFoundExc # will be raised otherwise swh_id_parsed = swh_id_resolved['swh_id_parsed'] object_type = swh_id_parsed.object_type object_id = swh_id_parsed.object_id if object_type == CONTENT: service.lookup_content('sha1_git:%s' % object_id) elif object_type == DIRECTORY: service.lookup_directory(object_id) elif object_type == RELEASE: service.lookup_release(object_id) elif object_type == REVISION: service.lookup_revision(object_id) elif object_type == SNAPSHOT: service.lookup_snapshot(object_id) # id is well-formed and the pointed object exists swh_id_data = swh_id_parsed._asdict() swh_id_data['browse_url'] = swh_id_resolved['browse_url'] return swh_id_data diff --git a/swh/web/api/views/origin.py b/swh/web/api/views/origin.py index 5a0f12b3c..223cce980 100644 --- a/swh/web/api/views/origin.py +++ b/swh/web/api/views/origin.py @@ -1,330 +1,332 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from distutils.util import strtobool from swh.web.common import service from swh.web.common.utils import ( reverse, get_origin_visits ) from swh.web.api.apidoc import api_doc from swh.web.api.apiurls import api_route from swh.web.api.views.utils import api_lookup def _enrich_origin(origin): if 'id' in origin: o = origin.copy() o['origin_visits_url'] = \ reverse('api-origin-visits', url_args={'origin_id': origin['id']}) return o return origin @api_route(r'/origin/(?P[0-9]+)/', 'api-origin') @api_route(r'/origin/(?P[a-z]+)/url/(?P.+)/', 'api-origin') @api_doc('/origin/') def api_origin(request, origin_id=None, origin_type=None, origin_url=None): """ .. http:get:: /api/1/origin/(origin_id)/ Get information about a software origin. - :param int origin_id: a SWH origin identifier + :param int origin_id: a software origin identifier :>json number id: the origin unique identifier - :>json string origin_visits_url: link to in order to get information about the SWH + :>json string origin_visits_url: link to in order to get information about the visits for that origin - :>json string type: the type of software origin (*git*, *svn*, *hg*, *deb*, *ftp*, ...) + :>json string type: the type of software origin (possible values are ``git``, ``svn``, + ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :>json string url: the origin canonical url :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive + :statuscode 404: requested origin can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`origin/1/` .. http:get:: /api/1/origin/(origin_type)/url/(origin_url)/ Get information about a software origin. - :param string origin_type: the origin type (*git*, *svn*, *hg*, *deb*, *ftp*, ...) + :param string origin_type: the origin type (possible values are ``git``, ``svn``, + ``hg``, ``deb``, ``pypi``, ``ftp`` or ``deposit``) :param string origin_url: the origin url :>json number id: the origin unique identifier - :>json string origin_visits_url: link to in order to get information about the SWH + :>json string origin_visits_url: link to in order to get information about the visits for that origin - :>json string type: the type of software origin (*git*, *svn*, *hg*, *deb*, *ftp*, ...) + :>json string type: the type of software origin :>json string url: the origin canonical url :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive + :statuscode 404: requested origin can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`origin/git/url/https://github.com/python/cpython/` """ # noqa ori_dict = { 'id': origin_id, 'type': origin_type, 'url': origin_url } ori_dict = {k: v for k, v in ori_dict.items() if ori_dict[k]} if 'id' in ori_dict: error_msg = 'Origin with id %s not found.' % ori_dict['id'] else: error_msg = 'Origin with type %s and URL %s not found' % ( ori_dict['type'], ori_dict['url']) return api_lookup( service.lookup_origin, ori_dict, notfound_msg=error_msg, enrich_fn=_enrich_origin) @api_route(r'/origin/search/(?P.+)/', 'api-origin-search') @api_doc('/origin/search/') def api_origin_search(request, url_pattern): """ .. http:get:: /api/1/origin/search/(url_pattern)/ Search for software origins whose urls contain a provided string pattern or match a provided regular expression. The search is performed in a case insensitive way. :param string url_pattern: a string pattern or a regular expression :query int offset: the number of found origins to skip before returning results :query int limit: the maximum number of found origins to return :query boolean regexp: if true, consider provided pattern as a regular expression and search origins whose urls match it :query boolean with_visit: if true, only return origins with at least one visit by Software heritage :>jsonarr number id: the origin unique identifier - :>jsonarr string origin_visits_url: link to in order to get information about the SWH + :>jsonarr string origin_visits_url: link to in order to get information about the visits for that origin - :>jsonarr string type: the type of software origin (*git*, *svn*, *hg*, *deb*, *ftp*, ...) + :>jsonarr string type: the type of software origin :>jsonarr string url: the origin canonical url :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error **Example:** .. parsed-literal:: :swh_web_api:`origin/search/python/?limit=2` """ # noqa result = {} offset = int(request.query_params.get('offset', '0')) limit = int(request.query_params.get('limit', '70')) regexp = request.query_params.get('regexp', 'false') with_visit = request.query_params.get('with_visit', 'false') results = api_lookup(service.search_origin, url_pattern, offset, limit, bool(strtobool(regexp)), bool(strtobool(with_visit)), enrich_fn=_enrich_origin) nb_results = len(results) if nb_results == limit: query_params = {} query_params['offset'] = offset + limit query_params['limit'] = limit query_params['regexp'] = regexp result['headers'] = { 'link-next': reverse('api-origin-search', url_args={'url_pattern': url_pattern}, query_params=query_params) } result.update({ 'results': results }) return result @api_route(r'/origin/(?P[0-9]+)/visits/', 'api-origin-visits') @api_doc('/origin/visits/') def api_origin_visits(request, origin_id): """ .. http:get:: /api/1/origin/(origin_id)/visits/ Get information about all visits of a software origin. Visits are returned sorted in descending order according to their date. - :param int origin_id: a SWH origin identifier + :param int origin_id: a software origin identifier :query int per_page: specify the number of visits to list, for pagination purposes :query int last_visit: visit to start listing from, for pagination purposes :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :resheader Link: indicates that a subsequent result page is available and contains the url pointing to it :>jsonarr string date: ISO representation of the visit date (in UTC) :>jsonarr number id: the unique identifier of the origin :>jsonarr string origin_visit_url: link to :http:get:`/api/1/origin/(origin_id)/visit/(visit_id)/` in order to get information about the visit :>jsonarr string snapshot: the snapshot identifier of the visit :>jsonarr string snapshot_url: link to :http:get:`/api/1/snapshot/(snapshot_id)/` in order to get information about the snapshot of the visit - :>jsonarr string status: status of the visit (either *full*, *partial* or *ongoing*) + :>jsonarr string status: status of the visit (either **full**, **partial** or **ongoing**) :>jsonarr number visit: the unique identifier of the visit **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 404: requested origin can not be found in the SWH archive + :statuscode 404: requested origin can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`origin/1/visits/` """ # noqa result = {} per_page = int(request.query_params.get('per_page', '10')) last_visit = request.query_params.get('last_visit') if last_visit: last_visit = int(last_visit) def _lookup_origin_visits( origin_id, last_visit=last_visit, per_page=per_page): all_visits = get_origin_visits({'id': origin_id}) all_visits.reverse() visits = [] if not last_visit: visits = all_visits[:per_page] else: for i, v in enumerate(all_visits): if v['visit'] == last_visit: visits = all_visits[i+1:i+1+per_page] break for v in visits: yield v def _enrich_origin_visit(origin_visit): ov = origin_visit.copy() ov['origin_visit_url'] = reverse('api-origin-visit', url_args={'origin_id': origin_id, 'visit_id': ov['visit']}) snapshot = ov['snapshot'] if snapshot: ov['snapshot_url'] = reverse('api-snapshot', url_args={'snapshot_id': snapshot}) else: ov['snapshot_url'] = None return ov results = api_lookup(_lookup_origin_visits, origin_id, notfound_msg='No origin {} found'.format(origin_id), enrich_fn=_enrich_origin_visit) if results: nb_results = len(results) if nb_results == per_page: new_last_visit = results[-1]['visit'] query_params = {} query_params['last_visit'] = new_last_visit if request.query_params.get('per_page'): query_params['per_page'] = per_page result['headers'] = { 'link-next': reverse('api-origin-visits', url_args={'origin_id': origin_id}, query_params=query_params) } result.update({ 'results': results }) return result @api_route(r'/origin/(?P[0-9]+)/visit/(?P[0-9]+)/', 'api-origin-visit') @api_doc('/origin/visit/') def api_origin_visit(request, origin_id, visit_id): """ .. http:get:: /api/1/origin/(origin_id)/visit/(visit_id)/ Get information about a specific visit of a software origin. - :param int origin_id: a SWH origin identifier + :param int origin_id: a software origin identifier :param int visit_id: a visit identifier :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>json string date: ISO representation of the visit date (in UTC) :>json number origin: the origin unique identifier :>json string origin_url: link to get information about the origin :>jsonarr string snapshot: the snapshot identifier of the visit :>jsonarr string snapshot_url: link to :http:get:`/api/1/snapshot/(snapshot_id)/` in order to get information about the snapshot of the visit - :>json string status: status of the visit (either *full*, *partial* or *ongoing*) + :>json string status: status of the visit (either **full**, **partial** or **ongoing**) :>json number visit: the unique identifier of the visit **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 404: requested origin or visit can not be found in the SWH archive + :statuscode 404: requested origin or visit can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`origin/1500/visit/1/` """ # noqa def _enrich_origin_visit(origin_visit): ov = origin_visit.copy() ov['origin_url'] = reverse('api-origin', url_args={'origin_id': ov['origin']}) snapshot = ov['snapshot'] if snapshot: ov['snapshot_url'] = reverse('api-snapshot', url_args={'snapshot_id': snapshot}) else: ov['snapshot_url'] = None return ov return api_lookup( service.lookup_origin_visit, origin_id, visit_id, notfound_msg=('No visit {} for origin {} found' .format(visit_id, origin_id)), enrich_fn=_enrich_origin_visit) diff --git a/swh/web/api/views/origin_save.py b/swh/web/api/views/origin_save.py index bf694908c..7c74324fb 100644 --- a/swh/web/api/views/origin_save.py +++ b/swh/web/api/views/origin_save.py @@ -1,79 +1,79 @@ # Copyright (C) 2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from django.views.decorators.cache import never_cache from swh.web.api.apidoc import api_doc from swh.web.api.apiurls import api_route from swh.web.common.origin_save import ( create_save_origin_request, get_save_origin_requests ) @api_route(r'/origin/save/(?P.+)/url/(?P.+)/', 'api-save-origin', methods=['GET', 'POST'], throttle_scope='swh_save_origin') @never_cache @api_doc('/origin/save/') def api_save_origin(request, origin_type, origin_url): """ .. http:get:: /api/1/origin/save/(origin_type)/url/(origin_url)/ .. http:post:: /api/1/origin/save/(origin_type)/url/(origin_url)/ Request the saving of a software origin into the archive or check the status of previously created save requests. That endpoint enables to create a saving task for a software origin through a POST request. Depending of the provided origin url, the save request can either be: * immediately **accepted**, for well known code hosting providers like for instance GitHub or GitLab * **rejected**, in case the url is blacklisted by Software Heritage * **put in pending state** until a manual check is done in order to determine if it can be loaded or not Once a saving request has been accepted, its associated saving task status can then be checked through a GET request on the same url. Returned status can either be: * **not created**: no saving task has been created * **not yet scheduled**: saving task has been created but its execution has not yet been scheduled * **scheduled**: the task execution has been scheduled * **succeed**: the saving task has been successfully executed * **failed**: the saving task has been executed but it failed When issuing a POST request an object will be returned while a GET request will return an array of objects (as multiple save requests might have been submitted for the same origin). :param string origin_type: the type of origin to save - (currently only *git* but *hg* and *svn* will soon be available) + (currently only ``git`` but ``hg`` and ``svn`` will soon be available) :param string origin_url: the url of the origin to save :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>json string origin_url: the url of the origin to save :>json string origin_type: the type of the origin to save :>json string save_request_date: the date (in iso format) the save request was issued - :>json string save_request_status: the status of the save request, either *accepted*, - *rejected* or *pending* - :>json string save_task_status: the status of the origin saving task, either *not created*, - *not yet scheduled*, *scheduled*, *succeed* or *failed* + :>json string save_request_status: the status of the save request, either **accepted**, + **rejected** or **pending** + :>json string save_task_status: the status of the origin saving task, either **not created**, + **not yet scheduled**, **scheduled**, **succeed** or **failed** **Allowed HTTP Methods:** :http:method:`get`, :http:method:`post`, :http:method:`head`, :http:method:`options` :statuscode 200: no error :statuscode 400: an invalid origin type or url has been provided :statuscode 403: the provided origin url is blacklisted """ # noqa if request.method == 'POST': return create_save_origin_request(origin_type, origin_url) else: return get_save_origin_requests(origin_type, origin_url) diff --git a/swh/web/api/views/person.py b/swh/web/api/views/person.py index 0792dace8..628ef7a9d 100644 --- a/swh/web/api/views/person.py +++ b/swh/web/api/views/person.py @@ -1,44 +1,44 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from swh.web.common import service from swh.web.api.apidoc import api_doc from swh.web.api.apiurls import api_route from swh.web.api.views.utils import api_lookup @api_route(r'/person/(?P[0-9]+)/', 'api-person') @api_doc('/person/') def api_person(request, person_id): """ .. http:get:: /api/1/person/(person_id)/ - Get information about a person in the SWH archive. + Get information about a person in the archive. - :param int person_id: a SWH person identifier + :param int person_id: a person identifier :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>json string email: the email of the person :>json string fullname: the full name of the person: combination of its name and email :>json number id: the unique identifier of the person :>json string name: the name of the person **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 404: requested person can not be found in the SWH archive + :statuscode 404: requested person can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`person/8275/` """ # noqa return api_lookup( service.lookup_person, person_id, notfound_msg='Person with id {} not found.'.format(person_id)) diff --git a/swh/web/api/views/release.py b/swh/web/api/views/release.py index 8abfcf8ed..c0e168ec9 100644 --- a/swh/web/api/views/release.py +++ b/swh/web/api/views/release.py @@ -1,58 +1,58 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from swh.web.common import service from swh.web.api import utils from swh.web.api.apidoc import api_doc from swh.web.api.apiurls import api_route from swh.web.api.views.utils import api_lookup @api_route(r'/release/(?P[0-9a-f]+)/', 'api-release') @api_doc('/release/') def api_release(request, sha1_git): """ .. http:get:: /api/1/release/(sha1_git)/ - Get information about a release in the SWH archive. - Releases are identified by *sha1* checksums, compatible with Git tag identifiers. + Get information about a release in the archive. + Releases are identified by **sha1** checksums, compatible with Git tag identifiers. See :func:`swh.model.identifiers.release_identifier` in our data model module for details about how they are computed. - :param string sha1_git: hexadecimal representation of the release *sha1_git* identifier + :param string sha1_git: hexadecimal representation of the release **sha1_git** identifier :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>json object author: information about the author of the release :>json string author_url: link to :http:get:`/api/1/person/(person_id)/` to get information about the author of the release :>json string date: ISO representation of the release date (in UTC) :>json string id: the release unique identifier :>json string message: the message associated to the release :>json string name: the name of the release :>json string target: the target identifier of the release - :>json string target_type: the type of the target, can be either *release*, - *revision*, *content*, *directory* + :>json string target_type: the type of the target, can be either **release**, + **revision**, **content**, **directory** :>json string target_url: a link to the adequate api url based on the target type **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *sha1_git* value has been provided - :statuscode 404: requested release can not be found in the SWH archive + :statuscode 400: an invalid **sha1_git** value has been provided + :statuscode 404: requested release can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`release/208f61cc7a5dbc9879ae6e5c2f95891e270f09ef/` """ # noqa error_msg = 'Release with sha1_git %s not found.' % sha1_git return api_lookup( service.lookup_release, sha1_git, notfound_msg=error_msg, enrich_fn=utils.enrich_release) diff --git a/swh/web/api/views/revision.py b/swh/web/api/views/revision.py index f15606fff..891c60740 100644 --- a/swh/web/api/views/revision.py +++ b/swh/web/api/views/revision.py @@ -1,505 +1,505 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from django.http import HttpResponse from swh.web.common import service from swh.web.common.utils import reverse from swh.web.common.utils import parse_timestamp from swh.web.api import utils from swh.web.api.apidoc import api_doc from swh.web.api.apiurls import api_route from swh.web.api.views.utils import api_lookup def _revision_directory_by(revision, path, request_path, limit=100, with_data=False): """ Compute the revision matching criterion's directory or content data. Args: revision: dictionary of criterions representing a revision to lookup path: directory's path to lookup request_path: request path which holds the original context to limit: optional query parameter to limit the revisions log (default to 100). For now, note that this limit could impede the transitivity conclusion about sha1_git not being an ancestor of with_data: indicate to retrieve the content's raw data if path resolves to a content. """ def enrich_directory_local(dir, context_url=request_path): return utils.enrich_directory(dir, context_url) rev_id, result = service.lookup_directory_through_revision( revision, path, limit=limit, with_data=with_data) content = result['content'] if result['type'] == 'dir': # dir_entries result['content'] = list(map(enrich_directory_local, content)) else: # content result['content'] = utils.enrich_content(content) return result @api_route(r'/revision/origin/(?P[0-9]+)' r'/branch/(?P.+)/log/', 'api-revision-origin-log') @api_route(r'/revision/origin/(?P[0-9]+)/log/', 'api-revision-origin-log') @api_route(r'/revision/origin/(?P[0-9]+)' r'/ts/(?P.+)/log/', 'api-revision-origin-log') @api_route(r'/revision/origin/(?P[0-9]+)' r'/branch/(?P.+)' r'/ts/(?P.+)/log/', 'api-revision-origin-log') @api_doc('/revision/origin/log/') def api_revision_log_by(request, origin_id, branch_name='refs/heads/master', ts=None): """ .. http:get:: /api/1/revision/origin/(origin_id)[/branch/(branch_name)][/ts/(timestamp)]/log Show the commit log for a revision, searching for it based on software origin, branch name, and/or visit timestamp. This endpoint behaves like :http:get:`/api/1/revision/(sha1_git)[/prev/(prev_sha1s)]/log/`, but operates on the revision that has been found at a given software origin, close to a given point in time, pointed by a given branch. - :param int origin_id: a SWH origin identifier + :param int origin_id: a software origin identifier :param string branch_name: optional parameter specifying a fully-qualified branch name associated to the software origin, e.g., "refs/heads/master". Defaults to the master branch. :param string timestamp: optional parameter specifying a timestamp close to which the revision pointed by the given branch should be looked up. The timestamp can be expressed either as an ISO date or as a Unix one (in UTC). Defaults to now. :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>jsonarr object author: information about the author of the revision :>jsonarr string author_url: link to :http:get:`/api/1/person/(person_id)/` to get information about the author of the revision :>jsonarr object committer: information about the committer of the revision :>jsonarr string committer_url: link to :http:get:`/api/1/person/(person_id)/` to get information about the committer of the revision :>jsonarr string committer_date: ISO representation of the commit date (in UTC) :>jsonarr string date: ISO representation of the revision date (in UTC) :>jsonarr string directory: the unique identifier that revision points to :>jsonarr string directory_url: link to :http:get:`/api/1/directory/(sha1_git)/[(path)/]` to get information about the directory associated to the revision :>jsonarr string id: the revision unique identifier :>jsonarr boolean merge: whether or not the revision corresponds to a merge commit :>jsonarr string message: the message associated to the revision :>jsonarr array parents: the parents of the revision, i.e. the previous revisions that head directly to it, each entry of that array contains an unique parent revision identifier but also a link to :http:get:`/api/1/revision/(sha1_git)/` to get more information about it :>jsonarr string type: the type of the revision **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 404: no revision matching the given criteria could be found in the SWH archive + :statuscode 404: no revision matching the given criteria could be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`revision/origin/723566/ts/2016-01-17T00:00:00+00:00/log/` """ # noqa result = {} per_page = int(request.query_params.get('per_page', '10')) if ts: ts = parse_timestamp(ts) def lookup_revision_log_by_with_limit(o_id, br, ts, limit=per_page+1): return service.lookup_revision_log_by(o_id, br, ts, limit) error_msg = 'No revision matching origin %s ' % origin_id error_msg += ', branch name %s' % branch_name error_msg += (' and time stamp %s.' % ts) if ts else '.' rev_get = api_lookup( lookup_revision_log_by_with_limit, origin_id, branch_name, ts, notfound_msg=error_msg, enrich_fn=utils.enrich_revision) nb_rev = len(rev_get) if nb_rev == per_page+1: revisions = rev_get[:-1] last_sha1_git = rev_get[-1]['id'] params = {k: v for k, v in {'origin_id': origin_id, 'branch_name': branch_name, 'ts': ts, }.items() if v is not None} query_params = {} query_params['sha1_git'] = last_sha1_git if request.query_params.get('per_page'): query_params['per_page'] = per_page result['headers'] = { 'link-next': reverse('api-revision-origin-log', url_args=params, query_params=query_params) } else: revisions = rev_get result.update({'results': revisions}) return result @api_route(r'/revision/origin/(?P[0-9]+)/directory/', 'api-revision-origin-directory') @api_route(r'/revision/origin/(?P[0-9]+)/directory/(?P.+)/', 'api-revision-origin-directory') @api_route(r'/revision/origin/(?P[0-9]+)' r'/branch/(?P.+)/directory/', 'api-revision-origin-directory') @api_route(r'/revision/origin/(?P[0-9]+)' r'/branch/(?P.+)/ts/(?P.+)/directory/', 'api-revision-origin-directory') @api_route(r'/revision/origin/(?P[0-9]+)' r'/branch/(?P.+)/directory/(?P.+)/', 'api-revision-origin-directory') @api_route(r'/revision/origin/(?P[0-9]+)' r'/branch/(?P.+)/ts/(?P.+)' r'/directory/(?P.+)/', 'api-revision-origin-directory') @api_doc('/revision/origin/directory/', tags=['hidden']) def api_directory_through_revision_origin(request, origin_id, branch_name="refs/heads/master", ts=None, path=None, with_data=False): """ Display directory or content information through a revision identified by origin/branch/timestamp. """ if ts: ts = parse_timestamp(ts) return _revision_directory_by({'origin_id': origin_id, 'branch_name': branch_name, 'ts': ts }, path, request.path, with_data=with_data) @api_route(r'/revision/origin/(?P[0-9]+)/', 'api-revision-origin') @api_route(r'/revision/origin/(?P[0-9]+)' r'/branch/(?P.+)/', 'api-revision-origin') @api_route(r'/revision/origin/(?P[0-9]+)' r'/branch/(?P.+)/ts/(?P.+)/', 'api-revision-origin') @api_route(r'/revision/origin/(?P[0-9]+)/ts/(?P.+)/', 'api-revision-origin') @api_doc('/revision/origin/') def api_revision_with_origin(request, origin_id, branch_name="refs/heads/master", ts=None): """ .. http:get:: /api/1/revision/origin/(origin_id)/[branch/(branch_name)/][ts/(timestamp)/] Get information about a revision, searching for it based on software origin, branch name, and/or visit timestamp. This endpoint behaves like :http:get:`/api/1/revision/(sha1_git)/`, but operates on the revision that has been found at a given software origin, close to a given point in time, pointed by a given branch. - :param int origin_id: a SWH origin identifier + :param int origin_id: a software origin identifier :param string branch_name: optional parameter specifying a fully-qualified branch name associated to the software origin, e.g., "refs/heads/master". Defaults to the master branch. :param string timestamp: optional parameter specifying a timestamp close to which the revision pointed by the given branch should be looked up. The timestamp can be expressed either as an ISO date or as a Unix one (in UTC). Defaults to now. :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>json object author: information about the author of the revision :>json string author_url: link to :http:get:`/api/1/person/(person_id)/` to get information about the author of the revision :>json object committer: information about the committer of the revision :>json string committer_url: link to :http:get:`/api/1/person/(person_id)/` to get information about the committer of the revision :>json string committer_date: ISO representation of the commit date (in UTC) :>json string date: ISO representation of the revision date (in UTC) :>json string directory: the unique identifier that revision points to :>json string directory_url: link to :http:get:`/api/1/directory/(sha1_git)/[(path)/]` to get information about the directory associated to the revision :>json string id: the revision unique identifier :>json boolean merge: whether or not the revision corresponds to a merge commit :>json string message: the message associated to the revision :>json array parents: the parents of the revision, i.e. the previous revisions that head directly to it, each entry of that array contains an unique parent revision identifier but also a link to :http:get:`/api/1/revision/(sha1_git)/` to get more information about it :>json string type: the type of the revision **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 404: no revision matching the given criteria could be found in the SWH archive + :statuscode 404: no revision matching the given criteria could be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`revision/origin/13706355/branch/refs/heads/2.7/` """ # noqa ts = parse_timestamp(ts) return api_lookup( service.lookup_revision_by, origin_id, branch_name, ts, notfound_msg=('Revision with (origin_id: {}, branch_name: {}' ', ts: {}) not found.'.format(origin_id, branch_name, ts)), enrich_fn=utils.enrich_revision) @api_route(r'/revision/(?P[0-9a-f]+)/prev/(?P[0-9a-f/]+)/', 'api-revision-context') @api_doc('/revision/prev/', tags=['hidden']) def api_revision_with_context(request, sha1_git, context): """ Return information about revision with id sha1_git. """ def _enrich_revision(revision, context=context): return utils.enrich_revision(revision, context) return api_lookup( service.lookup_revision, sha1_git, notfound_msg='Revision with sha1_git %s not found.' % sha1_git, enrich_fn=_enrich_revision) @api_route(r'/revision/(?P[0-9a-f]+)/', 'api-revision') @api_doc('/revision/') def api_revision(request, sha1_git): """ .. http:get:: /api/1/revision/(sha1_git)/ - Get information about a revision in the SWH archive. - Revisions are identified by *sha1* checksums, compatible with Git commit identifiers. + Get information about a revision in the archive. + Revisions are identified by **sha1** checksums, compatible with Git commit identifiers. See :func:`swh.model.identifiers.revision_identifier` in our data model module for details about how they are computed. - :param string sha1_git: hexadecimal representation of the revision *sha1_git* identifier + :param string sha1_git: hexadecimal representation of the revision **sha1_git** identifier :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>json object author: information about the author of the revision :>json string author_url: link to :http:get:`/api/1/person/(person_id)/` to get information about the author of the revision :>json object committer: information about the committer of the revision :>json string committer_url: link to :http:get:`/api/1/person/(person_id)/` to get information about the committer of the revision :>json string committer_date: ISO representation of the commit date (in UTC) :>json string date: ISO representation of the revision date (in UTC) :>json string directory: the unique identifier that revision points to :>json string directory_url: link to :http:get:`/api/1/directory/(sha1_git)/[(path)/]` to get information about the directory associated to the revision :>json string id: the revision unique identifier :>json boolean merge: whether or not the revision corresponds to a merge commit :>json string message: the message associated to the revision :>json array parents: the parents of the revision, i.e. the previous revisions that head directly to it, each entry of that array contains an unique parent revision identifier but also a link to :http:get:`/api/1/revision/(sha1_git)/` to get more information about it :>json string type: the type of the revision **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *sha1_git* value has been provided - :statuscode 404: requested revision can not be found in the SWH archive + :statuscode 400: an invalid **sha1_git** value has been provided + :statuscode 404: requested revision can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`revision/aafb16d69fd30ff58afdd69036a26047f3aebdc6/` """ # noqa return api_lookup( service.lookup_revision, sha1_git, notfound_msg='Revision with sha1_git {} not found.'.format(sha1_git), enrich_fn=utils.enrich_revision) @api_route(r'/revision/(?P[0-9a-f]+)/raw/', 'api-revision-raw-message') @api_doc('/revision/raw/', tags=['hidden'], handle_response=True) def api_revision_raw_message(request, sha1_git): """Return the raw data of the message of revision identified by sha1_git """ raw = service.lookup_revision_message(sha1_git) response = HttpResponse(raw['message'], content_type='application/octet-stream') response['Content-disposition'] = \ 'attachment;filename=rev_%s_raw' % sha1_git return response @api_route(r'/revision/(?P[0-9a-f]+)/directory/', 'api-revision-directory') @api_route(r'/revision/(?P[0-9a-f]+)/directory/(?P.+)/', 'api-revision-directory') @api_doc('/revision/directory/') def api_revision_directory(request, sha1_git, dir_path=None, with_data=False): """ .. http:get:: /api/1/revision/(sha1_git)/directory/[(path)/] Get information about directory (entry) objects associated to revisions. Each revision is associated to a single "root" directory. This endpoint behaves like :http:get:`/api/1/directory/(sha1_git)/[(path)/]`, but operates on the root directory associated to a given revision. - :param string sha1_git: hexadecimal representation of the revision *sha1_git* identifier + :param string sha1_git: hexadecimal representation of the revision **sha1_git** identifier :param string path: optional parameter to get information about the directory entry pointed by that relative path :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>json array content: directory entries as returned by :http:get:`/api/1/directory/(sha1_git)/[(path)/]` :>json string path: path of directory from the revision root one :>json string revision: the unique revision identifier :>json string type: the type of the directory **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *sha1_git* value has been provided - :statuscode 404: requested revision can not be found in the SWH archive + :statuscode 400: an invalid **sha1_git** value has been provided + :statuscode 404: requested revision can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`revision/f1b94134a4b879bc55c3dacdb496690c8ebdc03f/directory/` """ # noqa return _revision_directory_by({'sha1_git': sha1_git}, dir_path, request.path, with_data=with_data) @api_route(r'/revision/(?P[0-9a-f]+)/log/', 'api-revision-log') @api_route(r'/revision/(?P[0-9a-f]+)' r'/prev/(?P[0-9a-f/]+)/log/', 'api-revision-log') @api_doc('/revision/log/') def api_revision_log(request, sha1_git, prev_sha1s=None): """ .. http:get:: /api/1/revision/(sha1_git)[/prev/(prev_sha1s)]/log/ - Get a list of all SWH revisions heading to a given one, in other words show the commit log. + Get a list of all revisions heading to a given one, in other words show the commit log. - :param string sha1_git: hexadecimal representation of the revision *sha1_git* identifier + :param string sha1_git: hexadecimal representation of the revision **sha1_git** identifier :param string prev_sha1s: optional parameter representing the navigation breadcrumbs (descendant revisions previously visited). If multiple values, use / as delimiter. If provided, revisions information will be added at the beginning of the returned list. :query int per_page: number of elements in the returned list, for pagination purpose :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :resheader Link: indicates that a subsequent result page is available and contains the url pointing to it :>jsonarr object author: information about the author of the revision :>jsonarr string author_url: link to :http:get:`/api/1/person/(person_id)/` to get information about the author of the revision :>jsonarr object committer: information about the committer of the revision :>jsonarr string committer_url: link to :http:get:`/api/1/person/(person_id)/` to get information about the committer of the revision :>jsonarr string committer_date: ISO representation of the commit date (in UTC) :>jsonarr string date: ISO representation of the revision date (in UTC) :>jsonarr string directory: the unique identifier that revision points to :>jsonarr string directory_url: link to :http:get:`/api/1/directory/(sha1_git)/[(path)/]` to get information about the directory associated to the revision :>jsonarr string id: the revision unique identifier :>jsonarr boolean merge: whether or not the revision corresponds to a merge commit :>jsonarr string message: the message associated to the revision :>jsonarr array parents: the parents of the revision, i.e. the previous revisions that head directly to it, each entry of that array contains an unique parent revision identifier but also a link to :http:get:`/api/1/revision/(sha1_git)/` to get more information about it :>jsonarr string type: the type of the revision **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *sha1_git* value has been provided - :statuscode 404: requested revision can not be found in the SWH archive + :statuscode 400: an invalid **sha1_git** value has been provided + :statuscode 404: requested revision can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`revision/e1a315fa3fa734e2a6154ed7b5b9ae0eb8987aad/log/` """ # noqa result = {} per_page = int(request.query_params.get('per_page', '10')) def lookup_revision_log_with_limit(s, limit=per_page+1): return service.lookup_revision_log(s, limit) error_msg = 'Revision with sha1_git %s not found.' % sha1_git rev_get = api_lookup(lookup_revision_log_with_limit, sha1_git, notfound_msg=error_msg, enrich_fn=utils.enrich_revision) nb_rev = len(rev_get) if nb_rev == per_page+1: rev_backward = rev_get[:-1] new_last_sha1 = rev_get[-1]['id'] query_params = {} if request.query_params.get('per_page'): query_params['per_page'] = per_page result['headers'] = { 'link-next': reverse('api-revision-log', url_args={'sha1_git': new_last_sha1}, query_params=query_params) } else: rev_backward = rev_get if not prev_sha1s: # no nav breadcrumbs, so we're done revisions = rev_backward else: rev_forward_ids = prev_sha1s.split('/') rev_forward = api_lookup( service.lookup_revision_multiple, rev_forward_ids, notfound_msg=error_msg, enrich_fn=utils.enrich_revision) revisions = rev_forward + rev_backward result.update({ 'results': revisions }) return result diff --git a/swh/web/api/views/snapshot.py b/swh/web/api/views/snapshot.py index 0fa3f0746..43d56197d 100644 --- a/swh/web/api/views/snapshot.py +++ b/swh/web/api/views/snapshot.py @@ -1,107 +1,107 @@ # Copyright (C) 2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from swh.web.common import service from swh.web.common.utils import reverse from swh.web.config import get_config from swh.web.api.apidoc import api_doc from swh.web.api import utils from swh.web.api.apiurls import api_route from swh.web.api.views.utils import api_lookup @api_route(r'/snapshot/(?P[0-9a-f]+)/', 'api-snapshot') @api_doc('/snapshot/') def api_snapshot(request, snapshot_id): """ .. http:get:: /api/1/snapshot/(snapshot_id)/ - Get information about a snapshot in the SWH archive. + Get information about a snapshot in the archive. A snapshot is a set of named branches, which are pointers to objects at any level of the Software Heritage DAG. It represents a full picture of an origin at a given time. As well as pointing to other objects in the Software Heritage DAG, branches can also be aliases, in which case their target is the name of another branch in the same snapshot, or dangling, in which case the target is unknown. A snapshot identifier is a salted sha1. See :func:`swh.model.identifiers.snapshot_identifier` in our data model module for details about how they are computed. - :param sha1 snapshot_id: a SWH snapshot identifier + :param sha1 snapshot_id: a snapshot identifier :query str branches_from: optional parameter used to skip branches whose name is lesser than it before returning them :query int branches_count: optional parameter used to restrain the amount of returned branches (default to 1000) :query str target_types: optional comma separated list parameter used to filter the target types of branch to return (possible values - that can be contained in that list are `'content', 'directory', - 'revision', 'release', 'snapshot', 'alias'`) + that can be contained in that list are ``content``, ``directory``, + ``revision``, ``release``, ``snapshot`` or ``alias``) :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :resheader Link: indicates that a subsequent result page is available and contains the url pointing to it :>json object branches: object containing all branches associated to the snapshot, - for each of them the associated SWH target type and id are given but also + for each of them the associated target type and id are given but also a link to get information about that target :>json string id: the unique identifier of the snapshot **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error :statuscode 400: an invalid snapshot identifier has been provided - :statuscode 404: requested snapshot can not be found in the SWH archive + :statuscode 404: requested snapshot can not be found in the archive **Example:** .. parsed-literal:: :swh_web_api:`snapshot/6a3a2cf0b2b90ce7ae1cf0a221ed68035b686f5a/` """ # noqa def _enrich_snapshot(snapshot): s = snapshot.copy() if 'branches' in s: s['branches'] = { k: utils.enrich_object(v) if v else None for k, v in s['branches'].items() } return s snapshot_content_max_size = get_config()['snapshot_content_max_size'] branches_from = request.GET.get('branches_from', '') branches_count = int(request.GET.get('branches_count', snapshot_content_max_size)) target_types = request.GET.get('target_types', None) target_types = target_types.split(',') if target_types else None results = api_lookup( service.lookup_snapshot, snapshot_id, branches_from, branches_count+1, target_types, notfound_msg='Snapshot with id {} not found.'.format(snapshot_id), enrich_fn=_enrich_snapshot) next_branch = None if len(results['branches']) > branches_count: next_branch = sorted(results['branches'].keys())[-1] del results['branches'][next_branch] response = {'results': results, 'headers': {}} if next_branch: response['headers']['link-next'] = \ reverse('api-snapshot', url_args={'snapshot_id': snapshot_id}, query_params={'branches_from': next_branch, 'branches_count': branches_count, 'target_types': target_types}) return response diff --git a/swh/web/api/views/stat.py b/swh/web/api/views/stat.py index 019d1d3a4..e8a3f9c88 100644 --- a/swh/web/api/views/stat.py +++ b/swh/web/api/views/stat.py @@ -1,53 +1,53 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from swh.web.common import service from swh.web.api.apidoc import api_doc from swh.web.api.apiurls import api_route @api_route(r'/stat/counters/', 'api-stat-counters') @api_doc('/stat/counters/', noargs=True) def api_stats(request): """ .. http:get:: /api/1/stat/counters/ Get statistics about the content of the archive. - :>json number content: current number of content objects (aka files) in the SWH archive - :>json number directory: current number of directory objects in the SWH archive - :>json number directory_entry_dir: current number of SWH directory entries - pointing to others SWH directories in the SWH archive - :>json number directory_entry_file: current number of SWH directory entries - pointing to SWH content objects in the SWH archive - :>json number directory_entry_rev: current number of SWH directory entries - pointing to SWH revision objects (e.g. git submodules) in the SWH archive - :>json number entity: current number of SWH entities (a SWH entity is either - a *group_of_entities*, a *group_of_persons*, a *project*, a *person*, an *organization*, - or a *hosting* service) in the SWH archive - :>json number origin: current number of SWH origins (an origin is a "place" where code - source can be found, e.g. a git repository, a tarball, ...) in the SWH archive - :>json number person: current number of SWH persons (code source authors or committers) - in the SWH archive - :>json number release: current number of SWH releases objects in the SWH archive - :>json number revision: current number of SWH revision objects (aka commits) in the SWH archive + :>json number content: current number of content objects (aka files) in the archive + :>json number directory: current number of directory objects in the archive + :>json number directory_entry_dir: current number of directory entries + pointing to others directories in the archive + :>json number directory_entry_file: current number of directory entries + pointing to content objects in the archive + :>json number directory_entry_rev: current number of directory entries + pointing to revision objects (e.g. git submodules) in the archive + :>json number entity: current number of entities (an entity is either + a **group_of_entities**, a **group_of_persons**, a **project**, a **person**, an **organization**, + or a **hosting** service) in the archive + :>json number origin: current number of software origins (an origin is a "place" where code + source can be found, e.g. a git repository, a tarball, ...) in the archive + :>json number person: current number of persons (code source authors or committers) + in the archive + :>json number release: current number of releases objects in the archive + :>json number revision: current number of revision objects (aka commits) in the archive :>json number skipped_content: current number of content objects (aka files) which where - not inserted in the SWH archive + not inserted in the archive :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error **Example:** .. parsed-literal:: :swh_web_api:`stat/counters/` """ # noqa return service.stat_counters() diff --git a/swh/web/api/views/vault.py b/swh/web/api/views/vault.py index 207b63866..e5e3803a2 100644 --- a/swh/web/api/views/vault.py +++ b/swh/web/api/views/vault.py @@ -1,203 +1,203 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from django.http import HttpResponse from django.views.decorators.cache import never_cache from swh.model import hashutil from swh.web.common import service, query from swh.web.common.utils import reverse from swh.web.api.apidoc import api_doc from swh.web.api.apiurls import api_route from swh.web.api.views.utils import api_lookup # XXX: a bit spaghetti. Would be better with class-based views. def _dispatch_cook_progress(request, obj_type, obj_id): hex_id = hashutil.hash_to_hex(obj_id) object_name = obj_type.split('_')[0].title() if request.method == 'GET': return api_lookup( service.vault_progress, obj_type, obj_id, notfound_msg=("{} '{}' was never requested." .format(object_name, hex_id))) elif request.method == 'POST': email = request.POST.get('email', request.GET.get('email', None)) return api_lookup( service.vault_cook, obj_type, obj_id, email, notfound_msg=("{} '{}' not found." .format(object_name, hex_id))) @api_route(r'/vault/directory/(?P[a-fA-F0-9]+)/', 'api-vault-cook-directory', methods=['GET', 'POST'], throttle_scope='swh_vault_cooking') @never_cache @api_doc('/vault/directory/', tags=['hidden']) def api_vault_cook_directory(request, dir_id): """ .. http:get:: /api/1/vault/directory/(dir_id)/ .. http:post:: /api/1/vault/directory/(dir_id)/ Request the cooking of an archive for a directory or check its cooking status. That endpoint enables to create a vault cooking task for a directory through a POST request or check the status of a previously created one through a GET request. To import the directory in the current directory, use:: $ tar xvf path/to/directory.tar.gz :param string dir_id: the directory's sha1 identifier :query string email: e-mail to notify when the archive is ready :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>json string fetch_url: the url from which to download the archive once it has been cooked (see :http:get:`/api/1/vault/directory/(dir_id)/raw/`) :>json string obj_type: the type of object to cook (directory or revision) :>json string progress_message: message describing the cooking task progress :>json number id: the cooking task id :>json string status: the cooking task status (new/pending/done/failed) :>json string obj_id: the identifier of the object to cook **Allowed HTTP Methods:** :http:method:`get`, :http:method:`post`, :http:method:`head`, :http:method:`options` :statuscode 200: no error :statuscode 400: an invalid directory identifier has been provided - :statuscode 404: requested directory can not be found in the SWH archive + :statuscode 404: requested directory can not be found in the archive """ # noqa _, obj_id = query.parse_hash_with_algorithms_or_throws( dir_id, ['sha1'], 'Only sha1_git is supported.') res = _dispatch_cook_progress(request, 'directory', obj_id) res['fetch_url'] = reverse('api-vault-fetch-directory', url_args={'dir_id': dir_id}) return res @api_route(r'/vault/directory/(?P[a-fA-F0-9]+)/raw/', 'api-vault-fetch-directory') @api_doc('/vault/directory/raw/', tags=['hidden'], handle_response=True) def api_vault_fetch_directory(request, dir_id): """ .. http:get:: /api/1/vault/directory/(dir_id)/raw/ Fetch the cooked archive for a directory. See :http:get:`/api/1/vault/directory/(dir_id)/` to get more details on directory cooking. :param string dir_id: the directory's sha1 identifier :resheader Content-Type: application/octet-stream **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error :statuscode 400: an invalid directory identifier has been provided - :statuscode 404: requested directory can not be found in the SWH archive + :statuscode 404: requested directory can not be found in the archive """ # noqa _, obj_id = query.parse_hash_with_algorithms_or_throws( dir_id, ['sha1'], 'Only sha1_git is supported.') res = api_lookup( service.vault_fetch, 'directory', obj_id, notfound_msg="Directory with ID '{}' not found.".format(dir_id)) fname = '{}.tar.gz'.format(dir_id) response = HttpResponse(res, content_type='application/gzip') response['Content-disposition'] = 'attachment; filename={}'.format(fname) return response @api_route(r'/vault/revision/(?P[a-fA-F0-9]+)/gitfast/', 'api-vault-cook-revision_gitfast', methods=['GET', 'POST'], throttle_scope='swh_vault_cooking') @never_cache @api_doc('/vault/revision/gitfast/', tags=['hidden']) def api_vault_cook_revision_gitfast(request, rev_id): """ .. http:get:: /api/1/vault/revision/(rev_id)/gitfast/ .. http:post:: /api/1/vault/revision/(rev_id)/gitfast/ Request the cooking of a gitfast archive for a revision or check its cooking status. That endpoint enables to create a vault cooking task for a revision through a POST request or check the status of a previously created one through a GET request. To import the revision in the current directory, use:: $ git init $ zcat path/to/revision.gitfast.gz | git fast-import $ git checkout HEAD :param string rev_id: the revision's sha1 identifier :query string email: e-mail to notify when the gitfast archive is ready :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>json string fetch_url: the url from which to download the archive once it has been cooked (see :http:get:`/api/1/vault/revision/(rev_id)/gitfast/raw/`) :>json string obj_type: the type of object to cook (directory or revision) :>json string progress_message: message describing the cooking task progress :>json number id: the cooking task id :>json string status: the cooking task status (new/pending/done/failed) :>json string obj_id: the identifier of the object to cook **Allowed HTTP Methods:** :http:method:`get`, :http:method:`post`, :http:method:`head`, :http:method:`options` :statuscode 200: no error :statuscode 400: an invalid revision identifier has been provided - :statuscode 404: requested revision can not be found in the SWH archive + :statuscode 404: requested revision can not be found in the archive """ # noqa _, obj_id = query.parse_hash_with_algorithms_or_throws( rev_id, ['sha1'], 'Only sha1_git is supported.') res = _dispatch_cook_progress(request, 'revision_gitfast', obj_id) res['fetch_url'] = reverse('api-vault-fetch-revision_gitfast', url_args={'rev_id': rev_id}) return res @api_route(r'/vault/revision/(?P[a-fA-F0-9]+)/gitfast/raw/', 'api-vault-fetch-revision_gitfast') @api_doc('/vault/revision/gitfast/raw/', tags=['hidden'], handle_response=True) def api_vault_fetch_revision_gitfast(request, rev_id): """ .. http:get:: /api/1/vault/revision/(rev_id)/gitfast/raw/ Fetch the cooked gitfast archive for a revision. See :http:get:`/api/1/vault/revision/(rev_id)/gitfast/` to get more details on directory cooking. :param string rev_id: the revision's sha1 identifier :resheader Content-Type: application/octet-stream **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error :statuscode 400: an invalid revision identifier has been provided - :statuscode 404: requested revision can not be found in the SWH archive + :statuscode 404: requested revision can not be found in the archive """ # noqa _, obj_id = query.parse_hash_with_algorithms_or_throws( rev_id, ['sha1'], 'Only sha1_git is supported.') res = api_lookup( service.vault_fetch, 'revision_gitfast', obj_id, notfound_msg="Revision with ID '{}' not found.".format(rev_id)) fname = '{}.gitfast.gz'.format(rev_id) response = HttpResponse(res, content_type='application/gzip') response['Content-disposition'] = 'attachment; filename={}'.format(fname) return response diff --git a/swh/web/assets/src/bundles/origin/visits-histogram.js b/swh/web/assets/src/bundles/origin/visits-histogram.js index 5c59b92c1..f0613810c 100644 --- a/swh/web/assets/src/bundles/origin/visits-histogram.js +++ b/swh/web/assets/src/bundles/origin/visits-histogram.js @@ -1,337 +1,337 @@ /** * Copyright (C) 2018 The Software Heritage developers * See the AUTHORS file at the top-level directory of this distribution * License: GNU Affero General Public License version 3, or any later version * See top-level LICENSE file for more information */ -// Creation of a stacked histogram with D3.js for SWH origin visits history +// Creation of a stacked histogram with D3.js for software origin visits history // Parameters description: // - container: selector for the div that will contain the histogram // - visitsData: raw swh origin visits data // - currentYear: the visits year to display by default // - yearClickCallback: callback when the user selects a year through the histogram import * as d3 from 'd3'; export function createVisitsHistogram(container, visitsData, currentYear, yearClickCallback) { // remove previously created histogram and tooltip if any d3.select(container).select('svg').remove(); d3.select('div.d3-tooltip').remove(); // histogram size and margins let width = 1000; let height = 300; let margin = {top: 20, right: 80, bottom: 30, left: 50}; // create responsive svg let svg = d3.select(container) .attr('style', 'padding-bottom: ' + Math.ceil(height * 100 / width) + '%') .append('svg') .attr('viewBox', '0 0 ' + width + ' ' + height); // create tooltip div let tooltip = d3.select('body') .append('div') .attr('class', 'd3-tooltip') .style('opacity', 0); // update width and height without margins width = width - margin.left - margin.right; height = height - margin.top - margin.bottom; // create main svg group element let g = svg.append('g').attr('transform', 'translate(' + margin.left + ',' + margin.top + ')'); // create x scale let x = d3.scaleTime().rangeRound([0, width]); // create y scale let y = d3.scaleLinear().range([height, 0]); // create ordinal colorscale mapping visit status let colors = d3.scaleOrdinal() .domain(['full', 'partial', 'failed', 'ongoing']) .range(['#008000', '#edc344', '#ff0000', '#0000ff']); - // first SWH crawls were made in 2015 + // first swh crawls were made in 2015 let startYear = 2015; // set latest display year as the current one let now = new Date(); let endYear = now.getUTCFullYear() + 1; let monthExtent = [new Date(Date.UTC(startYear, 0, 1)), new Date(Date.UTC(endYear, 0, 1))]; // create months bins based on setup extent let monthBins = d3.timeMonths(d3.timeMonth.offset(monthExtent[0], -1), monthExtent[1]); // create years bins based on setup extent let yearBins = d3.timeYears(monthExtent[0], monthExtent[1]); // set x scale domain x.domain(d3.extent(monthBins)); // use D3 histogram layout to create a function that will bin the visits by month let binByMonth = d3.histogram() .value(d => d.date) .domain(x.domain()) .thresholds(monthBins); // use D3 nest function to group the visits by status let visitsByStatus = d3.nest() .key(d => d['status']) .sortKeys(d3.ascending) .entries(visitsData); // prepare data in order to be able to stack visit statuses by month let statuses = []; let histData = []; for (let i = 0; i < monthBins.length; ++i) { histData[i] = {}; } visitsByStatus.forEach(entry => { statuses.push(entry.key); let monthsData = binByMonth(entry.values); for (let i = 0; i < monthsData.length; ++i) { histData[i]['x0'] = monthsData[i]['x0']; histData[i]['x1'] = monthsData[i]['x1']; histData[i][entry.key] = monthsData[i]; } }); // create function to stack visits statuses by month let stacked = d3.stack() .keys(statuses) .value((d, key) => d[key].length); // compute the maximum amount of visits by month let yMax = d3.max(histData, d => { let total = 0; for (let i = 0; i < statuses.length; ++i) { total += d[statuses[i]].length; } return total; }); // set y scale domain y.domain([0, yMax]); // compute ticks values for the y axis let step = 5; let yTickValues = []; for (let i = 0; i <= yMax / step; ++i) { yTickValues.push(i * step); } if (yTickValues.length === 0) { for (let i = 0; i <= yMax; ++i) { yTickValues.push(i); } } else if (yMax % step !== 0) { yTickValues.push(yMax); } // add histogram background grid g.append('g') .attr('class', 'grid') .call(d3.axisLeft(y) .tickValues(yTickValues) .tickSize(-width) .tickFormat('')); // create one fill only rectangle by displayed year // each rectangle will be made visible when hovering the mouse over a year range // user will then be able to select a year by clicking in the rectangle g.append('g') .selectAll('rect') .data(yearBins) .enter().append('rect') .attr('class', d => 'year' + d.getUTCFullYear()) .attr('fill', 'red') .attr('fill-opacity', d => d.getUTCFullYear() === currentYear ? 0.3 : 0) .attr('stroke', 'none') .attr('x', d => { let date = new Date(Date.UTC(d.getUTCFullYear(), 0, 1)); return x(date); }) .attr('y', 0) .attr('height', height) .attr('width', d => { let date = new Date(Date.UTC(d.getUTCFullYear(), 0, 1)); let yearWidth = x(d3.timeYear.offset(date, 1)) - x(date); return yearWidth; }) // mouse event callbacks used to show rectangle years // when hovering the mouse over the histograms .on('mouseover', d => { svg.selectAll('rect.year' + d.getUTCFullYear()) .attr('fill-opacity', 0.5); }) .on('mouseout', d => { svg.selectAll('rect.year' + d.getUTCFullYear()) .attr('fill-opacity', 0); svg.selectAll('rect.year' + currentYear) .attr('fill-opacity', 0.3); }) // callback to select a year after a mouse click // in a rectangle year .on('click', d => { svg.selectAll('rect.year' + currentYear) .attr('fill-opacity', 0); svg.selectAll('rect.yearoutline' + currentYear) .attr('stroke', 'none'); currentYear = d.getUTCFullYear(); svg.selectAll('rect.year' + currentYear) .attr('fill-opacity', 0.5); svg.selectAll('rect.yearoutline' + currentYear) .attr('stroke', 'black'); yearClickCallback(currentYear); }); // create the stacked histogram of visits g.append('g') .selectAll('g') .data(stacked(histData)) .enter().append('g') .attr('fill', d => colors(d.key)) .selectAll('rect') .data(d => d) .enter().append('rect') .attr('class', d => 'month' + d.data.x1.getMonth()) .attr('x', d => x(d.data.x0)) .attr('y', d => y(d[1])) .attr('height', d => y(d[0]) - y(d[1])) .attr('width', d => x(d.data.x1) - x(d.data.x0) - 1) // mouse event callbacks used to show rectangle years // but also to show tooltip when hovering the mouse // over the histogram bars .on('mouseover', d => { svg.selectAll('rect.year' + d.data.x1.getUTCFullYear()) .attr('fill-opacity', 0.5); tooltip.transition() .duration(200) .style('opacity', 1); let ds = d.data.x1.toISOString().substr(0, 7).split('-'); let tooltipText = '' + ds[1] + ' / ' + ds[0] + ':
'; for (let i = 0; i < statuses.length; ++i) { let visitStatus = statuses[i]; let nbVisits = d.data[visitStatus].length; if (nbVisits === 0) continue; tooltipText += nbVisits + ' ' + visitStatus + ' visits'; if (i !== statuses.length - 1) tooltipText += '
'; } tooltip.html(tooltipText) .style('left', d3.event.pageX + 15 + 'px') .style('top', d3.event.pageY + 'px'); }) .on('mouseout', d => { svg.selectAll('rect.year' + d.data.x1.getUTCFullYear()) .attr('fill-opacity', 0); svg.selectAll('rect.year' + currentYear) .attr('fill-opacity', 0.3); tooltip.transition() .duration(500) .style('opacity', 0); }) .on('mousemove', () => { tooltip.style('left', d3.event.pageX + 15 + 'px') .style('top', d3.event.pageY + 'px'); }) // callback to select a year after a mouse click // inside a histogram bar .on('click', d => { svg.selectAll('rect.year' + currentYear) .attr('fill-opacity', 0); svg.selectAll('rect.yearoutline' + currentYear) .attr('stroke', 'none'); currentYear = d.data.x1.getUTCFullYear(); svg.selectAll('rect.year' + currentYear) .attr('fill-opacity', 0.5); svg.selectAll('rect.yearoutline' + currentYear) .attr('stroke', 'black'); yearClickCallback(currentYear); }); // create one stroke only rectangle by displayed year // that will be displayed on top of the histogram when the user has selected a year g.append('g') .selectAll('rect') .data(yearBins) .enter().append('rect') .attr('class', d => 'yearoutline' + d.getUTCFullYear()) .attr('fill', 'none') .attr('stroke', d => d.getUTCFullYear() === currentYear ? 'black' : 'none') .attr('x', d => { let date = new Date(Date.UTC(d.getUTCFullYear(), 0, 1)); return x(date); }) .attr('y', 0) .attr('height', height) .attr('width', d => { let date = new Date(Date.UTC(d.getUTCFullYear(), 0, 1)); let yearWidth = x(d3.timeYear.offset(date, 1)) - x(date); return yearWidth; }); // add x axis with a tick for every 1st day of each year let xAxis = g.append('g') .attr('class', 'axis') .attr('transform', 'translate(0,' + height + ')') .call( d3.axisBottom(x) .ticks(d3.timeYear.every(1)) .tickFormat(d => d.getUTCFullYear()) ); // shift tick labels in order to display them at the middle // of each year range xAxis.selectAll('text') .attr('transform', d => { let year = d.getUTCFullYear(); let date = new Date(Date.UTC(year, 0, 1)); let yearWidth = x(d3.timeYear.offset(date, 1)) - x(date); return 'translate(' + -yearWidth / 2 + ', 0)'; }); // add y axis for the number of visits g.append('g') .attr('class', 'axis') .call(d3.axisLeft(y).tickValues(yTickValues)); // add legend for visit statuses let legendGroup = g.append('g') .attr('font-family', 'sans-serif') .attr('font-size', 10) .attr('text-anchor', 'end'); legendGroup.append('text') .attr('x', width + margin.right - 5) .attr('y', 9.5) .attr('dy', '0.32em') .text('visit status:'); let legend = legendGroup.selectAll('g') .data(statuses.slice().reverse()) .enter().append('g') .attr('transform', (d, i) => 'translate(0,' + (i + 1) * 20 + ')'); legend.append('rect') .attr('x', width + 2 * margin.right / 3) .attr('width', 19) .attr('height', 19) .attr('fill', colors); legend.append('text') .attr('x', width + 2 * margin.right / 3 - 5) .attr('y', 9.5) .attr('dy', '0.32em') .text(d => d); // add text label for the y axis g.append('text') .attr('transform', 'rotate(-90)') .attr('y', -margin.left) .attr('x', -(height / 2)) .attr('dy', '1em') .style('text-anchor', 'middle') .text('Number of visits'); } diff --git a/swh/web/browse/browseurls.py b/swh/web/browse/browseurls.py index c5968158b..d1d4547be 100644 --- a/swh/web/browse/browseurls.py +++ b/swh/web/browse/browseurls.py @@ -1,38 +1,38 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from swh.web.common.urlsindex import UrlsIndex class BrowseUrls(UrlsIndex): """ - Class to manage SWH web browse application urls. + Class to manage swh-web browse application urls. """ scope = 'browse' class browse_route(object): # noqa: N801 """ - Decorator to ease the registration of a SWH web browse endpoint + Decorator to ease the registration of a swh-web browse endpoint Args: url_patterns: list of url patterns used by Django to identify the browse routes view_name: the name of the Django view associated to the routes used to reverse the url """ # noqa def __init__(self, *url_patterns, view_name=None): super().__init__() self.url_patterns = [] for url_pattern in url_patterns: self.url_patterns.append('^' + url_pattern + '$') self.view_name = view_name def __call__(self, f): # register the route and its view in the browse endpoints index for url_pattern in self.url_patterns: BrowseUrls.add_url_pattern(url_pattern, f, self.view_name) return f diff --git a/swh/web/browse/identifiers.py b/swh/web/browse/identifiers.py index 22fd98b12..c7d822851 100644 --- a/swh/web/browse/identifiers.py +++ b/swh/web/browse/identifiers.py @@ -1,25 +1,25 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from django.shortcuts import redirect from swh.web.common.utils import resolve_swh_persistent_id from swh.web.common.exc import handle_view_exception def swh_id_browse(request, swh_id): """ - Django view enabling to browse the SWH archive using + Django view enabling to browse the archive using :ref:`persistent-identifiers`. The url that points to it is :http:get:`/(swh_id)/`. """ try: swh_id_resolved = resolve_swh_persistent_id( swh_id, query_params=request.GET) except Exception as exc: return handle_view_exception(request, exc) return redirect(swh_id_resolved['browse_url']) diff --git a/swh/web/browse/utils.py b/swh/web/browse/utils.py index ef0a1ee92..604d4ea9f 100644 --- a/swh/web/browse/utils.py +++ b/swh/web/browse/utils.py @@ -1,1173 +1,1173 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import base64 from collections import defaultdict import magic import math import pypandoc import stat import textwrap from django.core.cache import cache from django.utils.safestring import mark_safe from importlib import reload from swh.model.identifiers import persistent_identifier from swh.web.common import highlightjs, service from swh.web.common.exc import NotFoundExc, http_status_code_message from swh.web.common.utils import ( reverse, format_utc_iso_date, parse_timestamp, get_origin_visits, get_swh_persistent_id, swh_object_icons ) from swh.web.config import get_config def get_directory_entries(sha1_git): - """Function that retrieves the content of a SWH directory - from the SWH archive. + """Function that retrieves the content of a directory + from the archive. The directories entries are first sorted in lexicographical order. Sub-directories and regular files are then extracted. Args: sha1_git: sha1_git identifier of the directory Returns: A tuple whose first member corresponds to the sub-directories list and second member the regular files list Raises: NotFoundExc if the directory is not found """ cache_entry_id = 'directory_entries_%s' % sha1_git cache_entry = cache.get(cache_entry_id) if cache_entry: return cache_entry entries = list(service.lookup_directory(sha1_git)) for e in entries: e['perms'] = stat.filemode(e['perms']) if e['type'] == 'rev': # modify dir entry name to explicitly show it points # to a revision e['name'] = '%s @ %s' % (e['name'], e['target'][:7]) dirs = [e for e in entries if e['type'] in ('dir', 'rev')] files = [e for e in entries if e['type'] == 'file'] dirs = sorted(dirs, key=lambda d: d['name']) files = sorted(files, key=lambda f: f['name']) cache.set(cache_entry_id, (dirs, files)) return dirs, files def get_mimetype_and_encoding_for_content(content): """Function that returns the mime type and the encoding associated to a content buffer using the magic module under the hood. Args: content (bytes): a content buffer Returns: A tuple (mimetype, encoding), for instance ('text/plain', 'us-ascii'), associated to the provided content. """ while True: try: magic_result = magic.detect_from_content(content) mime_type = magic_result.mime_type encoding = magic_result.encoding break except Exception: # workaround an issue with the magic module who can fail # if detect_from_content is called multiple times in # a short amount of time reload(magic) return mime_type, encoding # maximum authorized content size in bytes for HTML display # with code highlighting content_display_max_size = get_config()['content_display_max_size'] snapshot_content_max_size = get_config()['snapshot_content_max_size'] def request_content(query_string, max_size=content_display_max_size, raise_if_unavailable=True, reencode=True): - """Function that retrieves a SWH content from the SWH archive. + """Function that retrieves a content from the archive. Raw bytes content is first retrieved, then the content mime type. If the mime type is not stored in the archive, it will be computed using Python magic module. Args: query_string: a string of the form "[ALGO_HASH:]HASH" where - optional ALGO_HASH can be either *sha1*, *sha1_git*, *sha256*, - or *blake2s256* (default to *sha1*) and HASH the hexadecimal - representation of the hash value + optional ALGO_HASH can be either ``sha1``, ``sha1_git``, + ``sha256``, or ``blake2s256`` (default to ``sha1``) and HASH + the hexadecimal representation of the hash value max_size: the maximum size for a content to retrieve (default to 1MB, no size limit if None) Returns: A tuple whose first member corresponds to the content raw bytes and second member the content mime type Raises: NotFoundExc if the content is not found """ content_data = service.lookup_content(query_string) filetype = None language = None license = None # requests to the indexer db may fail so properly handle # those cases in order to avoid content display errors try: filetype = service.lookup_content_filetype(query_string) language = service.lookup_content_language(query_string) license = service.lookup_content_license(query_string) except Exception: pass mimetype = 'unknown' encoding = 'unknown' if filetype: mimetype = filetype['mimetype'] encoding = filetype['encoding'] content_data['error_code'] = 200 content_data['error_message'] = '' content_data['error_description'] = '' if not max_size or content_data['length'] < max_size: try: content_raw = service.lookup_content_raw(query_string) except Exception as e: if raise_if_unavailable: raise e else: content_data['raw_data'] = None content_data['error_code'] = 404 content_data['error_description'] = \ 'The bytes of the content are currently not available in the archive.' # noqa content_data['error_message'] = \ http_status_code_message[content_data['error_code']] else: content_data['raw_data'] = content_raw['data'] if not filetype: mimetype, encoding = \ get_mimetype_and_encoding_for_content(content_data['raw_data']) # noqa # encode textual content to utf-8 if needed if reencode and mimetype.startswith('text/'): # probably a malformed UTF-8 content, re-encode it # by replacing invalid chars with a substitution one if encoding == 'unknown-8bit': content_data['raw_data'] = \ content_data['raw_data'].decode('utf-8', 'replace')\ .encode('utf-8') elif 'ascii' not in encoding and encoding not in ['utf-8', 'binary']: # noqa content_data['raw_data'] = \ content_data['raw_data'].decode(encoding, 'replace')\ .encode('utf-8') elif reencode and mimetype.startswith('application/octet-stream'): # file may detect a text content as binary # so try to decode it for display encodings = ['us-ascii'] encodings += ['iso-8859-%s' % i for i in range(1, 17)] for encoding in encodings: try: content_data['raw_data'] = \ content_data['raw_data'].decode(encoding)\ .encode('utf-8') except Exception: pass else: # ensure display in content view mimetype = 'text/plain' break else: content_data['raw_data'] = None content_data['mimetype'] = mimetype content_data['encoding'] = encoding if language: content_data['language'] = language['lang'] else: content_data['language'] = 'not detected' if license: content_data['licenses'] = ', '.join(license['facts'][0]['licenses']) else: content_data['licenses'] = 'not detected' return content_data _browsers_supported_image_mimes = set(['image/gif', 'image/png', 'image/jpeg', 'image/bmp', 'image/webp', 'image/svg', 'image/svg+xml']) def prepare_content_for_display(content_data, mime_type, path): """Function that prepares a content for HTML display. The function tries to associate a programming language to a content in order to perform syntax highlighting client-side using highlightjs. The language is determined using either the content filename or its mime type. If the mime type corresponds to an image format supported by web browsers, the content will be encoded in base64 for displaying the image. Args: content_data (bytes): raw bytes of the content mime_type (string): mime type of the content path (string): path of the content including filename Returns: A dict containing the content bytes (possibly different from the one provided as parameter if it is an image) under the key 'content_data and the corresponding highlightjs language class under the key 'language'. """ language = highlightjs.get_hljs_language_from_filename(path) if not language: language = highlightjs.get_hljs_language_from_mime_type(mime_type) if not language: language = 'nohighlight' elif mime_type.startswith('application/'): mime_type = mime_type.replace('application/', 'text/') if mime_type.startswith('image/'): if mime_type in _browsers_supported_image_mimes: content_data = base64.b64encode(content_data) else: content_data = None if mime_type.startswith('image/svg'): mime_type = 'image/svg+xml' return {'content_data': content_data, 'language': language, 'mimetype': mime_type} def get_origin_visit(origin_info, visit_ts=None, visit_id=None, snapshot_id=None): - """Function that returns information about a SWH visit for + """Function that returns information about a visit for a given origin. The visit is retrieved from a provided timestamp. The closest visit from that timestamp is selected. Args: origin_info (dict): a dict filled with origin information (id, url, type) visit_ts (int or str): an ISO date string or Unix timestamp to parse Returns: A dict containing the visit info as described below:: {'origin': 2, 'date': '2017-10-08T11:54:25.582463+00:00', 'metadata': {}, 'visit': 25, 'status': 'full'} """ visits = get_origin_visits(origin_info) if not visits: - raise NotFoundExc('No SWH visit associated to origin with' + raise NotFoundExc('No visit associated to origin with' ' type %s and url %s!' % (origin_info['type'], origin_info['url'])) if snapshot_id: visit = [v for v in visits if v['snapshot'] == snapshot_id] if len(visit) == 0: raise NotFoundExc( 'Visit for snapshot with id %s for origin with type %s' ' and url %s not found!' % (snapshot_id, origin_info['type'], origin_info['url'])) return visit[0] if visit_id: visit = [v for v in visits if v['visit'] == int(visit_id)] if len(visit) == 0: raise NotFoundExc( 'Visit with id %s for origin with type %s' ' and url %s not found!' % (visit_id, origin_info['type'], origin_info['url'])) return visit[0] if not visit_ts: # returns the latest full visit when no timestamp is provided for v in reversed(visits): if v['status'] == 'full': return v return visits[-1] parsed_visit_ts = math.floor(parse_timestamp(visit_ts).timestamp()) visit_idx = None for i, visit in enumerate(visits): ts = math.floor(parse_timestamp(visit['date']).timestamp()) if i == 0 and parsed_visit_ts <= ts: return visit elif i == len(visits) - 1: if parsed_visit_ts >= ts: return visit else: next_ts = math.floor( parse_timestamp(visits[i+1]['date']).timestamp()) if parsed_visit_ts >= ts and parsed_visit_ts < next_ts: if (parsed_visit_ts - ts) < (next_ts - parsed_visit_ts): visit_idx = i break else: visit_idx = i+1 break if visit_idx is not None: visit = visits[visit_idx] while visit_idx < len(visits) - 1 and \ visit['date'] == visits[visit_idx+1]['date']: visit_idx = visit_idx + 1 visit = visits[visit_idx] return visit else: raise NotFoundExc( 'Visit with timestamp %s for origin with type %s and url %s not found!' % # noqa (visit_ts, origin_info['type'], origin_info['url'])) def process_snapshot_branches(snapshot_branches): """ Process a dictionary describing snapshot branches: extract those targeting revisions and releases, put them in two different lists, then sort those lists in lexicographical order of the branches' names. Args: snapshot_branches (dict): A dict describing the branches of a snapshot as returned for instance by :func:`swh.web.common.service.lookup_snapshot` Returns: tuple: A tuple whose first member is the sorted list of branches targeting revisions and second member the sorted list of branches targeting releases """ # noqa branches = {} releases = {} revision_to_branch = defaultdict(set) revision_to_release = defaultdict(set) release_to_branch = defaultdict(set) for branch_name, target in snapshot_branches.items(): if not target: # FIXME: display branches with an unknown target anyway continue target_id = target['target'] target_type = target['target_type'] if target_type == 'revision': branches[branch_name] = { 'name': branch_name, 'revision': target_id, } revision_to_branch[target_id].add(branch_name) elif target_type == 'release': release_to_branch[target_id].add(branch_name) # FIXME: handle pointers to other object types # FIXME: handle branch aliases releases_info = service.lookup_release_multiple( release_to_branch.keys() ) for release in releases_info: branches_to_update = release_to_branch[release['id']] for branch in branches_to_update: releases[branch] = { 'name': release['name'], 'branch_name': branch, 'date': format_utc_iso_date(release['date']), 'id': release['id'], 'message': release['message'], 'target_type': release['target_type'], 'target': release['target'], } if release['target_type'] == 'revision': revision_to_release[release['target']].update( branches_to_update ) revisions = service.lookup_revision_multiple( set(revision_to_branch.keys()) | set(revision_to_release.keys()) ) for revision in revisions: if not revision: continue revision_data = { 'directory': revision['directory'], 'date': format_utc_iso_date(revision['date']), 'message': revision['message'], } for branch in revision_to_branch[revision['id']]: branches[branch].update(revision_data) for release in revision_to_release[revision['id']]: releases[release]['directory'] = revision['directory'] ret_branches = list(sorted(branches.values(), key=lambda b: b['name'])) ret_releases = list(sorted(releases.values(), key=lambda b: b['name'])) return ret_branches, ret_releases def get_snapshot_content(snapshot_id): """Returns the lists of branches and releases associated to a swh snapshot. That list is put in cache in order to speedup the navigation in the swh-web/browse ui. .. warning:: At most 1000 branches contained in the snapshot will be returned for performance reasons. Args: snapshot_id (str): hexadecimal representation of the snapshot identifier Returns: A tuple with two members. The first one is a list of dict describing the snapshot branches. The second one is a list of dict describing the snapshot releases. Raises: NotFoundExc if the snapshot does not exist """ cache_entry_id = 'swh_snapshot_%s' % snapshot_id cache_entry = cache.get(cache_entry_id) if cache_entry: return cache_entry['branches'], cache_entry['releases'] branches = [] releases = [] if snapshot_id: snapshot = service.lookup_snapshot( snapshot_id, branches_count=snapshot_content_max_size) branches, releases = process_snapshot_branches(snapshot['branches']) cache.set(cache_entry_id, { 'branches': branches, 'releases': releases, }) return branches, releases def get_origin_visit_snapshot(origin_info, visit_ts=None, visit_id=None, snapshot_id=None): """Returns the lists of branches and releases associated to a swh origin for a given visit. The visit is expressed by a timestamp. In the latter case, the closest visit from the provided timestamp will be used. If no visit parameter is provided, it returns the list of branches found for the latest visit. That list is put in cache in order to speedup the navigation in the swh-web/browse ui. .. warning:: At most 1000 branches contained in the snapshot will be returned for performance reasons. Args: origin_info (dict): a dict filled with origin information (id, url, type) visit_ts (int or str): an ISO date string or Unix timestamp to parse visit_id (int): optional visit id for disambiguation in case several visits have the same timestamp Returns: A tuple with two members. The first one is a list of dict describing the origin branches for the given visit. The second one is a list of dict describing the origin releases for the given visit. Raises: NotFoundExc if the origin or its visit are not found """ visit_info = get_origin_visit(origin_info, visit_ts, visit_id, snapshot_id) return get_snapshot_content(visit_info['snapshot']) def gen_link(url, link_text=None, link_attrs={}): """ Utility function for generating an HTML link to insert in Django templates. Args: url (str): an url link_text (str): optional text for the produced link, if not provided the url will be used link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ attrs = ' ' for k, v in link_attrs.items(): attrs += '%s="%s" ' % (k, v) if not link_text: link_text = url link = '%s' % (attrs, url, link_text) return mark_safe(link) def gen_person_link(person_id, person_name, snapshot_context=None, link_attrs={}): """ - Utility function for generating a link to a SWH person HTML view + Utility function for generating a link to a person HTML view to insert in Django templates. Args: - person_id (int): a SWH person id + person_id (int): a person id person_name (str): the associated person name link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'person_name' """ query_params = None if snapshot_context and snapshot_context['origin_info']: origin_info = snapshot_context['origin_info'] query_params = {'origin_type': origin_info['type'], 'origin': origin_info['url']} if 'timestamp' in snapshot_context['url_args']: query_params['timestamp'] = \ snapshot_context['url_args']['timestamp'] if 'visit_id' in snapshot_context['query_params']: query_params['visit_id'] = \ snapshot_context['query_params']['visit_id'] elif snapshot_context: query_params = {'snapshot_id': snapshot_context['snapshot_id']} person_url = reverse('browse-person', url_args={'person_id': person_id}, query_params=query_params) return gen_link(person_url, person_name or 'None', link_attrs) def gen_revision_url(revision_id, snapshot_context=None): """ - Utility function for generating an url to a SWH revision. + Utility function for generating an url to a revision. Args: - revision_id (str): a SWH revision id + revision_id (str): a revision id snapshot_context (dict): if provided, generate snapshot-dependent browsing url Returns: str: The url to browse the revision """ query_params = None if snapshot_context and snapshot_context['origin_info']: origin_info = snapshot_context['origin_info'] origin_type = snapshot_context['origin_type'] query_params = {'origin_type': origin_type, 'origin': origin_info['url']} if 'timestamp' in snapshot_context['url_args']: query_params['timestamp'] = \ snapshot_context['url_args']['timestamp'] if 'visit_id' in snapshot_context['query_params']: query_params['visit_id'] = \ snapshot_context['query_params']['visit_id'] elif snapshot_context: query_params = {'snapshot_id': snapshot_context['snapshot_id']} return reverse('browse-revision', url_args={'sha1_git': revision_id}, query_params=query_params) def gen_revision_link(revision_id, shorten_id=False, snapshot_context=None, link_text=None, link_attrs={}): """ - Utility function for generating a link to a SWH revision HTML view + Utility function for generating a link to a revision HTML view to insert in Django templates. Args: - revision_id (str): a SWH revision id + revision_id (str): a revision id shorten_id (boolean): whether to shorten the revision id to 7 characters for the link text snapshot_context (dict): if provided, generate snapshot-dependent browsing link link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: str: An HTML link in the form 'revision_id' """ if not revision_id: return None revision_url = gen_revision_url(revision_id, snapshot_context) if shorten_id: return gen_link(revision_url, revision_id[:7], link_attrs) else: if not link_text: link_text = revision_id return gen_link(revision_url, link_text, link_attrs) def gen_origin_link(origin_info, link_attrs={}): """ - Utility function for generating a link to a SWH origin HTML view + Utility function for generating a link to a software origin HTML view to insert in Django templates. Args: origin_info (dict): a dict filled with origin information (id, type, url) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'Origin: origin_url' """ # noqa origin_browse_url = reverse('browse-origin', url_args={'origin_type': origin_info['type'], 'origin_url': origin_info['url']}) return gen_link(origin_browse_url, 'Origin: ' + origin_info['url'], link_attrs) def gen_directory_link(sha1_git, link_text=None, link_attrs={}): """ - Utility function for generating a link to a SWH directory HTML view + Utility function for generating a link to a directory HTML view to insert in Django templates. Args: sha1_git (str): directory identifier link_text (str): optional text for the generated link (the generated url will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ if not sha1_git: return None directory_url = reverse('browse-directory', url_args={'sha1_git': sha1_git}) if not link_text: link_text = directory_url return gen_link(directory_url, link_text, link_attrs) def gen_snapshot_link(snapshot_id, link_text=None, link_attrs={}): """ - Utility function for generating a link to a SWH snapshot HTML view + Utility function for generating a link to a snapshot HTML view to insert in Django templates. Args: snapshot_id (str): snapshot identifier link_text (str): optional text for the generated link (the generated url will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ snapshot_url = reverse('browse-snapshot', url_args={'snapshot_id': snapshot_id}) if not link_text: link_text = snapshot_url return gen_link(snapshot_url, link_text, link_attrs) def gen_snapshot_directory_link(snapshot_context, revision_id=None, link_text=None, link_attrs={}): """ - Utility function for generating a link to a SWH directory HTML view + Utility function for generating a link to a directory HTML view in the context of a snapshot to insert in Django templates. Args: snapshot_context (dict): the snapshot information revision_id (str): optional revision identifier in order to use the associated directory link_text (str): optional text to use for the generated link link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'origin_directory_view_url' """ query_params = {'revision': revision_id} if snapshot_context['origin_info']: origin_info = snapshot_context['origin_info'] url_args = {'origin_url': origin_info['url']} if 'timestamp' in snapshot_context['url_args']: url_args['timestamp'] = \ snapshot_context['url_args']['timestamp'] if 'visit_id' in snapshot_context['query_params']: query_params['visit_id'] = \ snapshot_context['query_params']['visit_id'] directory_url = reverse('browse-origin-directory', url_args=url_args, query_params=query_params) else: url_args = {'snapshot_id': snapshot_context['snapshot_id']} directory_url = reverse('browse-snapshot-directory', url_args=url_args, query_params=query_params) if not link_text: link_text = directory_url return gen_link(directory_url, link_text, link_attrs) def gen_content_link(sha1_git, link_text=None, link_attrs={}): """ - Utility function for generating a link to a SWH content HTML view + Utility function for generating a link to a content HTML view to insert in Django templates. Args: sha1_git (str): content identifier link_text (str): optional text for the generated link (the generated url will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ if not sha1_git: return None content_url = reverse('browse-content', url_args={'query_string': 'sha1_git:' + sha1_git}) if not link_text: link_text = content_url return gen_link(content_url, link_text, link_attrs) def get_revision_log_url(revision_id, snapshot_context=None): """ - Utility function for getting the URL for a SWH revision log HTML view + Utility function for getting the URL for a revision log HTML view (possibly in the context of an origin). Args: revision_id (str): revision identifier the history heads to snapshot_context (dict): if provided, generate snapshot-dependent browsing link Returns: - The SWH revision log view URL + The revision log view URL """ query_params = {'revision': revision_id} if snapshot_context and snapshot_context['origin_info']: origin_info = snapshot_context['origin_info'] url_args = {'origin_url': origin_info['url']} if 'timestamp' in snapshot_context['url_args']: url_args['timestamp'] = \ snapshot_context['url_args']['timestamp'] if 'visit_id' in snapshot_context['query_params']: query_params['visit_id'] = \ snapshot_context['query_params']['visit_id'] revision_log_url = reverse('browse-origin-log', url_args=url_args, query_params=query_params) elif snapshot_context: url_args = {'snapshot_id': snapshot_context['snapshot_id']} revision_log_url = reverse('browse-snapshot-log', url_args=url_args, query_params=query_params) else: revision_log_url = reverse('browse-revision-log', url_args={'sha1_git': revision_id}) return revision_log_url def gen_revision_log_link(revision_id, snapshot_context=None, link_text=None, link_attrs={}): """ - Utility function for generating a link to a SWH revision log HTML view + Utility function for generating a link to a revision log HTML view (possibly in the context of an origin) to insert in Django templates. Args: revision_id (str): revision identifier the history heads to snapshot_context (dict): if provided, generate snapshot-dependent browsing link link_text (str): optional text to use for the generated link link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ if not revision_id: return None revision_log_url = get_revision_log_url(revision_id, snapshot_context) if not link_text: link_text = revision_log_url return gen_link(revision_log_url, link_text, link_attrs) def format_log_entries(revision_log, per_page, snapshot_context=None): """ Utility functions that process raw revision log data for HTML display. Its purpose is to: - * add links to relevant SWH browse views + * add links to relevant browse views * format date in human readable format * truncate the message log Args: - revision_log (list): raw revision log as returned by the SWH web api + revision_log (list): raw revision log as returned by the swh-web api per_page (int): number of log entries per page snapshot_context (dict): if provided, generate snapshot-dependent browsing link """ revision_log_data = [] for i, rev in enumerate(revision_log): if i == per_page: break author_name = 'None' author_fullname = 'None' committer_fullname = 'None' if rev['author']: author_name = rev['author']['name'] or rev['author']['fullname'] author_fullname = rev['author']['fullname'] if rev['committer']: committer_fullname = rev['committer']['fullname'] author_date = format_utc_iso_date(rev['date']) committer_date = format_utc_iso_date(rev['committer_date']) tooltip = 'revision %s\n' % rev['id'] tooltip += 'author: %s\n' % author_fullname tooltip += 'author date: %s\n' % author_date tooltip += 'committer: %s\n' % committer_fullname tooltip += 'committer date: %s\n\n' % committer_date tooltip += textwrap.indent(rev['message'], ' '*4) revision_log_data.append({ 'author': author_name, 'id': rev['id'][:7], 'message': rev['message'], 'date': author_date, 'commit_date': committer_date, 'url': gen_revision_url(rev['id'], snapshot_context), 'tooltip': tooltip }) return revision_log_data # list of origin types that can be found in the swh archive # TODO: retrieve it dynamically in an efficient way instead # of hardcoding it _swh_origin_types = ['git', 'svn', 'deb', 'hg', 'ftp', 'deposit', 'pypi'] def get_origin_info(origin_url, origin_type=None): """ - Get info about a SWH origin. + Get info about a software origin. Its main purpose is to automatically find an origin type when it is not provided as parameter. Args: - origin_url (str): complete url of a SWH origin + origin_url (str): complete url of a software origin origin_type (str): optional origin type Returns: A dict with the following entries: * type: the origin type * url: the origin url - * id: the SWH internal id of the origin + * id: the internal id of the origin """ if origin_type: return service.lookup_origin({'type': origin_type, 'url': origin_url}) else: for origin_type in _swh_origin_types: try: origin_info = service.lookup_origin({'type': origin_type, 'url': origin_url}) return origin_info except Exception: pass raise NotFoundExc('Origin with url %s not found!' % origin_url) def get_snapshot_context(snapshot_id=None, origin_type=None, origin_url=None, timestamp=None, visit_id=None): """ Utility function to compute relevant information when navigating - the SWH archive in a snapshot context. The snapshot is either + the archive in a snapshot context. The snapshot is either referenced by its id or it will be retrieved from an origin visit. Args: snapshot_id (str): hexadecimal representation of a snapshot identifier, all other parameters will be ignored if it is provided origin_type (str): the origin type (git, svn, deposit, ...) origin_url (str): the origin_url (e.g. https://github.com/(user)/(repo)/) timestamp (str): a datetime string for retrieving the closest - SWH visit of the origin + visit of the origin visit_id (int): optional visit id for disambiguation in case of several visits with the same timestamp Returns: A dict with the following entries: * origin_info: dict containing origin information - * visit_info: dict containing SWH visit information + * visit_info: dict containing visit information * branches: the list of branches for the origin found during the visit * releases: the list of releases for the origin found during the visit * origin_browse_url: the url to browse the origin * origin_branches_url: the url to browse the origin branches * origin_releases_url': the url to browse the origin releases * origin_visit_url: the url to browse the snapshot of the origin found during the visit * url_args: dict containing url arguments to use when browsing in the context of the origin and its visit Raises: NotFoundExc: if no snapshot is found for the visit of an origin. """ # noqa origin_info = None visit_info = None url_args = None query_params = {} branches = [] releases = [] browse_url = None visit_url = None branches_url = None releases_url = None swh_type = 'snapshot' if origin_url: swh_type = 'origin' origin_info = get_origin_info(origin_url, origin_type) visit_info = get_origin_visit(origin_info, timestamp, visit_id, snapshot_id) fmt_date = format_utc_iso_date(visit_info['date']) visit_info['fmt_date'] = fmt_date snapshot_id = visit_info['snapshot'] if not snapshot_id: raise NotFoundExc('No snapshot associated to the visit of origin ' '%s on %s' % (origin_url, fmt_date)) # provided timestamp is not necessarily equals to the one # of the retrieved visit, so get the exact one in order # use it in the urls generated below if timestamp: timestamp = visit_info['date'] branches, releases = \ get_origin_visit_snapshot(origin_info, timestamp, visit_id, snapshot_id) url_args = {'origin_type': origin_type, 'origin_url': origin_info['url']} query_params = {'visit_id': visit_id} browse_url = reverse('browse-origin-visits', url_args=url_args) if timestamp: url_args['timestamp'] = format_utc_iso_date(timestamp, '%Y-%m-%dT%H:%M:%S') visit_url = reverse('browse-origin-directory', url_args=url_args, query_params=query_params) visit_info['url'] = visit_url branches_url = reverse('browse-origin-branches', url_args=url_args, query_params=query_params) releases_url = reverse('browse-origin-releases', url_args=url_args, query_params=query_params) elif snapshot_id: branches, releases = get_snapshot_content(snapshot_id) url_args = {'snapshot_id': snapshot_id} browse_url = reverse('browse-snapshot', url_args=url_args) branches_url = reverse('browse-snapshot-branches', url_args=url_args) releases_url = reverse('browse-snapshot-releases', url_args=url_args) releases = list(reversed(releases)) snapshot_size = service.lookup_snapshot_size(snapshot_id) is_empty = sum(snapshot_size.values()) == 0 swh_snp_id = persistent_identifier('snapshot', snapshot_id) return { 'swh_type': swh_type, 'swh_object_id': swh_snp_id, 'snapshot_id': snapshot_id, 'snapshot_size': snapshot_size, 'is_empty': is_empty, 'origin_info': origin_info, # keep track if the origin type was provided as url argument 'origin_type': origin_type, 'visit_info': visit_info, 'branches': branches, 'releases': releases, 'branch': None, 'release': None, 'browse_url': browse_url, 'branches_url': branches_url, 'releases_url': releases_url, 'url_args': url_args, 'query_params': query_params } # list of common readme names ordered by preference # (lower indices have higher priority) _common_readme_names = [ "readme.markdown", "readme.md", "readme.rst", "readme.txt", "readme" ] def get_readme_to_display(readmes): """ Process a list of readme files found in a directory in order to find the adequate one to display. Args: readmes: a list of dict where keys are readme file names and values are readme sha1s Returns: A tuple (readme_name, readme_sha1) """ readme_name = None readme_url = None readme_sha1 = None readme_html = None lc_readmes = {k.lower(): {'orig_name': k, 'sha1': v} for k, v in readmes.items()} # look for readme names according to the preference order # defined by the _common_readme_names list for common_readme_name in _common_readme_names: if common_readme_name in lc_readmes: readme_name = lc_readmes[common_readme_name]['orig_name'] readme_sha1 = lc_readmes[common_readme_name]['sha1'] readme_url = reverse('browse-content-raw', url_args={'query_string': readme_sha1}) break # otherwise pick the first readme like file if any if not readme_name and len(readmes.items()) > 0: readme_name = next(iter(readmes)) readme_sha1 = readmes[readme_name] readme_url = reverse('browse-content-raw', url_args={'query_string': readme_sha1}) # convert rst README to html server side as there is # no viable solution to perform that task client side if readme_name and readme_name.endswith('.rst'): cache_entry_id = 'readme_%s' % readme_sha1 cache_entry = cache.get(cache_entry_id) if cache_entry: readme_html = cache_entry else: try: rst_doc = request_content(readme_sha1) readme_html = pypandoc.convert_text(rst_doc['raw_data'], 'html', format='rst') cache.set(cache_entry_id, readme_html) except Exception: readme_html = 'Readme bytes are not available' return readme_name, readme_url, readme_html def get_swh_persistent_ids(swh_objects, snapshot_context=None): """ Returns a list of dict containing info related to persistent identifiers of swh objects. Args: swh_objects (list): a list of dict with the following keys: * type: swh object type (content/directory/release/revision/snapshot) * id: swh object id snapshot_context (dict): optional parameter describing the snapshot in which the object has been found Returns: list: a list of dict with the following keys: * object_type: the swh object type (content/directory/release/revision/snapshot) * object_icon: the swh object icon to use in HTML views * swh_id: the computed swh object persistent identifier * swh_id_url: the url resolving the persistent identifier * show_options: boolean indicating if the persistent id options must be displayed in persistent ids HTML view """ # noqa swh_ids = [] for swh_object in swh_objects: if not swh_object['id']: continue swh_id = get_swh_persistent_id(swh_object['type'], swh_object['id']) show_options = swh_object['type'] == 'content' or \ (snapshot_context and snapshot_context['origin_info'] is not None) object_icon = swh_object_icons[swh_object['type']] swh_ids.append({ 'object_type': swh_object['type'], 'object_icon': object_icon, 'swh_id': swh_id, 'swh_id_url': reverse('browse-swh-id', url_args={'swh_id': swh_id}), 'show_options': show_options }) return swh_ids diff --git a/swh/web/browse/views/content.py b/swh/web/browse/views/content.py index 492c26c83..d9c30bcb9 100644 --- a/swh/web/browse/views/content.py +++ b/swh/web/browse/views/content.py @@ -1,298 +1,299 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import difflib import json from distutils.util import strtobool from django.http import HttpResponse from django.shortcuts import render from django.template.defaultfilters import filesizeformat from swh.model.hashutil import hash_to_hex from swh.web.common import query from swh.web.common.utils import ( reverse, gen_path_info, swh_object_icons ) from swh.web.common.exc import NotFoundExc, handle_view_exception from swh.web.browse.utils import ( request_content, prepare_content_for_display, content_display_max_size, get_snapshot_context, get_swh_persistent_ids, gen_link ) from swh.web.browse.browseurls import browse_route @browse_route(r'content/(?P.+)/raw/', view_name='browse-content-raw') def content_raw(request, query_string): - """Django view that produces a raw display of a SWH content identified + """Django view that produces a raw display of a content identified by its hash value. The url that points to it is :http:get:`/browse/content/[(algo_hash):](hash)/raw/` """ # noqa try: algo, checksum = query.parse_hash(query_string) checksum = hash_to_hex(checksum) content_data = request_content(query_string, max_size=None, reencode=False) except Exception as exc: return handle_view_exception(request, exc) filename = request.GET.get('filename', None) if not filename: filename = '%s_%s' % (algo, checksum) if content_data['mimetype'].startswith('text/') or \ content_data['mimetype'] == 'inode/x-empty': response = HttpResponse(content_data['raw_data'], content_type="text/plain") response['Content-disposition'] = 'filename=%s' % filename else: response = HttpResponse(content_data['raw_data'], content_type='application/octet-stream') response['Content-disposition'] = 'attachment; filename=%s' % filename return response _auto_diff_size_limit = 20000 @browse_route(r'content/(?P.*)/diff/(?P.*)', # noqa view_name='diff-contents') def _contents_diff(request, from_query_string, to_query_string): """ Browse endpoint used to compute unified diffs between two contents. Diffs are generated only if the two contents are textual. By default, diffs whose size are greater than 20 kB will not be generated. To force the generation of large diffs, the 'force' boolean query parameter must be used. Args: request: input django http request from_query_string: a string of the form "[ALGO_HASH:]HASH" where - optional ALGO_HASH can be either *sha1*, *sha1_git*, *sha256*, - or *blake2s256* (default to *sha1*) and HASH the hexadecimal - representation of the hash value identifying the first content + optional ALGO_HASH can be either ``sha1``, ``sha1_git``, + ``sha256``, or ``blake2s256`` (default to ``sha1``) and HASH + the hexadecimal representation of the hash value identifying + the first content to_query_string: same as above for identifying the second content Returns: A JSON object containing the unified diff. """ diff_data = {} content_from = None content_to = None content_from_size = 0 content_to_size = 0 content_from_lines = [] content_to_lines = [] force = request.GET.get('force', 'false') path = request.GET.get('path', None) language = 'nohighlight' force = bool(strtobool(force)) if from_query_string == to_query_string: diff_str = 'File renamed without changes' else: try: text_diff = True if from_query_string: content_from = \ request_content(from_query_string, max_size=None) content_from_display_data = \ prepare_content_for_display(content_from['raw_data'], content_from['mimetype'], path) language = content_from_display_data['language'] content_from_size = content_from['length'] if not (content_from['mimetype'].startswith('text/') or content_from['mimetype'] == 'inode/x-empty'): text_diff = False if text_diff and to_query_string: content_to = request_content(to_query_string, max_size=None) content_to_display_data = prepare_content_for_display( content_to['raw_data'], content_to['mimetype'], path) language = content_to_display_data['language'] content_to_size = content_to['length'] if not (content_to['mimetype'].startswith('text/') or content_to['mimetype'] == 'inode/x-empty'): text_diff = False diff_size = abs(content_to_size - content_from_size) if not text_diff: diff_str = 'Diffs are not generated for non textual content' language = 'nohighlight' elif not force and diff_size > _auto_diff_size_limit: diff_str = 'Large diffs are not automatically computed' language = 'nohighlight' else: if content_from: content_from_lines = \ content_from['raw_data'].decode('utf-8')\ .splitlines(True) if content_from_lines and \ content_from_lines[-1][-1] != '\n': content_from_lines[-1] += '[swh-no-nl-marker]\n' if content_to: content_to_lines = content_to['raw_data'].decode('utf-8')\ .splitlines(True) if content_to_lines and content_to_lines[-1][-1] != '\n': content_to_lines[-1] += '[swh-no-nl-marker]\n' diff_lines = difflib.unified_diff(content_from_lines, content_to_lines) diff_str = ''.join(list(diff_lines)[2:]) except Exception as e: diff_str = str(e) diff_data['diff_str'] = diff_str diff_data['language'] = language diff_data_json = json.dumps(diff_data, separators=(',', ': ')) return HttpResponse(diff_data_json, content_type='application/json') @browse_route(r'content/(?P.+)/', view_name='browse-content') def content_display(request, query_string): - """Django view that produces an HTML display of a SWH content identified + """Django view that produces an HTML display of a content identified by its hash value. The url that points to it is :http:get:`/browse/content/[(algo_hash):](hash)/` """ # noqa try: algo, checksum = query.parse_hash(query_string) checksum = hash_to_hex(checksum) content_data = request_content(query_string, raise_if_unavailable=False) origin_type = request.GET.get('origin_type', None) origin_url = request.GET.get('origin_url', None) if not origin_url: origin_url = request.GET.get('origin', None) snapshot_context = None if origin_url: try: snapshot_context = get_snapshot_context(None, origin_type, origin_url) except Exception: raw_cnt_url = reverse('browse-content', url_args={'query_string': query_string}) error_message = \ ('The Software Heritage archive has a content ' 'with the hash you provided but the origin ' 'mentioned in your request appears broken: %s. ' 'Please check the URL and try again.\n\n' 'Nevertheless, you can still browse the content ' 'without origin information: %s' % (gen_link(origin_url), gen_link(raw_cnt_url))) raise NotFoundExc(error_message) if snapshot_context: snapshot_context['visit_info'] = None except Exception as exc: return handle_view_exception(request, exc) path = request.GET.get('path', None) content = None language = None mimetype = None if content_data['raw_data'] is not None: content_display_data = prepare_content_for_display( content_data['raw_data'], content_data['mimetype'], path) content = content_display_data['content_data'] language = content_display_data['language'] mimetype = content_display_data['mimetype'] root_dir = None filename = None path_info = None query_params = {'origin': origin_url} breadcrumbs = [] if path: split_path = path.split('/') root_dir = split_path[0] filename = split_path[-1] path = path.replace(root_dir + '/', '') path = path[:-len(filename)] path_info = gen_path_info(path) breadcrumbs.append({'name': root_dir[:7], 'url': reverse('browse-directory', url_args={'sha1_git': root_dir}, query_params=query_params)}) for pi in path_info: breadcrumbs.append({'name': pi['name'], 'url': reverse('browse-directory', url_args={'sha1_git': root_dir, 'path': pi['path']}, query_params=query_params)}) breadcrumbs.append({'name': filename, 'url': None}) query_params = {'filename': filename} content_raw_url = reverse('browse-content-raw', url_args={'query_string': query_string}, query_params=query_params) content_metadata = { 'sha1 checksum': content_data['checksums']['sha1'], 'sha1_git checksum': content_data['checksums']['sha1_git'], 'sha256 checksum': content_data['checksums']['sha256'], 'blake2s256 checksum': content_data['checksums']['blake2s256'], 'mime type': content_data['mimetype'], 'encoding': content_data['encoding'], 'size': filesizeformat(content_data['length']), 'language': content_data['language'], 'licenses': content_data['licenses'], 'filename': filename } if filename: content_metadata['filename'] = filename sha1_git = content_data['checksums']['sha1_git'] swh_ids = get_swh_persistent_ids([{'type': 'content', 'id': sha1_git}]) heading = 'Content - %s' % sha1_git if breadcrumbs: content_path = '/'.join([bc['name'] for bc in breadcrumbs]) heading += ' - %s' % content_path return render(request, 'browse/content.html', {'heading': heading, 'swh_object_id': swh_ids[0]['swh_id'], 'swh_object_name': 'Content', 'swh_object_metadata': content_metadata, 'content': content, 'content_size': content_data['length'], 'max_content_size': content_display_max_size, 'mimetype': mimetype, 'language': language, 'breadcrumbs': breadcrumbs, 'top_right_link': { 'url': content_raw_url, 'icon': swh_object_icons['content'], 'text': 'Raw File' }, 'snapshot_context': snapshot_context, 'vault_cooking': None, 'show_actions_menu': True, 'swh_ids': swh_ids, 'error_code': content_data['error_code'], 'error_message': content_data['error_message'], 'error_description': content_data['error_description']}, status=content_data['error_code']) diff --git a/swh/web/browse/views/directory.py b/swh/web/browse/views/directory.py index c7cd2a028..723fd2a6a 100644 --- a/swh/web/browse/views/directory.py +++ b/swh/web/browse/views/directory.py @@ -1,160 +1,160 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from django.shortcuts import render, redirect from django.template.defaultfilters import filesizeformat from swh.web.common import service from swh.web.common.utils import ( reverse, gen_path_info ) from swh.web.common.exc import handle_view_exception, NotFoundExc from swh.web.browse.utils import ( get_directory_entries, get_snapshot_context, get_readme_to_display, get_swh_persistent_ids, gen_link ) from swh.web.browse.browseurls import browse_route @browse_route(r'directory/(?P[0-9a-f]+)/', r'directory/(?P[0-9a-f]+)/(?P.+)/', view_name='browse-directory') def directory_browse(request, sha1_git, path=None): - """Django view for browsing the content of a SWH directory identified + """Django view for browsing the content of a directory identified by its sha1_git value. The url that points to it is :http:get:`/browse/directory/(sha1_git)/[(path)/]` """ # noqa root_sha1_git = sha1_git try: if path: dir_info = service.lookup_directory_with_path(sha1_git, path) # some readme files can reference assets reachable from the # browsed directory, handle that special case in order to # correctly displayed them if dir_info and dir_info['type'] == 'file': file_raw_url = reverse( 'browse-content-raw', url_args={'query_string': dir_info['checksums']['sha1']}) return redirect(file_raw_url) sha1_git = dir_info['target'] dirs, files = get_directory_entries(sha1_git) origin_type = request.GET.get('origin_type', None) origin_url = request.GET.get('origin_url', None) if not origin_url: origin_url = request.GET.get('origin', None) snapshot_context = None if origin_url: try: snapshot_context = get_snapshot_context(None, origin_type, origin_url) except Exception: raw_dir_url = reverse('browse-directory', url_args={'sha1_git': sha1_git}) error_message = \ ('The Software Heritage archive has a directory ' 'with the hash you provided but the origin ' 'mentioned in your request appears broken: %s. ' 'Please check the URL and try again.\n\n' 'Nevertheless, you can still browse the directory ' 'without origin information: %s' % (gen_link(origin_url), gen_link(raw_dir_url))) raise NotFoundExc(error_message) if snapshot_context: snapshot_context['visit_info'] = None except Exception as exc: return handle_view_exception(request, exc) path_info = gen_path_info(path) query_params = {'origin': origin_url} breadcrumbs = [] breadcrumbs.append({'name': root_sha1_git[:7], 'url': reverse('browse-directory', url_args={'sha1_git': root_sha1_git}, query_params=query_params)}) for pi in path_info: breadcrumbs.append({'name': pi['name'], 'url': reverse('browse-directory', url_args={'sha1_git': root_sha1_git, 'path': pi['path']}, query_params=query_params)}) path = '' if path is None else (path + '/') for d in dirs: if d['type'] == 'rev': d['url'] = reverse('browse-revision', url_args={'sha1_git': d['target']}, query_params=query_params) else: d['url'] = reverse('browse-directory', url_args={'sha1_git': root_sha1_git, 'path': path + d['name']}, query_params=query_params) sum_file_sizes = 0 readmes = {} for f in files: query_string = 'sha1_git:' + f['target'] f['url'] = reverse('browse-content', url_args={'query_string': query_string}, query_params={'path': root_sha1_git + '/' + path + f['name'], 'origin': origin_url}) if f['length'] is not None: sum_file_sizes += f['length'] f['length'] = filesizeformat(f['length']) if f['name'].lower().startswith('readme'): readmes[f['name']] = f['checksums']['sha1'] readme_name, readme_url, readme_html = get_readme_to_display(readmes) sum_file_sizes = filesizeformat(sum_file_sizes) dir_metadata = {'id': sha1_git, 'number of regular files': len(files), 'number of subdirectories': len(dirs), 'sum of regular file sizes': sum_file_sizes} vault_cooking = { 'directory_context': True, 'directory_id': sha1_git, 'revision_context': False, 'revision_id': None } swh_ids = get_swh_persistent_ids([{'type': 'directory', 'id': sha1_git}]) heading = 'Directory - %s' % sha1_git if breadcrumbs: dir_path = '/'.join([bc['name'] for bc in breadcrumbs]) + '/' heading += ' - %s' % dir_path return render(request, 'browse/directory.html', {'heading': heading, 'swh_object_id': swh_ids[0]['swh_id'], 'swh_object_name': 'Directory', 'swh_object_metadata': dir_metadata, 'dirs': dirs, 'files': files, 'breadcrumbs': breadcrumbs, 'top_right_link': None, 'readme_name': readme_name, 'readme_url': readme_url, 'readme_html': readme_html, 'snapshot_context': snapshot_context, 'vault_cooking': vault_cooking, 'show_actions_menu': True, 'swh_ids': swh_ids}) diff --git a/swh/web/browse/views/origin.py b/swh/web/browse/views/origin.py index f475b27c6..72238f53d 100644 --- a/swh/web/browse/views/origin.py +++ b/swh/web/browse/views/origin.py @@ -1,240 +1,240 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import json from distutils.util import strtobool from django.http import HttpResponse from django.shortcuts import render, redirect from swh.web.common import service from swh.web.common.utils import ( reverse, format_utc_iso_date, parse_timestamp, get_origin_visits ) from swh.web.common.exc import handle_view_exception from swh.web.browse.utils import ( get_origin_info, get_snapshot_context ) from swh.web.browse.browseurls import browse_route from .utils.snapshot_context import ( browse_snapshot_directory, browse_snapshot_content, browse_snapshot_log, browse_snapshot_branches, browse_snapshot_releases ) @browse_route(r'origin/(?P[a-z]+)/url/(?P.+)/visit/(?P.+)/directory/', # noqa r'origin/(?P[a-z]+)/url/(?P.+)/visit/(?P.+)/directory/(?P.+)/', # noqa r'origin/(?P[a-z]+)/url/(?P.+)/directory/', # noqa r'origin/(?P[a-z]+)/url/(?P.+)/directory/(?P.+)/', # noqa r'origin/(?P.+)/visit/(?P.+)/directory/', # noqa r'origin/(?P.+)/visit/(?P.+)/directory/(?P.+)/', # noqa r'origin/(?P.+)/directory/', # noqa r'origin/(?P.+)/directory/(?P.+)/', # noqa view_name='browse-origin-directory') def origin_directory_browse(request, origin_url, origin_type=None, timestamp=None, path=None): - """Django view for browsing the content of a SWH directory associated + """Django view for browsing the content of a directory associated to an origin for a given visit. The url scheme that points to it is the following: * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/directory/[(path)/]` * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/directory/[(path)/]` """ # noqa return browse_snapshot_directory( request, origin_type=origin_type, origin_url=origin_url, timestamp=timestamp, path=path) @browse_route(r'origin/(?P[a-z]+)/url/(?P.+)/visit/(?P.+)/content/(?P.+)/', # noqa r'origin/(?P[a-z]+)/url/(?P.+)/content/(?P.+)/', # noqa r'origin/(?P.+)/visit/(?P.+)/content/(?P.+)/', # noqa r'origin/(?P.+)/content/(?P.+)/', # noqa view_name='browse-origin-content') def origin_content_browse(request, origin_url, origin_type=None, path=None, timestamp=None): - """Django view that produces an HTML display of a SWH content + """Django view that produces an HTML display of a content associated to an origin for a given visit. The url scheme that points to it is the following: * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/content/(path)/` * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/content/(path)/` """ # noqa return browse_snapshot_content(request, origin_type=origin_type, origin_url=origin_url, timestamp=timestamp, path=path) PER_PAGE = 20 @browse_route(r'origin/(?P[a-z]+)/url/(?P.+)/visit/(?P.+)/log/', # noqa r'origin/(?P[a-z]+)/url/(?P.+)/log/', r'origin/(?P.+)/visit/(?P.+)/log/', # noqa r'origin/(?P.+)/log/', view_name='browse-origin-log') def origin_log_browse(request, origin_url, origin_type=None, timestamp=None): """Django view that produces an HTML display of revisions history (aka - the commit log) associated to a SWH origin. + the commit log) associated to a software origin. The url scheme that points to it is the following: * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/log/` * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/log/` """ # noqa return browse_snapshot_log(request, origin_type=origin_type, origin_url=origin_url, timestamp=timestamp) @browse_route(r'origin/(?P[a-z]+)/url/(?P.+)/visit/(?P.+)/branches/', # noqa r'origin/(?P[a-z]+)/url/(?P.+)/branches/', # noqa r'origin/(?P.+)/visit/(?P.+)/branches/', # noqa r'origin/(?P.+)/branches/', # noqa view_name='browse-origin-branches') def origin_branches_browse(request, origin_url, origin_type=None, timestamp=None): """Django view that produces an HTML display of the list of branches associated to an origin for a given visit. The url scheme that points to it is the following: * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/branches/` * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/branches/` """ # noqa return browse_snapshot_branches(request, origin_type=origin_type, origin_url=origin_url, timestamp=timestamp) @browse_route(r'origin/(?P[a-z]+)/url/(?P.+)/visit/(?P.+)/releases/', # noqa r'origin/(?P[a-z]+)/url/(?P.+)/releases/', # noqa r'origin/(?P.+)/visit/(?P.+)/releases/', # noqa r'origin/(?P.+)/releases/', # noqa view_name='browse-origin-releases') def origin_releases_browse(request, origin_url, origin_type=None, timestamp=None): """Django view that produces an HTML display of the list of releases associated to an origin for a given visit. The url scheme that points to it is the following: * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/releases/` * :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/releases/` """ # noqa return browse_snapshot_releases(request, origin_type=origin_type, origin_url=origin_url, timestamp=timestamp) @browse_route(r'origin/(?P[a-z]+)/url/(?P.+)/visits/', r'origin/(?P.+)/visits/', view_name='browse-origin-visits') def origin_visits_browse(request, origin_url, origin_type=None): """Django view that produces an HTML display of visits reporting for a swh origin identified by its id or its url. The url that points to it is :http:get:`/browse/origin/[(origin_type)/url/](origin_url)/visits/`. """ # noqa try: origin_info = get_origin_info(origin_url, origin_type) origin_visits = get_origin_visits(origin_info) snapshot_context = get_snapshot_context(origin_type=origin_type, origin_url=origin_url) except Exception as exc: return handle_view_exception(request, exc) for i, visit in enumerate(origin_visits): url_date = format_utc_iso_date(visit['date'], '%Y-%m-%dT%H:%M:%SZ') visit['fmt_date'] = format_utc_iso_date(visit['date']) query_params = {} if i < len(origin_visits) - 1: if visit['date'] == origin_visits[i+1]['date']: query_params = {'visit_id': visit['visit']} if i > 0: if visit['date'] == origin_visits[i-1]['date']: query_params = {'visit_id': visit['visit']} snapshot = visit['snapshot'] if visit['snapshot'] else '' visit['browse_url'] = reverse('browse-origin-directory', url_args={'origin_type': origin_type, 'origin_url': origin_url, 'timestamp': url_date}, query_params=query_params) if not snapshot: visit['snapshot'] = '' visit['date'] = parse_timestamp(visit['date']).timestamp() heading = 'Origin visits - %s' % origin_url return render(request, 'browse/origin-visits.html', {'heading': heading, 'swh_object_name': 'Visits', 'swh_object_metadata': origin_info, 'origin_visits': origin_visits, 'origin_info': origin_info, 'snapshot_context': snapshot_context, 'vault_cooking': None, 'show_actions_menu': False}) @browse_route(r'origin/search/(?P.+)/', view_name='browse-origin-search') def _origin_search(request, url_pattern): """Internal browse endpoint to search for origins whose urls contain a provided string pattern or match a provided regular expression. The search is performed in a case insensitive way. """ offset = int(request.GET.get('offset', '0')) limit = int(request.GET.get('limit', '50')) regexp = request.GET.get('regexp', 'false') with_visit = request.GET.get('with_visit', 'false') url_pattern = url_pattern.replace('///', '\\') try: results = service.search_origin(url_pattern, offset, limit, bool(strtobool(regexp)), bool(strtobool(with_visit))) results = json.dumps(list(results), sort_keys=True, indent=4, separators=(',', ': ')) except Exception as exc: return handle_view_exception(request, exc, html_response=False) return HttpResponse(results, content_type='application/json') @browse_route(r'origin/(?P[0-9]+)/latest_snapshot/', view_name='browse-origin-latest-snapshot') def _origin_latest_snapshot(request, origin_id): """ Internal browse endpoint used to check if an origin has already been visited by Software Heritage and has at least one full visit. """ result = service.lookup_latest_origin_snapshot(origin_id, allowed_statuses=['full']) result = json.dumps(result, sort_keys=True, indent=4, separators=(',', ': ')) return HttpResponse(result, content_type='application/json') @browse_route(r'origin/(?P[a-z]+)/url/(?P.+)/', r'origin/(?P.+)/', view_name='browse-origin') def origin_browse(request, origin_url, origin_type=None): """Django view that redirects to the display of the latest archived snapshot for a given software origin. """ # noqa last_snapshot_url = reverse('browse-origin-directory', url_args={'origin_type': origin_type, 'origin_url': origin_url}) return redirect(last_snapshot_url) diff --git a/swh/web/browse/views/release.py b/swh/web/browse/views/release.py index d2a9176b2..8a01e917b 100644 --- a/swh/web/browse/views/release.py +++ b/swh/web/browse/views/release.py @@ -1,212 +1,212 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from django.shortcuts import render from swh.web.common import service from swh.web.common.utils import ( reverse, format_utc_iso_date ) from swh.web.common.exc import NotFoundExc, handle_view_exception from swh.web.browse.browseurls import browse_route from swh.web.browse.utils import ( gen_person_link, gen_revision_link, get_snapshot_context, gen_link, gen_snapshot_link, get_swh_persistent_ids ) @browse_route(r'release/(?P[0-9a-f]+)/', view_name='browse-release') def release_browse(request, sha1_git): """ - Django view that produces an HTML display of a SWH release + Django view that produces an HTML display of a release identified by its id. The url that points to it is :http:get:`/browse/release/(sha1_git)/`. """ try: release = service.lookup_release(sha1_git) snapshot_context = None origin_info = None snapshot_id = request.GET.get('snapshot_id', None) origin_type = request.GET.get('origin_type', None) origin_url = request.GET.get('origin_url', None) if not origin_url: origin_url = request.GET.get('origin', None) timestamp = request.GET.get('timestamp', None) visit_id = request.GET.get('visit_id', None) if origin_url: try: snapshot_context = \ get_snapshot_context(snapshot_id, origin_type, origin_url, timestamp, visit_id) except Exception: raw_rel_url = reverse('browse-release', url_args={'sha1_git': sha1_git}) error_message = \ ('The Software Heritage archive has a release ' 'with the hash you provided but the origin ' 'mentioned in your request appears broken: %s. ' 'Please check the URL and try again.\n\n' 'Nevertheless, you can still browse the release ' 'without origin information: %s' % (gen_link(origin_url), gen_link(raw_rel_url))) raise NotFoundExc(error_message) origin_info = snapshot_context['origin_info'] elif snapshot_id: snapshot_context = get_snapshot_context(snapshot_id) except Exception as exc: return handle_view_exception(request, exc) release_data = {} author_name = 'None' release_data['author'] = 'None' if release['author']: author_name = release['author']['name'] or \ release['author']['fullname'] release_data['author'] = \ gen_person_link(release['author']['id'], author_name, snapshot_context) release_data['date'] = format_utc_iso_date(release['date']) release_data['id'] = sha1_git release_data['name'] = release['name'] release_data['synthetic'] = release['synthetic'] release_data['target type'] = release['target_type'] if release['target_type'] == 'revision': release_data['target'] = \ gen_revision_link(release['target'], snapshot_context=snapshot_context) elif release['target_type'] == 'content': content_url = \ reverse('browse-content', url_args={'query_string': 'sha1_git:' + release['target']}) release_data['target'] = gen_link(content_url, release['target']) elif release['target_type'] == 'directory': directory_url = \ reverse('browse-directory', url_args={'sha1_git': release['target']}) release_data['target'] = gen_link(directory_url, release['target']) elif release['target_type'] == 'release': release_url = \ reverse('browse-release', url_args={'sha1_git': release['target']}) release_data['target'] = gen_link(release_url, release['target']) release_note_lines = [] if release['message']: release_note_lines = release['message'].split('\n') vault_cooking = None query_params = {} if snapshot_id: query_params = {'snapshot_id': snapshot_id} elif origin_info: query_params = {'origin': origin_info['url']} target_url = '' if release['target_type'] == 'revision': target_url = reverse('browse-revision', url_args={'sha1_git': release['target']}, query_params=query_params) try: revision = service.lookup_revision(release['target']) vault_cooking = { 'directory_context': True, 'directory_id': revision['directory'], 'revision_context': True, 'revision_id': release['target'] } except Exception: pass elif release['target_type'] == 'directory': target_url = reverse('browse-directory', url_args={'sha1_git': release['target']}, query_params=query_params) try: revision = service.lookup_directory(release['target']) vault_cooking = { 'directory_context': True, 'directory_id': revision['directory'], 'revision_context': False, 'revision_id': None } except Exception: pass elif release['target_type'] == 'content': target_url = reverse('browse-content', url_args={'sha1_git': release['target']}, query_params=query_params) elif release['target_type'] == 'release': target_url = reverse('browse-release', url_args={'sha1_git': release['target']}, query_params=query_params) release['target_url'] = target_url if snapshot_context: release_data['snapshot id'] = snapshot_context['snapshot_id'] if origin_info: release_url = reverse('browse-release', url_args={'sha1_git': release['id']}) release_data['context-independent release'] = \ gen_link(release_url, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) release_data['origin id'] = origin_info['id'] release_data['origin type'] = origin_info['type'] release_data['origin url'] = gen_link(origin_info['url'], origin_info['url']) browse_snapshot_link = \ gen_snapshot_link(snapshot_context['snapshot_id'], link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) release_data['snapshot'] = browse_snapshot_link swh_objects = [{'type': 'release', 'id': sha1_git}] if snapshot_context: snapshot_id = snapshot_context['snapshot_id'] if snapshot_id: swh_objects.append({'type': 'snapshot', 'id': snapshot_id}) swh_ids = get_swh_persistent_ids(swh_objects, snapshot_context) note_header = 'None' if len(release_note_lines) > 0: note_header = release_note_lines[0] release['note_header'] = note_header release['note_body'] = '\n'.join(release_note_lines[1:]) heading = 'Release - %s' % release['name'] if snapshot_context: context_found = 'snapshot: %s' % snapshot_context['snapshot_id'] if origin_info: context_found = 'origin: %s' % origin_info['url'] heading += ' - %s' % context_found return render(request, 'browse/release.html', {'heading': heading, 'swh_object_id': swh_ids[0]['swh_id'], 'swh_object_name': 'Release', 'swh_object_metadata': release_data, 'release': release, 'snapshot_context': snapshot_context, 'show_actions_menu': True, 'breadcrumbs': None, 'vault_cooking': vault_cooking, 'top_right_link': None, 'swh_ids': swh_ids}) diff --git a/swh/web/browse/views/revision.py b/swh/web/browse/views/revision.py index 38ac23bf7..c4a6ca89b 100644 --- a/swh/web/browse/views/revision.py +++ b/swh/web/browse/views/revision.py @@ -1,551 +1,551 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import hashlib import json import textwrap from django.http import HttpResponse from django.shortcuts import render, redirect from django.template.defaultfilters import filesizeformat from django.utils.safestring import mark_safe from swh.model.identifiers import persistent_identifier from swh.web.common import service from swh.web.common.utils import ( reverse, format_utc_iso_date, gen_path_info, swh_object_icons ) from swh.web.common.exc import NotFoundExc, handle_view_exception from swh.web.browse.browseurls import browse_route from swh.web.browse.utils import ( gen_link, gen_person_link, gen_revision_link, gen_revision_url, get_snapshot_context, gen_snapshot_directory_link, get_revision_log_url, get_directory_entries, gen_directory_link, request_content, prepare_content_for_display, content_display_max_size, gen_snapshot_link, get_readme_to_display, get_swh_persistent_ids, format_log_entries ) def _gen_content_url(revision, query_string, path, snapshot_context): if snapshot_context: url_args = snapshot_context['url_args'] url_args['path'] = path query_params = snapshot_context['query_params'] query_params['revision'] = revision['id'] content_url = reverse('browse-origin-content', url_args=url_args, query_params=query_params) else: content_path = '%s/%s' % (revision['directory'], path) content_url = reverse('browse-content', url_args={'query_string': query_string}, query_params={'path': content_path}) return content_url def _gen_diff_link(idx, diff_anchor, link_text): if idx < _max_displayed_file_diffs: return gen_link(diff_anchor, link_text) else: return link_text # TODO: put in conf _max_displayed_file_diffs = 1000 def _gen_revision_changes_list(revision, changes, snapshot_context): """ Returns a HTML string describing the file changes introduced in a revision. As this string will be displayed in the browse revision view, links to adequate file diffs are also generated. Args: revision (str): hexadecimal representation of a revision identifier changes (list): list of file changes in the revision snapshot_context (dict): optional origin context used to reverse the content urls Returns: A string to insert in a revision HTML view. """ changes_msg = [] for i, change in enumerate(changes): hasher = hashlib.sha1() from_query_string = '' to_query_string = '' diff_id = 'diff-' if change['from']: from_query_string = 'sha1_git:' + change['from']['target'] diff_id += change['from']['target'] + '-' + change['from_path'] diff_id += '-' if change['to']: to_query_string = 'sha1_git:' + change['to']['target'] diff_id += change['to']['target'] + change['to_path'] change['path'] = change['to_path'] or change['from_path'] url_args = {'from_query_string': from_query_string, 'to_query_string': to_query_string} query_params = {'path': change['path']} change['diff_url'] = reverse('diff-contents', url_args=url_args, query_params=query_params) hasher.update(diff_id.encode('utf-8')) diff_id = hasher.hexdigest() change['id'] = diff_id panel_diff_link = '#panel_' + diff_id if change['type'] == 'modify': change['content_url'] = \ _gen_content_url(revision, to_query_string, change['to_path'], snapshot_context) changes_msg.append('modified: %s' % _gen_diff_link(i, panel_diff_link, change['to_path'])) elif change['type'] == 'insert': change['content_url'] = \ _gen_content_url(revision, to_query_string, change['to_path'], snapshot_context) changes_msg.append('new file: %s' % _gen_diff_link(i, panel_diff_link, change['to_path'])) elif change['type'] == 'delete': parent = service.lookup_revision(revision['parents'][0]) change['content_url'] = \ _gen_content_url(parent, from_query_string, change['from_path'], snapshot_context) changes_msg.append('deleted: %s' % _gen_diff_link(i, panel_diff_link, change['from_path'])) elif change['type'] == 'rename': change['content_url'] = \ _gen_content_url(revision, to_query_string, change['to_path'], snapshot_context) link_text = change['from_path'] + ' → ' + change['to_path'] changes_msg.append('renamed: %s' % _gen_diff_link(i, panel_diff_link, link_text)) if not changes: changes_msg.append('No changes') return mark_safe('\n'.join(changes_msg)) @browse_route(r'revision/(?P[0-9a-f]+)/diff/', view_name='diff-revision') def _revision_diff(request, sha1_git): """ Browse internal endpoint to compute revision diff """ try: revision = service.lookup_revision(sha1_git) snapshot_context = None origin_type = request.GET.get('origin_type', None) origin_url = request.GET.get('origin_url', None) if not origin_url: origin_url = request.GET.get('origin', None) timestamp = request.GET.get('timestamp', None) visit_id = request.GET.get('visit_id', None) if origin_url: snapshot_context = get_snapshot_context(None, origin_type, origin_url, timestamp, visit_id) except Exception as exc: return handle_view_exception(request, exc) changes = service.diff_revision(sha1_git) changes_msg = _gen_revision_changes_list(revision, changes, snapshot_context) diff_data = { 'total_nb_changes': len(changes), 'changes': changes[:_max_displayed_file_diffs], 'changes_msg': changes_msg } diff_data_json = json.dumps(diff_data, separators=(',', ': ')) return HttpResponse(diff_data_json, content_type='application/json') NB_LOG_ENTRIES = 100 @browse_route(r'revision/(?P[0-9a-f]+)/log/', view_name='browse-revision-log') def revision_log_browse(request, sha1_git): """ Django view that produces an HTML display of the history - log for a SWH revision identified by its id. + log for a revision identified by its id. The url that points to it is :http:get:`/browse/revision/(sha1_git)/log/` """ # noqa try: per_page = int(request.GET.get('per_page', NB_LOG_ENTRIES)) offset = int(request.GET.get('offset', 0)) revs_ordering = request.GET.get('revs_ordering', 'committer_date') session_key = 'rev_%s_log_ordering_%s' % (sha1_git, revs_ordering) rev_log_session = request.session.get(session_key, None) rev_log = [] revs_walker_state = None if rev_log_session: rev_log = rev_log_session['rev_log'] revs_walker_state = rev_log_session['revs_walker_state'] if len(rev_log) < offset+per_page: revs_walker = \ service.get_revisions_walker(revs_ordering, sha1_git, max_revs=offset+per_page+1, state=revs_walker_state) rev_log += list(revs_walker) revs_walker_state = revs_walker.export_state() revision_log = rev_log[offset:offset+per_page] request.session[session_key] = { 'rev_log': rev_log, 'revs_walker_state': revs_walker_state } except Exception as exc: return handle_view_exception(request, exc) revs_ordering = request.GET.get('revs_ordering', '') prev_log_url = None if len(rev_log) > offset + per_page: prev_log_url = reverse('browse-revision-log', url_args={'sha1_git': sha1_git}, query_params={'per_page': per_page, 'offset': offset + per_page, 'revs_ordering': revs_ordering}) next_log_url = None if offset != 0: next_log_url = reverse('browse-revision-log', url_args={'sha1_git': sha1_git}, query_params={'per_page': per_page, 'offset': offset - per_page, 'revs_ordering': revs_ordering}) revision_log_data = format_log_entries(revision_log, per_page) swh_rev_id = persistent_identifier('revision', sha1_git) return render(request, 'browse/revision-log.html', {'heading': 'Revision history', 'swh_object_id': swh_rev_id, 'swh_object_name': 'Revisions history', 'swh_object_metadata': None, 'revision_log': revision_log_data, 'revs_ordering': revs_ordering, 'next_log_url': next_log_url, 'prev_log_url': prev_log_url, 'breadcrumbs': None, 'top_right_link': None, 'snapshot_context': None, 'vault_cooking': None, 'show_actions_menu': True, 'swh_ids': None}) @browse_route(r'revision/(?P[0-9a-f]+)/', r'revision/(?P[0-9a-f]+)/(?P.+)/', view_name='browse-revision') def revision_browse(request, sha1_git, extra_path=None): """ - Django view that produces an HTML display of a SWH revision + Django view that produces an HTML display of a revision identified by its id. The url that points to it is :http:get:`/browse/revision/(sha1_git)/`. """ try: revision = service.lookup_revision(sha1_git) # some readme files can reference assets reachable from the # browsed directory, handle that special case in order to # correctly displayed them if extra_path: dir_info = \ service.lookup_directory_with_path(revision['directory'], extra_path) if dir_info and dir_info['type'] == 'file': file_raw_url = reverse( 'browse-content-raw', url_args={'query_string': dir_info['checksums']['sha1']}) return redirect(file_raw_url) origin_info = None snapshot_context = None origin_type = request.GET.get('origin_type', None) origin_url = request.GET.get('origin_url', None) if not origin_url: origin_url = request.GET.get('origin', None) timestamp = request.GET.get('timestamp', None) visit_id = request.GET.get('visit_id', None) snapshot_id = request.GET.get('snapshot_id', None) path = request.GET.get('path', None) dir_id = None dirs, files = None, None content_data = None if origin_url: try: snapshot_context = get_snapshot_context(None, origin_type, origin_url, timestamp, visit_id) except Exception: raw_rev_url = reverse('browse-revision', url_args={'sha1_git': sha1_git}) error_message = \ ('The Software Heritage archive has a revision ' 'with the hash you provided but the origin ' 'mentioned in your request appears broken: %s. ' 'Please check the URL and try again.\n\n' 'Nevertheless, you can still browse the revision ' 'without origin information: %s' % (gen_link(origin_url), gen_link(raw_rev_url))) raise NotFoundExc(error_message) origin_info = snapshot_context['origin_info'] snapshot_id = snapshot_context['snapshot_id'] elif snapshot_id: snapshot_context = get_snapshot_context(snapshot_id) if path: file_info = \ service.lookup_directory_with_path(revision['directory'], path) if file_info['type'] == 'dir': dir_id = file_info['target'] else: query_string = 'sha1_git:' + file_info['target'] content_data = request_content(query_string, raise_if_unavailable=False) else: dir_id = revision['directory'] if dir_id: path = '' if path is None else (path + '/') dirs, files = get_directory_entries(dir_id) except Exception as exc: return handle_view_exception(request, exc) revision_data = {} author_name = 'None' revision_data['author'] = 'None' if revision['author']: author_name = revision['author']['name'] or \ revision['author']['fullname'] revision_data['author'] = \ gen_person_link(revision['author']['id'], author_name, snapshot_context) revision_data['committer'] = 'None' if revision['committer']: revision_data['committer'] = \ gen_person_link(revision['committer']['id'], revision['committer']['name'], snapshot_context) revision_data['committer date'] = \ format_utc_iso_date(revision['committer_date']) revision_data['date'] = format_utc_iso_date(revision['date']) if snapshot_context: revision_data['snapshot id'] = snapshot_id revision_data['directory'] = \ gen_snapshot_directory_link(snapshot_context, sha1_git, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', # noqa 'role': 'button'}) else: revision_data['directory'] = \ gen_directory_link(revision['directory'], link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) revision_data['id'] = sha1_git revision_data['merge'] = revision['merge'] revision_data['metadata'] = json.dumps(revision['metadata'], sort_keys=True, indent=4, separators=(',', ': ')) if origin_info: revision_data['context-independent revision'] = \ gen_revision_link(sha1_git, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) revision_data['origin id'] = origin_info['id'] revision_data['origin type'] = origin_info['type'] revision_data['origin url'] = gen_link(origin_info['url'], origin_info['url']) browse_snapshot_link = \ gen_snapshot_link(snapshot_id, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) revision_data['snapshot'] = browse_snapshot_link parents = '' for p in revision['parents']: parent_link = gen_revision_link(p, snapshot_context=snapshot_context) parents += parent_link + '
' revision_data['parents'] = mark_safe(parents) revision_data['synthetic'] = revision['synthetic'] revision_data['type'] = revision['type'] message_lines = ['None'] if revision['message']: message_lines = revision['message'].split('\n') parents = [] for p in revision['parents']: parent_url = gen_revision_url(p, snapshot_context) parents.append({'id': p, 'url': parent_url}) path_info = gen_path_info(path) query_params = {'snapshot_id': snapshot_id, 'origin_type': origin_type, 'origin': origin_url, 'timestamp': timestamp, 'visit_id': visit_id} breadcrumbs = [] breadcrumbs.append({'name': revision['directory'][:7], 'url': reverse('browse-revision', url_args={'sha1_git': sha1_git}, query_params=query_params)}) for pi in path_info: query_params['path'] = pi['path'] breadcrumbs.append({'name': pi['name'], 'url': reverse('browse-revision', url_args={'sha1_git': sha1_git}, query_params=query_params)}) vault_cooking = { 'directory_context': False, 'directory_id': None, 'revision_context': True, 'revision_id': sha1_git } swh_objects = [{'type': 'revision', 'id': sha1_git}] content = None content_size = None mimetype = None language = None readme_name = None readme_url = None readme_html = None readmes = {} error_code = 200 error_message = '' error_description = '' if content_data: breadcrumbs[-1]['url'] = None content_size = content_data['length'] mimetype = content_data['mimetype'] if content_data['raw_data']: content_display_data = prepare_content_for_display( content_data['raw_data'], content_data['mimetype'], path) content = content_display_data['content_data'] language = content_display_data['language'] query_params = {} if path: query_params['filename'] = path_info[-1]['name'] top_right_link = { 'url': reverse('browse-content-raw', url_args={'query_string': query_string}, query_params=query_params), 'icon': swh_object_icons['content'], 'text': 'Raw File' } swh_objects.append({'type': 'content', 'id': file_info['target']}) error_code = content_data['error_code'] error_message = content_data['error_message'] error_description = content_data['error_description'] else: for d in dirs: if d['type'] == 'rev': d['url'] = reverse('browse-revision', url_args={'sha1_git': d['target']}) else: query_params['path'] = path + d['name'] d['url'] = reverse('browse-revision', url_args={'sha1_git': sha1_git}, query_params=query_params) for f in files: query_params['path'] = path + f['name'] f['url'] = reverse('browse-revision', url_args={'sha1_git': sha1_git}, query_params=query_params) if f['length'] is not None: f['length'] = filesizeformat(f['length']) if f['name'].lower().startswith('readme'): readmes[f['name']] = f['checksums']['sha1'] readme_name, readme_url, readme_html = get_readme_to_display(readmes) top_right_link = { 'url': get_revision_log_url(sha1_git, snapshot_context), 'icon': swh_object_icons['revisions history'], 'text': 'History' } vault_cooking['directory_context'] = True vault_cooking['directory_id'] = dir_id swh_objects.append({'type': 'directory', 'id': dir_id}) diff_revision_url = reverse('diff-revision', url_args={'sha1_git': sha1_git}, query_params={'origin_type': origin_type, 'origin': origin_url, 'timestamp': timestamp, 'visit_id': visit_id}) if snapshot_id: swh_objects.append({'type': 'snapshot', 'id': snapshot_id}) swh_ids = get_swh_persistent_ids(swh_objects, snapshot_context) heading = 'Revision - %s - %s' %\ (sha1_git[:7], textwrap.shorten(message_lines[0], width=70)) if snapshot_context: context_found = 'snapshot: %s' % snapshot_context['snapshot_id'] if origin_info: context_found = 'origin: %s' % origin_info['url'] heading += ' - %s' % context_found return render(request, 'browse/revision.html', {'heading': heading, 'swh_object_id': swh_ids[0]['swh_id'], 'swh_object_name': 'Revision', 'swh_object_metadata': revision_data, 'message_header': message_lines[0], 'message_body': '\n'.join(message_lines[1:]), 'parents': parents, 'snapshot_context': snapshot_context, 'dirs': dirs, 'files': files, 'content': content, 'content_size': content_size, 'max_content_size': content_display_max_size, 'mimetype': mimetype, 'language': language, 'readme_name': readme_name, 'readme_url': readme_url, 'readme_html': readme_html, 'breadcrumbs': breadcrumbs, 'top_right_link': top_right_link, 'vault_cooking': vault_cooking, 'diff_revision_url': diff_revision_url, 'show_actions_menu': True, 'swh_ids': swh_ids, 'error_code': error_code, 'error_message': error_message, 'error_description': error_description}, status=error_code) diff --git a/swh/web/browse/views/snapshot.py b/swh/web/browse/views/snapshot.py index 1bde5eeea..f79afe657 100644 --- a/swh/web/browse/views/snapshot.py +++ b/swh/web/browse/views/snapshot.py @@ -1,91 +1,91 @@ # Copyright (C) 2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from django.shortcuts import redirect from swh.web.browse.browseurls import browse_route from swh.web.common.utils import reverse from .utils.snapshot_context import ( browse_snapshot_directory, browse_snapshot_content, browse_snapshot_log, browse_snapshot_branches, browse_snapshot_releases ) @browse_route(r'snapshot/(?P[0-9a-f]+)/', view_name='browse-snapshot') def snapshot_browse(request, snapshot_id): - """Django view for browsing the content of a SWH snapshot. + """Django view for browsing the content of a snapshot. The url that points to it is :http:get:`/browse/snapshot/(snapshot_id)/` """ browse_snapshot_url = reverse('browse-snapshot-directory', url_args={'snapshot_id': snapshot_id}, query_params=request.GET) return redirect(browse_snapshot_url) @browse_route(r'snapshot/(?P[0-9a-f]+)/directory/', r'snapshot/(?P[0-9a-f]+)/directory/(?P.+)/', view_name='browse-snapshot-directory') def snapshot_directory_browse(request, snapshot_id, path=None): - """Django view for browsing the content of a SWH directory collected - in a SWH snapshot. + """Django view for browsing the content of a directory collected + in a snapshot. The url that points to it is :http:get:`/browse/snapshot/(snapshot_id)/directory/[(path)/]` """ # noqa origin_type = request.GET.get('origin_type', None) origin_url = request.GET.get('origin_url', None) if not origin_url: origin_url = request.GET.get('origin', None) return browse_snapshot_directory(request, snapshot_id=snapshot_id, path=path, origin_type=origin_type, origin_url=origin_url) @browse_route(r'snapshot/(?P[0-9a-f]+)/content/(?P.+)/', view_name='browse-snapshot-content') def snapshot_content_browse(request, snapshot_id, path): - """Django view that produces an HTML display of a SWH content - collected in a SWH snapshot. + """Django view that produces an HTML display of a content + collected in a snapshot. The url that points to it is :http:get:`/browse/snapshot/(snapshot_id)/content/(path)/` """ # noqa return browse_snapshot_content(request, snapshot_id=snapshot_id, path=path) @browse_route(r'snapshot/(?P[0-9a-f]+)/log/', view_name='browse-snapshot-log') def snapshot_log_browse(request, snapshot_id): """Django view that produces an HTML display of revisions history (aka - the commit log) collected in a SWH snapshot. + the commit log) collected in a snapshot. The url that points to it is :http:get:`/browse/snapshot/(snapshot_id)/log/` """ # noqa return browse_snapshot_log(request, snapshot_id=snapshot_id) @browse_route(r'snapshot/(?P[0-9a-f]+)/branches/', view_name='browse-snapshot-branches') def snapshot_branches_browse(request, snapshot_id): """Django view that produces an HTML display of the list of releases - collected in a SWH snapshot. + collected in a snapshot. The url that points to it is :http:get:`/browse/snapshot/(snapshot_id)/branches/` """ # noqa return browse_snapshot_branches(request, snapshot_id=snapshot_id) @browse_route(r'snapshot/(?P[0-9a-f]+)/releases/', view_name='browse-snapshot-releases') def snapshot_releases_browse(request, snapshot_id): """Django view that produces an HTML display of the list of releases - collected in a SWH snapshot. + collected in a snapshot. The url that points to it is :http:get:`/browse/snapshot/(snapshot_id)/releases/` """ # noqa return browse_snapshot_releases(request, snapshot_id=snapshot_id) diff --git a/swh/web/browse/views/utils/snapshot_context.py b/swh/web/browse/views/utils/snapshot_context.py index ffbf512d4..cef896018 100644 --- a/swh/web/browse/views/utils/snapshot_context.py +++ b/swh/web/browse/views/utils/snapshot_context.py @@ -1,945 +1,945 @@ # Copyright (C) 2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information -# Utility module implementing Django views for browsing the SWH archive +# Utility module implementing Django views for browsing the archive # in a snapshot context. # Its purpose is to factorize code for the views reachable from the # /origin/.* and /snapshot/.* endpoints. from django.shortcuts import render, redirect from django.template.defaultfilters import filesizeformat from swh.model.identifiers import snapshot_identifier from swh.web.browse.utils import ( get_snapshot_context, get_directory_entries, gen_directory_link, gen_revision_link, request_content, gen_content_link, prepare_content_for_display, content_display_max_size, format_log_entries, gen_revision_log_link, gen_link, get_readme_to_display, get_swh_persistent_ids, process_snapshot_branches ) from swh.web.common import service from swh.web.common.exc import ( handle_view_exception, NotFoundExc ) from swh.web.common.utils import ( reverse, gen_path_info, format_utc_iso_date, swh_object_icons ) _empty_snapshot_id = snapshot_identifier({'branches': {}}) def _get_branch(branches, branch_name, snapshot_id): """ Utility function to get a specific branch from a branches list. - Its purpose is to get the default HEAD branch as some SWH origin + Its purpose is to get the default HEAD branch as some software origin (e.g those with svn type) does not have it. In that latter case, check if there is a master branch instead and returns it. """ filtered_branches = \ [b for b in branches if b['name'].endswith(branch_name)] if len(filtered_branches) > 0: return filtered_branches[0] elif branch_name == 'HEAD': filtered_branches = \ [b for b in branches if b['name'].endswith('master')] if len(filtered_branches) > 0: return filtered_branches[0] elif len(branches) > 0: return branches[0] else: # case where a large branches list has been truncated snp_branch = service.lookup_snapshot(snapshot_id, branches_from=branch_name, branches_count=1, target_types=['revision']) snp_branch, _ = process_snapshot_branches(snp_branch['branches']) if snp_branch: branches.append(snp_branch[0]) return snp_branch[0] return None def _get_release(releases, release_name): """ Utility function to get a specific release from a releases list. Returns None if the release can not be found in the list. """ filtered_releases = \ [r for r in releases if r['name'] == release_name] if len(filtered_releases) > 0: return filtered_releases[0] else: return None def _branch_not_found(branch_type, branch, branches, snapshot_id=None, origin_info=None, timestamp=None, visit_id=None): """ Utility function to raise an exception when a specified branch/release can not be found. """ if branch_type == 'branch': branch_type = 'Branch' branch_type_plural = 'branches' else: branch_type = 'Release' branch_type_plural = 'releases' if snapshot_id and len(branches) == 0: msg = 'Snapshot with id %s has an empty list' \ ' of %s!' % (snapshot_id, branch_type_plural) elif snapshot_id: msg = '%s %s for snapshot with id %s' \ ' not found!' % (branch_type, branch, snapshot_id) elif visit_id and len(branches) == 0: msg = 'Origin with type %s and url %s' \ ' for visit with id %s has an empty list' \ ' of %s!' % (origin_info['type'], origin_info['url'], visit_id, branch_type_plural) elif visit_id: msg = '%s %s associated to visit with' \ ' id %s for origin with type %s and url %s' \ ' not found!' % (branch_type, branch, visit_id, origin_info['type'], origin_info['url']) elif len(branches) == 0: msg = 'Origin with type %s and url %s' \ ' for visit with timestamp %s has an empty list' \ ' of %s!' % (origin_info['type'], origin_info['url'], timestamp, branch_type_plural) else: msg = '%s %s associated to visit with' \ ' timestamp %s for origin with type %s' \ ' and url %s not found!' % (branch_type, branch, timestamp, origin_info['type'], origin_info['url']) raise NotFoundExc(msg) def _process_snapshot_request(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None, path=None, browse_context='directory'): """ Utility function to perform common input request processing for snapshot context views. """ visit_id = request.GET.get('visit_id', None) snapshot_context = get_snapshot_context(snapshot_id, origin_type, origin_url, timestamp, visit_id) swh_type = snapshot_context['swh_type'] origin_info = snapshot_context['origin_info'] branches = snapshot_context['branches'] releases = snapshot_context['releases'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] if snapshot_context['visit_info']: timestamp = format_utc_iso_date(snapshot_context['visit_info']['date'], '%Y-%m-%dT%H:%M:%SZ') snapshot_context['timestamp'] = \ format_utc_iso_date(snapshot_context['visit_info']['date']) browse_view_name = 'browse-' + swh_type + '-' + browse_context root_sha1_git = None revision_id = request.GET.get('revision', None) release_name = request.GET.get('release', None) release_id = None branch_name = None snapshot_total_size = sum(snapshot_context['snapshot_size'].values()) if snapshot_total_size and revision_id: revision = service.lookup_revision(revision_id) root_sha1_git = revision['directory'] branches.append({'name': revision_id, 'revision': revision_id, 'directory': root_sha1_git, 'url': None}) branch_name = revision_id query_params['revision'] = revision_id elif snapshot_total_size and release_name: release = _get_release(releases, release_name) try: root_sha1_git = release['directory'] revision_id = release['target'] release_id = release['id'] query_params['release'] = release_name except Exception: _branch_not_found("release", release_name, releases, snapshot_id, origin_info, timestamp, visit_id) elif snapshot_total_size: branch_name = request.GET.get('branch', None) if branch_name: query_params['branch'] = branch_name branch = _get_branch(branches, branch_name or 'HEAD', snapshot_context['snapshot_id']) try: branch_name = branch['name'] revision_id = branch['revision'] root_sha1_git = branch['directory'] except Exception: _branch_not_found("branch", branch_name, branches, snapshot_id, origin_info, timestamp, visit_id) for b in branches: branch_url_args = dict(url_args) branch_query_params = dict(query_params) if 'release' in branch_query_params: del branch_query_params['release'] branch_query_params['branch'] = b['name'] if path: b['path'] = path branch_url_args['path'] = path b['url'] = reverse(browse_view_name, url_args=branch_url_args, query_params=branch_query_params) for r in releases: release_url_args = dict(url_args) release_query_params = dict(query_params) if 'branch' in release_query_params: del release_query_params['branch'] release_query_params['release'] = r['name'] if path: r['path'] = path release_url_args['path'] = path r['url'] = reverse(browse_view_name, url_args=release_url_args, query_params=release_query_params) snapshot_context['query_params'] = query_params snapshot_context['root_sha1_git'] = root_sha1_git snapshot_context['revision_id'] = revision_id snapshot_context['branch'] = branch_name snapshot_context['release'] = release_name snapshot_context['release_id'] = release_id return snapshot_context def browse_snapshot_directory(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None, path=None): """ Django view implementation for browsing a directory in a snapshot context. """ try: snapshot_context = _process_snapshot_request(request, snapshot_id, origin_type, origin_url, timestamp, path, browse_context='directory') # noqa root_sha1_git = snapshot_context['root_sha1_git'] sha1_git = root_sha1_git if root_sha1_git and path: dir_info = service.lookup_directory_with_path(root_sha1_git, path) # some readme files can reference assets reachable from the # browsed directory, handle that special case in order to # correctly displayed them if dir_info and dir_info['type'] == 'file': file_raw_url = reverse( 'browse-content-raw', url_args={'query_string': dir_info['checksums']['sha1']}) return redirect(file_raw_url) sha1_git = dir_info['target'] dirs = [] files = [] if sha1_git: dirs, files = get_directory_entries(sha1_git) except Exception as exc: return handle_view_exception(request, exc) swh_type = snapshot_context['swh_type'] origin_info = snapshot_context['origin_info'] visit_info = snapshot_context['visit_info'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] revision_id = snapshot_context['revision_id'] snapshot_id = snapshot_context['snapshot_id'] path_info = gen_path_info(path) browse_view_name = 'browse-' + swh_type + '-directory' breadcrumbs = [] if root_sha1_git: breadcrumbs.append({'name': root_sha1_git[:7], 'url': reverse(browse_view_name, url_args=url_args, query_params=query_params)}) for pi in path_info: bc_url_args = dict(url_args) bc_url_args['path'] = pi['path'] breadcrumbs.append({'name': pi['name'], 'url': reverse(browse_view_name, url_args=bc_url_args, query_params=query_params)}) path = '' if path is None else (path + '/') for d in dirs: if d['type'] == 'rev': d['url'] = reverse('browse-revision', url_args={'sha1_git': d['target']}) else: bc_url_args = dict(url_args) bc_url_args['path'] = path + d['name'] d['url'] = reverse(browse_view_name, url_args=bc_url_args, query_params=query_params) sum_file_sizes = 0 readmes = {} browse_view_name = 'browse-' + swh_type + '-content' for f in files: bc_url_args = dict(url_args) bc_url_args['path'] = path + f['name'] f['url'] = reverse(browse_view_name, url_args=bc_url_args, query_params=query_params) if f['length'] is not None: sum_file_sizes += f['length'] f['length'] = filesizeformat(f['length']) if f['name'].lower().startswith('readme'): readmes[f['name']] = f['checksums']['sha1'] readme_name, readme_url, readme_html = get_readme_to_display(readmes) browse_view_name = 'browse-' + swh_type + '-log' history_url = None if snapshot_id != _empty_snapshot_id: history_url = reverse(browse_view_name, url_args=url_args, query_params=query_params) nb_files = None nb_dirs = None sum_file_sizes = None dir_path = None if root_sha1_git: nb_files = len(files) nb_dirs = len(dirs) sum_file_sizes = filesizeformat(sum_file_sizes) dir_path = '/' + path browse_dir_link = \ gen_directory_link(sha1_git, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) browse_rev_link = \ gen_revision_link(revision_id, snapshot_context=snapshot_context, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) dir_metadata = {'id': sha1_git, 'context-independent directory': browse_dir_link, 'number of regular files': nb_files, 'number of subdirectories': nb_dirs, 'sum of regular file sizes': sum_file_sizes, 'path': dir_path, 'revision id': revision_id, 'revision': browse_rev_link, 'snapshot id': snapshot_id} if origin_info: dir_metadata['origin id'] = origin_info['id'] dir_metadata['origin type'] = origin_info['type'] dir_metadata['origin url'] = origin_info['url'] dir_metadata['origin visit date'] = format_utc_iso_date(visit_info['date']) # noqa dir_metadata['origin visit id'] = visit_info['visit'] snapshot_context_url = reverse('browse-snapshot-directory', url_args={'snapshot_id': snapshot_id}, query_params=request.GET) browse_snapshot_link = \ gen_link(snapshot_context_url, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) dir_metadata['snapshot context'] = browse_snapshot_link vault_cooking = { 'directory_context': True, 'directory_id': sha1_git, 'revision_context': True, 'revision_id': revision_id } swh_objects = [{'type': 'directory', 'id': sha1_git}, {'type': 'revision', 'id': revision_id}, {'type': 'snapshot', 'id': snapshot_id}] release_id = snapshot_context['release_id'] if release_id: swh_objects.append({'type': 'release', 'id': release_id}) swh_ids = get_swh_persistent_ids(swh_objects, snapshot_context) dir_path = '/'.join([bc['name'] for bc in breadcrumbs]) + '/' context_found = 'snapshot: %s' % snapshot_context['snapshot_id'] if origin_info: context_found = 'origin: %s' % origin_info['url'] heading = 'Directory - %s - %s - %s' %\ (dir_path, snapshot_context['branch'], context_found) return render(request, 'browse/directory.html', {'heading': heading, 'swh_object_name': 'Directory', 'swh_object_metadata': dir_metadata, 'dirs': dirs, 'files': files, 'breadcrumbs': breadcrumbs if root_sha1_git else [], 'top_right_link': { 'url': history_url, 'icon': swh_object_icons['revisions history'], 'text': 'History' }, 'readme_name': readme_name, 'readme_url': readme_url, 'readme_html': readme_html, 'snapshot_context': snapshot_context, 'vault_cooking': vault_cooking, 'show_actions_menu': True, 'swh_ids': swh_ids}) def browse_snapshot_content(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None, path=None): """ Django view implementation for browsing a content in a snapshot context. """ try: snapshot_context = _process_snapshot_request(request, snapshot_id, origin_type, origin_url, timestamp, path, browse_context='content') root_sha1_git = snapshot_context['root_sha1_git'] sha1_git = None query_string = None content_data = None if root_sha1_git: content_info = service.lookup_directory_with_path(root_sha1_git, path) sha1_git = content_info['target'] query_string = 'sha1_git:' + sha1_git content_data = request_content(query_string, raise_if_unavailable=False) except Exception as exc: return handle_view_exception(request, exc) swh_type = snapshot_context['swh_type'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] revision_id = snapshot_context['revision_id'] origin_info = snapshot_context['origin_info'] visit_info = snapshot_context['visit_info'] snapshot_id = snapshot_context['snapshot_id'] content = None language = None mimetype = None if content_data and content_data['raw_data'] is not None: content_display_data = prepare_content_for_display( content_data['raw_data'], content_data['mimetype'], path) content = content_display_data['content_data'] language = content_display_data['language'] mimetype = content_display_data['mimetype'] filename = None path_info = None browse_view_name = 'browse-' + swh_type + '-directory' breadcrumbs = [] split_path = path.split('/') filename = split_path[-1] path_info = gen_path_info(path[:-len(filename)]) if root_sha1_git: breadcrumbs.append({'name': root_sha1_git[:7], 'url': reverse(browse_view_name, url_args=url_args, query_params=query_params)}) for pi in path_info: bc_url_args = dict(url_args) bc_url_args['path'] = pi['path'] breadcrumbs.append({'name': pi['name'], 'url': reverse(browse_view_name, url_args=bc_url_args, query_params=query_params)}) breadcrumbs.append({'name': filename, 'url': None}) browse_content_link = \ gen_content_link(sha1_git, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) content_raw_url = None if query_string: content_raw_url = reverse('browse-content-raw', url_args={'query_string': query_string}, query_params={'filename': filename}) browse_rev_link = \ gen_revision_link(revision_id, snapshot_context=snapshot_context, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) content_metadata = { 'context-independent content': browse_content_link, 'path': None, 'filename': None, 'revision id': revision_id, 'revision': browse_rev_link, 'snapshot id': snapshot_id } cnt_sha1_git = None content_size = None error_code = 200 error_description = '' error_message = '' if content_data: content_metadata['sha1 checksum'] = \ content_data['checksums']['sha1'] content_metadata['sha1_git checksum'] = \ content_data['checksums']['sha1_git'] content_metadata['sha256 checksum'] = \ content_data['checksums']['sha256'] content_metadata['blake2s256 checksum'] = \ content_data['checksums']['blake2s256'] content_metadata['mime type'] = content_data['mimetype'] content_metadata['encoding'] = content_data['encoding'] content_metadata['size'] = filesizeformat(content_data['length']) content_metadata['language'] = content_data['language'] content_metadata['licenses'] = content_data['licenses'] content_metadata['path'] = '/' + path[:-len(filename)] content_metadata['filename'] = filename cnt_sha1_git = content_data['checksums']['sha1_git'] content_size = content_data['length'] error_code = content_data['error_code'] error_message = content_data['error_message'] error_description = content_data['error_description'] if origin_info: content_metadata['origin id'] = origin_info['id'] content_metadata['origin type'] = origin_info['type'] content_metadata['origin url'] = origin_info['url'] content_metadata['origin visit date'] = format_utc_iso_date(visit_info['date']) # noqa content_metadata['origin visit id'] = visit_info['visit'] browse_snapshot_url = reverse('browse-snapshot-content', url_args={'snapshot_id': snapshot_id, 'path': path}, query_params=request.GET) browse_snapshot_link = \ gen_link(browse_snapshot_url, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) content_metadata['snapshot context'] = browse_snapshot_link swh_objects = [{'type': 'content', 'id': cnt_sha1_git}, {'type': 'revision', 'id': revision_id}, {'type': 'snapshot', 'id': snapshot_id}] release_id = snapshot_context['release_id'] if release_id: swh_objects.append({'type': 'release', 'id': release_id}) swh_ids = get_swh_persistent_ids(swh_objects, snapshot_context) content_path = '/'.join([bc['name'] for bc in breadcrumbs]) context_found = 'snapshot: %s' % snapshot_context['snapshot_id'] if origin_info: context_found = 'origin: %s' % origin_info['url'] heading = 'Content - %s - %s - %s' %\ (content_path, snapshot_context['branch'], context_found) return render(request, 'browse/content.html', {'heading': heading, 'swh_object_name': 'Content', 'swh_object_metadata': content_metadata, 'content': content, 'content_size': content_size, 'max_content_size': content_display_max_size, 'mimetype': mimetype, 'language': language, 'breadcrumbs': breadcrumbs if root_sha1_git else [], 'top_right_link': { 'url': content_raw_url, 'icon': swh_object_icons['content'], 'text': 'Raw File' }, 'snapshot_context': snapshot_context, 'vault_cooking': None, 'show_actions_menu': True, 'swh_ids': swh_ids, 'error_code': error_code, 'error_message': error_message, 'error_description': error_description}, status=error_code) PER_PAGE = 100 def browse_snapshot_log(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None): """ Django view implementation for browsing a revision history in a snapshot context. """ try: snapshot_context = _process_snapshot_request(request, snapshot_id, origin_type, origin_url, timestamp, browse_context='log') # noqa revision_id = snapshot_context['revision_id'] per_page = int(request.GET.get('per_page', PER_PAGE)) offset = int(request.GET.get('offset', 0)) revs_ordering = request.GET.get('revs_ordering', 'committer_date') session_key = 'rev_%s_log_ordering_%s' % (revision_id, revs_ordering) rev_log_session = request.session.get(session_key, None) rev_log = [] revs_walker_state = None if rev_log_session: rev_log = rev_log_session['rev_log'] revs_walker_state = rev_log_session['revs_walker_state'] if len(rev_log) < offset+per_page: revs_walker = \ service.get_revisions_walker(revs_ordering, revision_id, max_revs=offset+per_page+1, state=revs_walker_state) rev_log += list(revs_walker) revs_walker_state = revs_walker.export_state() revision_log = rev_log[offset:offset+per_page] request.session[session_key] = { 'rev_log': rev_log, 'revs_walker_state': revs_walker_state } except Exception as exc: return handle_view_exception(request, exc) swh_type = snapshot_context['swh_type'] origin_info = snapshot_context['origin_info'] visit_info = snapshot_context['visit_info'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] snapshot_id = snapshot_context['snapshot_id'] query_params['per_page'] = per_page revs_ordering = request.GET.get('revs_ordering', '') query_params['revs_ordering'] = revs_ordering browse_view_name = 'browse-' + swh_type + '-log' prev_log_url = None if len(rev_log) > offset + per_page: query_params['offset'] = offset + per_page prev_log_url = reverse(browse_view_name, url_args=url_args, query_params=query_params) next_log_url = None if offset != 0: query_params['offset'] = offset - per_page next_log_url = reverse(browse_view_name, url_args=url_args, query_params=query_params) revision_log_data = format_log_entries(revision_log, per_page, snapshot_context) browse_log_link = \ gen_revision_log_link(revision_id, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) revision_metadata = { 'context-independent revision history': browse_log_link, 'snapshot id': snapshot_id } if origin_info: revision_metadata['origin id'] = origin_info['id'] revision_metadata['origin type'] = origin_info['type'] revision_metadata['origin url'] = origin_info['url'] revision_metadata['origin visit date'] = format_utc_iso_date(visit_info['date']) # noqa revision_metadata['origin visit id'] = visit_info['visit'] browse_snapshot_url = reverse('browse-snapshot-log', url_args={'snapshot_id': snapshot_id}, query_params=request.GET) browse_snapshot_link = \ gen_link(browse_snapshot_url, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) revision_metadata['snapshot context'] = browse_snapshot_link swh_objects = [{'type': 'revision', 'id': revision_id}, {'type': 'snapshot', 'id': snapshot_id}] release_id = snapshot_context['release_id'] if release_id: swh_objects.append({'type': 'release', 'id': release_id}) swh_ids = get_swh_persistent_ids(swh_objects, snapshot_context) context_found = 'snapshot: %s' % snapshot_context['snapshot_id'] if origin_info: context_found = 'origin: %s' % origin_info['url'] heading = 'Revision history - %s - %s' %\ (snapshot_context['branch'], context_found) return render(request, 'browse/revision-log.html', {'heading': heading, 'swh_object_name': 'Revisions history', 'swh_object_metadata': revision_metadata, 'revision_log': revision_log_data, 'revs_ordering': revs_ordering, 'next_log_url': next_log_url, 'prev_log_url': prev_log_url, 'breadcrumbs': None, 'top_right_link': None, 'snapshot_context': snapshot_context, 'vault_cooking': None, 'show_actions_menu': True, 'swh_ids': swh_ids}) def browse_snapshot_branches(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None): """ Django view implementation for browsing a list of branches in a snapshot context. """ try: snapshot_context = _process_snapshot_request(request, snapshot_id, origin_type, origin_url, timestamp) branches_bc = request.GET.get('branches_breadcrumbs', '') branches_bc = \ branches_bc.split(',') if branches_bc else [] branches_from = branches_bc[-1] if branches_bc else '' swh_type = snapshot_context['swh_type'] origin_info = snapshot_context['origin_info'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] browse_view_name = 'browse-' + swh_type + '-directory' displayed_branches = \ service.lookup_snapshot(snapshot_context['snapshot_id'], branches_from, PER_PAGE+1, target_types=['revision'])['branches'] except Exception as exc: return handle_view_exception(request, exc) displayed_branches, _ = process_snapshot_branches(displayed_branches) for branch in displayed_branches: if snapshot_id: revision_url = reverse('browse-revision', url_args={'sha1_git': branch['revision']}, query_params={'snapshot_id': snapshot_id}) else: revision_url = reverse('browse-revision', url_args={'sha1_git': branch['revision']}, query_params={'origin_type': origin_type, 'origin': origin_info['url']}) query_params['branch'] = branch['name'] directory_url = reverse(browse_view_name, url_args=url_args, query_params=query_params) del query_params['branch'] branch['revision_url'] = revision_url branch['directory_url'] = directory_url browse_view_name = 'browse-' + swh_type + '-branches' prev_branches_url = None next_branches_url = None if branches_bc: query_params_prev = dict(query_params) query_params_prev['branches_breadcrumbs'] = \ ','.join(branches_bc[:-1]) prev_branches_url = reverse(browse_view_name, url_args=url_args, query_params=query_params_prev) elif branches_from: prev_branches_url = reverse(browse_view_name, url_args=url_args, query_params=query_params) if len(displayed_branches) > PER_PAGE: query_params_next = dict(query_params) next_branch = displayed_branches[-1]['name'] del displayed_branches[-1] branches_bc.append(next_branch) query_params_next['branches_breadcrumbs'] = \ ','.join(branches_bc) next_branches_url = reverse(browse_view_name, url_args=url_args, query_params=query_params_next) heading = 'Branches - ' if origin_info: heading += 'origin: %s' % origin_info['url'] else: heading += 'snapshot: %s' % snapshot_id return render(request, 'browse/branches.html', {'heading': heading, 'swh_object_name': 'Branches', 'swh_object_metadata': {}, 'top_right_link': None, 'displayed_branches': displayed_branches, 'prev_branches_url': prev_branches_url, 'next_branches_url': next_branches_url, 'snapshot_context': snapshot_context}) def browse_snapshot_releases(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None): """ Django view implementation for browsing a list of releases in a snapshot context. """ try: snapshot_context = _process_snapshot_request(request, snapshot_id, origin_type, origin_url, timestamp) rel_bc = request.GET.get('releases_breadcrumbs', '') rel_bc = \ rel_bc.split(',') if rel_bc else [] rel_from = rel_bc[-1] if rel_bc else '' swh_type = snapshot_context['swh_type'] origin_info = snapshot_context['origin_info'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] displayed_releases = \ service.lookup_snapshot(snapshot_context['snapshot_id'], rel_from, PER_PAGE+1, target_types=['release'])['branches'] except Exception as exc: return handle_view_exception(request, exc) _, displayed_releases = process_snapshot_branches(displayed_releases) for release in displayed_releases: if snapshot_id: query_params_tgt = {'snapshot_id': snapshot_id} else: query_params_tgt = {'origin': origin_info['url']} release_url = reverse('browse-release', url_args={'sha1_git': release['id']}, query_params=query_params_tgt) target_url = '' if release['target_type'] == 'revision': target_url = reverse('browse-revision', url_args={'sha1_git': release['target']}, query_params=query_params_tgt) elif release['target_type'] == 'directory': target_url = reverse('browse-directory', url_args={'sha1_git': release['target']}, query_params=query_params_tgt) elif release['target_type'] == 'content': target_url = reverse('browse-content', url_args={'sha1_git': release['target']}, query_params=query_params_tgt) elif release['target_type'] == 'release': target_url = reverse('browse-release', url_args={'sha1_git': release['target']}, query_params=query_params_tgt) release['release_url'] = release_url release['target_url'] = target_url browse_view_name = 'browse-' + swh_type + '-releases' prev_releases_url = None next_releases_url = None if rel_bc: query_params_prev = dict(query_params) query_params_prev['releases_breadcrumbs'] = \ ','.join(rel_bc[:-1]) prev_releases_url = reverse(browse_view_name, url_args=url_args, query_params=query_params_prev) elif rel_from: prev_releases_url = reverse(browse_view_name, url_args=url_args, query_params=query_params) if len(displayed_releases) > PER_PAGE: query_params_next = dict(query_params) next_rel = displayed_releases[-1]['branch_name'] del displayed_releases[-1] rel_bc.append(next_rel) query_params_next['releases_breadcrumbs'] = \ ','.join(rel_bc) next_releases_url = reverse(browse_view_name, url_args=url_args, query_params=query_params_next) heading = 'Releases - ' if origin_info: heading += 'origin: %s' % origin_info['url'] else: heading += 'snapshot: %s' % snapshot_id return render(request, 'browse/releases.html', {'heading': heading, 'top_panel_visible': False, 'top_panel_collapsible': False, 'swh_object_name': 'Releases', 'swh_object_metadata': {}, 'top_right_link': None, 'displayed_releases': displayed_releases, 'prev_releases_url': prev_releases_url, 'next_releases_url': next_releases_url, 'snapshot_context': snapshot_context, 'vault_cooking': None, 'show_actions_menu': False}) diff --git a/swh/web/common/converters.py b/swh/web/common/converters.py index 0e988bdd8..08d53e24f 100644 --- a/swh/web/common/converters.py +++ b/swh/web/common/converters.py @@ -1,382 +1,382 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import datetime import json from swh.model import hashutil from swh.core.utils import decode_with_escape def _group_checksums(data): """Groups checksums values computed from hash functions used in swh and stored in data dict under a single entry 'checksums' """ if data: checksums = {} for hash in hashutil.ALGORITHMS: if hash in data and data[hash]: checksums[hash] = data[hash] del data[hash] if len(checksums) > 0: data['checksums'] = checksums def fmap(f, data): """Map f to data at each level. This must keep the origin data structure type: - map -> map - dict -> dict - list -> list - None -> None Args: f: function that expects one argument. data: data to traverse to apply the f function. list, map, dict or bare value. Returns: The same data-structure with modified values by the f function. """ if data is None: return data if isinstance(data, map): return map(lambda y: fmap(f, y), (x for x in data)) if isinstance(data, list): return [fmap(f, x) for x in data] if isinstance(data, dict): return {k: fmap(f, v) for (k, v) in data.items()} return f(data) def from_swh(dict_swh, hashess={}, bytess={}, dates={}, blacklist={}, removables_if_empty={}, empty_dict={}, empty_list={}, convert={}, convert_fn=lambda x: x): - """Convert from an swh dictionary to something reasonably json + """Convert from a swh dictionary to something reasonably json serializable. Args: dict_swh: the origin dictionary needed to be transformed hashess: list/set of keys representing hashes values (sha1, sha256, sha1_git, etc...) as bytes. Those need to be transformed in hexadecimal string bytess: list/set of keys representing bytes values which needs to be decoded blacklist: set of keys to filter out from the conversion convert: set of keys whose associated values need to be converted using convert_fn convert_fn: the conversion function to apply on the value of key in 'convert' The remaining keys are copied as is in the output. Returns: dictionary equivalent as dict_swh only with its keys converted. """ def convert_hashes_bytes(v): """v is supposedly a hash as bytes, returns it converted in hex. """ if isinstance(v, bytes): return hashutil.hash_to_hex(v) return v def convert_bytes(v): """v is supposedly a bytes string, decode as utf-8. FIXME: Improve decoding policy. If not utf-8, break! """ if isinstance(v, bytes): return v.decode('utf-8') return v def convert_date(v): """ Args: v (dict or datatime): either: - a dict with three keys: - timestamp (dict or integer timestamp) - offset - negative_utc - or, a datetime We convert it to a human-readable string """ if not v: return v if isinstance(v, datetime.datetime): return v.isoformat() tz = datetime.timezone(datetime.timedelta(minutes=v['offset'])) swh_timestamp = v['timestamp'] if isinstance(swh_timestamp, dict): date = datetime.datetime.fromtimestamp( swh_timestamp['seconds'], tz=tz) else: date = datetime.datetime.fromtimestamp( swh_timestamp, tz=tz) datestr = date.isoformat() if v['offset'] == 0 and v['negative_utc']: # remove the rightmost + and replace it with a - return '-'.join(datestr.rsplit('+', 1)) return datestr if not dict_swh: return dict_swh new_dict = {} for key, value in dict_swh.items(): if key in blacklist or (key in removables_if_empty and not value): continue if key in dates: new_dict[key] = convert_date(value) elif key in convert: new_dict[key] = convert_fn(value) elif isinstance(value, dict): new_dict[key] = from_swh(value, hashess=hashess, bytess=bytess, dates=dates, blacklist=blacklist, removables_if_empty=removables_if_empty, empty_dict=empty_dict, empty_list=empty_list, convert=convert, convert_fn=convert_fn) elif key in hashess: new_dict[key] = fmap(convert_hashes_bytes, value) elif key in bytess: try: new_dict[key] = fmap(convert_bytes, value) except UnicodeDecodeError: if 'decoding_failures' not in new_dict: new_dict['decoding_failures'] = [key] else: new_dict['decoding_failures'].append(key) new_dict[key] = fmap(decode_with_escape, value) elif key in empty_dict and not value: new_dict[key] = {} elif key in empty_list and not value: new_dict[key] = [] else: new_dict[key] = value _group_checksums(new_dict) return new_dict def from_provenance(provenance): - """Convert from a provenance information to a provenance dictionary. + """Convert from a swh provenance information to a provenance dictionary. Args: provenance (dict): Dictionary with the following keys: - content (sha1_git): the content's identifier - revision (sha1_git): the revision the content was seen - origin (int): the origin the content was seen - visit (int): the visit it occurred - path (bytes): the path the content was seen at """ return from_swh(provenance, hashess={'content', 'revision'}, bytess={'path'}) def from_origin(origin): - """Convert from an SWH origin to an origin dictionary. + """Convert from a swh origin to an origin dictionary. """ return from_swh(origin) def from_release(release): - """Convert from an SWH release to a json serializable release dictionary. + """Convert from a swh release to a json serializable release dictionary. Args: release (dict): dictionary with keys: - id: identifier of the revision (sha1 in bytes) - revision: identifier of the revision the release points to (sha1 in bytes) comment: release's comment message (bytes) name: release's name (string) author: release's author identifier (swh's id) synthetic: the synthetic property (boolean) Returns: dict: Release dictionary with the following keys: - id: hexadecimal sha1 (string) - revision: hexadecimal sha1 (string) - comment: release's comment message (string) - name: release's name (string) - author: release's author identifier (swh's id) - synthetic: the synthetic property (boolean) """ return from_swh( release, hashess={'id', 'target'}, bytess={'message', 'name', 'fullname', 'email'}, dates={'date'}, ) class SWHMetadataEncoder(json.JSONEncoder): """Special json encoder for metadata field which can contain bytes encoded value. """ def default(self, obj): if isinstance(obj, bytes): return obj.decode('utf-8') # Let the base class default method raise the TypeError return json.JSONEncoder.default(self, obj) def convert_revision_metadata(metadata): """Convert json specific dict to a json serializable one. """ if not metadata: return {} return json.loads(json.dumps(metadata, cls=SWHMetadataEncoder)) def from_revision(revision): - """Convert from an SWH revision to a json serializable revision dictionary. + """Convert from a swh revision to a json serializable revision dictionary. Args: revision (dict): dict with keys: - id: identifier of the revision (sha1 in bytes) - directory: identifier of the directory the revision points to (sha1 in bytes) - author_name, author_email: author's revision name and email - committer_name, committer_email: committer's revision name and email - message: revision's message - date, date_offset: revision's author date - committer_date, committer_date_offset: revision's commit date - parents: list of parents for such revision - synthetic: revision's property nature - type: revision's type (git, tar or dsc at the moment) - metadata: if the revision is synthetic, this can reference dynamic properties. Returns: dict: Revision dictionary with the same keys as inputs, except: - sha1s are in hexadecimal strings (id, directory) - bytes are decoded in string (author_name, committer_name, author_email, committer_email) Remaining keys are left as is """ revision = from_swh(revision, hashess={'id', 'directory', 'parents', 'children'}, bytess={'name', 'fullname', 'email'}, convert={'metadata'}, convert_fn=convert_revision_metadata, dates={'date', 'committer_date'}) if revision: if 'parents' in revision: revision['merge'] = len(revision['parents']) > 1 if 'message' in revision: try: revision['message'] = revision['message'].decode('utf-8') except UnicodeDecodeError: revision['message_decoding_failed'] = True revision['message'] = None return revision def from_content(content): """Convert swh content to serializable content dictionary. """ return from_swh(content, hashess={'sha1', 'sha1_git', 'sha256', 'blake2s256'}, blacklist={'ctime'}, convert={'status'}, convert_fn=lambda v: 'absent' if v == 'hidden' else v) def from_person(person): """Convert swh person to serializable person dictionary. """ return from_swh(person, bytess={'name', 'fullname', 'email'}) def from_origin_visit(visit): """Convert swh origin_visit to serializable origin_visit dictionary. """ ov = from_swh(visit, hashess={'target', 'snapshot'}, bytess={'branch'}, dates={'date'}, empty_dict={'metadata'}) return ov def from_snapshot(snapshot): """Convert swh snapshot to serializable snapshot dictionary. """ sv = from_swh(snapshot, hashess={'id', 'target'}, bytess={'next_branch'}) if sv and 'branches' in sv: sv['branches'] = { decode_with_escape(k): v for k, v in sv['branches'].items() } return sv def from_directory_entry(dir_entry): """Convert swh person to serializable person dictionary. """ return from_swh(dir_entry, hashess={'dir_id', 'sha1_git', 'sha1', 'sha256', 'blake2s256', 'target'}, bytess={'name'}, removables_if_empty={ 'sha1', 'sha1_git', 'sha256', 'blake2s256', 'status'}, convert={'status'}, convert_fn=lambda v: 'absent' if v == 'hidden' else v) def from_filetype(content_entry): """Convert swh person to serializable person dictionary. """ return from_swh(content_entry, hashess={'id'}, bytess={'mimetype', 'encoding'}) diff --git a/swh/web/common/origin_save.py b/swh/web/common/origin_save.py index 24529f60d..2682f40a7 100644 --- a/swh/web/common/origin_save.py +++ b/swh/web/common/origin_save.py @@ -1,354 +1,355 @@ # Copyright (C) 2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from bisect import bisect_right from django.core.exceptions import ObjectDoesNotExist from django.core.exceptions import ValidationError from django.core.validators import URLValidator from swh.web import config from swh.web.common import service from swh.web.common.exc import BadInputExc, ForbiddenExc from swh.web.common.models import ( SaveUnauthorizedOrigin, SaveAuthorizedOrigin, SaveOriginRequest, SAVE_REQUEST_ACCEPTED, SAVE_REQUEST_REJECTED, SAVE_REQUEST_PENDING, SAVE_TASK_NOT_YET_SCHEDULED, SAVE_TASK_SCHEDULED, SAVE_TASK_SUCCEED, SAVE_TASK_FAILED ) from swh.web.common.utils import get_origin_visits, parse_timestamp from swh.scheduler.utils import create_oneshot_task_dict scheduler = config.scheduler() def get_origin_save_authorized_urls(): """ Get the list of origin url prefixes authorized to be immediately loaded into the archive (whitelist). Returns: list: The list of authorized origin url prefix """ return [origin.url for origin in SaveAuthorizedOrigin.objects.all()] def get_origin_save_unauthorized_urls(): """ Get the list of origin url prefixes forbidden to be loaded into the archive (blacklist). Returns: list: the list of unauthorized origin url prefix """ return [origin.url for origin in SaveUnauthorizedOrigin.objects.all()] def can_save_origin(origin_url): """ Check if a software origin can be saved into the archive. Based on the origin url, the save request will be either: * immediately accepted if the url is whitelisted * rejected if the url is blacklisted * put in pending state for manual review otherwise Args: origin_url (str): the software origin url to check Returns: - str: the origin save request status, either *accepted*, - *rejected* or *pending* + str: the origin save request status, either **accepted**, + **rejected** or **pending** """ # origin url may be blacklisted for url_prefix in get_origin_save_unauthorized_urls(): if origin_url.startswith(url_prefix): return SAVE_REQUEST_REJECTED # if the origin url is in the white list, it can be immediately saved for url_prefix in get_origin_save_authorized_urls(): if origin_url.startswith(url_prefix): return SAVE_REQUEST_ACCEPTED # otherwise, the origin url needs to be manually verified return SAVE_REQUEST_PENDING # map origin type to scheduler task # TODO: do not hardcode the task name here # TODO: unlock hg and svn loading once the scheduler # loading tasks are available in production _origin_type_task = { 'git': 'origin-update-git', # 'hg': 'origin-load-hg', # 'svn': 'origin-load-svn' } # map scheduler task status to origin save status _save_task_status = { 'next_run_not_scheduled': SAVE_TASK_NOT_YET_SCHEDULED, 'next_run_scheduled': SAVE_TASK_SCHEDULED, 'completed': SAVE_TASK_SUCCEED, 'disabled': SAVE_TASK_FAILED } def get_savable_origin_types(): return sorted(list(_origin_type_task.keys())) def _check_origin_type_savable(origin_type): """ Get the list of software origin types that can be loaded through a save request. Returns: list: the list of saveable origin types """ allowed_origin_types = ', '.join(get_savable_origin_types()) if origin_type not in _origin_type_task: raise BadInputExc('Origin of type %s can not be saved! ' 'Allowed types are the following: %s' % (origin_type, allowed_origin_types)) _validate_url = URLValidator(schemes=['http', 'https', 'svn', 'git']) def _check_origin_url_valid(origin_url): try: _validate_url(origin_url) except ValidationError: raise BadInputExc('The provided origin url (%s) is not valid!' % origin_url) def _get_visit_date_for_save_request(save_request): visit_date = None try: origin = {'type': save_request.origin_type, 'url': save_request.origin_url} origin_info = service.lookup_origin(origin) origin_visits = get_origin_visits(origin_info) visit_dates = [parse_timestamp(v['date']) for v in origin_visits] i = bisect_right(visit_dates, save_request.request_date) if i != len(visit_dates): visit_date = visit_dates[i] except Exception: pass return visit_date def _save_request_dict(save_request, task=None): must_save = False visit_date = save_request.visit_date if task: save_task_status = _save_task_status[task['status']] if save_task_status in (SAVE_TASK_FAILED, SAVE_TASK_SUCCEED) \ and not visit_date: visit_date = _get_visit_date_for_save_request(save_request) save_request.visit_date = visit_date must_save = True # Ensure last origin visit is available in database # before reporting the task execution as successful if save_task_status == SAVE_TASK_SUCCEED and not visit_date: save_task_status = SAVE_TASK_SCHEDULED if save_request.loading_task_status != save_task_status: save_request.loading_task_status = save_task_status must_save = True if must_save: save_request.save() else: save_task_status = save_request.loading_task_status return {'origin_type': save_request.origin_type, 'origin_url': save_request.origin_url, 'save_request_date': save_request.request_date.isoformat(), 'save_request_status': save_request.status, 'save_task_status': save_task_status, 'visit_date': visit_date.isoformat() if visit_date else None} def create_save_origin_request(origin_type, origin_url): """ Create a loading task to save a software origin into the archive. This function aims to create a software origin loading task trough the use of the swh-scheduler component. First, some checks are performed to see if the origin type and url are valid but also if the the save request can be accepted. If those checks passed, the loading task is then created. Otherwise, the save request is put in pending or rejected state. All the submitted save requests are logged into the swh-web database to keep track of them. Args: - origin_type (str): the type of origin to save (*git*, *hg*, *svn*, ...) + origin_type (str): the type of origin to save (currently only + ``git`` but ``svn`` and ``hg`` will soon be available) origin_url (str): the url of the origin to save Raises: BadInputExc: the origin type or url is invalid ForbiddenExc: the provided origin url is blacklisted Returns: dict: A dict describing the save request with the following keys: * **origin_type**: the type of the origin to save * **origin_url**: the url of the origin * **save_request_date**: the date the request was submitted - * **save_request_status**: the request status, either *accepted*, - *rejected* or *pending* + * **save_request_status**: the request status, either **accepted**, + **rejected** or **pending** * **save_task_status**: the origin loading task status, either - *not created*, *not yet scheduled*, *scheduled*, *succeed* or - *failed* + **not created**, **not yet scheduled**, **scheduled**, + **succeed** or **failed** """ _check_origin_type_savable(origin_type) _check_origin_url_valid(origin_url) save_request_status = can_save_origin(origin_url) task = None # if the origin save request is accepted, create a scheduler # task to load it into the archive if save_request_status == SAVE_REQUEST_ACCEPTED: # create a task with high priority kwargs = {'priority': 'high'} # set task parameters according to the origin type if origin_type == 'git': kwargs['repo_url'] = origin_url elif origin_type == 'hg': kwargs['origin_url'] = origin_url elif origin_type == 'svn': kwargs['origin_url'] = origin_url kwargs['svn_url'] = origin_url sor = None # get list of previously sumitted save requests current_sors = \ list(SaveOriginRequest.objects.filter(origin_type=origin_type, origin_url=origin_url)) can_create_task = False # if no save requests previously submitted, create the scheduler task if not current_sors: can_create_task = True else: # get the latest submitted save request sor = current_sors[0] # if it was in pending state, we need to create the scheduler task # and update the save request info in the database if sor.status == SAVE_REQUEST_PENDING: can_create_task = True # a task has already been created to load the origin elif sor.loading_task_id != -1: # get the scheduler task and its status tasks = scheduler.get_tasks([sor.loading_task_id]) task = tasks[0] if tasks else None task_status = _save_request_dict(sor, task)['save_task_status'] # create a new scheduler task only if the previous one has been # already executed if task_status == SAVE_TASK_FAILED or \ task_status == SAVE_TASK_SUCCEED: can_create_task = True sor = None else: can_create_task = False if can_create_task: # effectively create the scheduler task task_dict = create_oneshot_task_dict( _origin_type_task[origin_type], **kwargs) task = scheduler.create_tasks([task_dict])[0] # pending save request has been accepted if sor: sor.status = SAVE_REQUEST_ACCEPTED sor.loading_task_id = task['id'] sor.save() else: sor = SaveOriginRequest.objects.create(origin_type=origin_type, origin_url=origin_url, status=save_request_status, # noqa loading_task_id=task['id']) # noqa # save request must be manually reviewed for acceptation elif save_request_status == SAVE_REQUEST_PENDING: # check if there is already such a save request already submitted, # no need to add it to the database in that case try: sor = SaveOriginRequest.objects.get(origin_type=origin_type, origin_url=origin_url, status=save_request_status) # if not add it to the database except ObjectDoesNotExist: sor = SaveOriginRequest.objects.create(origin_type=origin_type, origin_url=origin_url, status=save_request_status) # origin can not be saved as its url is blacklisted, # log the request to the database anyway else: sor = SaveOriginRequest.objects.create(origin_type=origin_type, origin_url=origin_url, status=save_request_status) if save_request_status == SAVE_REQUEST_REJECTED: raise ForbiddenExc('The origin url is blacklisted and will not be ' 'loaded into the archive.') return _save_request_dict(sor, task) def get_save_origin_requests_from_queryset(requests_queryset): """ Get all save requests from a SaveOriginRequest queryset. Args: requests_queryset (django.db.models.QuerySet): input SaveOriginRequest queryset Returns: list: A list of save origin requests dict as described in :func:`swh.web.common.origin_save.create_save_origin_request` """ task_ids = [] for sor in requests_queryset: task_ids.append(sor.loading_task_id) tasks = scheduler.get_tasks(task_ids) tasks = {task['id']: task for task in tasks} requests = [] for sor in requests_queryset: sr_dict = _save_request_dict(sor, tasks.get(sor.loading_task_id, None)) requests.append(sr_dict) return requests def get_save_origin_requests(origin_type, origin_url): """ Get all save requests for a given software origin. Args: origin_type (str): the type of the origin origin_url (str): the url of the origin Raises: BadInputExc: the origin type or url is invalid Returns: list: A list of save origin requests dict as described in :func:`swh.web.common.origin_save.create_save_origin_request` """ _check_origin_type_savable(origin_type) _check_origin_url_valid(origin_url) sors = SaveOriginRequest.objects.filter(origin_type=origin_type, origin_url=origin_url) return get_save_origin_requests_from_queryset(sors) diff --git a/swh/web/common/service.py b/swh/web/common/service.py index ddcc78181..32597a5f0 100644 --- a/swh/web/common/service.py +++ b/swh/web/common/service.py @@ -1,1029 +1,1029 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import os from collections import defaultdict from swh.model import hashutil from swh.storage.algos import revisions_walker from swh.web.common import converters from swh.web.common import query from swh.web.common.exc import NotFoundExc from swh.web import config storage = config.storage() vault = config.vault() idx_storage = config.indexer_storage() MAX_LIMIT = 50 # Top limit the users can ask for def _first_element(l): """Returns the first element in the provided list or None if it is empty or None""" return next(iter(l or []), None) def lookup_multiple_hashes(hashes): """Lookup the passed hashes in a single DB connection, using batch processing. Args: An array of {filename: X, sha1: Y}, string X, hex sha1 string Y. Returns: The same array with elements updated with elem['found'] = true if the hash is present in storage, elem['found'] = false if not. """ hashlist = [hashutil.hash_to_bytes(elem['sha1']) for elem in hashes] content_missing = storage.content_missing_per_sha1(hashlist) missing = [hashutil.hash_to_hex(x) for x in content_missing] for x in hashes: x.update({'found': True}) for h in hashes: if h['sha1'] in missing: h['found'] = False return hashes def lookup_expression(expression, last_sha1, per_page): """Lookup expression in raw content. Args: expression (str): An expression to lookup through raw indexed content last_sha1 (str): Last sha1 seen per_page (int): Number of results per page Returns: List of ctags whose content match the expression """ limit = min(per_page, MAX_LIMIT) ctags = idx_storage.content_ctags_search(expression, last_sha1=last_sha1, limit=limit) for ctag in ctags: ctag = converters.from_swh(ctag, hashess={'id'}) ctag['sha1'] = ctag['id'] ctag.pop('id') yield ctag def lookup_hash(q): """Checks if the storage contains a given content checksum Args: query string of the form Returns: Dict with key found containing the hash info if the hash is present, None if not. """ algo, hash = query.parse_hash(q) found = storage.content_find({algo: hash}) return {'found': found, 'algo': algo} def search_hash(q): """Checks if the storage contains a given content checksum Args: query string of the form Returns: Dict with key found to True or False, according to whether the checksum is present or not """ algo, hash = query.parse_hash(q) found = storage.content_find({algo: hash}) return {'found': found is not None} def lookup_content_provenance(q): """Return provenance information from a specified content. Args: q: query string of the form Yields: provenance information (dict) list if the content is found. """ algo, hash = query.parse_hash(q) provenances = storage.content_find_provenance({algo: hash}) if not provenances: return None return (converters.from_provenance(p) for p in provenances) def _lookup_content_sha1(q): """Given a possible input, query for the content's sha1. Args: q: query string of the form Returns: binary sha1 if found or None """ algo, hash = query.parse_hash(q) if algo != 'sha1': hashes = storage.content_find({algo: hash}) if not hashes: return None return hashes['sha1'] return hash def lookup_content_ctags(q): """Return ctags information from a specified content. Args: q: query string of the form Yields: ctags information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None ctags = list(idx_storage.content_ctags_get([sha1])) if not ctags: return None for ctag in ctags: yield converters.from_swh(ctag, hashess={'id'}) def lookup_content_filetype(q): """Return filetype information from a specified content. Args: q: query string of the form Yields: filetype information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None filetype = _first_element(list(idx_storage.content_mimetype_get([sha1]))) if not filetype: return None return converters.from_filetype(filetype) def lookup_content_language(q): """Return language information from a specified content. Args: q: query string of the form Yields: language information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None lang = _first_element(list(idx_storage.content_language_get([sha1]))) if not lang: return None return converters.from_swh(lang, hashess={'id'}) def lookup_content_license(q): """Return license information from a specified content. Args: q: query string of the form Yields: license information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None lic = _first_element(idx_storage.content_fossology_license_get([sha1])) if not lic: return None return converters.from_swh({'id': sha1, 'facts': lic[sha1]}, hashess={'id'}) def lookup_origin(origin): """Return information about the origin matching dict origin. Args: origin: origin's dict with keys either 'id' or ('type' AND 'url') Returns: origin information as dict. """ origin_info = storage.origin_get(origin) if not origin_info: if 'id' in origin and origin['id']: msg = 'Origin with id %s not found!' % origin['id'] else: msg = 'Origin with type %s and url %s not found!' % \ (origin['type'], origin['url']) raise NotFoundExc(msg) return converters.from_origin(origin_info) def search_origin(url_pattern, offset=0, limit=50, regexp=False, with_visit=False): """Search for origins whose urls contain a provided string pattern or match a provided regular expression. Args: url_pattern: the string pattern to search for in origin urls offset: number of found origins to skip before returning results limit: the maximum number of found origins to return Returns: list of origin information as dict. """ origins = storage.origin_search(url_pattern, offset, limit, regexp, with_visit) return map(converters.from_origin, origins) def lookup_person(person_id): """Return information about the person with id person_id. Args: person_id as string Returns: person information as dict. Raises: NotFoundExc if there is no person with the provided id. """ person = _first_element(storage.person_get([person_id])) if not person: raise NotFoundExc('Person with id %s not found' % person_id) return converters.from_person(person) def _to_sha1_bin(sha1_hex): _, sha1_git_bin = query.parse_hash_with_algorithms_or_throws( sha1_hex, ['sha1'], # HACK: sha1_git really 'Only sha1_git is supported.') return sha1_git_bin def lookup_directory(sha1_git): """Return information about the directory with id sha1_git. Args: sha1_git as string Returns: directory information as dict. """ empty_dir_sha1 = '4b825dc642cb6eb9a060e54bf8d69288fbee4904' if sha1_git == empty_dir_sha1: return [] sha1_git_bin = _to_sha1_bin(sha1_git) directory_entries = storage.directory_ls(sha1_git_bin) if directory_entries: return map(converters.from_directory_entry, directory_entries) else: raise NotFoundExc('Directory with sha1_git %s not found' % sha1_git) def lookup_directory_with_path(directory_sha1_git, path_string): """Return directory information for entry with path path_string w.r.t. root directory pointed by directory_sha1_git Args: - directory_sha1_git: sha1_git corresponding to the directory to which we append paths to (hopefully) find the entry - the relative path to the entry starting from the directory pointed by directory_sha1_git Raises: NotFoundExc if the directory entry is not found """ sha1_git_bin = _to_sha1_bin(directory_sha1_git) paths = path_string.strip(os.path.sep).split(os.path.sep) queried_dir = storage.directory_entry_get_by_path( sha1_git_bin, list(map(lambda p: p.encode('utf-8'), paths))) if not queried_dir: raise NotFoundExc(('Directory entry with path %s from %s not found') % (path_string, directory_sha1_git)) return converters.from_directory_entry(queried_dir) def lookup_release(release_sha1_git): """Return information about the release with sha1 release_sha1_git. Args: release_sha1_git: The release's sha1 as hexadecimal Returns: Release information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. """ sha1_git_bin = _to_sha1_bin(release_sha1_git) release = _first_element(storage.release_get([sha1_git_bin])) if not release: raise NotFoundExc('Release with sha1_git %s not found.' % release_sha1_git) return converters.from_release(release) def lookup_release_multiple(sha1_git_list): """Return information about the revisions identified with their sha1_git identifiers. Args: sha1_git_list: A list of revision sha1_git identifiers Returns: Release information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. """ sha1_bin_list = (_to_sha1_bin(sha1_git) for sha1_git in sha1_git_list) releases = storage.release_get(sha1_bin_list) or [] return (converters.from_release(r) for r in releases) def lookup_revision(rev_sha1_git): """Return information about the revision with sha1 revision_sha1_git. Args: revision_sha1_git: The revision's sha1 as hexadecimal Returns: Revision information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. NotFoundExc if there is no revision with the provided sha1_git. """ sha1_git_bin = _to_sha1_bin(rev_sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision with sha1_git %s not found.' % rev_sha1_git) return converters.from_revision(revision) def lookup_revision_multiple(sha1_git_list): """Return information about the revisions identified with their sha1_git identifiers. Args: sha1_git_list: A list of revision sha1_git identifiers Returns: Generator of revisions information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. """ sha1_bin_list = (_to_sha1_bin(sha1_git) for sha1_git in sha1_git_list) revisions = storage.revision_get(sha1_bin_list) or [] return (converters.from_revision(r) for r in revisions) def lookup_revision_message(rev_sha1_git): """Return the raw message of the revision with sha1 revision_sha1_git. Args: revision_sha1_git: The revision's sha1 as hexadecimal Returns: Decoded revision message as dict {'message': } Raises: ValueError if the identifier provided is not of sha1 nature. NotFoundExc if the revision is not found, or if it has no message """ sha1_git_bin = _to_sha1_bin(rev_sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision with sha1_git %s not found.' % rev_sha1_git) if 'message' not in revision: raise NotFoundExc('No message for revision with sha1_git %s.' % rev_sha1_git) res = {'message': revision['message']} return res def lookup_revision_by(origin_id, branch_name="refs/heads/master", timestamp=None): """Lookup revisions by origin_id, branch_name and timestamp. If: - branch_name is not provided, lookup using 'refs/heads/master' as default. - ts is not provided, use the most recent Args: - origin_id: origin of the revision. - branch_name: revision's branch. - timestamp: revision's time frame. Yields: The revisions matching the criterions. Raises: NotFoundExc if no revision corresponds to the criterion """ res = _first_element(storage.revision_get_by(origin_id, branch_name, timestamp=timestamp, limit=1)) if not res: raise NotFoundExc('Revision for origin %s and branch %s not found.' % (origin_id, branch_name)) return converters.from_revision(res) def lookup_revision_log(rev_sha1_git, limit): """Return information about the revision with sha1 revision_sha1_git. Args: revision_sha1_git: The revision's sha1 as hexadecimal limit: the maximum number of revisions returned Returns: Revision information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. NotFoundExc if there is no revision with the provided sha1_git. """ sha1_git_bin = _to_sha1_bin(rev_sha1_git) revision_entries = storage.revision_log([sha1_git_bin], limit) if not revision_entries: raise NotFoundExc('Revision with sha1_git %s not found.' % rev_sha1_git) return map(converters.from_revision, revision_entries) def lookup_revision_log_by(origin_id, branch_name, timestamp, limit): """Return information about the revision with sha1 revision_sha1_git. Args: origin_id: origin of the revision branch_name: revision's branch timestamp: revision's time frame limit: the maximum number of revisions returned Returns: Revision information as dict. Raises: NotFoundExc if no revision corresponds to the criterion """ revision_entries = storage.revision_log_by(origin_id, branch_name, timestamp, limit=limit) if not revision_entries: return None return map(converters.from_revision, revision_entries) def lookup_revision_with_context_by(origin_id, branch_name, ts, sha1_git, limit=100): """Return information about revision sha1_git, limited to the sub-graph of all transitive parents of sha1_git_root. sha1_git_root being resolved through the lookup of a revision by origin_id, branch_name and ts. In other words, sha1_git is an ancestor of sha1_git_root. Args: - origin_id: origin of the revision. - branch_name: revision's branch. - timestamp: revision's time frame. - sha1_git: one of sha1_git_root's ancestors. - limit: limit the lookup to 100 revisions back. Returns: Pair of (root_revision, revision). Information on sha1_git if it is an ancestor of sha1_git_root including children leading to sha1_git_root Raises: - BadInputExc in case of unknown algo_hash or bad hash. - NotFoundExc if either revision is not found or if sha1_git is not an ancestor of sha1_git_root. """ rev_root = _first_element(storage.revision_get_by(origin_id, branch_name, timestamp=ts, limit=1)) if not rev_root: raise NotFoundExc('Revision with (origin_id: %s, branch_name: %s' ', ts: %s) not found.' % (origin_id, branch_name, ts)) return (converters.from_revision(rev_root), lookup_revision_with_context(rev_root, sha1_git, limit)) def lookup_revision_with_context(sha1_git_root, sha1_git, limit=100): """Return information about revision sha1_git, limited to the sub-graph of all transitive parents of sha1_git_root. In other words, sha1_git is an ancestor of sha1_git_root. Args: sha1_git_root: latest revision. The type is either a sha1 (as an hex string) or a non converted dict. sha1_git: one of sha1_git_root's ancestors limit: limit the lookup to 100 revisions back Returns: Information on sha1_git if it is an ancestor of sha1_git_root including children leading to sha1_git_root Raises: BadInputExc in case of unknown algo_hash or bad hash NotFoundExc if either revision is not found or if sha1_git is not an ancestor of sha1_git_root """ sha1_git_bin = _to_sha1_bin(sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision %s not found' % sha1_git) if isinstance(sha1_git_root, str): sha1_git_root_bin = _to_sha1_bin(sha1_git_root) revision_root = _first_element(storage.revision_get([sha1_git_root_bin])) # noqa if not revision_root: raise NotFoundExc('Revision root %s not found' % sha1_git_root) else: sha1_git_root_bin = sha1_git_root['id'] revision_log = storage.revision_log([sha1_git_root_bin], limit) parents = {} children = defaultdict(list) for rev in revision_log: rev_id = rev['id'] parents[rev_id] = [] for parent_id in rev['parents']: parents[rev_id].append(parent_id) children[parent_id].append(rev_id) if revision['id'] not in parents: raise NotFoundExc('Revision %s is not an ancestor of %s' % (sha1_git, sha1_git_root)) revision['children'] = children[revision['id']] return converters.from_revision(revision) def lookup_directory_with_revision(sha1_git, dir_path=None, with_data=False): """Return information on directory pointed by revision with sha1_git. If dir_path is not provided, display top level directory. Otherwise, display the directory pointed by dir_path (if it exists). Args: sha1_git: revision's hash. dir_path: optional directory pointed to by that revision. with_data: boolean that indicates to retrieve the raw data if the path resolves to a content. Default to False (for the api) Returns: Information on the directory pointed to by that revision. Raises: BadInputExc in case of unknown algo_hash or bad hash. NotFoundExc either if the revision is not found or the path referenced does not exist. NotImplementedError in case of dir_path exists but do not reference a type 'dir' or 'file'. """ sha1_git_bin = _to_sha1_bin(sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision %s not found' % sha1_git) dir_sha1_git_bin = revision['directory'] if dir_path: paths = dir_path.strip(os.path.sep).split(os.path.sep) entity = storage.directory_entry_get_by_path( dir_sha1_git_bin, list(map(lambda p: p.encode('utf-8'), paths))) if not entity: raise NotFoundExc( "Directory or File '%s' pointed to by revision %s not found" % (dir_path, sha1_git)) else: entity = {'type': 'dir', 'target': dir_sha1_git_bin} if entity['type'] == 'dir': directory_entries = storage.directory_ls(entity['target']) or [] return {'type': 'dir', 'path': '.' if not dir_path else dir_path, 'revision': sha1_git, 'content': map(converters.from_directory_entry, directory_entries)} elif entity['type'] == 'file': # content content = storage.content_find({'sha1_git': entity['target']}) if with_data: c = _first_element(storage.content_get([content['sha1']])) content['data'] = c['data'] return {'type': 'file', 'path': '.' if not dir_path else dir_path, 'revision': sha1_git, 'content': converters.from_content(content)} else: raise NotImplementedError('Entity of type %s not implemented.' % entity['type']) def lookup_content(q): """Lookup the content designed by q. Args: q: The release's sha1 as hexadecimal Raises: NotFoundExc if the requested content is not found """ algo, hash = query.parse_hash(q) c = storage.content_find({algo: hash}) if not c: raise NotFoundExc('Content with %s checksum equals to %s not found!' % (algo, hashutil.hash_to_hex(hash))) return converters.from_content(c) def lookup_content_raw(q): """Lookup the content defined by q. Args: q: query string of the form Returns: dict with 'sha1' and 'data' keys. data representing its raw data decoded. Raises: NotFoundExc if the requested content is not found or if the content bytes are not available in the storage """ c = lookup_content(q) content = _first_element(storage.content_get([c['checksums']['sha1']])) if not content: algo, hash = query.parse_hash(q) raise NotFoundExc('Bytes of content with %s checksum equals to %s ' 'are not available!' % (algo, hashutil.hash_to_hex(hash))) return converters.from_content(content) def stat_counters(): """Return the stat counters for Software Heritage Returns: A dict mapping textual labels to integer values. """ return storage.stat_counters() def _lookup_origin_visits(origin_id, last_visit=None, limit=10): """Yields the origin origin_ids' visits. Args: origin_id (int): origin to list visits for last_visit (int): last visit to lookup from limit (int): Number of elements max to display Yields: Dictionaries of origin_visit for that origin """ limit = min(limit, MAX_LIMIT) yield from storage.origin_visit_get( origin_id, last_visit=last_visit, limit=limit) def lookup_origin_visits(origin_id, last_visit=None, per_page=10): """Yields the origin origin_ids' visits. Args: origin_id: origin to list visits for Yields: Dictionaries of origin_visit for that origin """ visits = _lookup_origin_visits(origin_id, last_visit=last_visit, limit=per_page) for visit in visits: yield converters.from_origin_visit(visit) def lookup_origin_visit(origin_id, visit_id): """Return information about visit visit_id with origin origin_id. Args: origin_id: origin concerned by the visit visit_id: the visit identifier to lookup Yields: The dict origin_visit concerned """ visit = storage.origin_visit_get_by(origin_id, visit_id) if not visit: raise NotFoundExc('Origin with id %s or its visit ' 'with id %s not found!' % (origin_id, visit_id)) return converters.from_origin_visit(visit) def lookup_snapshot_size(snapshot_id): """Count the number of branches in the snapshot with the given id Args: snapshot_id (str): sha1 identifier of the snapshot Returns: dict: A dict whose keys are the target types of branches and values their corresponding amount """ snapshot_id_bin = _to_sha1_bin(snapshot_id) snapshot_size = storage.snapshot_count_branches(snapshot_id_bin) if 'revision' not in snapshot_size: snapshot_size['revision'] = 0 if 'release' not in snapshot_size: snapshot_size['release'] = 0 return snapshot_size def lookup_snapshot(snapshot_id, branches_from='', branches_count=None, target_types=None): """Return information about a snapshot, aka the list of named branches found during a specific visit of an origin. Args: snapshot_id (str): sha1 identifier of the snapshot branches_from (str): optional parameter used to skip branches whose name is lesser than it before returning them branches_count (int): optional parameter used to restrain the amount of returned branches target_types (list): optional parameter used to filter the target types of branch to return (possible values that can be contained in that list are `'content', 'directory', 'revision', 'release', 'snapshot', 'alias'`) Returns: A dict filled with the snapshot content. """ snapshot_id_bin = _to_sha1_bin(snapshot_id) snapshot = storage.snapshot_get_branches(snapshot_id_bin, branches_from.encode(), branches_count, target_types) if not snapshot: raise NotFoundExc('Snapshot with id %s not found!' % snapshot_id) return converters.from_snapshot(snapshot) def lookup_latest_origin_snapshot(origin_id, allowed_statuses=None): """Return information about the latest snapshot of an origin. .. warning:: At most 1000 branches contained in the snapshot will be returned for performance reasons. Args: origin_id: integer identifier of the origin allowed_statuses: list of visit statuses considered to find the latest snapshot for the visit. For instance, ``allowed_statuses=['full']`` will only consider visits that have successfully run to completion. Returns: A dict filled with the snapshot content. """ snapshot = storage.snapshot_get_latest(origin_id, allowed_statuses) return converters.from_snapshot(snapshot) def lookup_entity_by_uuid(uuid): """Return the entity's hierarchy from its uuid. Args: uuid: entity's identifier. Returns: List of hierarchy entities from the entity with uuid. """ uuid = query.parse_uuid4(uuid) for entity in storage.entity_get(uuid): entity = converters.from_swh(entity, convert={'last_seen', 'uuid'}, convert_fn=lambda x: str(x)) yield entity def lookup_revision_through(revision, limit=100): """Retrieve a revision from the criterion stored in revision dictionary. Args: revision: Dictionary of criterion to lookup the revision with. Here are the supported combination of possible values: - origin_id, branch_name, ts, sha1_git - origin_id, branch_name, ts - sha1_git_root, sha1_git - sha1_git Returns: None if the revision is not found or the actual revision. """ if 'origin_id' in revision and \ 'branch_name' in revision and \ 'ts' in revision and \ 'sha1_git' in revision: return lookup_revision_with_context_by(revision['origin_id'], revision['branch_name'], revision['ts'], revision['sha1_git'], limit) if 'origin_id' in revision and \ 'branch_name' in revision and \ 'ts' in revision: return lookup_revision_by(revision['origin_id'], revision['branch_name'], revision['ts']) if 'sha1_git_root' in revision and \ 'sha1_git' in revision: return lookup_revision_with_context(revision['sha1_git_root'], revision['sha1_git'], limit) if 'sha1_git' in revision: return lookup_revision(revision['sha1_git']) # this should not happen raise NotImplementedError('Should not happen!') def lookup_directory_through_revision(revision, path=None, limit=100, with_data=False): """Retrieve the directory information from the revision. Args: revision: dictionary of criterion representing a revision to lookup path: directory's path to lookup. limit: optional query parameter to limit the revisions log (default to 100). For now, note that this limit could impede the transitivity conclusion about sha1_git not being an ancestor of. with_data: indicate to retrieve the content's raw data if path resolves to a content. Returns: The directory pointing to by the revision criterions at path. """ rev = lookup_revision_through(revision, limit) if not rev: raise NotFoundExc('Revision with criterion %s not found!' % revision) return (rev['id'], lookup_directory_with_revision(rev['id'], path, with_data)) def vault_cook(obj_type, obj_id, email=None): """Cook a vault bundle. """ return vault.cook(obj_type, obj_id, email=email) def vault_fetch(obj_type, obj_id): """Fetch a vault bundle. """ return vault.fetch(obj_type, obj_id) def vault_progress(obj_type, obj_id): """Get the current progress of a vault bundle. """ return vault.progress(obj_type, obj_id) def diff_revision(rev_id): """Get the list of file changes (insertion / deletion / modification / renaming) for a particular revision. """ rev_sha1_git_bin = _to_sha1_bin(rev_id) changes = storage.diff_revision(rev_sha1_git_bin, track_renaming=True) for change in changes: change['from'] = converters.from_directory_entry(change['from']) change['to'] = converters.from_directory_entry(change['to']) if change['from_path']: change['from_path'] = change['from_path'].decode('utf-8') if change['to_path']: change['to_path'] = change['to_path'].decode('utf-8') return changes class _RevisionsWalkerProxy(object): """ Proxy class wrapping a revisions walker iterator from swh-storage and performing needed conversions. """ def __init__(self, rev_walker_type, rev_start, *args, **kwargs): rev_start_bin = hashutil.hash_to_bytes(rev_start) self.revisions_walker = \ revisions_walker.get_revisions_walker(rev_walker_type, storage, rev_start_bin, *args, **kwargs) def export_state(self): return self.revisions_walker.export_state() def __next__(self): return converters.from_revision(next(self.revisions_walker)) def __iter__(self): return self def get_revisions_walker(rev_walker_type, rev_start, *args, **kwargs): """ Utility function to instantiate a revisions walker of a given type, see :mod:`swh.storage.algos.revisions_walker`. Args: rev_walker_type (str): the type of revisions walker to return, - possible values are: *committer_date*, *dfs*, *dfs_post*, - *bfs* and *path* + possible values are: ``committer_date``, ``dfs``, ``dfs_post``, + ``bfs`` and ``path`` rev_start (str): hexadecimal representation of a revision identifier args (list): position arguments to pass to the revisions walker constructor kwargs (dict): keyword arguments to pass to the revisions walker constructor """ # first check if the provided revision is valid lookup_revision(rev_start) return _RevisionsWalkerProxy(rev_walker_type, rev_start, *args, **kwargs) diff --git a/swh/web/common/utils.py b/swh/web/common/utils.py index 017e1d8bd..a9336bfc8 100644 --- a/swh/web/common/utils.py +++ b/swh/web/common/utils.py @@ -1,412 +1,413 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import docutils.parsers.rst import docutils.utils import re import requests from datetime import datetime, timezone from dateutil import parser as date_parser from dateutil import tz from django.core.cache import cache from django.urls import reverse as django_reverse from django.http import QueryDict from swh.model.exceptions import ValidationError from swh.model.identifiers import ( persistent_identifier, parse_persistent_identifier, CONTENT, DIRECTORY, RELEASE, REVISION, SNAPSHOT ) from swh.web.common import service from swh.web.common.exc import BadInputExc from swh.web.config import get_config swh_object_icons = { 'branch': 'fa fa-code-fork', 'branches': 'fa fa-code-fork', 'content': 'fa fa-file-text', 'directory': 'fa fa-folder', 'person': 'fa fa-user', 'revisions history': 'fa fa-history', 'release': 'fa fa-tag', 'releases': 'fa fa-tag', 'revision': 'octicon octicon-git-commit', 'snapshot': 'fa fa-camera', 'visits': 'fa fa-calendar', } def reverse(viewname, url_args=None, query_params=None, current_app=None, urlconf=None): """An override of django reverse function supporting query parameters. Args: viewname (str): the name of the django view from which to compute a url url_args (dict): dictionary of url arguments indexed by their names query_params (dict): dictionary of query parameters to append to the reversed url current_app (str): the name of the django app tighten to the view urlconf (str): url configuration module Returns: str: the url of the requested view with processed arguments and query parameters """ if url_args: url_args = {k: v for k, v in url_args.items() if v is not None} url = django_reverse(viewname, urlconf=urlconf, kwargs=url_args, current_app=current_app) if query_params: query_params = {k: v for k, v in query_params.items() if v} if query_params and len(query_params) > 0: query_dict = QueryDict('', mutable=True) for k in sorted(query_params.keys()): query_dict[k] = query_params[k] url += ('?' + query_dict.urlencode(safe='/;:')) return url def datetime_to_utc(date): """Returns datetime in UTC without timezone info Args: date (datetime.datetime): input datetime with timezone info Returns: datetime.datetime: datetime in UTC without timezone info """ if date.tzinfo: return date.astimezone(tz.gettz('UTC')).replace(tzinfo=timezone.utc) else: return date def parse_timestamp(timestamp): """Given a time or timestamp (as string), parse the result as UTC datetime. Returns: datetime.datetime: a timezone-aware datetime representing the parsed value or None if the parsing fails. Samples: - 2016-01-12 - 2016-01-12T09:19:12+0100 - Today is January 1, 2047 at 8:21:00AM - 1452591542 """ if not timestamp: return None try: date = date_parser.parse(timestamp, ignoretz=False, fuzzy=True) return datetime_to_utc(date) except Exception: try: return datetime.utcfromtimestamp(float(timestamp)).replace( tzinfo=timezone.utc) except (ValueError, OverflowError) as e: raise BadInputExc(e) def shorten_path(path): """Shorten the given path: for each hash present, only return the first 8 characters followed by an ellipsis""" sha256_re = r'([0-9a-f]{8})[0-9a-z]{56}' sha1_re = r'([0-9a-f]{8})[0-9a-f]{32}' ret = re.sub(sha256_re, r'\1...', path) return re.sub(sha1_re, r'\1...', ret) def format_utc_iso_date(iso_date, fmt='%d %B %Y, %H:%M UTC'): """Turns a string representation of an ISO 8601 date string to UTC and format it into a more human readable one. For instance, from the following input string: '2017-05-04T13:27:13+02:00' the following one is returned: '04 May 2017, 11:27 UTC'. Custom format string may also be provided as parameter Args: iso_date (str): a string representation of an ISO 8601 date fmt (str): optional date formatting string Returns: str: a formatted string representation of the input iso date """ if not iso_date: return iso_date date = parse_timestamp(iso_date) return date.strftime(fmt) def gen_path_info(path): """Function to generate path data navigation for use with a breadcrumb in the swh web ui. For instance, from a path /folder1/folder2/folder3, it returns the following list:: [{'name': 'folder1', 'path': 'folder1'}, {'name': 'folder2', 'path': 'folder1/folder2'}, {'name': 'folder3', 'path': 'folder1/folder2/folder3'}] Args: path: a filesystem path Returns: list: a list of path data for navigation as illustrated above. """ path_info = [] if path: sub_paths = path.strip('/').split('/') path_from_root = '' for p in sub_paths: path_from_root += '/' + p path_info.append({'name': p, 'path': path_from_root.strip('/')}) return path_info def get_origin_visits(origin_info): """Function that returns the list of visits for a swh origin. That list is put in cache in order to speedup the navigation in the swh web browse ui. Args: origin_id (int): the id of the swh origin to fetch visits from Returns: list: A list of dict describing the origin visits with the following keys: * **date**: UTC visit date in ISO format, * **origin**: the origin id - * **status**: the visit status, either *full* or *partial* + * **status**: the visit status, either **full**, **partial** + or **ongoing** * **visit**: the visit id Raises: NotFoundExc: if the origin is not found """ cache_entry_id = 'origin_%s_visits' % origin_info['id'] cache_entry = cache.get(cache_entry_id) last_snapshot = service.lookup_latest_origin_snapshot(origin_info['id']) if cache_entry and \ (not last_snapshot or last_snapshot['id'] == cache_entry[-1]['snapshot']): return cache_entry origin_visits = [] per_page = service.MAX_LIMIT last_visit = None while 1: visits = list(service.lookup_origin_visits(origin_info['id'], last_visit=last_visit, per_page=per_page)) origin_visits += visits if len(visits) < per_page: break else: if not last_visit: last_visit = per_page else: last_visit += per_page def _visit_sort_key(visit): ts = parse_timestamp(visit['date']).timestamp() return ts + (float(visit['visit']) / 10e3) for v in origin_visits: if 'metadata' in v: del v['metadata'] origin_visits = [dict(t) for t in set([tuple(d.items()) for d in origin_visits])] origin_visits = sorted(origin_visits, key=lambda v: _visit_sort_key(v)) cache.set(cache_entry_id, origin_visits) return origin_visits def get_swh_persistent_id(object_type, object_id, scheme_version=1): """ Returns the persistent identifier for a swh object based on: * the object type * the object id * the swh identifiers scheme version Args: object_type (str): the swh object type (content/directory/release/revision/snapshot) object_id (str): the swh object id (hexadecimal representation of its hash value) scheme_version (int): the scheme version of the swh persistent identifiers Returns: str: the swh object persistent identifier Raises: BadInputExc: if the provided parameters do not enable to generate a valid identifier """ try: swh_id = persistent_identifier(object_type, object_id, scheme_version) except ValidationError as e: raise BadInputExc('Invalid object (%s) for swh persistent id. %s' % (object_id, e)) else: return swh_id def resolve_swh_persistent_id(swh_id, query_params=None): """ - Try to resolve a SWH persistent id into an url for + Try to resolve a Software Heritage persistent id into an url for browsing the pointed object. Args: - swh_id (str): a SWH persistent identifier + swh_id (str): a Software Heritage persistent identifier query_params (django.http.QueryDict): optional dict filled with query parameters to append to the browse url Returns: dict: a dict with the following keys: * **swh_id_parsed (swh.model.identifiers.PersistentId)**: the parsed identifier * **browse_url (str)**: the url for browsing the pointed object Raises: BadInputExc: if the provided identifier can not be parsed """ # noqa try: swh_id_parsed = parse_persistent_identifier(swh_id) object_type = swh_id_parsed.object_type object_id = swh_id_parsed.object_id browse_url = None query_dict = QueryDict('', mutable=True) if query_params and len(query_params) > 0: for k in sorted(query_params.keys()): query_dict[k] = query_params[k] if 'origin' in swh_id_parsed.metadata: query_dict['origin'] = swh_id_parsed.metadata['origin'] if object_type == CONTENT: query_string = 'sha1_git:' + object_id fragment = '' if 'lines' in swh_id_parsed.metadata: lines = swh_id_parsed.metadata['lines'].split('-') fragment += '#L' + lines[0] if len(lines) > 1: fragment += '-L' + lines[1] browse_url = reverse('browse-content', url_args={'query_string': query_string}, query_params=query_dict) + fragment elif object_type == DIRECTORY: browse_url = reverse('browse-directory', url_args={'sha1_git': object_id}, query_params=query_dict) elif object_type == RELEASE: browse_url = reverse('browse-release', url_args={'sha1_git': object_id}, query_params=query_dict) elif object_type == REVISION: browse_url = reverse('browse-revision', url_args={'sha1_git': object_id}, query_params=query_dict) elif object_type == SNAPSHOT: browse_url = reverse('browse-snapshot', url_args={'snapshot_id': object_id}, query_params=query_dict) except ValidationError as ve: raise BadInputExc('Error when parsing identifier. %s' % ' '.join(ve.messages)) else: return {'swh_id_parsed': swh_id_parsed, 'browse_url': browse_url} def parse_rst(text, report_level=2): """ Parse a reStructuredText string with docutils. Args: text (str): string with reStructuredText markups in it report_level (int): level of docutils report messages to print (1 info 2 warning 3 error 4 severe 5 none) Returns: docutils.nodes.document: a parsed docutils document """ parser = docutils.parsers.rst.Parser() components = (docutils.parsers.rst.Parser,) settings = docutils.frontend.OptionParser( components=components).get_default_values() settings.report_level = report_level document = docutils.utils.new_document('rst-doc', settings=settings) parser.parse(text, document) return document def get_client_ip(request): """ Return the client IP address from an incoming HTTP request. Args: request (django.http.HttpRequest): the incoming HTTP request Returns: str: The client IP address """ x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR') if x_forwarded_for: ip = x_forwarded_for.split(',')[0] else: ip = request.META.get('REMOTE_ADDR') return ip def is_recaptcha_valid(request, recaptcha_response): """ Verify if the response for Google reCAPTCHA is valid. Args: request (django.http.HttpRequest): the incoming HTTP request recaptcha_response (str): the reCAPTCHA response Returns: bool: Whether the reCAPTCHA response is valid or not """ config = get_config() return requests.post( config['grecaptcha']['validation_url'], data={ 'secret': config['grecaptcha']['private_key'], 'response': recaptcha_response, 'remoteip': get_client_ip(request) }, verify=True ).json().get("success", False) def context_processor(request): """ Django context processor used to inject variables in all swh-web templates. """ return {'swh_object_icons': swh_object_icons} diff --git a/swh/web/config.py b/swh/web/config.py index 2f0b40f5e..1c1199ae3 100644 --- a/swh/web/config.py +++ b/swh/web/config.py @@ -1,130 +1,130 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from swh.core import config from swh.storage import get_storage from swh.indexer.storage import get_indexer_storage from swh.vault.api.client import RemoteVaultClient from swh.scheduler import get_scheduler DEFAULT_CONFIG = { 'allowed_hosts': ('list', []), 'storage': ('dict', { 'cls': 'remote', 'args': { 'url': 'http://127.0.0.1:5002/', 'timeout': 10, }, }), 'indexer_storage': ('dict', { 'cls': 'remote', 'args': { 'url': 'http://127.0.0.1:5007/', 'timeout': 1, } }), 'vault': ('string', 'http://127.0.0.1:5005/'), 'log_dir': ('string', '/tmp/swh/log'), 'debug': ('bool', False), 'serve_assets': ('bool', False), 'host': ('string', '127.0.0.1'), 'port': ('int', 5004), 'secret_key': ('string', 'development key'), # do not display code highlighting for content > 1MB 'content_display_max_size': ('int', 1024 * 1024), 'snapshot_content_max_size': ('int', 1000), 'throttling': ('dict', { 'cache_uri': None, # production: memcached as cache (127.0.0.1:11211) # development: in-memory cache so None 'scopes': { 'swh_api': { 'limiter_rate': { 'default': '120/h' }, 'exempted_networks': ['127.0.0.0/8'] }, 'swh_vault_cooking': { 'limiter_rate': { 'default': '120/h', 'GET': '60/m' }, 'exempted_networks': ['127.0.0.0/8'] }, 'swh_save_origin': { 'limiter_rate': { 'default': '120/h', 'POST': '10/h' }, 'exempted_networks': ['127.0.0.0/8'] } } }), 'scheduler': ('dict', { 'cls': 'remote', 'args': { 'url': 'http://localhost:5008/' } }), 'grecaptcha': ('dict', { 'validation_url': 'https://www.google.com/recaptcha/api/siteverify', 'site_key': '', 'private_key': '' }), 'production_db': ('string', '/var/lib/swh/web.sqlite3'), 'deposit': ('dict', { 'private_api_url': 'https://deposit.softwareheritage.org/1/private/', 'private_api_user': 'swhworker', 'private_api_password': '' }) } swhweb_config = {} def get_config(config_file='web/web'): """Read the configuration file `config_file`, update the app with parameters (secret_key, conf) and return the parsed configuration as a dict. If no configuration file is provided, return a default configuration.""" if not swhweb_config: cfg = config.load_named_config(config_file, DEFAULT_CONFIG) swhweb_config.update(cfg) config.prepare_folders(swhweb_config, 'log_dir') swhweb_config['storage'] = get_storage(**swhweb_config['storage']) swhweb_config['vault'] = RemoteVaultClient(swhweb_config['vault']) swhweb_config['indexer_storage'] = \ get_indexer_storage(**swhweb_config['indexer_storage']) swhweb_config['scheduler'] = get_scheduler(**swhweb_config['scheduler']) # noqa return swhweb_config def storage(): - """Return the current application's SWH storage. + """Return the current application's storage. """ return get_config()['storage'] def vault(): - """Return the current application's SWH vault. + """Return the current application's vault. """ return get_config()['vault'] def indexer_storage(): - """Return the current application's SWH indexer storage. + """Return the current application's indexer storage. """ return get_config()['indexer_storage'] def scheduler(): - """Return the current application's SWH scheduler. + """Return the current application's scheduler. """ return get_config()['scheduler'] diff --git a/swh/web/tests/api/test_apidoc.py b/swh/web/tests/api/test_apidoc.py index f84de924a..9bbfc790c 100644 --- a/swh/web/tests/api/test_apidoc.py +++ b/swh/web/tests/api/test_apidoc.py @@ -1,279 +1,279 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from rest_framework.test import APITestCase from rest_framework.response import Response from swh.web.api.apidoc import api_doc, _parse_httpdomain_doc from swh.web.api.apiurls import api_route from swh.web.tests.testcase import SWHWebTestCase # flake8: noqa httpdomain_doc = """ .. http:get:: /api/1/revision/(sha1_git)/ - Get information about a revision in the SWH archive. - Revisions are identified by *sha1* checksums, compatible with Git commit identifiers. + Get information about a revision in the archive. + Revisions are identified by **sha1** checksums, compatible with Git commit identifiers. See :func:`swh.model.identifiers.revision_identifier` in our data model module for details about how they are computed. - :param string sha1_git: hexadecimal representation of the revision *sha1_git* identifier + :param string sha1_git: hexadecimal representation of the revision **sha1_git** identifier :reqheader Accept: the requested response content type, - either *application/json* (default) or *application/yaml* + either ``application/json`` (default) or ``application/yaml`` :resheader Content-Type: this depends on :http:header:`Accept` header of request :>json object author: information about the author of the revision :>json string author_url: link to :http:get:`/api/1/person/(person_id)/` to get information about the author of the revision :>json object committer: information about the committer of the revision :>json string committer_url: link to :http:get:`/api/1/person/(person_id)/` to get information about the committer of the revision :>json string committer_date: ISO representation of the commit date (in UTC) :>json string date: ISO representation of the revision date (in UTC) :>json string directory: the unique identifier that revision points to :>json string directory_url: link to :http:get:`/api/1/directory/(sha1_git)/[(path)/]` to get information about the directory associated to the revision :>json string id: the revision unique identifier :>json boolean merge: whether or not the revision corresponds to a merge commit :>json string message: the message associated to the revision :>json array parents: the parents of the revision, i.e. the previous revisions that head directly to it, each entry of that array contains an unique parent revision identifier but also a link to :http:get:`/api/1/revision/(sha1_git)/` to get more information about it :>json string type: the type of the revision **Allowed HTTP Methods:** :http:method:`get`, :http:method:`head`, :http:method:`options` :statuscode 200: no error - :statuscode 400: an invalid *sha1_git* value has been provided - :statuscode 404: requested revision can not be found in the SWH archive + :statuscode 400: an invalid **sha1_git** value has been provided + :statuscode 404: requested revision can not be found in the archive **Request:** .. parsed-literal:: $ curl -i :swh_web_api:`revision/aafb16d69fd30ff58afdd69036a26047f3aebdc6/` """ class APIDocTestCase(SWHWebTestCase, APITestCase): def test_apidoc_nodoc_failure(self): with self.assertRaises(Exception): @api_doc('/my/nodoc/url/') def apidoc_nodoc_tester(request, arga=0, argb=0): return Response(arga + argb) @staticmethod @api_route(r'/some/(?P[0-9]+)/(?P[0-9]+)/', 'some-doc-route') @api_doc('/some/doc/route/') def apidoc_route(request, myarg, myotherarg, akw=0): """ Sample doc """ return {'result': int(myarg) + int(myotherarg) + akw} def test_apidoc_route_doc(self): # when rv = self.client.get('/api/1/some/doc/route/') # then self.assertEqual(rv.status_code, 200) self.assertTemplateUsed('api/apidoc.html') def test_apidoc_route_fn(self): # when rv = self.client.get('/api/1/some/1/1/') # then self.assertEqual(rv.status_code, 200) @staticmethod @api_route(r'/some/full/(?P[0-9]+)/(?P[0-9]+)/', 'some-complete-doc-route') @api_doc('/some/complete/doc/route/') def apidoc_full_stack(request, myarg, myotherarg, akw=0): """ Sample doc """ return {'result': int(myarg) + int(myotherarg) + akw} def test_apidoc_full_stack_doc(self): # when rv = self.client.get('/api/1/some/complete/doc/route/') # then self.assertEqual(rv.status_code, 200) self.assertTemplateUsed('api/apidoc.html') def test_apidoc_full_stack_fn(self): # when rv = self.client.get('/api/1/some/full/1/1/') # then self.assertEqual(rv.status_code, 200) def test_api_doc_parse_httpdomain(self): doc_data = { 'description': '', 'urls': [], 'args': [], 'params': [], 'resheaders': [], 'reqheaders': [], 'return_type': '', 'returns': [], 'status_codes': [], 'examples': [] } _parse_httpdomain_doc(httpdomain_doc, doc_data) expected_urls = [{ 'rule': '/api/1/revision/ **\\(sha1_git\\)** /', 'methods': ['GET', 'HEAD', 'OPTIONS'] }] self.assertIn('urls', doc_data) self.assertEqual(doc_data['urls'], expected_urls) - expected_description = 'Get information about a revision in the SWH archive. \ -Revisions are identified by *sha1* checksums, compatible with Git commit \ + expected_description = 'Get information about a revision in the archive. \ +Revisions are identified by **sha1** checksums, compatible with Git commit \ identifiers. See **swh.model.identifiers.revision_identifier** in our data \ model module for details about how they are computed.' self.assertIn('description', doc_data) self.assertEqual(doc_data['description'], expected_description) expected_args = [{ 'name': 'sha1_git', 'type': 'string', - 'doc': 'hexadecimal representation of the revision *sha1_git* identifier' + 'doc': 'hexadecimal representation of the revision **sha1_git** identifier' }] self.assertIn('args', doc_data) self.assertEqual(doc_data['args'], expected_args) expected_params = [] self.assertIn('params', doc_data) self.assertEqual(doc_data['params'], expected_params) expected_reqheaders = [{ - 'doc': 'the requested response content type, either *application/json* or *application/yaml*', + 'doc': 'the requested response content type, either ``application/json`` or ``application/yaml``', 'name': 'Accept' }] self.assertIn('reqheaders', doc_data) self.assertEqual(doc_data['reqheaders'], expected_reqheaders) expected_resheaders = [{ 'doc': 'this depends on **Accept** header of request', 'name': 'Content-Type' }] self.assertIn('resheaders', doc_data) self.assertEqual(doc_data['resheaders'], expected_resheaders) expected_statuscodes = [ { 'code': '200', 'doc': 'no error' }, { 'code': '400', - 'doc': 'an invalid *sha1_git* value has been provided' + 'doc': 'an invalid **sha1_git** value has been provided' }, { 'code': '404', - 'doc': 'requested revision can not be found in the SWH archive' + 'doc': 'requested revision can not be found in the archive' } ] self.assertIn('status_codes', doc_data) self.assertEqual(doc_data['status_codes'], expected_statuscodes) expected_return_type = 'object' self.assertIn('return_type', doc_data) self.assertEqual(doc_data['return_type'], expected_return_type) expected_returns = [ { 'name': 'author', 'type': 'object', 'doc': 'information about the author of the revision' }, { 'name': 'author_url', 'type': 'string', 'doc': 'link to ``_ to get information about the author of the revision' }, { 'name': 'committer', 'type': 'object', 'doc': 'information about the committer of the revision' }, { 'name': 'committer_url', 'type': 'string', 'doc': 'link to ``_ to get information about the committer of the revision' }, { 'name': 'committer_date', 'type': 'string', 'doc': 'ISO representation of the commit date (in UTC)' }, { 'name': 'date', 'type': 'string', 'doc': 'ISO representation of the revision date (in UTC)' }, { 'name': 'directory', 'type': 'string', 'doc': 'the unique identifier that revision points to' }, { 'name': 'directory_url', 'type': 'string', 'doc': 'link to ``_ to get information about the directory associated to the revision' }, { 'name': 'id', 'type': 'string', 'doc': 'the revision unique identifier' }, { 'name': 'merge', 'type': 'boolean', 'doc': 'whether or not the revision corresponds to a merge commit' }, { 'name': 'message', 'type': 'string', 'doc': 'the message associated to the revision' }, { 'name': 'parents', 'type': 'array', 'doc': 'the parents of the revision, i.e. the previous revisions that head directly to it, each entry of that array contains an unique parent revision identifier but also a link to ``_ to get more information about it' }, { 'name': 'type', 'type': 'string', 'doc': 'the type of the revision' } ] self.assertIn('returns', doc_data) self.assertEqual(doc_data['returns'], expected_returns) expected_examples = ['/api/1/revision/aafb16d69fd30ff58afdd69036a26047f3aebdc6/'] self.assertIn('examples', doc_data) self.assertEqual(doc_data['examples'], expected_examples) diff --git a/swh/web/tests/api/views/test_origin.py b/swh/web/tests/api/views/test_origin.py index a59acc986..6950e4e60 100644 --- a/swh/web/tests/api/views/test_origin.py +++ b/swh/web/tests/api/views/test_origin.py @@ -1,246 +1,246 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from rest_framework.test import APITestCase from unittest.mock import patch from swh.storage.exc import StorageDBError, StorageAPIError from swh.web.tests.testcase import SWHWebTestCase class OriginApiTestCase(SWHWebTestCase, APITestCase): def setUp(self): self.origin_visit1 = { 'date': 1104616800.0, 'origin': 10, 'visit': 100, 'metadata': None, 'status': 'full', } self.origin1 = { 'id': 1234, 'url': 'ftp://some/url/to/origin/0', 'type': 'ftp' } @patch('swh.web.api.views.origin.get_origin_visits') def test_api_1_lookup_origin_visits_raise_error( self, mock_get_origin_visits, ): # given mock_get_origin_visits.side_effect = ValueError( 'voluntary error to check the bad request middleware.') # when rv = self.client.get('/api/1/origin/2/visits/') # then self.assertEqual(rv.status_code, 400) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'ValueError', 'reason': 'voluntary error to check the bad request middleware.'}) @patch('swh.web.common.utils.service') def test_api_1_lookup_origin_visits_raise_swh_storage_error_db( self, mock_service): # given mock_service.lookup_origin_visits.side_effect = StorageDBError( - 'SWH Storage exploded! Will be back online shortly!') + 'Storage exploded! Will be back online shortly!') # when rv = self.client.get('/api/1/origin/2/visits/') # then self.assertEqual(rv.status_code, 503) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'StorageDBError', 'reason': 'An unexpected error occurred in the backend: ' - 'SWH Storage exploded! Will be back online shortly!'}) + 'Storage exploded! Will be back online shortly!'}) @patch('swh.web.common.utils.service') def test_api_1_lookup_origin_visits_raise_swh_storage_error_api( self, mock_service): # given mock_service.lookup_origin_visits.side_effect = StorageAPIError( - 'SWH Storage API dropped dead! Will resurrect from its ashes asap!' + 'Storage API dropped dead! Will resurrect from its ashes asap!' ) # when rv = self.client.get('/api/1/origin/2/visits/') # then self.assertEqual(rv.status_code, 503) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'StorageAPIError', 'reason': 'An unexpected error occurred in the api backend: ' - 'SWH Storage API dropped dead! Will resurrect from its ashes asap!' + 'Storage API dropped dead! Will resurrect from its ashes asap!' }) @patch('swh.web.api.views.origin.get_origin_visits') def test_api_1_lookup_origin_visits(self, mock_get_origin_visits): # given stub_visits = [ { 'date': 1293919200.0, 'origin': 2, 'snapshot': '1234', 'visit': 1 }, { 'date': 1293919200.0, 'origin': 2, 'snapshot': '1234', 'visit': 2 }, { 'date': 1420149600.0, 'origin': 2, 'snapshot': '5678', 'visit': 3 }, { 'date': 1420149600.0, 'origin': 2, 'snapshot': '5678', 'visit': 4 } ] mock_get_origin_visits.return_value = stub_visits # when rv = self.client.get('/api/1/origin/2/visits/?per_page=2&last_visit=3') self.assertEqual(rv.status_code, 200) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, [ { 'date': 1293919200.0, 'origin': 2, 'snapshot': '1234', 'visit': 2, 'origin_visit_url': '/api/1/origin/2/visit/2/', 'snapshot_url': '/api/1/snapshot/1234/' }, { 'date': 1293919200.0, 'origin': 2, 'snapshot': '1234', 'visit': 1, 'origin_visit_url': '/api/1/origin/2/visit/1/', 'snapshot_url': '/api/1/snapshot/1234/' }, ]) @patch('swh.web.api.views.origin.service') def test_api_1_lookup_origin_visit(self, mock_service): # given origin_visit = self.origin_visit1.copy() origin_visit.update({ 'snapshot': '57478754' }) mock_service.lookup_origin_visit.return_value = origin_visit expected_origin_visit = self.origin_visit1.copy() expected_origin_visit.update({ 'origin_url': '/api/1/origin/10/', 'snapshot': '57478754', 'snapshot_url': '/api/1/snapshot/57478754/' }) # when rv = self.client.get('/api/1/origin/10/visit/100/') self.assertEqual(rv.status_code, 200) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, expected_origin_visit) mock_service.lookup_origin_visit.assert_called_once_with('10', '100') @patch('swh.web.api.views.origin.service') def test_api_1_lookup_origin_visit_not_found(self, mock_service): # given mock_service.lookup_origin_visit.return_value = None # when rv = self.client.get('/api/1/origin/1/visit/1000/') self.assertEqual(rv.status_code, 404) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'NotFoundExc', 'reason': 'No visit 1000 for origin 1 found' }) mock_service.lookup_origin_visit.assert_called_once_with('1', '1000') @patch('swh.web.api.views.origin.service') def test_api_origin_by_id(self, mock_service): # given mock_service.lookup_origin.return_value = self.origin1 expected_origin = self.origin1.copy() expected_origin.update({ 'origin_visits_url': '/api/1/origin/1234/visits/' }) # when rv = self.client.get('/api/1/origin/1234/') # then self.assertEqual(rv.status_code, 200) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, expected_origin) mock_service.lookup_origin.assert_called_with({'id': '1234'}) @patch('swh.web.api.views.origin.service') def test_api_origin_by_type_url(self, mock_service): # given stub_origin = self.origin1.copy() stub_origin.update({ 'id': 987 }) mock_service.lookup_origin.return_value = stub_origin expected_origin = stub_origin.copy() expected_origin.update({ 'origin_visits_url': '/api/1/origin/987/visits/' }) # when rv = self.client.get('/api/1/origin/ftp/url' '/ftp://some/url/to/origin/0/') # then self.assertEqual(rv.status_code, 200) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, expected_origin) mock_service.lookup_origin.assert_called_with( {'url': 'ftp://some/url/to/origin/0', 'type': 'ftp'}) @patch('swh.web.api.views.origin.service') def test_api_origin_not_found(self, mock_service): # given mock_service.lookup_origin.return_value = None # when rv = self.client.get('/api/1/origin/4321/') # then self.assertEqual(rv.status_code, 404) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'NotFoundExc', 'reason': 'Origin with id 4321 not found.' }) mock_service.lookup_origin.assert_called_with({'id': '4321'}) diff --git a/swh/web/tests/api/views/test_stat.py b/swh/web/tests/api/views/test_stat.py index 5e179c339..afe9b9755 100644 --- a/swh/web/tests/api/views/test_stat.py +++ b/swh/web/tests/api/views/test_stat.py @@ -1,90 +1,90 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from rest_framework.test import APITestCase from unittest.mock import patch from swh.storage.exc import StorageDBError, StorageAPIError from swh.web.tests.testcase import SWHWebTestCase class StatApiTestCase(SWHWebTestCase, APITestCase): @patch('swh.web.api.views.stat.service') def test_api_1_stat_counters_raise_error(self, mock_service): # given mock_service.stat_counters.side_effect = ValueError( 'voluntary error to check the bad request middleware.') # when rv = self.client.get('/api/1/stat/counters/') # then self.assertEqual(rv.status_code, 400) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'ValueError', 'reason': 'voluntary error to check the bad request middleware.'}) @patch('swh.web.api.views.stat.service') def test_api_1_stat_counters_raise_from_db(self, mock_service): # given mock_service.stat_counters.side_effect = StorageDBError( - 'SWH Storage exploded! Will be back online shortly!') + 'Storage exploded! Will be back online shortly!') # when rv = self.client.get('/api/1/stat/counters/') # then self.assertEqual(rv.status_code, 503) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'StorageDBError', 'reason': 'An unexpected error occurred in the backend: ' - 'SWH Storage exploded! Will be back online shortly!'}) + 'Storage exploded! Will be back online shortly!'}) @patch('swh.web.api.views.stat.service') def test_api_1_stat_counters_raise_from_api(self, mock_service): # given mock_service.stat_counters.side_effect = StorageAPIError( - 'SWH Storage API dropped dead! Will resurrect from its ashes asap!' + 'Storage API dropped dead! Will resurrect from its ashes asap!' ) # when rv = self.client.get('/api/1/stat/counters/') # then self.assertEqual(rv.status_code, 503) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'StorageAPIError', 'reason': 'An unexpected error occurred in the api backend: ' - 'SWH Storage API dropped dead! Will resurrect from its ashes asap!' + 'Storage API dropped dead! Will resurrect from its ashes asap!' }) @patch('swh.web.api.views.stat.service') def test_api_1_stat_counters(self, mock_service): # given stub_stats = { "content": 1770830, "directory": 211683, "directory_entry_dir": 209167, "directory_entry_file": 1807094, "directory_entry_rev": 0, "entity": 0, "entity_history": 0, "origin": 1096, "person": 0, "release": 8584, "revision": 7792, "revision_history": 0, "skipped_content": 0 } mock_service.stat_counters.return_value = stub_stats # when rv = self.client.get('/api/1/stat/counters/') self.assertEqual(rv.status_code, 200) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, stub_stats) mock_service.stat_counters.assert_called_once_with() diff --git a/swh/web/tests/browse/test_utils.py b/swh/web/tests/browse/test_utils.py index a03b41827..903e834cc 100644 --- a/swh/web/tests/browse/test_utils.py +++ b/swh/web/tests/browse/test_utils.py @@ -1,244 +1,244 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information # flake8: noqa from unittest.mock import patch from swh.web.browse import utils from swh.web.common.exc import NotFoundExc from swh.web.common.utils import reverse from swh.web.tests.testcase import SWHWebTestCase from .views.data.revision_test_data import revision_history_log_test class SwhBrowseUtilsTestCase(SWHWebTestCase): def test_get_mimetype_and_encoding_for_content(self): text = b'Hello world!' self.assertEqual(utils.get_mimetype_and_encoding_for_content(text), ('text/plain', 'us-ascii')) @patch('swh.web.browse.utils.get_origin_visits') def test_get_origin_visit(self, mock_origin_visits): origin_info = { 'id': 2, 'type': 'git', 'url': 'https://github.com/foo/bar', } visits = \ [{'status': 'full', 'date': '2015-07-09T21:09:24+00:00', 'visit': 1, 'origin': origin_info['id'] }, {'status': 'full', 'date': '2016-02-23T18:05:23.312045+00:00', 'visit': 2, 'origin': origin_info['id'] }, {'status': 'full', 'date': '2016-03-28T01:35:06.554111+00:00', 'visit': 3, 'origin': origin_info['id'] }, {'status': 'full', 'date': '2016-06-18T01:22:24.808485+00:00', 'visit': 4, 'origin': origin_info['id'] }, {'status': 'full', 'date': '2016-08-14T12:10:00.536702+00:00', 'visit': 5, 'origin': origin_info['id'] }] mock_origin_visits.return_value = visits visit_id = 12 with self.assertRaises(NotFoundExc) as cm: visit = utils.get_origin_visit(origin_info, visit_id=visit_id) exception_text = cm.exception.args[0] self.assertIn('Visit with id %s' % visit_id, exception_text) self.assertIn('type %s' % origin_info['type'], exception_text) self.assertIn('url %s' % origin_info['url'], exception_text) visit = utils.get_origin_visit(origin_info, visit_id=2) self.assertEqual(visit, visits[1]) visit = utils.get_origin_visit( origin_info, visit_ts='2016-02-23T18:05:23.312045+00:00') self.assertEqual(visit, visits[1]) visit = utils.get_origin_visit( origin_info, visit_ts='2016-02-20') self.assertEqual(visit, visits[1]) visit = utils.get_origin_visit( origin_info, visit_ts='2016-06-18T01:22') self.assertEqual(visit, visits[3]) visit = utils.get_origin_visit( origin_info, visit_ts='2016-06-18 01:22') self.assertEqual(visit, visits[3]) visit = utils.get_origin_visit( origin_info, visit_ts=1466208000) self.assertEqual(visit, visits[3]) visit = utils.get_origin_visit( origin_info, visit_ts='2014-01-01') self.assertEqual(visit, visits[0]) visit = utils.get_origin_visit( origin_info, visit_ts='2018-01-01') self.assertEqual(visit, visits[-1]) @patch('swh.web.browse.utils.service') @patch('swh.web.browse.utils.get_origin_visit') def test_get_origin_visit_snapshot(self, mock_get_origin_visit, mock_service): mock_get_origin_visit.return_value = \ {'status': 'full', 'date': '2015-08-04T22:26:14.804009+00:00', 'visit': 1, 'origin': 1, 'snapshot': '584b2fe3ce6218a96892e73bd76c2966bbc2a797'} mock_service.lookup_snapshot.return_value = \ {'branches': { 'HEAD': { 'target': '9fbd21adbac36be869514e82e2e98505dc47219c', 'target_type': 'revision', 'target_url': '/api/1/revision/9fbd21adbac36be869514e82e2e98505dc47219c/' }, 'refs/heads/master': { 'target': '9fbd21adbac36be869514e82e2e98505dc47219c', 'target_type': 'revision', 'target_url': '/api/1/revision/9fbd21adbac36be869514e82e2e98505dc47219c/' }, 'refs/tags/0.10.0': { 'target': '7045404f3d1c54e6473c71bbb716529fbad4be24', 'target_type': 'release', 'target_url': '/api/1/release/7045404f3d1c54e6473c71bbb716529fbad4be24/' }, 'refs/tags/0.10.1': { 'target': 'c893f4549c367e68288b0eb74595050410aa0de7', 'target_type': 'release', 'target_url': '/api/1/release/c893f4549c367e68288b0eb74595050410aa0de7/' } }, 'id': '584b2fe3ce6218a96892e73bd76c2966bbc2a797'} mock_service.lookup_release_multiple.return_value = \ [{'name': '0.10.0', 'message': '0.10: The "Oh fuck it\'s PyCon" release\n', 'id': '7045404f3d1c54e6473c71bbb716529fbad4be24', 'date': '2014-04-10T23:01:28-04:00', 'target_type': 'revision', 'target': '6072557b6c10cd9a21145781e26ad1f978ed14b9'}, {'name': '0.10.1', 'message': 'Tagging 0.10.1\n', 'id': 'c893f4549c367e68288b0eb74595050410aa0de7', 'date': '2014-10-10T09:45:52-04:00', 'target_type': 'revision', 'target': 'ecc003b43433e5b46511157598e4857a761007bf'}] mock_service.lookup_revision_multiple.return_value = \ [{'date': '2015-08-04T13:16:54+03:00', 'directory': '828da2b80e41aa958b2c98526f4a1d2cc7d298b7', 'id': '9fbd21adbac36be869514e82e2e98505dc47219c', 'message': 'Merge pull request #678 from algernon'}, {'date': '2014-04-10T23:01:11-04:00', 'directory': '2df4cd84ecc65b50b1d5318d3727e02a39b8a4cf', 'id': '6072557b6c10cd9a21145781e26ad1f978ed14b9', 'message': '0.10: The "Oh fuck it\'s PyCon" release\n'}, {'date': '2014-10-10T09:45:23-04:00', 'directory': '28ba64f97ef709e54838ae482c2da2619a74a0bd', 'id': 'ecc003b43433e5b46511157598e4857a761007bf', 'message': '0.10.1\n'}] expected_result = ( [{'name': 'HEAD', 'message': 'Merge pull request #678 from algernon', 'date': '04 August 2015, 10:16 UTC', 'revision': '9fbd21adbac36be869514e82e2e98505dc47219c', 'directory': '828da2b80e41aa958b2c98526f4a1d2cc7d298b7'}, {'name': 'refs/heads/master', 'message': 'Merge pull request #678 from algernon', 'date': '04 August 2015, 10:16 UTC', 'revision': '9fbd21adbac36be869514e82e2e98505dc47219c', 'directory': '828da2b80e41aa958b2c98526f4a1d2cc7d298b7'}], [{'name': '0.10.0', 'branch_name': 'refs/tags/0.10.0', 'id': '7045404f3d1c54e6473c71bbb716529fbad4be24', 'message': '0.10: The "Oh fuck it\'s PyCon" release\n', 'date': '11 April 2014, 03:01 UTC', 'target_type': 'revision', 'target': '6072557b6c10cd9a21145781e26ad1f978ed14b9', 'directory': '2df4cd84ecc65b50b1d5318d3727e02a39b8a4cf'}, {'name': '0.10.1', 'branch_name': 'refs/tags/0.10.1', 'id': 'c893f4549c367e68288b0eb74595050410aa0de7', 'message': 'Tagging 0.10.1\n', 'date': '10 October 2014, 13:45 UTC', 'target_type': 'revision', 'target': 'ecc003b43433e5b46511157598e4857a761007bf', 'directory': '28ba64f97ef709e54838ae482c2da2619a74a0bd'}] ) origin_info = { 'id': 1, 'type': 'git', 'url': 'https://github.com/hylang/hy' } origin_visit_branches = \ utils.get_origin_visit_snapshot(origin_info, visit_id=1) lookup_release_calls = mock_service.lookup_release_multiple.call_args_list self.assertEqual(len(lookup_release_calls), 1) # Check that we looked up the two expected releases self.assertCountEqual(set(lookup_release_calls[0][0][0]), { '7045404f3d1c54e6473c71bbb716529fbad4be24', 'c893f4549c367e68288b0eb74595050410aa0de7', }) lookup_revision_calls = mock_service.lookup_revision_multiple.call_args_list self.assertEqual(len(lookup_revision_calls), 1) # Check that we looked up the three expected revisions self.assertCountEqual(set(lookup_revision_calls[0][0][0]), { '9fbd21adbac36be869514e82e2e98505dc47219c', '6072557b6c10cd9a21145781e26ad1f978ed14b9', 'ecc003b43433e5b46511157598e4857a761007bf', }) self.assertEqual(origin_visit_branches, expected_result) def test_gen_link(self): - self.assertEqual(utils.gen_link('https://www.softwareheritage.org/', 'SWH'), - 'SWH') + self.assertEqual(utils.gen_link('https://www.softwareheritage.org/', 'swh'), + 'swh') def test_gen_person_link(self): person_id = 8221896 person_name = 'Antoine Lambert' person_url = reverse('browse-person', url_args={'person_id': person_id}) self.assertEqual(utils.gen_person_link(person_id, person_name), '%s' % (person_url, person_name)) def test_gen_revision_link(self): revision_id = '28a0bc4120d38a394499382ba21d6965a67a3703' revision_url = reverse('browse-revision', url_args={'sha1_git': revision_id}) self.assertEqual(utils.gen_revision_link(revision_id), '%s' % (revision_url, revision_id)) self.assertEqual(utils.gen_revision_link(revision_id, shorten_id=True), '%s' % (revision_url, revision_id[:7])) diff --git a/swh/web/tests/browse/views/test_origin.py b/swh/web/tests/browse/views/test_origin.py index 485a0e223..7a3dfb099 100644 --- a/swh/web/tests/browse/views/test_origin.py +++ b/swh/web/tests/browse/views/test_origin.py @@ -1,914 +1,914 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information # flake8: noqa from unittest.mock import patch from django.utils.html import escape from swh.web.common.exc import NotFoundExc from swh.web.common.utils import ( reverse, gen_path_info, format_utc_iso_date, parse_timestamp, get_swh_persistent_id ) from swh.web.tests.testcase import SWHWebTestCase from .data.origin_test_data import ( origin_info_test_data, origin_visits_test_data, stub_content_origin_info, stub_content_origin_visit_id, stub_content_origin_visit_unix_ts, stub_content_origin_visit_iso_date, stub_content_origin_branch, stub_content_origin_visits, stub_content_origin_snapshot, stub_origin_info, stub_visit_id, stub_origin_visits, stub_origin_snapshot, stub_origin_root_directory_entries, stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_sub_directory_path, stub_origin_sub_directory_entries, stub_visit_unix_ts, stub_visit_iso_date ) from .data.content_test_data import ( stub_content_root_dir, stub_content_text_data, stub_content_text_path ) stub_origin_info_no_type = dict(stub_origin_info) stub_origin_info_no_type['type'] = None def _to_snapshot_dict(branches=None, releases=None): snp = {'branches': {}} if branches: for b in branches: snp['branches'][b['name']] = { 'target': b['revision'], 'target_type': 'revision' } if releases: for r in releases: snp['branches'][r['branch_name']] = { 'target': r['id'], 'target_type': 'release' } return snp class SwhBrowseOriginTest(SWHWebTestCase): @patch('swh.web.browse.utils.service') @patch('swh.web.browse.utils.get_origin_visit_snapshot') @patch('swh.web.browse.utils.get_origin_visits') @patch('swh.web.browse.utils.get_origin_info') @patch('swh.web.browse.views.origin.get_origin_info') @patch('swh.web.browse.views.origin.get_origin_visits') @patch('swh.web.browse.views.origin.service') def test_origin_visits_browse(self, mock_service, mock_get_origin_visits, mock_get_origin_info, mock_get_origin_info_utils, mock_get_origin_visits_utils, mock_get_origin_visit_snapshot, mock_utils_service): mock_service.lookup_origin.return_value = origin_info_test_data mock_get_origin_info.return_value = origin_info_test_data mock_get_origin_info_utils.return_value = origin_info_test_data mock_get_origin_visits.return_value = origin_visits_test_data mock_get_origin_visits_utils.return_value = origin_visits_test_data mock_get_origin_visit_snapshot.return_value = stub_content_origin_snapshot mock_utils_service.lookup_snapshot_size.return_value = { 'revision': len(stub_content_origin_snapshot[0]), 'release': len(stub_content_origin_snapshot[1]) } url = reverse('browse-origin-visits', url_args={'origin_type': origin_info_test_data['type'], 'origin_url': origin_info_test_data['url']}) resp = self.client.get(url) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('origin-visits.html') url = reverse('browse-origin-visits', url_args={'origin_url': origin_info_test_data['url']}) resp = self.client.get(url) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('origin-visits.html') def origin_content_view_helper(self, origin_info, origin_visits, origin_branches, origin_releases, origin_branch, root_dir_sha1, content_sha1, content_sha1_git, content_path, content_data, content_language, visit_id=None, timestamp=None): url_args = {'origin_type': origin_info['type'], 'origin_url': origin_info['url'], 'path': content_path} if not visit_id: visit_id = origin_visits[-1]['visit'] query_params = {} if timestamp: url_args['timestamp'] = timestamp if visit_id: query_params['visit_id'] = visit_id url = reverse('browse-origin-content', url_args=url_args, query_params=query_params) resp = self.client.get(url) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('content.html') self.assertContains(resp, '' % content_language) self.assertContains(resp, escape(content_data)) split_path = content_path.split('/') filename = split_path[-1] path = content_path.replace(filename, '')[:-1] path_info = gen_path_info(path) del url_args['path'] if timestamp: url_args['timestamp'] = \ format_utc_iso_date(parse_timestamp(timestamp).isoformat(), '%Y-%m-%dT%H:%M:%S') root_dir_url = reverse('browse-origin-directory', url_args=url_args, query_params=query_params) self.assertContains(resp, '
  • ', count=len(path_info)+1) self.assertContains(resp, '%s' % (root_dir_url, root_dir_sha1[:7])) for p in path_info: url_args['path'] = p['path'] dir_url = reverse('browse-origin-directory', url_args=url_args, query_params=query_params) self.assertContains(resp, '%s' % (dir_url, p['name'])) self.assertContains(resp, '
  • %s
  • ' % filename) query_string = 'sha1_git:' + content_sha1 url_raw = reverse('browse-content-raw', url_args={'query_string': query_string}, query_params={'filename': filename}) self.assertContains(resp, url_raw) del url_args['path'] origin_branches_url = \ reverse('browse-origin-branches', url_args=url_args, query_params=query_params) self.assertContains(resp, 'Branches (%s)' % (origin_branches_url, len(origin_branches))) origin_releases_url = \ reverse('browse-origin-releases', url_args=url_args, query_params=query_params) self.assertContains(resp, 'Releases (%s)' % (origin_releases_url, len(origin_releases))) self.assertContains(resp, '
  • ', count=len(origin_branches)) url_args['path'] = content_path for branch in origin_branches: query_params['branch'] = branch['name'] root_dir_branch_url = \ reverse('browse-origin-content', url_args=url_args, query_params=query_params) self.assertContains(resp, '' % root_dir_branch_url) self.assertContains(resp, '
  • ', count=len(origin_releases)) query_params['branch'] = None for release in origin_releases: query_params['release'] = release['name'] root_dir_release_url = \ reverse('browse-origin-content', url_args=url_args, query_params=query_params) self.assertContains(resp, '' % root_dir_release_url) del url_args['origin_type'] url = reverse('browse-origin-content', url_args=url_args, query_params=query_params) resp = self.client.get(url) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('content.html') swh_cnt_id = get_swh_persistent_id('content', content_sha1_git) swh_cnt_id_url = reverse('browse-swh-id', url_args={'swh_id': swh_cnt_id}) self.assertContains(resp, swh_cnt_id) self.assertContains(resp, swh_cnt_id_url) @patch('swh.web.browse.utils.get_origin_visits') @patch('swh.web.browse.utils.get_origin_visit_snapshot') @patch('swh.web.browse.views.utils.snapshot_context.service') @patch('swh.web.browse.utils.service') @patch('swh.web.browse.views.utils.snapshot_context.request_content') def test_origin_content_view(self, mock_request_content, mock_utils_service, mock_service, mock_get_origin_visit_snapshot, mock_get_origin_visits): stub_content_text_sha1 = stub_content_text_data['checksums']['sha1'] stub_content_text_sha1_git = stub_content_text_data['checksums']['sha1_git'] mock_get_origin_visits.return_value = stub_content_origin_visits mock_get_origin_visit_snapshot.return_value = stub_content_origin_snapshot mock_service.lookup_directory_with_path.return_value = \ {'target': stub_content_text_sha1} mock_request_content.return_value = stub_content_text_data mock_utils_service.lookup_origin.return_value = stub_content_origin_info mock_utils_service.lookup_snapshot_size.return_value = { 'revision': len(stub_content_origin_snapshot[0]), 'release': len(stub_content_origin_snapshot[1]) } self.origin_content_view_helper(stub_content_origin_info, stub_content_origin_visits, stub_content_origin_snapshot[0], stub_content_origin_snapshot[1], stub_content_origin_branch, stub_content_root_dir, stub_content_text_sha1, stub_content_text_sha1_git, stub_content_text_path, stub_content_text_data['raw_data'], 'cpp') self.origin_content_view_helper(stub_content_origin_info, stub_content_origin_visits, stub_content_origin_snapshot[0], stub_content_origin_snapshot[1], stub_content_origin_branch, stub_content_root_dir, stub_content_text_sha1, stub_content_text_sha1_git, stub_content_text_path, stub_content_text_data['raw_data'], 'cpp', visit_id=stub_content_origin_visit_id) self.origin_content_view_helper(stub_content_origin_info, stub_content_origin_visits, stub_content_origin_snapshot[0], stub_content_origin_snapshot[1], stub_content_origin_branch, stub_content_root_dir, stub_content_text_sha1, stub_content_text_sha1_git, stub_content_text_path, stub_content_text_data['raw_data'], 'cpp', timestamp=stub_content_origin_visit_unix_ts) self.origin_content_view_helper(stub_content_origin_info, stub_content_origin_visits, stub_content_origin_snapshot[0], stub_content_origin_snapshot[1], stub_content_origin_branch, stub_content_root_dir, stub_content_text_sha1, stub_content_text_sha1_git, stub_content_text_path, stub_content_text_data['raw_data'], 'cpp', timestamp=stub_content_origin_visit_iso_date) def origin_directory_view_helper(self, origin_info, origin_visits, origin_branches, origin_releases, origin_branch, root_directory_sha1, directory_entries, visit_id=None, timestamp=None, path=None): dirs = [e for e in directory_entries if e['type'] in ('dir', 'rev')] files = [e for e in directory_entries if e['type'] == 'file'] if not visit_id: visit_id = origin_visits[-1]['visit'] url_args = {'origin_url': origin_info['url']} query_params = {} if timestamp: url_args['timestamp'] = timestamp else: query_params['visit_id'] = visit_id if path: url_args['path'] = path url = reverse('browse-origin-directory', url_args=url_args, query_params=query_params) resp = self.client.get(url) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('directory.html') self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('directory.html') self.assertContains(resp, '', count=len(dirs)) self.assertContains(resp, '', count=len(files)) if timestamp: url_args['timestamp'] = \ format_utc_iso_date(parse_timestamp(timestamp).isoformat(), '%Y-%m-%dT%H:%M:%S') for d in dirs: if d['type'] == 'rev': dir_url = reverse('browse-revision', url_args={'sha1_git': d['target']}) else: dir_path = d['name'] if path: dir_path = "%s/%s" % (path, d['name']) dir_url_args = dict(url_args) dir_url_args['path'] = dir_path dir_url = reverse('browse-origin-directory', url_args=dir_url_args, query_params=query_params) self.assertContains(resp, dir_url) for f in files: file_path = f['name'] if path: file_path = "%s/%s" % (path, f['name']) file_url_args = dict(url_args) file_url_args['path'] = file_path file_url = reverse('browse-origin-content', url_args=file_url_args, query_params=query_params) self.assertContains(resp, file_url) if 'path' in url_args: del url_args['path'] root_dir_branch_url = \ reverse('browse-origin-directory', url_args=url_args, query_params=query_params) nb_bc_paths = 1 if path: nb_bc_paths = len(path.split('/')) + 1 self.assertContains(resp, '
  • ', count=nb_bc_paths) self.assertContains(resp, '%s' % (root_dir_branch_url, root_directory_sha1[:7])) origin_branches_url = \ reverse('browse-origin-branches', url_args=url_args, query_params=query_params) self.assertContains(resp, 'Branches (%s)' % (origin_branches_url, len(origin_branches))) origin_releases_url = \ reverse('browse-origin-releases', url_args=url_args, query_params=query_params) self.assertContains(resp, 'Releases (%s)' % (origin_releases_url, len(origin_releases))) if path: url_args['path'] = path self.assertContains(resp, '
  • ', count=len(origin_branches)) for branch in origin_branches: query_params['branch'] = branch['name'] root_dir_branch_url = \ reverse('browse-origin-directory', url_args=url_args, query_params=query_params) self.assertContains(resp, '' % root_dir_branch_url) self.assertContains(resp, '
  • ', count=len(origin_releases)) query_params['branch'] = None for release in origin_releases: query_params['release'] = release['name'] root_dir_release_url = \ reverse('browse-origin-directory', url_args=url_args, query_params=query_params) self.assertContains(resp, '' % root_dir_release_url) self.assertContains(resp, 'vault-cook-directory') self.assertContains(resp, 'vault-cook-revision') swh_dir_id = get_swh_persistent_id('directory', directory_entries[0]['dir_id']) # noqa swh_dir_id_url = reverse('browse-swh-id', url_args={'swh_id': swh_dir_id}) self.assertContains(resp, swh_dir_id) self.assertContains(resp, swh_dir_id_url) @patch('swh.web.browse.utils.get_origin_visits') @patch('swh.web.browse.utils.get_origin_visit_snapshot') @patch('swh.web.browse.utils.service') @patch('swh.web.browse.views.origin.service') def test_origin_root_directory_view(self, mock_origin_service, mock_utils_service, mock_get_origin_visit_snapshot, mock_get_origin_visits): mock_get_origin_visits.return_value = stub_origin_visits mock_get_origin_visit_snapshot.return_value = stub_origin_snapshot mock_utils_service.lookup_directory.return_value = \ stub_origin_root_directory_entries mock_utils_service.lookup_origin.return_value = stub_origin_info mock_utils_service.lookup_snapshot_size.return_value = { 'revision': len(stub_origin_snapshot[0]), 'release': len(stub_origin_snapshot[1]) } self.origin_directory_view_helper(stub_origin_info, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_root_directory_entries) self.origin_directory_view_helper(stub_origin_info, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_root_directory_entries, visit_id=stub_visit_id) self.origin_directory_view_helper(stub_origin_info, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_root_directory_entries, timestamp=stub_visit_unix_ts) self.origin_directory_view_helper(stub_origin_info, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_root_directory_entries, timestamp=stub_visit_iso_date) self.origin_directory_view_helper(stub_origin_info_no_type, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_root_directory_entries) self.origin_directory_view_helper(stub_origin_info_no_type, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_root_directory_entries, visit_id=stub_visit_id) self.origin_directory_view_helper(stub_origin_info_no_type, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_root_directory_entries, timestamp=stub_visit_unix_ts) self.origin_directory_view_helper(stub_origin_info_no_type, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_root_directory_entries, timestamp=stub_visit_iso_date) @patch('swh.web.browse.utils.get_origin_visits') @patch('swh.web.browse.utils.get_origin_visit_snapshot') @patch('swh.web.browse.utils.service') @patch('swh.web.browse.views.utils.snapshot_context.service') def test_origin_sub_directory_view(self, mock_origin_service, mock_utils_service, mock_get_origin_visit_snapshot, mock_get_origin_visits): mock_get_origin_visits.return_value = stub_origin_visits mock_get_origin_visit_snapshot.return_value = stub_origin_snapshot mock_utils_service.lookup_directory.return_value = \ stub_origin_sub_directory_entries mock_origin_service.lookup_directory_with_path.return_value = \ {'target': stub_origin_sub_directory_entries[0]['dir_id'], 'type' : 'dir'} mock_utils_service.lookup_origin.return_value = stub_origin_info mock_utils_service.lookup_snapshot_size.return_value = { 'revision': len(stub_origin_snapshot[0]), 'release': len(stub_origin_snapshot[1]) } self.origin_directory_view_helper(stub_origin_info, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_sub_directory_entries, path=stub_origin_sub_directory_path) self.origin_directory_view_helper(stub_origin_info, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_sub_directory_entries, visit_id=stub_visit_id, path=stub_origin_sub_directory_path) self.origin_directory_view_helper(stub_origin_info, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_sub_directory_entries, timestamp=stub_visit_unix_ts, path=stub_origin_sub_directory_path) self.origin_directory_view_helper(stub_origin_info, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_sub_directory_entries, timestamp=stub_visit_iso_date, path=stub_origin_sub_directory_path) self.origin_directory_view_helper(stub_origin_info_no_type, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_sub_directory_entries, path=stub_origin_sub_directory_path) self.origin_directory_view_helper(stub_origin_info_no_type, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_sub_directory_entries, visit_id=stub_visit_id, path=stub_origin_sub_directory_path) self.origin_directory_view_helper(stub_origin_info_no_type, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_sub_directory_entries, timestamp=stub_visit_unix_ts, path=stub_origin_sub_directory_path) self.origin_directory_view_helper(stub_origin_info_no_type, stub_origin_visits, stub_origin_snapshot[0], stub_origin_snapshot[1], stub_origin_master_branch, stub_origin_root_directory_sha1, stub_origin_sub_directory_entries, timestamp=stub_visit_iso_date, path=stub_origin_sub_directory_path) @patch('swh.web.browse.views.utils.snapshot_context.request_content') @patch('swh.web.browse.utils.get_origin_visits') @patch('swh.web.browse.utils.get_origin_visit_snapshot') @patch('swh.web.browse.utils.service') @patch('swh.web.browse.views.origin.service') @patch('swh.web.browse.views.utils.snapshot_context.service') @patch('swh.web.browse.views.origin.get_origin_info') def test_origin_request_errors(self, mock_get_origin_info, mock_snapshot_service, mock_origin_service, mock_utils_service, mock_get_origin_visit_snapshot, mock_get_origin_visits, mock_request_content): mock_get_origin_info.side_effect = \ NotFoundExc('origin not found') url = reverse('browse-origin-visits', url_args={'origin_type': 'foo', 'origin_url': 'bar'}) resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') self.assertContains(resp, 'origin not found', status_code=404) mock_utils_service.lookup_origin.side_effect = None mock_utils_service.lookup_origin.return_value = origin_info_test_data mock_get_origin_visits.return_value = [] url = reverse('browse-origin-directory', url_args={'origin_type': 'foo', 'origin_url': 'bar'}) resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') - self.assertContains(resp, "No SWH visit", status_code=404) + self.assertContains(resp, "No visit", status_code=404) mock_get_origin_visits.return_value = stub_origin_visits mock_get_origin_visit_snapshot.side_effect = \ NotFoundExc('visit not found') url = reverse('browse-origin-directory', url_args={'origin_type': 'foo', 'origin_url': 'bar'}, query_params={'visit_id': len(stub_origin_visits)+1}) resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') self.assertRegex(resp.content.decode('utf-8'), 'Visit.*not found') mock_get_origin_visits.return_value = stub_origin_visits mock_get_origin_visit_snapshot.side_effect = None mock_get_origin_visit_snapshot.return_value = stub_origin_snapshot mock_utils_service.lookup_snapshot_size.return_value = { 'revision': len(stub_origin_snapshot[0]), 'release': len(stub_origin_snapshot[1]) } mock_utils_service.lookup_directory.side_effect = \ NotFoundExc('Directory not found') url = reverse('browse-origin-directory', url_args={'origin_type': 'foo', 'origin_url': 'bar'}) resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') self.assertContains(resp, 'Directory not found', status_code=404) with patch('swh.web.browse.views.utils.snapshot_context.get_snapshot_context') \ as mock_get_snapshot_context: mock_get_snapshot_context.side_effect = \ NotFoundExc('Snapshot not found') url = reverse('browse-origin-directory', url_args={'origin_type': 'foo', 'origin_url': 'bar'}) resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') self.assertContains(resp, 'Snapshot not found', status_code=404) mock_origin_service.lookup_origin.side_effect = None mock_origin_service.lookup_origin.return_value = origin_info_test_data mock_get_origin_visits.return_value = [] url = reverse('browse-origin-content', url_args={'origin_type': 'foo', 'origin_url': 'bar', 'path': 'foo'}) resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') - self.assertContains(resp, "No SWH visit", status_code=404) + self.assertContains(resp, "No visit", status_code=404) mock_get_origin_visits.return_value = stub_origin_visits mock_get_origin_visit_snapshot.side_effect = \ NotFoundExc('visit not found') url = reverse('browse-origin-content', url_args={'origin_type': 'foo', 'origin_url': 'bar', 'path': 'foo'}, query_params={'visit_id': len(stub_origin_visits)+1}) resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') self.assertRegex(resp.content.decode('utf-8'), 'Visit.*not found') mock_get_origin_visits.return_value = stub_origin_visits mock_get_origin_visit_snapshot.side_effect = None mock_get_origin_visit_snapshot.return_value = ([], []) url = reverse('browse-origin-content', url_args={'origin_type': 'foo', 'origin_url': 'bar', 'path': 'baz'}) resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') self.assertRegex(resp.content.decode('utf-8'), 'Origin.*has an empty list of branches') mock_get_origin_visit_snapshot.return_value = stub_origin_snapshot mock_snapshot_service.lookup_directory_with_path.return_value = \ {'target': stub_content_text_data['checksums']['sha1']} mock_request_content.side_effect = \ NotFoundExc('Content not found') url = reverse('browse-origin-content', url_args={'origin_type': 'foo', 'origin_url': 'bar', 'path': 'baz'}) resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') self.assertContains(resp, 'Content not found', status_code=404) @patch('swh.web.browse.utils.get_origin_visits') @patch('swh.web.browse.utils.get_origin_visit_snapshot') @patch('swh.web.browse.utils.service') def test_origin_empty_snapshot(self, mock_utils_service, mock_get_origin_visit_snapshot, mock_get_origin_visits): mock_get_origin_visits.return_value = stub_origin_visits mock_get_origin_visit_snapshot.return_value = ([], []) mock_utils_service.lookup_snapshot_size.return_value = { 'revision': 0, 'release': 0 } url = reverse('browse-origin-directory', url_args={'origin_type': 'foo', 'origin_url': 'bar'}) resp = self.client.get(url) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('content.html') self.assertRegex(resp.content.decode('utf-8'), 'snapshot.*is empty') def origin_branches_helper(self, origin_info, origin_snapshot): url_args = {'origin_type': origin_info['type'], 'origin_url': origin_info['url']} url = reverse('browse-origin-branches', url_args=url_args) resp = self.client.get(url) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('branches.html') origin_branches = origin_snapshot[0] origin_releases = origin_snapshot[1] origin_branches_url = \ reverse('browse-origin-branches', url_args=url_args) self.assertContains(resp, 'Branches (%s)' % (origin_branches_url, len(origin_branches))) origin_releases_url = \ reverse('browse-origin-releases', url_args=url_args) self.assertContains(resp, 'Releases (%s)' % (origin_releases_url, len(origin_releases))) self.assertContains(resp, '' % escape(browse_branch_url)) browse_revision_url = reverse('browse-revision', url_args={'sha1_git': branch['revision']}, query_params={'origin_type': origin_info['type'], 'origin': origin_info['url']}) self.assertContains(resp, '' % escape(browse_revision_url)) @patch('swh.web.browse.views.utils.snapshot_context.process_snapshot_branches') @patch('swh.web.browse.views.utils.snapshot_context.service') @patch('swh.web.browse.utils.get_origin_visits') @patch('swh.web.browse.utils.get_origin_visit_snapshot') @patch('swh.web.browse.utils.service') @patch('swh.web.browse.views.origin.service') def test_origin_branches(self, mock_origin_service, mock_utils_service, mock_get_origin_visit_snapshot, mock_get_origin_visits, mock_snp_ctx_service, mock_snp_ctx_process_branches): mock_get_origin_visits.return_value = stub_origin_visits mock_get_origin_visit_snapshot.return_value = stub_origin_snapshot mock_utils_service.lookup_origin.return_value = stub_origin_info mock_utils_service.lookup_snapshot_size.return_value = \ {'revision': len(stub_origin_snapshot[0]), 'release': len(stub_origin_snapshot[1])} mock_snp_ctx_service.lookup_snapshot.return_value = \ _to_snapshot_dict(branches=stub_origin_snapshot[0]) mock_snp_ctx_process_branches.return_value = stub_origin_snapshot self.origin_branches_helper(stub_origin_info, stub_origin_snapshot) self.origin_branches_helper(stub_origin_info_no_type, stub_origin_snapshot) def origin_releases_helper(self, origin_info, origin_snapshot): url_args = {'origin_type': origin_info['type'], 'origin_url': origin_info['url']} url = reverse('browse-origin-releases', url_args=url_args) resp = self.client.get(url) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('releases.html') origin_branches = origin_snapshot[0] origin_releases = origin_snapshot[1] origin_branches_url = \ reverse('browse-origin-branches', url_args=url_args) self.assertContains(resp, 'Branches (%s)' % (origin_branches_url, len(origin_branches))) origin_releases_url = \ reverse('browse-origin-releases', url_args=url_args) self.assertContains(resp, 'Releases (%s)' % (origin_releases_url, len(origin_releases))) self.assertContains(resp, '' % escape(browse_release_url)) self.assertContains(resp, '' % escape(browse_revision_url)) @patch('swh.web.browse.views.utils.snapshot_context.process_snapshot_branches') @patch('swh.web.browse.views.utils.snapshot_context.service') @patch('swh.web.browse.utils.get_origin_visits') @patch('swh.web.browse.utils.get_origin_visit_snapshot') @patch('swh.web.browse.utils.service') @patch('swh.web.browse.views.origin.service') def test_origin_releases(self, mock_origin_service, mock_utils_service, mock_get_origin_visit_snapshot, mock_get_origin_visits, mock_snp_ctx_service, mock_snp_ctx_process_branches): mock_get_origin_visits.return_value = stub_origin_visits mock_get_origin_visit_snapshot.return_value = stub_origin_snapshot mock_utils_service.lookup_origin.return_value = stub_origin_info mock_utils_service.lookup_snapshot_size.return_value = \ {'revision': len(stub_origin_snapshot[0]), 'release': len(stub_origin_snapshot[1])} mock_snp_ctx_service.lookup_snapshot.return_value = \ _to_snapshot_dict(releases=stub_origin_snapshot[1]) mock_snp_ctx_process_branches.return_value = stub_origin_snapshot self.origin_releases_helper(stub_origin_info, stub_origin_snapshot) self.origin_releases_helper(stub_origin_info_no_type, stub_origin_snapshot)