diff --git a/MANIFEST.in b/MANIFEST.in index 1f4d4094..aef7206e 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,15 +1,16 @@ include Makefile include pytest.ini include README.md include requirements.txt include requirements-swh.txt include requirements-test.txt include tox.ini include version.txt +recursive-include swh py.typed recursive-include swh/web/assets * recursive-include swh/web/static * recursive-include swh/web/templates * recursive-include swh/web/tests/resources * include package.json include yarn.lock diff --git a/PKG-INFO b/PKG-INFO index c62070e3..1fcc2afe 100644 --- a/PKG-INFO +++ b/PKG-INFO @@ -1,127 +1,127 @@ Metadata-Version: 2.1 Name: swh.web -Version: 0.0.219 +Version: 0.0.220 Summary: Software Heritage Web UI Home-page: https://forge.softwareheritage.org/diffusion/DWUI/ Author: Software Heritage developers Author-email: swh-devel@inria.fr License: UNKNOWN +Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Project-URL: Funding, https://www.softwareheritage.org/donate Project-URL: Source, https://forge.softwareheritage.org/source/swh-web -Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Description: # swh-web This repository holds the development of Software Heritage web applications: * swh-web API (https://archive.softwareheritage.org/api): enables to query the content of the archive through HTTP requests and get responses in JSON or YAML. * swh-web browse (https://archive.softwareheritage.org/browse): graphical interface that eases the navigation in the archive. Documentation about how to use these components but also the details of their URI schemes can be found in the docs folder. The produced HTML documentation can be read and browsed at https://docs.softwareheritage.org/devel/swh-web/index.html. ## Technical details Those applications are powered by: * [Django Web Framework](https://www.djangoproject.com/) on the backend side with the following extensions enabled: * [django-rest-framework](http://www.django-rest-framework.org/) * [django-webpack-loader](https://github.com/owais/django-webpack-loader) * [django-js-reverse](http://django-js-reverse.readthedocs.io/en/latest/) * [webpack](https://webpack.js.org/) on the frontend side for better static assets management, including: * assets dependencies management and retrieval through [yarn](https://yarnpkg.com/en/) * linting of custom javascript code (through [eslint](https://eslint.org/)) and stylesheets (through [stylelint](https://stylelint.io/)) * use of [es6](http://es6-features.org) syntax and advanced javascript feature like [async/await](https://javascript.info/async-await) or [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) thanks to [babel](https://babeljs.io/) (es6 to es5 transpiler and polyfills provider) * assets minification (using [terser](https://github.com/terser-js/terser) and [cssnano](http://cssnano.co/)) but also dead code elimination for production use ## How to build, run and test ### Backend requirements First you will need [Python 3](https://www.python.org) and a complete [swh development environment](https://forge.softwareheritage.org/source/swh-environment/) installed. To run the backend, you need to have the following [Python 3 modules](requirements.txt) installed. To run the backend tests, the following [Python 3 modules](requirements-test.txt) are also required to be installed. One easy way to install them is to use the `pip` tool: ``` $ pip install -r requirements.txt -r requirements-test.txt ``` ### Frontend requirements To compile the frontend assets, you need to have [nodejs](https://nodejs.org/en/) >= 8.9.0 and [yarn](https://yarnpkg.com/en/) installed. If you are on Debian stretch, you can easily install an up to date nodejs from the [stretch-backports](https://backports.debian.org/Instructions/) repository. Packages for yarn can be installed by following [these instructions](https://yarnpkg.com/en/docs/install#debian-stable). Alternatively, you can install yarn with `npm install yarn`, and add `YARN=node_modules/yarn/bin/yarn` as argument whenever you run `make`. Please note that the static assets bundles generated by webpack are not stored in the git repository. Follow the instructions below in order to generate them in order to be able to run the frontend part of the web applications. ### Make targets to execute the applications Below is the list of available make targets that can be executed from the root directory of swh-web in order to build and/or execute the web applications under various configurations: * **run-django-webpack-devserver**: Compile and serve not optimized (without mignification and dead code elimination) frontend static assets using [webpack-dev-server](https://github.com/webpack/webpack-dev-server) and run django server with development settings. This is the recommended target to use when developing swh-web as it enables automatic reloading of backend and frontend part of the applications when modifying source files (*.py, *.js, *.css, *.html). * **run-django-webpack-dev**: Compile not optimized (no minification, no dead code elimination) frontend static assets using webpack and run django server with development settings. This is the recommended target when one only wants to develop the backend side of the application. * **run-django-webpack-prod**: Compile optimized (with minification and dead code elimination) frontend static assets using webpack and run django server with production settings. This is useful to test the applications in production mode (with the difference that static assets are served by django). Production settings notably enable advanced django caching and you will need to have [memcached](https://memcached.org/) installed for that feature to work. * **run-django-server-dev**: Run the django server with development settings but without compiling frontend static assets through webpack. * **run-django-server-prod**: Run the django server with production settings but without compiling frontend static assets through webpack. * **run-gunicorn-server**: Run the web applications with production settings in a [gunicorn](http://gunicorn.org/) worker as they will be in real production environment. Once one of these targets executed, the web applications can be executed by pointing your browser to http://localhost:5004. ### Make targets to test the applications Some make targets are also available to easily execute the backend and frontend tests of the Software Heritage web applications. The backend tests are powered by the [pytest](https://docs.pytest.org/en/latest/) and [hypothesis](https://hypothesis.readthedocs.io/en/latest/) frameworks while the frontend ones rely on the use of the [cypress](https://www.cypress.io/) tool. Below is the exhaustive list of those targets: * **test**: execute the backend tests using a fast hypothesis profile (only one input example will be provided for each test) * **test-full**: execute the backend tests using a slower hypothesis profile (one hundred of input examples will be provided for each test which helps spotting possible bugs) * **test-frontend**: execute the frontend tests using cypress in headless mode but with some slow test suites disabled * **test-frontend-full**: execute the frontend tests using cypress in headless mode with all test suites enabled * **test-frontend-ui**: execute the frontend tests using the cypress GUI but with some slow test suites disabled * **test-frontend-full-ui**: execute the frontend tests using the cypress GUI with all test suites enabled ### Yarn targets Below is a list of available yarn targets in order to only execute the frontend static assets compilation (no web server will be executed): * **build-dev**: compile not optimized (without mignification and dead code elimination) frontend static assets and store the results in the `swh/web/static` folder. * **build**: compile optimized (with mignification and dead code elimination) frontend static assets and store the results in the `swh/web/static` folder. **The build target must be executed prior performing the Debian packaging of swh-web** in order for the package to contain the optimized assets dedicated to production environment. To execute these targets, issue the following command: ``` $ yarn ``` Platform: UNKNOWN Classifier: Programming Language :: Python :: 3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+) Classifier: Operating System :: OS Independent Classifier: Development Status :: 5 - Production/Stable Classifier: Framework :: Django Description-Content-Type: text/markdown Provides-Extra: testing diff --git a/requirements-test.txt b/requirements-test.txt index 489af9b0..576fe193 100644 --- a/requirements-test.txt +++ b/requirements-test.txt @@ -1,7 +1,8 @@ hypothesis pytest pytest-django pytest-mock +django-stubs requests-mock swh.core[http] >= 0.0.61 swh.loader.git >= 0.0.47 diff --git a/swh.web.egg-info/PKG-INFO b/swh.web.egg-info/PKG-INFO index c62070e3..1fcc2afe 100644 --- a/swh.web.egg-info/PKG-INFO +++ b/swh.web.egg-info/PKG-INFO @@ -1,127 +1,127 @@ Metadata-Version: 2.1 Name: swh.web -Version: 0.0.219 +Version: 0.0.220 Summary: Software Heritage Web UI Home-page: https://forge.softwareheritage.org/diffusion/DWUI/ Author: Software Heritage developers Author-email: swh-devel@inria.fr License: UNKNOWN +Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Project-URL: Funding, https://www.softwareheritage.org/donate Project-URL: Source, https://forge.softwareheritage.org/source/swh-web -Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Description: # swh-web This repository holds the development of Software Heritage web applications: * swh-web API (https://archive.softwareheritage.org/api): enables to query the content of the archive through HTTP requests and get responses in JSON or YAML. * swh-web browse (https://archive.softwareheritage.org/browse): graphical interface that eases the navigation in the archive. Documentation about how to use these components but also the details of their URI schemes can be found in the docs folder. The produced HTML documentation can be read and browsed at https://docs.softwareheritage.org/devel/swh-web/index.html. ## Technical details Those applications are powered by: * [Django Web Framework](https://www.djangoproject.com/) on the backend side with the following extensions enabled: * [django-rest-framework](http://www.django-rest-framework.org/) * [django-webpack-loader](https://github.com/owais/django-webpack-loader) * [django-js-reverse](http://django-js-reverse.readthedocs.io/en/latest/) * [webpack](https://webpack.js.org/) on the frontend side for better static assets management, including: * assets dependencies management and retrieval through [yarn](https://yarnpkg.com/en/) * linting of custom javascript code (through [eslint](https://eslint.org/)) and stylesheets (through [stylelint](https://stylelint.io/)) * use of [es6](http://es6-features.org) syntax and advanced javascript feature like [async/await](https://javascript.info/async-await) or [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) thanks to [babel](https://babeljs.io/) (es6 to es5 transpiler and polyfills provider) * assets minification (using [terser](https://github.com/terser-js/terser) and [cssnano](http://cssnano.co/)) but also dead code elimination for production use ## How to build, run and test ### Backend requirements First you will need [Python 3](https://www.python.org) and a complete [swh development environment](https://forge.softwareheritage.org/source/swh-environment/) installed. To run the backend, you need to have the following [Python 3 modules](requirements.txt) installed. To run the backend tests, the following [Python 3 modules](requirements-test.txt) are also required to be installed. One easy way to install them is to use the `pip` tool: ``` $ pip install -r requirements.txt -r requirements-test.txt ``` ### Frontend requirements To compile the frontend assets, you need to have [nodejs](https://nodejs.org/en/) >= 8.9.0 and [yarn](https://yarnpkg.com/en/) installed. If you are on Debian stretch, you can easily install an up to date nodejs from the [stretch-backports](https://backports.debian.org/Instructions/) repository. Packages for yarn can be installed by following [these instructions](https://yarnpkg.com/en/docs/install#debian-stable). Alternatively, you can install yarn with `npm install yarn`, and add `YARN=node_modules/yarn/bin/yarn` as argument whenever you run `make`. Please note that the static assets bundles generated by webpack are not stored in the git repository. Follow the instructions below in order to generate them in order to be able to run the frontend part of the web applications. ### Make targets to execute the applications Below is the list of available make targets that can be executed from the root directory of swh-web in order to build and/or execute the web applications under various configurations: * **run-django-webpack-devserver**: Compile and serve not optimized (without mignification and dead code elimination) frontend static assets using [webpack-dev-server](https://github.com/webpack/webpack-dev-server) and run django server with development settings. This is the recommended target to use when developing swh-web as it enables automatic reloading of backend and frontend part of the applications when modifying source files (*.py, *.js, *.css, *.html). * **run-django-webpack-dev**: Compile not optimized (no minification, no dead code elimination) frontend static assets using webpack and run django server with development settings. This is the recommended target when one only wants to develop the backend side of the application. * **run-django-webpack-prod**: Compile optimized (with minification and dead code elimination) frontend static assets using webpack and run django server with production settings. This is useful to test the applications in production mode (with the difference that static assets are served by django). Production settings notably enable advanced django caching and you will need to have [memcached](https://memcached.org/) installed for that feature to work. * **run-django-server-dev**: Run the django server with development settings but without compiling frontend static assets through webpack. * **run-django-server-prod**: Run the django server with production settings but without compiling frontend static assets through webpack. * **run-gunicorn-server**: Run the web applications with production settings in a [gunicorn](http://gunicorn.org/) worker as they will be in real production environment. Once one of these targets executed, the web applications can be executed by pointing your browser to http://localhost:5004. ### Make targets to test the applications Some make targets are also available to easily execute the backend and frontend tests of the Software Heritage web applications. The backend tests are powered by the [pytest](https://docs.pytest.org/en/latest/) and [hypothesis](https://hypothesis.readthedocs.io/en/latest/) frameworks while the frontend ones rely on the use of the [cypress](https://www.cypress.io/) tool. Below is the exhaustive list of those targets: * **test**: execute the backend tests using a fast hypothesis profile (only one input example will be provided for each test) * **test-full**: execute the backend tests using a slower hypothesis profile (one hundred of input examples will be provided for each test which helps spotting possible bugs) * **test-frontend**: execute the frontend tests using cypress in headless mode but with some slow test suites disabled * **test-frontend-full**: execute the frontend tests using cypress in headless mode with all test suites enabled * **test-frontend-ui**: execute the frontend tests using the cypress GUI but with some slow test suites disabled * **test-frontend-full-ui**: execute the frontend tests using the cypress GUI with all test suites enabled ### Yarn targets Below is a list of available yarn targets in order to only execute the frontend static assets compilation (no web server will be executed): * **build-dev**: compile not optimized (without mignification and dead code elimination) frontend static assets and store the results in the `swh/web/static` folder. * **build**: compile optimized (with mignification and dead code elimination) frontend static assets and store the results in the `swh/web/static` folder. **The build target must be executed prior performing the Debian packaging of swh-web** in order for the package to contain the optimized assets dedicated to production environment. To execute these targets, issue the following command: ``` $ yarn ``` Platform: UNKNOWN Classifier: Programming Language :: Python :: 3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+) Classifier: Operating System :: OS Independent Classifier: Development Status :: 5 - Production/Stable Classifier: Framework :: Django Description-Content-Type: text/markdown Provides-Extra: testing diff --git a/swh.web.egg-info/SOURCES.txt b/swh.web.egg-info/SOURCES.txt index 3c26de44..96852092 100644 --- a/swh.web.egg-info/SOURCES.txt +++ b/swh.web.egg-info/SOURCES.txt @@ -1,1518 +1,1519 @@ MANIFEST.in Makefile README.md package.json pytest.ini requirements-swh.txt requirements-test.txt requirements.txt setup.py tox.ini version.txt yarn.lock swh/__init__.py swh.web.egg-info/PKG-INFO swh.web.egg-info/SOURCES.txt swh.web.egg-info/dependency_links.txt swh.web.egg-info/requires.txt swh.web.egg-info/top_level.txt swh/web/__init__.py swh/web/config.py swh/web/doc_config.py swh/web/manage.py +swh/web/py.typed swh/web/urls.py swh/web/admin/__init__.py swh/web/admin/adminurls.py swh/web/admin/deposit.py swh/web/admin/origin_save.py swh/web/admin/urls.py swh/web/api/__init__.py swh/web/api/apidoc.py swh/web/api/apiresponse.py swh/web/api/apiurls.py swh/web/api/renderers.py swh/web/api/urls.py swh/web/api/utils.py swh/web/api/views/__init__.py swh/web/api/views/content.py swh/web/api/views/directory.py swh/web/api/views/identifiers.py swh/web/api/views/origin.py swh/web/api/views/origin_save.py swh/web/api/views/release.py swh/web/api/views/revision.py swh/web/api/views/snapshot.py swh/web/api/views/stat.py swh/web/api/views/utils.py swh/web/api/views/vault.py swh/web/assets/config/.bootstraprc swh/web/assets/config/.eslintignore swh/web/assets/config/.eslintrc swh/web/assets/config/bootstrap-pre-customize.scss swh/web/assets/config/mathjax-js-files.js swh/web/assets/config/webpack.config.development.js swh/web/assets/config/webpack.config.production.js swh/web/assets/config/webpack-plugins/fix-swh-source-maps-webpack-plugin.js swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/README.md swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/index.js swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/jslicenses.ejs swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/plugin-options-schema.json swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/spdx-licenses-mapping.js swh/web/assets/src/bundles/admin/deposit.js swh/web/assets/src/bundles/admin/index.js swh/web/assets/src/bundles/admin/origin-save.js swh/web/assets/src/bundles/browse/breadcrumbs.css swh/web/assets/src/bundles/browse/browse-utils.js swh/web/assets/src/bundles/browse/browse.css swh/web/assets/src/bundles/browse/content.css swh/web/assets/src/bundles/browse/index.js swh/web/assets/src/bundles/browse/origin-search.js swh/web/assets/src/bundles/browse/snapshot-navigation.css swh/web/assets/src/bundles/browse/snapshot-navigation.js swh/web/assets/src/bundles/browse/swh-ids-utils.js swh/web/assets/src/bundles/origin/index.js swh/web/assets/src/bundles/origin/visits-calendar.js swh/web/assets/src/bundles/origin/visits-histogram.js swh/web/assets/src/bundles/origin/visits-reporting.css swh/web/assets/src/bundles/origin/visits-reporting.js swh/web/assets/src/bundles/revision/diff-utils.js swh/web/assets/src/bundles/revision/index.js swh/web/assets/src/bundles/revision/log-utils.js swh/web/assets/src/bundles/revision/revision.css swh/web/assets/src/bundles/save/index.js swh/web/assets/src/bundles/vault/index.js swh/web/assets/src/bundles/vault/vault-create-tasks.js swh/web/assets/src/bundles/vault/vault-ui.js swh/web/assets/src/bundles/vault/vault.css swh/web/assets/src/bundles/vendors/datatables.css swh/web/assets/src/bundles/vendors/index.js swh/web/assets/src/bundles/vendors/octicons.css swh/web/assets/src/bundles/webapp/breadcrumbs.css swh/web/assets/src/bundles/webapp/code-highlighting.js swh/web/assets/src/bundles/webapp/history-counters.css swh/web/assets/src/bundles/webapp/history-counters.js swh/web/assets/src/bundles/webapp/index.js swh/web/assets/src/bundles/webapp/notebook-rendering.js swh/web/assets/src/bundles/webapp/notebook.css swh/web/assets/src/bundles/webapp/pdf-rendering.js swh/web/assets/src/bundles/webapp/readme-rendering.js swh/web/assets/src/bundles/webapp/webapp-utils.js swh/web/assets/src/bundles/webapp/webapp.css swh/web/assets/src/bundles/webapp/xss-filtering.js swh/web/assets/src/thirdparty/jquery.tabSlideOut/LICENSE swh/web/assets/src/thirdparty/jquery.tabSlideOut/jquery.tabSlideOut.css swh/web/assets/src/thirdparty/jquery.tabSlideOut/jquery.tabSlideOut.js swh/web/assets/src/utils/constants.js swh/web/assets/src/utils/d3-custom.js swh/web/assets/src/utils/d3.js swh/web/assets/src/utils/functions.js swh/web/assets/src/utils/heaps-permute.js swh/web/assets/src/utils/highlightjs.css swh/web/assets/src/utils/highlightjs.js swh/web/assets/src/utils/org.css swh/web/assets/src/utils/org.js swh/web/assets/src/utils/showdown.css swh/web/assets/src/utils/showdown.js swh/web/browse/__init__.py swh/web/browse/browseurls.py swh/web/browse/identifiers.py swh/web/browse/urls.py swh/web/browse/utils.py swh/web/browse/views/__init__.py swh/web/browse/views/content.py swh/web/browse/views/directory.py swh/web/browse/views/origin.py swh/web/browse/views/release.py swh/web/browse/views/revision.py swh/web/browse/views/snapshot.py swh/web/browse/views/utils/__init__.py swh/web/browse/views/utils/snapshot_context.py swh/web/common/__init__.py swh/web/common/apps.py swh/web/common/converters.py swh/web/common/exc.py swh/web/common/highlightjs.py swh/web/common/middlewares.py swh/web/common/models.py swh/web/common/origin_save.py swh/web/common/origin_visits.py swh/web/common/query.py swh/web/common/service.py swh/web/common/swh_templatetags.py swh/web/common/throttling.py swh/web/common/urlsindex.py swh/web/common/utils.py swh/web/common/migrations/0001_initial.py swh/web/common/migrations/0002_saveoriginrequest_visit_date.py swh/web/common/migrations/0003_saveoriginrequest_loading_task_status.py swh/web/common/migrations/0004_auto_20190204_1324.py swh/web/common/migrations/0005_remove_duplicated_authorized_origins.py swh/web/common/migrations/0006_rename_origin_type.py swh/web/common/migrations/__init__.py swh/web/misc/__init__.py swh/web/misc/coverage.py swh/web/misc/origin_save.py swh/web/misc/urls.py swh/web/settings/__init__.py swh/web/settings/common.py swh/web/settings/development.py swh/web/settings/production.py swh/web/settings/tests.py swh/web/static/robots.txt swh/web/static/webpack-stats.json swh/web/static/css/browse.85c0b69e17ffd2e4a76d.css swh/web/static/css/browse.85c0b69e17ffd2e4a76d.css.map swh/web/static/css/highlightjs.5396a1c65059bf62e73c.css swh/web/static/css/highlightjs.5396a1c65059bf62e73c.css.map swh/web/static/css/org.7803b1b03dd8a213f240.css swh/web/static/css/org.7803b1b03dd8a213f240.css.map swh/web/static/css/origin.535108e895bd05a2d39c.css swh/web/static/css/origin.535108e895bd05a2d39c.css.map swh/web/static/css/revision.9654a4d587e88181e39c.css swh/web/static/css/revision.9654a4d587e88181e39c.css.map swh/web/static/css/showdown.4e75568b9b7500a9c4ae.css swh/web/static/css/showdown.4e75568b9b7500a9c4ae.css.map swh/web/static/css/vault.61b43667fb07c8dc8bf4.css swh/web/static/css/vault.61b43667fb07c8dc8bf4.css.map swh/web/static/css/vendors.428a95a7975d029360e1.css swh/web/static/css/vendors.428a95a7975d029360e1.css.map swh/web/static/css/webapp.abbc9a80b97c0b3027de.css swh/web/static/css/webapp.abbc9a80b97c0b3027de.css.map swh/web/static/fonts/alegreya-latin-400.woff swh/web/static/fonts/alegreya-latin-400.woff2 swh/web/static/fonts/alegreya-latin-400italic.woff swh/web/static/fonts/alegreya-latin-400italic.woff2 swh/web/static/fonts/alegreya-latin-500.woff swh/web/static/fonts/alegreya-latin-500.woff2 swh/web/static/fonts/alegreya-latin-500italic.woff swh/web/static/fonts/alegreya-latin-500italic.woff2 swh/web/static/fonts/alegreya-latin-700.woff swh/web/static/fonts/alegreya-latin-700.woff2 swh/web/static/fonts/alegreya-latin-700italic.woff swh/web/static/fonts/alegreya-latin-700italic.woff2 swh/web/static/fonts/alegreya-latin-800.woff swh/web/static/fonts/alegreya-latin-800.woff2 swh/web/static/fonts/alegreya-latin-800italic.woff swh/web/static/fonts/alegreya-latin-800italic.woff2 swh/web/static/fonts/alegreya-latin-900.woff swh/web/static/fonts/alegreya-latin-900.woff2 swh/web/static/fonts/alegreya-latin-900italic.woff swh/web/static/fonts/alegreya-latin-900italic.woff2 swh/web/static/fonts/alegreya-sans-latin-100.woff swh/web/static/fonts/alegreya-sans-latin-100.woff2 swh/web/static/fonts/alegreya-sans-latin-100italic.woff swh/web/static/fonts/alegreya-sans-latin-100italic.woff2 swh/web/static/fonts/alegreya-sans-latin-300.woff swh/web/static/fonts/alegreya-sans-latin-300.woff2 swh/web/static/fonts/alegreya-sans-latin-300italic.woff swh/web/static/fonts/alegreya-sans-latin-300italic.woff2 swh/web/static/fonts/alegreya-sans-latin-400.woff swh/web/static/fonts/alegreya-sans-latin-400.woff2 swh/web/static/fonts/alegreya-sans-latin-400italic.woff swh/web/static/fonts/alegreya-sans-latin-400italic.woff2 swh/web/static/fonts/alegreya-sans-latin-500.woff swh/web/static/fonts/alegreya-sans-latin-500.woff2 swh/web/static/fonts/alegreya-sans-latin-500italic.woff swh/web/static/fonts/alegreya-sans-latin-500italic.woff2 swh/web/static/fonts/alegreya-sans-latin-700.woff swh/web/static/fonts/alegreya-sans-latin-700.woff2 swh/web/static/fonts/alegreya-sans-latin-700italic.woff swh/web/static/fonts/alegreya-sans-latin-700italic.woff2 swh/web/static/fonts/alegreya-sans-latin-800.woff swh/web/static/fonts/alegreya-sans-latin-800.woff2 swh/web/static/fonts/alegreya-sans-latin-800italic.woff swh/web/static/fonts/alegreya-sans-latin-800italic.woff2 swh/web/static/fonts/alegreya-sans-latin-900.woff swh/web/static/fonts/alegreya-sans-latin-900.woff2 swh/web/static/fonts/alegreya-sans-latin-900italic.woff swh/web/static/fonts/alegreya-sans-latin-900italic.woff2 swh/web/static/fonts/fontawesome-webfont.eot swh/web/static/fonts/fontawesome-webfont.svg swh/web/static/fonts/fontawesome-webfont.ttf swh/web/static/fonts/fontawesome-webfont.woff swh/web/static/fonts/fontawesome-webfont.woff2 swh/web/static/fonts/git-commit.svg swh/web/static/img/arrow-up-small.png swh/web/static/img/swh-api.png swh/web/static/img/swh-browse.png swh/web/static/img/swh-logo.png swh/web/static/img/swh-logo.svg swh/web/static/img/swh-spinner-small.gif swh/web/static/img/swh-spinner.gif swh/web/static/img/swh-support.png swh/web/static/img/swh-vault.png swh/web/static/img/icons/swh-logo-32x32.png swh/web/static/img/icons/swh-logo-archive-180x180.png swh/web/static/img/icons/swh-logo-archive-192x192.png swh/web/static/img/icons/swh-logo-archive-270x270.png swh/web/static/img/logos/bitbucket.png swh/web/static/img/logos/debian.png swh/web/static/img/logos/framagit.png swh/web/static/img/logos/github.png swh/web/static/img/logos/gitlab.svg swh/web/static/img/logos/gitorious.png swh/web/static/img/logos/gnu.png swh/web/static/img/logos/googlecode.png swh/web/static/img/logos/hal.png swh/web/static/img/logos/inria.jpg swh/web/static/img/logos/npm.png swh/web/static/img/logos/pypi.svg swh/web/static/img/thirdParty/chosen-sprite.png swh/web/static/img/thirdParty/chosen-sprite@2x.png swh/web/static/js/admin.04ec02d78c3315c65a4c.js swh/web/static/js/admin.04ec02d78c3315c65a4c.js.map swh/web/static/js/browse.85c0b69e17ffd2e4a76d.js swh/web/static/js/browse.85c0b69e17ffd2e4a76d.js.LICENSE swh/web/static/js/browse.85c0b69e17ffd2e4a76d.js.map swh/web/static/js/d3.4925a71e9248edd5b6eb.js swh/web/static/js/d3.4925a71e9248edd5b6eb.js.map swh/web/static/js/highlightjs.5396a1c65059bf62e73c.js swh/web/static/js/highlightjs.5396a1c65059bf62e73c.js.map swh/web/static/js/org.7803b1b03dd8a213f240.js swh/web/static/js/org.7803b1b03dd8a213f240.js.map swh/web/static/js/origin.535108e895bd05a2d39c.js swh/web/static/js/origin.535108e895bd05a2d39c.js.map swh/web/static/js/pdf.worker.min.js swh/web/static/js/pdfjs.cb6f00dfe3efe93823a2.js swh/web/static/js/pdfjs.cb6f00dfe3efe93823a2.js.map swh/web/static/js/revision.9654a4d587e88181e39c.js swh/web/static/js/revision.9654a4d587e88181e39c.js.LICENSE swh/web/static/js/revision.9654a4d587e88181e39c.js.map swh/web/static/js/save.1afcb2cb3d3a92e94807.js swh/web/static/js/save.1afcb2cb3d3a92e94807.js.LICENSE swh/web/static/js/save.1afcb2cb3d3a92e94807.js.map swh/web/static/js/showdown.4e75568b9b7500a9c4ae.js swh/web/static/js/showdown.4e75568b9b7500a9c4ae.js.LICENSE swh/web/static/js/showdown.4e75568b9b7500a9c4ae.js.map swh/web/static/js/vault.61b43667fb07c8dc8bf4.js swh/web/static/js/vault.61b43667fb07c8dc8bf4.js.map swh/web/static/js/vendors.428a95a7975d029360e1.js swh/web/static/js/vendors.428a95a7975d029360e1.js.LICENSE swh/web/static/js/vendors.428a95a7975d029360e1.js.map swh/web/static/js/webapp.abbc9a80b97c0b3027de.js swh/web/static/js/webapp.abbc9a80b97c0b3027de.js.LICENSE swh/web/static/js/webapp.abbc9a80b97c0b3027de.js.map swh/web/static/jssources/LICENSE.txt swh/web/static/jssources/jslicenses.json swh/web/static/jssources/@babel/runtime/LICENSE.txt swh/web/static/jssources/@babel/runtime/helpers/asyncToGenerator.js swh/web/static/jssources/@babel/runtime/node_modules/regenerator-runtime/runtime-module.js swh/web/static/jssources/@babel/runtime/node_modules/regenerator-runtime/runtime.js swh/web/static/jssources/@babel/runtime/regenerator/index.js swh/web/static/jssources/admin-lte/LICENSE.txt swh/web/static/jssources/admin-lte/dist/js/adminlte.js swh/web/static/jssources/ansi_up/LICENSE.txt swh/web/static/jssources/ansi_up/ansi_up.js swh/web/static/jssources/bootstrap/LICENSE.txt swh/web/static/jssources/bootstrap/js/dist/alert.js swh/web/static/jssources/bootstrap/js/dist/button.js swh/web/static/jssources/bootstrap/js/dist/carousel.js swh/web/static/jssources/bootstrap/js/dist/collapse.js swh/web/static/jssources/bootstrap/js/dist/dropdown.js swh/web/static/jssources/bootstrap/js/dist/modal.js swh/web/static/jssources/bootstrap/js/dist/popover.js swh/web/static/jssources/bootstrap/js/dist/scrollspy.js swh/web/static/jssources/bootstrap/js/dist/tab.js swh/web/static/jssources/bootstrap/js/dist/tooltip.js swh/web/static/jssources/bootstrap/js/dist/util.js swh/web/static/jssources/chosen-js/LICENSE.md swh/web/static/jssources/chosen-js/chosen.jquery.js swh/web/static/jssources/clipboard/dist/clipboard.js swh/web/static/jssources/core-js/LICENSE.txt swh/web/static/jssources/core-js/es/index.js swh/web/static/jssources/core-js/internals/a-function.js swh/web/static/jssources/core-js/internals/a-possible-prototype.js swh/web/static/jssources/core-js/internals/add-to-unscopables.js swh/web/static/jssources/core-js/internals/advance-string-index.js swh/web/static/jssources/core-js/internals/an-instance.js swh/web/static/jssources/core-js/internals/an-object.js swh/web/static/jssources/core-js/internals/array-buffer-view-core.js swh/web/static/jssources/core-js/internals/array-buffer.js swh/web/static/jssources/core-js/internals/array-copy-within.js swh/web/static/jssources/core-js/internals/array-fill.js swh/web/static/jssources/core-js/internals/array-for-each.js swh/web/static/jssources/core-js/internals/array-from.js swh/web/static/jssources/core-js/internals/array-includes.js swh/web/static/jssources/core-js/internals/array-iteration.js swh/web/static/jssources/core-js/internals/array-last-index-of.js swh/web/static/jssources/core-js/internals/array-method-has-species-support.js swh/web/static/jssources/core-js/internals/array-reduce.js swh/web/static/jssources/core-js/internals/array-species-create.js swh/web/static/jssources/core-js/internals/bind-context.js swh/web/static/jssources/core-js/internals/call-with-safe-iteration-closing.js swh/web/static/jssources/core-js/internals/check-correctness-of-iteration.js swh/web/static/jssources/core-js/internals/classof-raw.js swh/web/static/jssources/core-js/internals/classof.js swh/web/static/jssources/core-js/internals/collection-strong.js swh/web/static/jssources/core-js/internals/collection-weak.js swh/web/static/jssources/core-js/internals/collection.js swh/web/static/jssources/core-js/internals/copy-constructor-properties.js swh/web/static/jssources/core-js/internals/correct-is-regexp-logic.js swh/web/static/jssources/core-js/internals/correct-prototype-getter.js swh/web/static/jssources/core-js/internals/create-html.js swh/web/static/jssources/core-js/internals/create-iterator-constructor.js swh/web/static/jssources/core-js/internals/create-non-enumerable-property.js swh/web/static/jssources/core-js/internals/create-property-descriptor.js swh/web/static/jssources/core-js/internals/create-property.js swh/web/static/jssources/core-js/internals/date-to-iso-string.js swh/web/static/jssources/core-js/internals/date-to-primitive.js swh/web/static/jssources/core-js/internals/define-iterator.js swh/web/static/jssources/core-js/internals/define-well-known-symbol.js swh/web/static/jssources/core-js/internals/descriptors.js swh/web/static/jssources/core-js/internals/document-create-element.js swh/web/static/jssources/core-js/internals/dom-iterables.js swh/web/static/jssources/core-js/internals/enum-bug-keys.js swh/web/static/jssources/core-js/internals/export.js swh/web/static/jssources/core-js/internals/fails.js swh/web/static/jssources/core-js/internals/fix-regexp-well-known-symbol-logic.js swh/web/static/jssources/core-js/internals/flatten-into-array.js swh/web/static/jssources/core-js/internals/forced-object-prototype-accessors-methods.js swh/web/static/jssources/core-js/internals/forced-string-html-method.js swh/web/static/jssources/core-js/internals/forced-string-trim-method.js swh/web/static/jssources/core-js/internals/freezing.js swh/web/static/jssources/core-js/internals/function-bind.js swh/web/static/jssources/core-js/internals/function-to-string.js swh/web/static/jssources/core-js/internals/get-built-in.js swh/web/static/jssources/core-js/internals/get-iterator-method.js swh/web/static/jssources/core-js/internals/get-iterator.js swh/web/static/jssources/core-js/internals/global.js swh/web/static/jssources/core-js/internals/has.js swh/web/static/jssources/core-js/internals/hidden-keys.js swh/web/static/jssources/core-js/internals/host-report-errors.js swh/web/static/jssources/core-js/internals/html.js swh/web/static/jssources/core-js/internals/ie8-dom-define.js swh/web/static/jssources/core-js/internals/indexed-object.js swh/web/static/jssources/core-js/internals/inherit-if-required.js swh/web/static/jssources/core-js/internals/internal-metadata.js swh/web/static/jssources/core-js/internals/internal-state.js swh/web/static/jssources/core-js/internals/is-array-iterator-method.js swh/web/static/jssources/core-js/internals/is-array.js swh/web/static/jssources/core-js/internals/is-forced.js swh/web/static/jssources/core-js/internals/is-integer.js swh/web/static/jssources/core-js/internals/is-object.js swh/web/static/jssources/core-js/internals/is-pure.js swh/web/static/jssources/core-js/internals/is-regexp.js swh/web/static/jssources/core-js/internals/iterate.js swh/web/static/jssources/core-js/internals/iterators-core.js swh/web/static/jssources/core-js/internals/iterators.js swh/web/static/jssources/core-js/internals/math-expm1.js swh/web/static/jssources/core-js/internals/math-fround.js swh/web/static/jssources/core-js/internals/math-log1p.js swh/web/static/jssources/core-js/internals/math-sign.js swh/web/static/jssources/core-js/internals/microtask.js swh/web/static/jssources/core-js/internals/native-promise-constructor.js swh/web/static/jssources/core-js/internals/native-symbol.js swh/web/static/jssources/core-js/internals/native-url.js swh/web/static/jssources/core-js/internals/native-weak-map.js swh/web/static/jssources/core-js/internals/new-promise-capability.js swh/web/static/jssources/core-js/internals/not-a-regexp.js swh/web/static/jssources/core-js/internals/number-is-finite.js swh/web/static/jssources/core-js/internals/object-assign.js swh/web/static/jssources/core-js/internals/object-create.js swh/web/static/jssources/core-js/internals/object-define-properties.js swh/web/static/jssources/core-js/internals/object-define-property.js swh/web/static/jssources/core-js/internals/object-get-own-property-descriptor.js swh/web/static/jssources/core-js/internals/object-get-own-property-names-external.js swh/web/static/jssources/core-js/internals/object-get-own-property-names.js swh/web/static/jssources/core-js/internals/object-get-own-property-symbols.js swh/web/static/jssources/core-js/internals/object-get-prototype-of.js swh/web/static/jssources/core-js/internals/object-keys-internal.js swh/web/static/jssources/core-js/internals/object-keys.js swh/web/static/jssources/core-js/internals/object-property-is-enumerable.js swh/web/static/jssources/core-js/internals/object-set-prototype-of.js swh/web/static/jssources/core-js/internals/object-to-array.js swh/web/static/jssources/core-js/internals/object-to-string.js swh/web/static/jssources/core-js/internals/own-keys.js swh/web/static/jssources/core-js/internals/parse-float.js swh/web/static/jssources/core-js/internals/parse-int.js swh/web/static/jssources/core-js/internals/path.js swh/web/static/jssources/core-js/internals/perform.js swh/web/static/jssources/core-js/internals/promise-resolve.js swh/web/static/jssources/core-js/internals/punycode-to-ascii.js swh/web/static/jssources/core-js/internals/redefine-all.js swh/web/static/jssources/core-js/internals/redefine.js swh/web/static/jssources/core-js/internals/regexp-exec-abstract.js swh/web/static/jssources/core-js/internals/regexp-exec.js swh/web/static/jssources/core-js/internals/regexp-flags.js swh/web/static/jssources/core-js/internals/require-object-coercible.js swh/web/static/jssources/core-js/internals/same-value.js swh/web/static/jssources/core-js/internals/set-global.js swh/web/static/jssources/core-js/internals/set-species.js swh/web/static/jssources/core-js/internals/set-to-string-tag.js swh/web/static/jssources/core-js/internals/shared-key.js swh/web/static/jssources/core-js/internals/shared-store.js swh/web/static/jssources/core-js/internals/shared.js swh/web/static/jssources/core-js/internals/sloppy-array-method.js swh/web/static/jssources/core-js/internals/species-constructor.js swh/web/static/jssources/core-js/internals/string-multibyte.js swh/web/static/jssources/core-js/internals/string-pad.js swh/web/static/jssources/core-js/internals/string-repeat.js swh/web/static/jssources/core-js/internals/string-trim.js swh/web/static/jssources/core-js/internals/task.js swh/web/static/jssources/core-js/internals/this-number-value.js swh/web/static/jssources/core-js/internals/to-absolute-index.js swh/web/static/jssources/core-js/internals/to-index.js swh/web/static/jssources/core-js/internals/to-indexed-object.js swh/web/static/jssources/core-js/internals/to-integer.js swh/web/static/jssources/core-js/internals/to-length.js swh/web/static/jssources/core-js/internals/to-object.js swh/web/static/jssources/core-js/internals/to-offset.js swh/web/static/jssources/core-js/internals/to-positive-integer.js swh/web/static/jssources/core-js/internals/to-primitive.js swh/web/static/jssources/core-js/internals/typed-array-constructor.js swh/web/static/jssources/core-js/internals/typed-array-from.js swh/web/static/jssources/core-js/internals/typed-arrays-constructors-requires-wrappers.js swh/web/static/jssources/core-js/internals/uid.js swh/web/static/jssources/core-js/internals/user-agent.js swh/web/static/jssources/core-js/internals/v8-version.js swh/web/static/jssources/core-js/internals/webkit-string-pad-bug.js swh/web/static/jssources/core-js/internals/well-known-symbol.js swh/web/static/jssources/core-js/internals/whitespaces.js swh/web/static/jssources/core-js/internals/wrapped-well-known-symbol.js swh/web/static/jssources/core-js/modules/es.array-buffer.constructor.js swh/web/static/jssources/core-js/modules/es.array-buffer.is-view.js swh/web/static/jssources/core-js/modules/es.array-buffer.slice.js swh/web/static/jssources/core-js/modules/es.array.concat.js swh/web/static/jssources/core-js/modules/es.array.copy-within.js swh/web/static/jssources/core-js/modules/es.array.every.js swh/web/static/jssources/core-js/modules/es.array.fill.js swh/web/static/jssources/core-js/modules/es.array.filter.js swh/web/static/jssources/core-js/modules/es.array.find-index.js swh/web/static/jssources/core-js/modules/es.array.find.js swh/web/static/jssources/core-js/modules/es.array.flat-map.js swh/web/static/jssources/core-js/modules/es.array.flat.js swh/web/static/jssources/core-js/modules/es.array.for-each.js swh/web/static/jssources/core-js/modules/es.array.from.js swh/web/static/jssources/core-js/modules/es.array.includes.js swh/web/static/jssources/core-js/modules/es.array.index-of.js swh/web/static/jssources/core-js/modules/es.array.is-array.js swh/web/static/jssources/core-js/modules/es.array.iterator.js swh/web/static/jssources/core-js/modules/es.array.join.js swh/web/static/jssources/core-js/modules/es.array.last-index-of.js swh/web/static/jssources/core-js/modules/es.array.map.js swh/web/static/jssources/core-js/modules/es.array.of.js swh/web/static/jssources/core-js/modules/es.array.reduce-right.js swh/web/static/jssources/core-js/modules/es.array.reduce.js swh/web/static/jssources/core-js/modules/es.array.reverse.js swh/web/static/jssources/core-js/modules/es.array.slice.js swh/web/static/jssources/core-js/modules/es.array.some.js swh/web/static/jssources/core-js/modules/es.array.sort.js swh/web/static/jssources/core-js/modules/es.array.species.js swh/web/static/jssources/core-js/modules/es.array.splice.js swh/web/static/jssources/core-js/modules/es.array.unscopables.flat-map.js swh/web/static/jssources/core-js/modules/es.array.unscopables.flat.js swh/web/static/jssources/core-js/modules/es.data-view.js swh/web/static/jssources/core-js/modules/es.date.now.js swh/web/static/jssources/core-js/modules/es.date.to-iso-string.js swh/web/static/jssources/core-js/modules/es.date.to-json.js swh/web/static/jssources/core-js/modules/es.date.to-primitive.js swh/web/static/jssources/core-js/modules/es.date.to-string.js swh/web/static/jssources/core-js/modules/es.function.bind.js swh/web/static/jssources/core-js/modules/es.function.has-instance.js swh/web/static/jssources/core-js/modules/es.function.name.js swh/web/static/jssources/core-js/modules/es.global-this.js swh/web/static/jssources/core-js/modules/es.json.to-string-tag.js swh/web/static/jssources/core-js/modules/es.map.js swh/web/static/jssources/core-js/modules/es.math.acosh.js swh/web/static/jssources/core-js/modules/es.math.asinh.js swh/web/static/jssources/core-js/modules/es.math.atanh.js swh/web/static/jssources/core-js/modules/es.math.cbrt.js swh/web/static/jssources/core-js/modules/es.math.clz32.js swh/web/static/jssources/core-js/modules/es.math.cosh.js swh/web/static/jssources/core-js/modules/es.math.expm1.js swh/web/static/jssources/core-js/modules/es.math.fround.js swh/web/static/jssources/core-js/modules/es.math.hypot.js swh/web/static/jssources/core-js/modules/es.math.imul.js swh/web/static/jssources/core-js/modules/es.math.log10.js swh/web/static/jssources/core-js/modules/es.math.log1p.js swh/web/static/jssources/core-js/modules/es.math.log2.js swh/web/static/jssources/core-js/modules/es.math.sign.js swh/web/static/jssources/core-js/modules/es.math.sinh.js swh/web/static/jssources/core-js/modules/es.math.tanh.js swh/web/static/jssources/core-js/modules/es.math.to-string-tag.js swh/web/static/jssources/core-js/modules/es.math.trunc.js swh/web/static/jssources/core-js/modules/es.number.constructor.js swh/web/static/jssources/core-js/modules/es.number.epsilon.js swh/web/static/jssources/core-js/modules/es.number.is-finite.js swh/web/static/jssources/core-js/modules/es.number.is-integer.js swh/web/static/jssources/core-js/modules/es.number.is-nan.js swh/web/static/jssources/core-js/modules/es.number.is-safe-integer.js swh/web/static/jssources/core-js/modules/es.number.max-safe-integer.js swh/web/static/jssources/core-js/modules/es.number.min-safe-integer.js swh/web/static/jssources/core-js/modules/es.number.parse-float.js swh/web/static/jssources/core-js/modules/es.number.parse-int.js swh/web/static/jssources/core-js/modules/es.number.to-fixed.js swh/web/static/jssources/core-js/modules/es.number.to-precision.js swh/web/static/jssources/core-js/modules/es.object.assign.js swh/web/static/jssources/core-js/modules/es.object.create.js swh/web/static/jssources/core-js/modules/es.object.define-getter.js swh/web/static/jssources/core-js/modules/es.object.define-properties.js swh/web/static/jssources/core-js/modules/es.object.define-property.js swh/web/static/jssources/core-js/modules/es.object.define-setter.js swh/web/static/jssources/core-js/modules/es.object.entries.js swh/web/static/jssources/core-js/modules/es.object.freeze.js swh/web/static/jssources/core-js/modules/es.object.from-entries.js swh/web/static/jssources/core-js/modules/es.object.get-own-property-descriptor.js swh/web/static/jssources/core-js/modules/es.object.get-own-property-descriptors.js swh/web/static/jssources/core-js/modules/es.object.get-own-property-names.js swh/web/static/jssources/core-js/modules/es.object.get-prototype-of.js swh/web/static/jssources/core-js/modules/es.object.is-extensible.js swh/web/static/jssources/core-js/modules/es.object.is-frozen.js swh/web/static/jssources/core-js/modules/es.object.is-sealed.js swh/web/static/jssources/core-js/modules/es.object.is.js swh/web/static/jssources/core-js/modules/es.object.keys.js swh/web/static/jssources/core-js/modules/es.object.lookup-getter.js swh/web/static/jssources/core-js/modules/es.object.lookup-setter.js swh/web/static/jssources/core-js/modules/es.object.prevent-extensions.js swh/web/static/jssources/core-js/modules/es.object.seal.js swh/web/static/jssources/core-js/modules/es.object.set-prototype-of.js swh/web/static/jssources/core-js/modules/es.object.to-string.js swh/web/static/jssources/core-js/modules/es.object.values.js swh/web/static/jssources/core-js/modules/es.parse-float.js swh/web/static/jssources/core-js/modules/es.parse-int.js swh/web/static/jssources/core-js/modules/es.promise.all-settled.js swh/web/static/jssources/core-js/modules/es.promise.finally.js swh/web/static/jssources/core-js/modules/es.promise.js swh/web/static/jssources/core-js/modules/es.reflect.apply.js swh/web/static/jssources/core-js/modules/es.reflect.construct.js swh/web/static/jssources/core-js/modules/es.reflect.define-property.js swh/web/static/jssources/core-js/modules/es.reflect.delete-property.js swh/web/static/jssources/core-js/modules/es.reflect.get-own-property-descriptor.js swh/web/static/jssources/core-js/modules/es.reflect.get-prototype-of.js swh/web/static/jssources/core-js/modules/es.reflect.get.js swh/web/static/jssources/core-js/modules/es.reflect.has.js swh/web/static/jssources/core-js/modules/es.reflect.is-extensible.js swh/web/static/jssources/core-js/modules/es.reflect.own-keys.js swh/web/static/jssources/core-js/modules/es.reflect.prevent-extensions.js swh/web/static/jssources/core-js/modules/es.reflect.set-prototype-of.js swh/web/static/jssources/core-js/modules/es.reflect.set.js swh/web/static/jssources/core-js/modules/es.regexp.constructor.js swh/web/static/jssources/core-js/modules/es.regexp.exec.js swh/web/static/jssources/core-js/modules/es.regexp.flags.js swh/web/static/jssources/core-js/modules/es.regexp.to-string.js swh/web/static/jssources/core-js/modules/es.set.js swh/web/static/jssources/core-js/modules/es.string.anchor.js swh/web/static/jssources/core-js/modules/es.string.big.js swh/web/static/jssources/core-js/modules/es.string.blink.js swh/web/static/jssources/core-js/modules/es.string.bold.js swh/web/static/jssources/core-js/modules/es.string.code-point-at.js swh/web/static/jssources/core-js/modules/es.string.ends-with.js swh/web/static/jssources/core-js/modules/es.string.fixed.js swh/web/static/jssources/core-js/modules/es.string.fontcolor.js swh/web/static/jssources/core-js/modules/es.string.fontsize.js swh/web/static/jssources/core-js/modules/es.string.from-code-point.js swh/web/static/jssources/core-js/modules/es.string.includes.js swh/web/static/jssources/core-js/modules/es.string.italics.js swh/web/static/jssources/core-js/modules/es.string.iterator.js swh/web/static/jssources/core-js/modules/es.string.link.js swh/web/static/jssources/core-js/modules/es.string.match-all.js swh/web/static/jssources/core-js/modules/es.string.match.js swh/web/static/jssources/core-js/modules/es.string.pad-end.js swh/web/static/jssources/core-js/modules/es.string.pad-start.js swh/web/static/jssources/core-js/modules/es.string.raw.js swh/web/static/jssources/core-js/modules/es.string.repeat.js swh/web/static/jssources/core-js/modules/es.string.replace.js swh/web/static/jssources/core-js/modules/es.string.search.js swh/web/static/jssources/core-js/modules/es.string.small.js swh/web/static/jssources/core-js/modules/es.string.split.js swh/web/static/jssources/core-js/modules/es.string.starts-with.js swh/web/static/jssources/core-js/modules/es.string.strike.js swh/web/static/jssources/core-js/modules/es.string.sub.js swh/web/static/jssources/core-js/modules/es.string.sup.js swh/web/static/jssources/core-js/modules/es.string.trim-end.js swh/web/static/jssources/core-js/modules/es.string.trim-start.js swh/web/static/jssources/core-js/modules/es.string.trim.js swh/web/static/jssources/core-js/modules/es.symbol.async-iterator.js swh/web/static/jssources/core-js/modules/es.symbol.description.js swh/web/static/jssources/core-js/modules/es.symbol.has-instance.js swh/web/static/jssources/core-js/modules/es.symbol.is-concat-spreadable.js swh/web/static/jssources/core-js/modules/es.symbol.iterator.js swh/web/static/jssources/core-js/modules/es.symbol.js swh/web/static/jssources/core-js/modules/es.symbol.match-all.js swh/web/static/jssources/core-js/modules/es.symbol.match.js swh/web/static/jssources/core-js/modules/es.symbol.replace.js swh/web/static/jssources/core-js/modules/es.symbol.search.js swh/web/static/jssources/core-js/modules/es.symbol.species.js swh/web/static/jssources/core-js/modules/es.symbol.split.js swh/web/static/jssources/core-js/modules/es.symbol.to-primitive.js swh/web/static/jssources/core-js/modules/es.symbol.to-string-tag.js swh/web/static/jssources/core-js/modules/es.symbol.unscopables.js swh/web/static/jssources/core-js/modules/es.typed-array.copy-within.js swh/web/static/jssources/core-js/modules/es.typed-array.every.js swh/web/static/jssources/core-js/modules/es.typed-array.fill.js swh/web/static/jssources/core-js/modules/es.typed-array.filter.js swh/web/static/jssources/core-js/modules/es.typed-array.find-index.js swh/web/static/jssources/core-js/modules/es.typed-array.find.js swh/web/static/jssources/core-js/modules/es.typed-array.float32-array.js swh/web/static/jssources/core-js/modules/es.typed-array.float64-array.js swh/web/static/jssources/core-js/modules/es.typed-array.for-each.js swh/web/static/jssources/core-js/modules/es.typed-array.from.js swh/web/static/jssources/core-js/modules/es.typed-array.includes.js swh/web/static/jssources/core-js/modules/es.typed-array.index-of.js swh/web/static/jssources/core-js/modules/es.typed-array.int16-array.js swh/web/static/jssources/core-js/modules/es.typed-array.int32-array.js swh/web/static/jssources/core-js/modules/es.typed-array.int8-array.js swh/web/static/jssources/core-js/modules/es.typed-array.iterator.js swh/web/static/jssources/core-js/modules/es.typed-array.join.js swh/web/static/jssources/core-js/modules/es.typed-array.last-index-of.js swh/web/static/jssources/core-js/modules/es.typed-array.map.js swh/web/static/jssources/core-js/modules/es.typed-array.of.js swh/web/static/jssources/core-js/modules/es.typed-array.reduce-right.js swh/web/static/jssources/core-js/modules/es.typed-array.reduce.js swh/web/static/jssources/core-js/modules/es.typed-array.reverse.js swh/web/static/jssources/core-js/modules/es.typed-array.set.js swh/web/static/jssources/core-js/modules/es.typed-array.slice.js swh/web/static/jssources/core-js/modules/es.typed-array.some.js swh/web/static/jssources/core-js/modules/es.typed-array.sort.js swh/web/static/jssources/core-js/modules/es.typed-array.subarray.js swh/web/static/jssources/core-js/modules/es.typed-array.to-locale-string.js swh/web/static/jssources/core-js/modules/es.typed-array.to-string.js swh/web/static/jssources/core-js/modules/es.typed-array.uint16-array.js swh/web/static/jssources/core-js/modules/es.typed-array.uint32-array.js swh/web/static/jssources/core-js/modules/es.typed-array.uint8-array.js swh/web/static/jssources/core-js/modules/es.typed-array.uint8-clamped-array.js swh/web/static/jssources/core-js/modules/es.weak-map.js swh/web/static/jssources/core-js/modules/es.weak-set.js swh/web/static/jssources/core-js/modules/web.dom-collections.for-each.js swh/web/static/jssources/core-js/modules/web.dom-collections.iterator.js swh/web/static/jssources/core-js/modules/web.immediate.js swh/web/static/jssources/core-js/modules/web.queue-microtask.js swh/web/static/jssources/core-js/modules/web.timers.js swh/web/static/jssources/core-js/modules/web.url-search-params.js swh/web/static/jssources/core-js/modules/web.url.js swh/web/static/jssources/core-js/modules/web.url.to-json.js swh/web/static/jssources/core-js/stable/index.js swh/web/static/jssources/core-js/web/index.js swh/web/static/jssources/d3-array/LICENSE.txt swh/web/static/jssources/d3-array/src/array.js swh/web/static/jssources/d3-array/src/ascending.js swh/web/static/jssources/d3-array/src/bisect.js swh/web/static/jssources/d3-array/src/bisector.js swh/web/static/jssources/d3-array/src/constant.js swh/web/static/jssources/d3-array/src/cross.js swh/web/static/jssources/d3-array/src/descending.js swh/web/static/jssources/d3-array/src/deviation.js swh/web/static/jssources/d3-array/src/extent.js swh/web/static/jssources/d3-array/src/histogram.js swh/web/static/jssources/d3-array/src/identity.js swh/web/static/jssources/d3-array/src/index.js swh/web/static/jssources/d3-array/src/max.js swh/web/static/jssources/d3-array/src/mean.js swh/web/static/jssources/d3-array/src/median.js swh/web/static/jssources/d3-array/src/merge.js swh/web/static/jssources/d3-array/src/min.js swh/web/static/jssources/d3-array/src/number.js swh/web/static/jssources/d3-array/src/pairs.js swh/web/static/jssources/d3-array/src/permute.js swh/web/static/jssources/d3-array/src/quantile.js swh/web/static/jssources/d3-array/src/range.js swh/web/static/jssources/d3-array/src/scan.js swh/web/static/jssources/d3-array/src/shuffle.js swh/web/static/jssources/d3-array/src/sum.js swh/web/static/jssources/d3-array/src/ticks.js swh/web/static/jssources/d3-array/src/transpose.js swh/web/static/jssources/d3-array/src/variance.js swh/web/static/jssources/d3-array/src/zip.js swh/web/static/jssources/d3-array/src/threshold/freedmanDiaconis.js swh/web/static/jssources/d3-array/src/threshold/scott.js swh/web/static/jssources/d3-array/src/threshold/sturges.js swh/web/static/jssources/d3-axis/LICENSE.txt swh/web/static/jssources/d3-axis/src/array.js swh/web/static/jssources/d3-axis/src/axis.js swh/web/static/jssources/d3-axis/src/identity.js swh/web/static/jssources/d3-axis/src/index.js swh/web/static/jssources/d3-collection/LICENSE.txt swh/web/static/jssources/d3-collection/src/entries.js swh/web/static/jssources/d3-collection/src/index.js swh/web/static/jssources/d3-collection/src/keys.js swh/web/static/jssources/d3-collection/src/map.js swh/web/static/jssources/d3-collection/src/nest.js swh/web/static/jssources/d3-collection/src/set.js swh/web/static/jssources/d3-collection/src/values.js swh/web/static/jssources/d3-color/LICENSE.txt swh/web/static/jssources/d3-color/src/color.js swh/web/static/jssources/d3-color/src/cubehelix.js swh/web/static/jssources/d3-color/src/define.js swh/web/static/jssources/d3-color/src/index.js swh/web/static/jssources/d3-color/src/lab.js swh/web/static/jssources/d3-color/src/math.js swh/web/static/jssources/d3-dispatch/LICENSE.txt swh/web/static/jssources/d3-dispatch/src/dispatch.js swh/web/static/jssources/d3-dispatch/src/index.js swh/web/static/jssources/d3-ease/LICENSE.txt swh/web/static/jssources/d3-ease/src/back.js swh/web/static/jssources/d3-ease/src/bounce.js swh/web/static/jssources/d3-ease/src/circle.js swh/web/static/jssources/d3-ease/src/cubic.js swh/web/static/jssources/d3-ease/src/elastic.js swh/web/static/jssources/d3-ease/src/exp.js swh/web/static/jssources/d3-ease/src/index.js swh/web/static/jssources/d3-ease/src/linear.js swh/web/static/jssources/d3-ease/src/poly.js swh/web/static/jssources/d3-ease/src/quad.js swh/web/static/jssources/d3-ease/src/sin.js swh/web/static/jssources/d3-format/LICENSE.txt swh/web/static/jssources/d3-format/src/defaultLocale.js swh/web/static/jssources/d3-format/src/exponent.js swh/web/static/jssources/d3-format/src/formatDecimal.js swh/web/static/jssources/d3-format/src/formatGroup.js swh/web/static/jssources/d3-format/src/formatNumerals.js swh/web/static/jssources/d3-format/src/formatPrefixAuto.js swh/web/static/jssources/d3-format/src/formatRounded.js swh/web/static/jssources/d3-format/src/formatSpecifier.js swh/web/static/jssources/d3-format/src/formatTrim.js swh/web/static/jssources/d3-format/src/formatTypes.js swh/web/static/jssources/d3-format/src/identity.js swh/web/static/jssources/d3-format/src/index.js swh/web/static/jssources/d3-format/src/locale.js swh/web/static/jssources/d3-format/src/precisionFixed.js swh/web/static/jssources/d3-format/src/precisionPrefix.js swh/web/static/jssources/d3-format/src/precisionRound.js swh/web/static/jssources/d3-interpolate/LICENSE.txt swh/web/static/jssources/d3-interpolate/src/array.js swh/web/static/jssources/d3-interpolate/src/basis.js swh/web/static/jssources/d3-interpolate/src/basisClosed.js swh/web/static/jssources/d3-interpolate/src/color.js swh/web/static/jssources/d3-interpolate/src/constant.js swh/web/static/jssources/d3-interpolate/src/cubehelix.js swh/web/static/jssources/d3-interpolate/src/date.js swh/web/static/jssources/d3-interpolate/src/discrete.js swh/web/static/jssources/d3-interpolate/src/hcl.js swh/web/static/jssources/d3-interpolate/src/hsl.js swh/web/static/jssources/d3-interpolate/src/hue.js swh/web/static/jssources/d3-interpolate/src/index.js swh/web/static/jssources/d3-interpolate/src/lab.js swh/web/static/jssources/d3-interpolate/src/number.js swh/web/static/jssources/d3-interpolate/src/object.js swh/web/static/jssources/d3-interpolate/src/piecewise.js swh/web/static/jssources/d3-interpolate/src/quantize.js swh/web/static/jssources/d3-interpolate/src/rgb.js swh/web/static/jssources/d3-interpolate/src/round.js swh/web/static/jssources/d3-interpolate/src/string.js swh/web/static/jssources/d3-interpolate/src/value.js swh/web/static/jssources/d3-interpolate/src/zoom.js swh/web/static/jssources/d3-interpolate/src/transform/decompose.js swh/web/static/jssources/d3-interpolate/src/transform/index.js swh/web/static/jssources/d3-interpolate/src/transform/parse.js swh/web/static/jssources/d3-path/LICENSE.txt swh/web/static/jssources/d3-path/src/index.js swh/web/static/jssources/d3-path/src/path.js swh/web/static/jssources/d3-scale/LICENSE.txt swh/web/static/jssources/d3-scale/src/array.js swh/web/static/jssources/d3-scale/src/band.js swh/web/static/jssources/d3-scale/src/constant.js swh/web/static/jssources/d3-scale/src/continuous.js swh/web/static/jssources/d3-scale/src/diverging.js swh/web/static/jssources/d3-scale/src/identity.js swh/web/static/jssources/d3-scale/src/index.js swh/web/static/jssources/d3-scale/src/init.js swh/web/static/jssources/d3-scale/src/linear.js swh/web/static/jssources/d3-scale/src/log.js swh/web/static/jssources/d3-scale/src/nice.js swh/web/static/jssources/d3-scale/src/number.js swh/web/static/jssources/d3-scale/src/ordinal.js swh/web/static/jssources/d3-scale/src/pow.js swh/web/static/jssources/d3-scale/src/quantile.js swh/web/static/jssources/d3-scale/src/quantize.js swh/web/static/jssources/d3-scale/src/sequential.js swh/web/static/jssources/d3-scale/src/sequentialQuantile.js swh/web/static/jssources/d3-scale/src/symlog.js swh/web/static/jssources/d3-scale/src/threshold.js swh/web/static/jssources/d3-scale/src/tickFormat.js swh/web/static/jssources/d3-scale/src/time.js swh/web/static/jssources/d3-scale/src/utcTime.js swh/web/static/jssources/d3-selection/LICENSE.txt swh/web/static/jssources/d3-selection/src/constant.js swh/web/static/jssources/d3-selection/src/create.js swh/web/static/jssources/d3-selection/src/creator.js swh/web/static/jssources/d3-selection/src/index.js swh/web/static/jssources/d3-selection/src/local.js swh/web/static/jssources/d3-selection/src/matcher.js swh/web/static/jssources/d3-selection/src/mouse.js swh/web/static/jssources/d3-selection/src/namespace.js swh/web/static/jssources/d3-selection/src/namespaces.js swh/web/static/jssources/d3-selection/src/point.js swh/web/static/jssources/d3-selection/src/select.js swh/web/static/jssources/d3-selection/src/selectAll.js swh/web/static/jssources/d3-selection/src/selector.js swh/web/static/jssources/d3-selection/src/selectorAll.js swh/web/static/jssources/d3-selection/src/sourceEvent.js swh/web/static/jssources/d3-selection/src/touch.js swh/web/static/jssources/d3-selection/src/touches.js swh/web/static/jssources/d3-selection/src/window.js swh/web/static/jssources/d3-selection/src/selection/append.js swh/web/static/jssources/d3-selection/src/selection/attr.js swh/web/static/jssources/d3-selection/src/selection/call.js swh/web/static/jssources/d3-selection/src/selection/classed.js swh/web/static/jssources/d3-selection/src/selection/clone.js swh/web/static/jssources/d3-selection/src/selection/data.js swh/web/static/jssources/d3-selection/src/selection/datum.js swh/web/static/jssources/d3-selection/src/selection/dispatch.js swh/web/static/jssources/d3-selection/src/selection/each.js swh/web/static/jssources/d3-selection/src/selection/empty.js swh/web/static/jssources/d3-selection/src/selection/enter.js swh/web/static/jssources/d3-selection/src/selection/exit.js swh/web/static/jssources/d3-selection/src/selection/filter.js swh/web/static/jssources/d3-selection/src/selection/html.js swh/web/static/jssources/d3-selection/src/selection/index.js swh/web/static/jssources/d3-selection/src/selection/insert.js swh/web/static/jssources/d3-selection/src/selection/join.js swh/web/static/jssources/d3-selection/src/selection/lower.js swh/web/static/jssources/d3-selection/src/selection/merge.js swh/web/static/jssources/d3-selection/src/selection/node.js swh/web/static/jssources/d3-selection/src/selection/nodes.js swh/web/static/jssources/d3-selection/src/selection/on.js swh/web/static/jssources/d3-selection/src/selection/order.js swh/web/static/jssources/d3-selection/src/selection/property.js swh/web/static/jssources/d3-selection/src/selection/raise.js swh/web/static/jssources/d3-selection/src/selection/remove.js swh/web/static/jssources/d3-selection/src/selection/select.js swh/web/static/jssources/d3-selection/src/selection/selectAll.js swh/web/static/jssources/d3-selection/src/selection/size.js swh/web/static/jssources/d3-selection/src/selection/sort.js swh/web/static/jssources/d3-selection/src/selection/sparse.js swh/web/static/jssources/d3-selection/src/selection/style.js swh/web/static/jssources/d3-selection/src/selection/text.js swh/web/static/jssources/d3-shape/LICENSE.txt swh/web/static/jssources/d3-shape/src/arc.js swh/web/static/jssources/d3-shape/src/area.js swh/web/static/jssources/d3-shape/src/areaRadial.js swh/web/static/jssources/d3-shape/src/array.js swh/web/static/jssources/d3-shape/src/constant.js swh/web/static/jssources/d3-shape/src/descending.js swh/web/static/jssources/d3-shape/src/identity.js swh/web/static/jssources/d3-shape/src/index.js swh/web/static/jssources/d3-shape/src/line.js swh/web/static/jssources/d3-shape/src/lineRadial.js swh/web/static/jssources/d3-shape/src/math.js swh/web/static/jssources/d3-shape/src/noop.js swh/web/static/jssources/d3-shape/src/pie.js swh/web/static/jssources/d3-shape/src/point.js swh/web/static/jssources/d3-shape/src/pointRadial.js swh/web/static/jssources/d3-shape/src/stack.js swh/web/static/jssources/d3-shape/src/symbol.js swh/web/static/jssources/d3-shape/src/curve/basis.js swh/web/static/jssources/d3-shape/src/curve/basisClosed.js swh/web/static/jssources/d3-shape/src/curve/basisOpen.js swh/web/static/jssources/d3-shape/src/curve/bundle.js swh/web/static/jssources/d3-shape/src/curve/cardinal.js swh/web/static/jssources/d3-shape/src/curve/cardinalClosed.js swh/web/static/jssources/d3-shape/src/curve/cardinalOpen.js swh/web/static/jssources/d3-shape/src/curve/catmullRom.js swh/web/static/jssources/d3-shape/src/curve/catmullRomClosed.js swh/web/static/jssources/d3-shape/src/curve/catmullRomOpen.js swh/web/static/jssources/d3-shape/src/curve/linear.js swh/web/static/jssources/d3-shape/src/curve/linearClosed.js swh/web/static/jssources/d3-shape/src/curve/monotone.js swh/web/static/jssources/d3-shape/src/curve/natural.js swh/web/static/jssources/d3-shape/src/curve/radial.js swh/web/static/jssources/d3-shape/src/curve/step.js swh/web/static/jssources/d3-shape/src/link/index.js swh/web/static/jssources/d3-shape/src/offset/diverging.js swh/web/static/jssources/d3-shape/src/offset/expand.js swh/web/static/jssources/d3-shape/src/offset/none.js swh/web/static/jssources/d3-shape/src/offset/silhouette.js swh/web/static/jssources/d3-shape/src/offset/wiggle.js swh/web/static/jssources/d3-shape/src/order/appearance.js swh/web/static/jssources/d3-shape/src/order/ascending.js swh/web/static/jssources/d3-shape/src/order/descending.js swh/web/static/jssources/d3-shape/src/order/insideOut.js swh/web/static/jssources/d3-shape/src/order/none.js swh/web/static/jssources/d3-shape/src/order/reverse.js swh/web/static/jssources/d3-shape/src/symbol/circle.js swh/web/static/jssources/d3-shape/src/symbol/cross.js swh/web/static/jssources/d3-shape/src/symbol/diamond.js swh/web/static/jssources/d3-shape/src/symbol/square.js swh/web/static/jssources/d3-shape/src/symbol/star.js swh/web/static/jssources/d3-shape/src/symbol/triangle.js swh/web/static/jssources/d3-shape/src/symbol/wye.js swh/web/static/jssources/d3-time/LICENSE.txt swh/web/static/jssources/d3-time-format/LICENSE.txt swh/web/static/jssources/d3-time-format/src/defaultLocale.js swh/web/static/jssources/d3-time-format/src/index.js swh/web/static/jssources/d3-time-format/src/isoFormat.js swh/web/static/jssources/d3-time-format/src/isoParse.js swh/web/static/jssources/d3-time-format/src/locale.js swh/web/static/jssources/d3-time/src/day.js swh/web/static/jssources/d3-time/src/duration.js swh/web/static/jssources/d3-time/src/hour.js swh/web/static/jssources/d3-time/src/index.js swh/web/static/jssources/d3-time/src/interval.js swh/web/static/jssources/d3-time/src/millisecond.js swh/web/static/jssources/d3-time/src/minute.js swh/web/static/jssources/d3-time/src/month.js swh/web/static/jssources/d3-time/src/second.js swh/web/static/jssources/d3-time/src/utcDay.js swh/web/static/jssources/d3-time/src/utcHour.js swh/web/static/jssources/d3-time/src/utcMinute.js swh/web/static/jssources/d3-time/src/utcMonth.js swh/web/static/jssources/d3-time/src/utcWeek.js swh/web/static/jssources/d3-time/src/utcYear.js swh/web/static/jssources/d3-time/src/week.js swh/web/static/jssources/d3-time/src/year.js swh/web/static/jssources/d3-timer/LICENSE.txt swh/web/static/jssources/d3-timer/src/index.js swh/web/static/jssources/d3-timer/src/interval.js swh/web/static/jssources/d3-timer/src/timeout.js swh/web/static/jssources/d3-timer/src/timer.js swh/web/static/jssources/d3-transition/LICENSE.txt swh/web/static/jssources/d3-transition/src/active.js swh/web/static/jssources/d3-transition/src/index.js swh/web/static/jssources/d3-transition/src/interrupt.js swh/web/static/jssources/d3-transition/src/selection/index.js swh/web/static/jssources/d3-transition/src/selection/interrupt.js swh/web/static/jssources/d3-transition/src/selection/transition.js swh/web/static/jssources/d3-transition/src/transition/attr.js swh/web/static/jssources/d3-transition/src/transition/attrTween.js swh/web/static/jssources/d3-transition/src/transition/delay.js swh/web/static/jssources/d3-transition/src/transition/duration.js swh/web/static/jssources/d3-transition/src/transition/ease.js swh/web/static/jssources/d3-transition/src/transition/end.js swh/web/static/jssources/d3-transition/src/transition/filter.js swh/web/static/jssources/d3-transition/src/transition/index.js swh/web/static/jssources/d3-transition/src/transition/interpolate.js swh/web/static/jssources/d3-transition/src/transition/merge.js swh/web/static/jssources/d3-transition/src/transition/on.js swh/web/static/jssources/d3-transition/src/transition/remove.js swh/web/static/jssources/d3-transition/src/transition/schedule.js swh/web/static/jssources/d3-transition/src/transition/select.js swh/web/static/jssources/d3-transition/src/transition/selectAll.js swh/web/static/jssources/d3-transition/src/transition/selection.js swh/web/static/jssources/d3-transition/src/transition/style.js swh/web/static/jssources/d3-transition/src/transition/styleTween.js swh/web/static/jssources/d3-transition/src/transition/text.js swh/web/static/jssources/d3-transition/src/transition/transition.js swh/web/static/jssources/d3-transition/src/transition/tween.js swh/web/static/jssources/datatables.net/License.txt swh/web/static/jssources/datatables.net-bs4/js/dataTables.bootstrap4.js swh/web/static/jssources/datatables.net-responsive/License.txt swh/web/static/jssources/datatables.net-responsive-bs4/js/responsive.bootstrap4.js swh/web/static/jssources/datatables.net-responsive/js/dataTables.responsive.js swh/web/static/jssources/datatables.net/js/jquery.dataTables.js swh/web/static/jssources/dompurify/LICENSE.txt swh/web/static/jssources/dompurify/dist/purify.js swh/web/static/jssources/elementsfrompoint-polyfill/LICENSE.txt swh/web/static/jssources/elementsfrompoint-polyfill/index.js swh/web/static/jssources/he/LICENSE-MIT.txt.txt swh/web/static/jssources/he/he.js swh/web/static/jssources/highlight.js/LICENSE.txt swh/web/static/jssources/highlight.js/lib/highlight.js swh/web/static/jssources/highlight.js/lib/index.js swh/web/static/jssources/highlight.js/lib/languages/1c.js swh/web/static/jssources/highlight.js/lib/languages/abnf.js swh/web/static/jssources/highlight.js/lib/languages/accesslog.js swh/web/static/jssources/highlight.js/lib/languages/actionscript.js swh/web/static/jssources/highlight.js/lib/languages/ada.js swh/web/static/jssources/highlight.js/lib/languages/angelscript.js swh/web/static/jssources/highlight.js/lib/languages/apache.js swh/web/static/jssources/highlight.js/lib/languages/applescript.js swh/web/static/jssources/highlight.js/lib/languages/arcade.js swh/web/static/jssources/highlight.js/lib/languages/arduino.js swh/web/static/jssources/highlight.js/lib/languages/armasm.js swh/web/static/jssources/highlight.js/lib/languages/asciidoc.js swh/web/static/jssources/highlight.js/lib/languages/aspectj.js swh/web/static/jssources/highlight.js/lib/languages/autohotkey.js swh/web/static/jssources/highlight.js/lib/languages/autoit.js swh/web/static/jssources/highlight.js/lib/languages/avrasm.js swh/web/static/jssources/highlight.js/lib/languages/awk.js swh/web/static/jssources/highlight.js/lib/languages/axapta.js swh/web/static/jssources/highlight.js/lib/languages/bash.js swh/web/static/jssources/highlight.js/lib/languages/basic.js swh/web/static/jssources/highlight.js/lib/languages/bnf.js swh/web/static/jssources/highlight.js/lib/languages/brainfuck.js swh/web/static/jssources/highlight.js/lib/languages/cal.js swh/web/static/jssources/highlight.js/lib/languages/capnproto.js swh/web/static/jssources/highlight.js/lib/languages/ceylon.js swh/web/static/jssources/highlight.js/lib/languages/clean.js swh/web/static/jssources/highlight.js/lib/languages/clojure-repl.js swh/web/static/jssources/highlight.js/lib/languages/clojure.js swh/web/static/jssources/highlight.js/lib/languages/cmake.js swh/web/static/jssources/highlight.js/lib/languages/coffeescript.js swh/web/static/jssources/highlight.js/lib/languages/coq.js swh/web/static/jssources/highlight.js/lib/languages/cos.js swh/web/static/jssources/highlight.js/lib/languages/cpp.js swh/web/static/jssources/highlight.js/lib/languages/crmsh.js swh/web/static/jssources/highlight.js/lib/languages/crystal.js swh/web/static/jssources/highlight.js/lib/languages/cs.js swh/web/static/jssources/highlight.js/lib/languages/csp.js swh/web/static/jssources/highlight.js/lib/languages/css.js swh/web/static/jssources/highlight.js/lib/languages/d.js swh/web/static/jssources/highlight.js/lib/languages/dart.js swh/web/static/jssources/highlight.js/lib/languages/delphi.js swh/web/static/jssources/highlight.js/lib/languages/diff.js swh/web/static/jssources/highlight.js/lib/languages/django.js swh/web/static/jssources/highlight.js/lib/languages/dns.js swh/web/static/jssources/highlight.js/lib/languages/dockerfile.js swh/web/static/jssources/highlight.js/lib/languages/dos.js swh/web/static/jssources/highlight.js/lib/languages/dsconfig.js swh/web/static/jssources/highlight.js/lib/languages/dts.js swh/web/static/jssources/highlight.js/lib/languages/dust.js swh/web/static/jssources/highlight.js/lib/languages/ebnf.js swh/web/static/jssources/highlight.js/lib/languages/elixir.js swh/web/static/jssources/highlight.js/lib/languages/elm.js swh/web/static/jssources/highlight.js/lib/languages/erb.js swh/web/static/jssources/highlight.js/lib/languages/erlang-repl.js swh/web/static/jssources/highlight.js/lib/languages/erlang.js swh/web/static/jssources/highlight.js/lib/languages/excel.js swh/web/static/jssources/highlight.js/lib/languages/fix.js swh/web/static/jssources/highlight.js/lib/languages/flix.js swh/web/static/jssources/highlight.js/lib/languages/fortran.js swh/web/static/jssources/highlight.js/lib/languages/fsharp.js swh/web/static/jssources/highlight.js/lib/languages/gams.js swh/web/static/jssources/highlight.js/lib/languages/gauss.js swh/web/static/jssources/highlight.js/lib/languages/gcode.js swh/web/static/jssources/highlight.js/lib/languages/gherkin.js swh/web/static/jssources/highlight.js/lib/languages/glsl.js swh/web/static/jssources/highlight.js/lib/languages/gml.js swh/web/static/jssources/highlight.js/lib/languages/go.js swh/web/static/jssources/highlight.js/lib/languages/golo.js swh/web/static/jssources/highlight.js/lib/languages/gradle.js swh/web/static/jssources/highlight.js/lib/languages/groovy.js swh/web/static/jssources/highlight.js/lib/languages/haml.js swh/web/static/jssources/highlight.js/lib/languages/handlebars.js swh/web/static/jssources/highlight.js/lib/languages/haskell.js swh/web/static/jssources/highlight.js/lib/languages/haxe.js swh/web/static/jssources/highlight.js/lib/languages/hsp.js swh/web/static/jssources/highlight.js/lib/languages/htmlbars.js swh/web/static/jssources/highlight.js/lib/languages/http.js swh/web/static/jssources/highlight.js/lib/languages/hy.js swh/web/static/jssources/highlight.js/lib/languages/inform7.js swh/web/static/jssources/highlight.js/lib/languages/ini.js swh/web/static/jssources/highlight.js/lib/languages/irpf90.js swh/web/static/jssources/highlight.js/lib/languages/isbl.js swh/web/static/jssources/highlight.js/lib/languages/java.js swh/web/static/jssources/highlight.js/lib/languages/javascript.js swh/web/static/jssources/highlight.js/lib/languages/jboss-cli.js swh/web/static/jssources/highlight.js/lib/languages/json.js swh/web/static/jssources/highlight.js/lib/languages/julia-repl.js swh/web/static/jssources/highlight.js/lib/languages/julia.js swh/web/static/jssources/highlight.js/lib/languages/kotlin.js swh/web/static/jssources/highlight.js/lib/languages/lasso.js swh/web/static/jssources/highlight.js/lib/languages/ldif.js swh/web/static/jssources/highlight.js/lib/languages/leaf.js swh/web/static/jssources/highlight.js/lib/languages/less.js swh/web/static/jssources/highlight.js/lib/languages/lisp.js swh/web/static/jssources/highlight.js/lib/languages/livecodeserver.js swh/web/static/jssources/highlight.js/lib/languages/livescript.js swh/web/static/jssources/highlight.js/lib/languages/llvm.js swh/web/static/jssources/highlight.js/lib/languages/lsl.js swh/web/static/jssources/highlight.js/lib/languages/lua.js swh/web/static/jssources/highlight.js/lib/languages/makefile.js swh/web/static/jssources/highlight.js/lib/languages/markdown.js swh/web/static/jssources/highlight.js/lib/languages/mathematica.js swh/web/static/jssources/highlight.js/lib/languages/matlab.js swh/web/static/jssources/highlight.js/lib/languages/maxima.js swh/web/static/jssources/highlight.js/lib/languages/mel.js swh/web/static/jssources/highlight.js/lib/languages/mercury.js swh/web/static/jssources/highlight.js/lib/languages/mipsasm.js swh/web/static/jssources/highlight.js/lib/languages/mizar.js swh/web/static/jssources/highlight.js/lib/languages/mojolicious.js swh/web/static/jssources/highlight.js/lib/languages/monkey.js swh/web/static/jssources/highlight.js/lib/languages/moonscript.js swh/web/static/jssources/highlight.js/lib/languages/n1ql.js swh/web/static/jssources/highlight.js/lib/languages/nginx.js swh/web/static/jssources/highlight.js/lib/languages/nimrod.js swh/web/static/jssources/highlight.js/lib/languages/nix.js swh/web/static/jssources/highlight.js/lib/languages/nsis.js swh/web/static/jssources/highlight.js/lib/languages/objectivec.js swh/web/static/jssources/highlight.js/lib/languages/ocaml.js swh/web/static/jssources/highlight.js/lib/languages/openscad.js swh/web/static/jssources/highlight.js/lib/languages/oxygene.js swh/web/static/jssources/highlight.js/lib/languages/parser3.js swh/web/static/jssources/highlight.js/lib/languages/perl.js swh/web/static/jssources/highlight.js/lib/languages/pf.js swh/web/static/jssources/highlight.js/lib/languages/pgsql.js swh/web/static/jssources/highlight.js/lib/languages/php.js swh/web/static/jssources/highlight.js/lib/languages/plaintext.js swh/web/static/jssources/highlight.js/lib/languages/pony.js swh/web/static/jssources/highlight.js/lib/languages/powershell.js swh/web/static/jssources/highlight.js/lib/languages/processing.js swh/web/static/jssources/highlight.js/lib/languages/profile.js swh/web/static/jssources/highlight.js/lib/languages/prolog.js swh/web/static/jssources/highlight.js/lib/languages/properties.js swh/web/static/jssources/highlight.js/lib/languages/protobuf.js swh/web/static/jssources/highlight.js/lib/languages/puppet.js swh/web/static/jssources/highlight.js/lib/languages/purebasic.js swh/web/static/jssources/highlight.js/lib/languages/python.js swh/web/static/jssources/highlight.js/lib/languages/q.js swh/web/static/jssources/highlight.js/lib/languages/qml.js swh/web/static/jssources/highlight.js/lib/languages/r.js swh/web/static/jssources/highlight.js/lib/languages/reasonml.js swh/web/static/jssources/highlight.js/lib/languages/rib.js swh/web/static/jssources/highlight.js/lib/languages/roboconf.js swh/web/static/jssources/highlight.js/lib/languages/routeros.js swh/web/static/jssources/highlight.js/lib/languages/rsl.js swh/web/static/jssources/highlight.js/lib/languages/ruby.js swh/web/static/jssources/highlight.js/lib/languages/ruleslanguage.js swh/web/static/jssources/highlight.js/lib/languages/rust.js swh/web/static/jssources/highlight.js/lib/languages/sas.js swh/web/static/jssources/highlight.js/lib/languages/scala.js swh/web/static/jssources/highlight.js/lib/languages/scheme.js swh/web/static/jssources/highlight.js/lib/languages/scilab.js swh/web/static/jssources/highlight.js/lib/languages/scss.js swh/web/static/jssources/highlight.js/lib/languages/shell.js swh/web/static/jssources/highlight.js/lib/languages/smali.js swh/web/static/jssources/highlight.js/lib/languages/smalltalk.js swh/web/static/jssources/highlight.js/lib/languages/sml.js swh/web/static/jssources/highlight.js/lib/languages/sqf.js swh/web/static/jssources/highlight.js/lib/languages/sql.js swh/web/static/jssources/highlight.js/lib/languages/stan.js swh/web/static/jssources/highlight.js/lib/languages/stata.js swh/web/static/jssources/highlight.js/lib/languages/step21.js swh/web/static/jssources/highlight.js/lib/languages/stylus.js swh/web/static/jssources/highlight.js/lib/languages/subunit.js swh/web/static/jssources/highlight.js/lib/languages/swift.js swh/web/static/jssources/highlight.js/lib/languages/taggerscript.js swh/web/static/jssources/highlight.js/lib/languages/tap.js swh/web/static/jssources/highlight.js/lib/languages/tcl.js swh/web/static/jssources/highlight.js/lib/languages/tex.js swh/web/static/jssources/highlight.js/lib/languages/thrift.js swh/web/static/jssources/highlight.js/lib/languages/tp.js swh/web/static/jssources/highlight.js/lib/languages/twig.js swh/web/static/jssources/highlight.js/lib/languages/typescript.js swh/web/static/jssources/highlight.js/lib/languages/vala.js swh/web/static/jssources/highlight.js/lib/languages/vbnet.js swh/web/static/jssources/highlight.js/lib/languages/vbscript-html.js swh/web/static/jssources/highlight.js/lib/languages/vbscript.js swh/web/static/jssources/highlight.js/lib/languages/verilog.js swh/web/static/jssources/highlight.js/lib/languages/vhdl.js swh/web/static/jssources/highlight.js/lib/languages/vim.js swh/web/static/jssources/highlight.js/lib/languages/x86asm.js swh/web/static/jssources/highlight.js/lib/languages/xl.js swh/web/static/jssources/highlight.js/lib/languages/xml.js swh/web/static/jssources/highlight.js/lib/languages/xquery.js swh/web/static/jssources/highlight.js/lib/languages/yaml.js swh/web/static/jssources/highlight.js/lib/languages/zephir.js swh/web/static/jssources/highlightjs-line-numbers.js/LICENSE.txt swh/web/static/jssources/highlightjs-line-numbers.js/src/highlightjs-line-numbers.js swh/web/static/jssources/html-encoder-decoder/LICENSE.txt swh/web/static/jssources/html-encoder-decoder/lib/index.js swh/web/static/jssources/iframe-resizer/LICENSE.txt swh/web/static/jssources/iframe-resizer/index.js swh/web/static/jssources/iframe-resizer/js/iframeResizer.contentWindow.js swh/web/static/jssources/iframe-resizer/js/iframeResizer.js swh/web/static/jssources/iframe-resizer/js/index.js swh/web/static/jssources/iterate-object/LICENSE.txt swh/web/static/jssources/iterate-object/lib/index.js swh/web/static/jssources/jquery/LICENSE.txt swh/web/static/jssources/jquery/dist/jquery.js swh/web/static/jssources/js-cookie/LICENSE.txt swh/web/static/jssources/js-cookie/src/js.cookie.js swh/web/static/jssources/js-year-calendar/LICENSE.txt swh/web/static/jssources/js-year-calendar/dist/js-year-calendar.js swh/web/static/jssources/notebookjs/LICENSE.txt swh/web/static/jssources/notebookjs/notebook.js swh/web/static/jssources/object-fit-images/license.txt swh/web/static/jssources/object-fit-images/dist/ofi.common-js.js swh/web/static/jssources/org/LICENSE.txt swh/web/static/jssources/org/lib/org.js swh/web/static/jssources/org/lib/org/lexer.js swh/web/static/jssources/org/lib/org/node.js swh/web/static/jssources/org/lib/org/parser.js swh/web/static/jssources/org/lib/org/stream.js swh/web/static/jssources/org/lib/org/converter/converter.js swh/web/static/jssources/org/lib/org/converter/html.js swh/web/static/jssources/pdfjs-dist/LICENSE.txt swh/web/static/jssources/pdfjs-dist/build/pdf.js swh/web/static/jssources/pdfjs-dist/build/pdf.worker.js swh/web/static/jssources/popper.js/dist/esm/popper.js swh/web/static/jssources/regenerator-runtime/LICENSE.txt swh/web/static/jssources/regenerator-runtime/runtime.js swh/web/static/jssources/regex-escape/LICENSE.txt swh/web/static/jssources/regex-escape/lib/index.js swh/web/static/jssources/script-loader/LICENSE.txt swh/web/static/jssources/script-loader/addScript.js swh/web/static/jssources/showdown/license.txt swh/web/static/jssources/showdown/dist/showdown.js swh/web/static/jssources/swh/web/assets/src/bundles/admin/deposit.js swh/web/static/jssources/swh/web/assets/src/bundles/admin/index.js swh/web/static/jssources/swh/web/assets/src/bundles/admin/origin-save.js swh/web/static/jssources/swh/web/assets/src/bundles/browse/browse-utils.js swh/web/static/jssources/swh/web/assets/src/bundles/browse/index.js swh/web/static/jssources/swh/web/assets/src/bundles/browse/origin-search.js swh/web/static/jssources/swh/web/assets/src/bundles/browse/snapshot-navigation.js swh/web/static/jssources/swh/web/assets/src/bundles/browse/swh-ids-utils.js swh/web/static/jssources/swh/web/assets/src/bundles/origin/index.js swh/web/static/jssources/swh/web/assets/src/bundles/origin/visits-calendar.js swh/web/static/jssources/swh/web/assets/src/bundles/origin/visits-histogram.js swh/web/static/jssources/swh/web/assets/src/bundles/origin/visits-reporting.js swh/web/static/jssources/swh/web/assets/src/bundles/revision/diff-utils.js swh/web/static/jssources/swh/web/assets/src/bundles/revision/index.js swh/web/static/jssources/swh/web/assets/src/bundles/revision/log-utils.js swh/web/static/jssources/swh/web/assets/src/bundles/save/index.js swh/web/static/jssources/swh/web/assets/src/bundles/vault/index.js swh/web/static/jssources/swh/web/assets/src/bundles/vault/vault-create-tasks.js swh/web/static/jssources/swh/web/assets/src/bundles/vault/vault-ui.js swh/web/static/jssources/swh/web/assets/src/bundles/vendors/index.js swh/web/static/jssources/swh/web/assets/src/bundles/webapp/code-highlighting.js swh/web/static/jssources/swh/web/assets/src/bundles/webapp/history-counters.js swh/web/static/jssources/swh/web/assets/src/bundles/webapp/index.js swh/web/static/jssources/swh/web/assets/src/bundles/webapp/notebook-rendering.js swh/web/static/jssources/swh/web/assets/src/bundles/webapp/pdf-rendering.js swh/web/static/jssources/swh/web/assets/src/bundles/webapp/readme-rendering.js swh/web/static/jssources/swh/web/assets/src/bundles/webapp/webapp-utils.js swh/web/static/jssources/swh/web/assets/src/bundles/webapp/xss-filtering.js swh/web/static/jssources/swh/web/assets/src/thirdparty/jquery.tabSlideOut/LICENSE.txt swh/web/static/jssources/swh/web/assets/src/thirdparty/jquery.tabSlideOut/jquery.tabSlideOut.js swh/web/static/jssources/swh/web/assets/src/utils/constants.js swh/web/static/jssources/swh/web/assets/src/utils/d3-custom.js swh/web/static/jssources/swh/web/assets/src/utils/d3.js swh/web/static/jssources/swh/web/assets/src/utils/functions.js swh/web/static/jssources/swh/web/assets/src/utils/heaps-permute.js swh/web/static/jssources/swh/web/assets/src/utils/highlightjs.js swh/web/static/jssources/swh/web/assets/src/utils/org.js swh/web/static/jssources/swh/web/assets/src/utils/showdown.js swh/web/static/jssources/validate.js/LICENSE.txt swh/web/static/jssources/validate.js/validate.js swh/web/static/jssources/waypoints/licenses.txt.txt swh/web/static/jssources/waypoints/lib/jquery.waypoints.js swh/web/static/jssources/whatwg-fetch/LICENSE.txt swh/web/static/jssources/whatwg-fetch/dist/fetch.umd.js swh/web/static/xml/swh-opensearch.xml swh/web/templates/error.html swh/web/templates/homepage.html swh/web/templates/layout.html swh/web/templates/login.html swh/web/templates/logout.html swh/web/templates/admin/deposit.html swh/web/templates/admin/origin-save.html swh/web/templates/api/api.html swh/web/templates/api/apidoc.html swh/web/templates/api/endpoints.html swh/web/templates/browse/branches.html swh/web/templates/browse/browse.html swh/web/templates/browse/content.html swh/web/templates/browse/directory.html swh/web/templates/browse/help.html swh/web/templates/browse/layout.html swh/web/templates/browse/origin-visits.html swh/web/templates/browse/person.html swh/web/templates/browse/release.html swh/web/templates/browse/releases.html swh/web/templates/browse/revision-log.html swh/web/templates/browse/revision.html swh/web/templates/browse/search.html swh/web/templates/browse/vault-ui.html swh/web/templates/includes/apidoc-header.html swh/web/templates/includes/apidoc-header.md swh/web/templates/includes/breadcrumbs.html swh/web/templates/includes/content-display.html swh/web/templates/includes/directory-display.html swh/web/templates/includes/empty-snapshot.html swh/web/templates/includes/global-modals.html swh/web/templates/includes/http-error.html swh/web/templates/includes/readme-display.html swh/web/templates/includes/show-metadata.html swh/web/templates/includes/show-swh-ids.html swh/web/templates/includes/snapshot-context.html swh/web/templates/includes/take-new-snapshot.html swh/web/templates/includes/top-navigation.html swh/web/templates/includes/vault-common.html swh/web/templates/includes/vault-create-tasks.html swh/web/templates/misc/coverage.html swh/web/templates/misc/jslicenses.html swh/web/templates/misc/origin-save.html swh/web/tests/__init__.py swh/web/tests/conftest.py swh/web/tests/create_test_admin.py swh/web/tests/data.py swh/web/tests/strategies.py swh/web/tests/testcase.py swh/web/tests/admin/__init__.py swh/web/tests/admin/test_origin_save.py swh/web/tests/api/__init__.py swh/web/tests/api/test_api_lookup.py swh/web/tests/api/test_apidoc.py swh/web/tests/api/test_apiresponse.py swh/web/tests/api/test_utils.py swh/web/tests/api/views/__init__.py swh/web/tests/api/views/test_content.py swh/web/tests/api/views/test_directory.py swh/web/tests/api/views/test_identifiers.py swh/web/tests/api/views/test_origin.py swh/web/tests/api/views/test_origin_save.py swh/web/tests/api/views/test_release.py swh/web/tests/api/views/test_revision.py swh/web/tests/api/views/test_snapshot.py swh/web/tests/api/views/test_stat.py swh/web/tests/api/views/test_vault.py swh/web/tests/browse/__init__.py swh/web/tests/browse/test_utils.py swh/web/tests/browse/views/__init__.py swh/web/tests/browse/views/test_content.py swh/web/tests/browse/views/test_directory.py swh/web/tests/browse/views/test_identifiers.py swh/web/tests/browse/views/test_origin.py swh/web/tests/browse/views/test_release.py swh/web/tests/browse/views/test_revision.py swh/web/tests/common/__init__.py swh/web/tests/common/test_converters.py swh/web/tests/common/test_highlightjs.py swh/web/tests/common/test_origin_save.py swh/web/tests/common/test_origin_visits.py swh/web/tests/common/test_query.py swh/web/tests/common/test_service.py swh/web/tests/common/test_templatetags.py swh/web/tests/common/test_throttling.py swh/web/tests/common/test_utils.py swh/web/tests/misc/__init__.py swh/web/tests/misc/test_origin_save.py swh/web/tests/resources/contents/code/LICENSE swh/web/tests/resources/contents/code/extensions/test.R swh/web/tests/resources/contents/code/extensions/test.abnf swh/web/tests/resources/contents/code/extensions/test.adb swh/web/tests/resources/contents/code/extensions/test.adoc swh/web/tests/resources/contents/code/extensions/test.ahk swh/web/tests/resources/contents/code/extensions/test.aj swh/web/tests/resources/contents/code/extensions/test.applescript swh/web/tests/resources/contents/code/extensions/test.as swh/web/tests/resources/contents/code/extensions/test.au3 swh/web/tests/resources/contents/code/extensions/test.awk swh/web/tests/resources/contents/code/extensions/test.bas swh/web/tests/resources/contents/code/extensions/test.bat swh/web/tests/resources/contents/code/extensions/test.bf swh/web/tests/resources/contents/code/extensions/test.bnf swh/web/tests/resources/contents/code/extensions/test.bsl swh/web/tests/resources/contents/code/extensions/test.cal swh/web/tests/resources/contents/code/extensions/test.capnp swh/web/tests/resources/contents/code/extensions/test.ceylon swh/web/tests/resources/contents/code/extensions/test.clj swh/web/tests/resources/contents/code/extensions/test.cls swh/web/tests/resources/contents/code/extensions/test.cmake swh/web/tests/resources/contents/code/extensions/test.coffee swh/web/tests/resources/contents/code/extensions/test.cpp swh/web/tests/resources/contents/code/extensions/test.cr swh/web/tests/resources/contents/code/extensions/test.cs swh/web/tests/resources/contents/code/extensions/test.css swh/web/tests/resources/contents/code/extensions/test.d swh/web/tests/resources/contents/code/extensions/test.dart swh/web/tests/resources/contents/code/extensions/test.dcl swh/web/tests/resources/contents/code/extensions/test.dfm swh/web/tests/resources/contents/code/extensions/test.diff swh/web/tests/resources/contents/code/extensions/test.do swh/web/tests/resources/contents/code/extensions/test.dts swh/web/tests/resources/contents/code/extensions/test.dust swh/web/tests/resources/contents/code/extensions/test.ebnf swh/web/tests/resources/contents/code/extensions/test.elm swh/web/tests/resources/contents/code/extensions/test.ep swh/web/tests/resources/contents/code/extensions/test.erb swh/web/tests/resources/contents/code/extensions/test.erl swh/web/tests/resources/contents/code/extensions/test.ex swh/web/tests/resources/contents/code/extensions/test.f90 swh/web/tests/resources/contents/code/extensions/test.feature swh/web/tests/resources/contents/code/extensions/test.flix swh/web/tests/resources/contents/code/extensions/test.fs swh/web/tests/resources/contents/code/extensions/test.gcode swh/web/tests/resources/contents/code/extensions/test.glsl swh/web/tests/resources/contents/code/extensions/test.gml swh/web/tests/resources/contents/code/extensions/test.gms swh/web/tests/resources/contents/code/extensions/test.go swh/web/tests/resources/contents/code/extensions/test.golo swh/web/tests/resources/contents/code/extensions/test.gradle swh/web/tests/resources/contents/code/extensions/test.groovy swh/web/tests/resources/contents/code/extensions/test.gss swh/web/tests/resources/contents/code/extensions/test.haml swh/web/tests/resources/contents/code/extensions/test.hbs swh/web/tests/resources/contents/code/extensions/test.hs swh/web/tests/resources/contents/code/extensions/test.hsp swh/web/tests/resources/contents/code/extensions/test.html swh/web/tests/resources/contents/code/extensions/test.hx swh/web/tests/resources/contents/code/extensions/test.hy swh/web/tests/resources/contents/code/extensions/test.ini swh/web/tests/resources/contents/code/extensions/test.ino swh/web/tests/resources/contents/code/extensions/test.java swh/web/tests/resources/contents/code/extensions/test.jl swh/web/tests/resources/contents/code/extensions/test.js swh/web/tests/resources/contents/code/extensions/test.json swh/web/tests/resources/contents/code/extensions/test.kt swh/web/tests/resources/contents/code/extensions/test.lasso swh/web/tests/resources/contents/code/extensions/test.lc swh/web/tests/resources/contents/code/extensions/test.ldif swh/web/tests/resources/contents/code/extensions/test.leaf swh/web/tests/resources/contents/code/extensions/test.less swh/web/tests/resources/contents/code/extensions/test.lisp swh/web/tests/resources/contents/code/extensions/test.ll swh/web/tests/resources/contents/code/extensions/test.ls swh/web/tests/resources/contents/code/extensions/test.lsl swh/web/tests/resources/contents/code/extensions/test.lua swh/web/tests/resources/contents/code/extensions/test.m swh/web/tests/resources/contents/code/extensions/test.md swh/web/tests/resources/contents/code/extensions/test.mel swh/web/tests/resources/contents/code/extensions/test.mk swh/web/tests/resources/contents/code/extensions/test.ml swh/web/tests/resources/contents/code/extensions/test.moon swh/web/tests/resources/contents/code/extensions/test.nim swh/web/tests/resources/contents/code/extensions/test.nix swh/web/tests/resources/contents/code/extensions/test.nsi swh/web/tests/resources/contents/code/extensions/test.p swh/web/tests/resources/contents/code/extensions/test.pbi swh/web/tests/resources/contents/code/extensions/test.pde swh/web/tests/resources/contents/code/extensions/test.php swh/web/tests/resources/contents/code/extensions/test.pl swh/web/tests/resources/contents/code/extensions/test.pony swh/web/tests/resources/contents/code/extensions/test.pp swh/web/tests/resources/contents/code/extensions/test.properties swh/web/tests/resources/contents/code/extensions/test.proto swh/web/tests/resources/contents/code/extensions/test.ps1 swh/web/tests/resources/contents/code/extensions/test.py swh/web/tests/resources/contents/code/extensions/test.q swh/web/tests/resources/contents/code/extensions/test.qml swh/web/tests/resources/contents/code/extensions/test.rb swh/web/tests/resources/contents/code/extensions/test.re swh/web/tests/resources/contents/code/extensions/test.rib swh/web/tests/resources/contents/code/extensions/test.rs swh/web/tests/resources/contents/code/extensions/test.rsc swh/web/tests/resources/contents/code/extensions/test.s swh/web/tests/resources/contents/code/extensions/test.sas swh/web/tests/resources/contents/code/extensions/test.scad swh/web/tests/resources/contents/code/extensions/test.scala swh/web/tests/resources/contents/code/extensions/test.sci swh/web/tests/resources/contents/code/extensions/test.scm swh/web/tests/resources/contents/code/extensions/test.scss swh/web/tests/resources/contents/code/extensions/test.sh swh/web/tests/resources/contents/code/extensions/test.sl swh/web/tests/resources/contents/code/extensions/test.smali swh/web/tests/resources/contents/code/extensions/test.sml swh/web/tests/resources/contents/code/extensions/test.sqf swh/web/tests/resources/contents/code/extensions/test.st swh/web/tests/resources/contents/code/extensions/test.stan swh/web/tests/resources/contents/code/extensions/test.styl swh/web/tests/resources/contents/code/extensions/test.subunit swh/web/tests/resources/contents/code/extensions/test.swift swh/web/tests/resources/contents/code/extensions/test.tap swh/web/tests/resources/contents/code/extensions/test.tcl swh/web/tests/resources/contents/code/extensions/test.tex swh/web/tests/resources/contents/code/extensions/test.thrift swh/web/tests/resources/contents/code/extensions/test.ts swh/web/tests/resources/contents/code/extensions/test.v swh/web/tests/resources/contents/code/extensions/test.vala swh/web/tests/resources/contents/code/extensions/test.vb swh/web/tests/resources/contents/code/extensions/test.vbs swh/web/tests/resources/contents/code/extensions/test.vhd swh/web/tests/resources/contents/code/extensions/test.vim swh/web/tests/resources/contents/code/extensions/test.wl swh/web/tests/resources/contents/code/extensions/test.xml swh/web/tests/resources/contents/code/extensions/test.xqy swh/web/tests/resources/contents/code/extensions/test.yml swh/web/tests/resources/contents/code/extensions/test.zep swh/web/tests/resources/contents/code/filenames/.htaccess swh/web/tests/resources/contents/code/filenames/CMakeLists.txt swh/web/tests/resources/contents/code/filenames/Dockerfile swh/web/tests/resources/contents/code/filenames/Makefile swh/web/tests/resources/contents/code/filenames/access.log swh/web/tests/resources/contents/code/filenames/httpd.conf swh/web/tests/resources/contents/code/filenames/nginx.conf swh/web/tests/resources/contents/code/filenames/nginx.log swh/web/tests/resources/contents/code/filenames/pf.conf swh/web/tests/resources/contents/code/filenames/resolv.conf swh/web/tests/resources/contents/other/extensions/bash-cheatsheet.pdf swh/web/tests/resources/contents/other/extensions/swh-logo.jpeg swh/web/tests/resources/contents/other/extensions/swh-logo.png swh/web/tests/resources/contents/other/extensions/swh-logo.webp swh/web/tests/resources/contents/other/extensions/swh-spinner.gif swh/web/tests/resources/contents/other/extensions/word2vec.ipynb swh/web/tests/resources/json/es_task_info_response.json swh/web/tests/resources/repos/highlightjs-line-numbers.js.zip swh/web/tests/resources/repos/highlightjs-line-numbers.js_visit2.zip swh/web/tests/resources/repos/libtess2.zip swh/web/tests/resources/repos/repo_with_submodules.tgz \ No newline at end of file diff --git a/swh.web.egg-info/requires.txt b/swh.web.egg-info/requires.txt index 21ab712e..f00fcca6 100644 --- a/swh.web.egg-info/requires.txt +++ b/swh.web.egg-info/requires.txt @@ -1,31 +1,32 @@ beautifulsoup4 Django<2.0,>=1.11.0 djangorestframework>=3.4.0 django_webpack_loader django_js_reverse docutils python-magic>=0.4.0 htmlmin lxml pygments pypandoc python-dateutil pyyaml requests python-memcached sphinx sphinxcontrib-httpdomain swh.model>=0.0.32 swh.storage>=0.0.145 swh.vault>=0.0.23 swh.indexer>=0.0.120 swh.scheduler>=0.0.31 [testing] hypothesis pytest pytest-django pytest-mock +django-stubs requests-mock swh.core[http]>=0.0.61 swh.loader.git>=0.0.47 diff --git a/swh/__init__.py b/swh/__init__.py index 69e3be50..f14e1965 100644 --- a/swh/__init__.py +++ b/swh/__init__.py @@ -1 +1,4 @@ -__path__ = __import__('pkgutil').extend_path(__path__, __name__) +from pkgutil import extend_path +from typing import Iterable + +__path__ = extend_path(__path__, __name__) # type: Iterable[str] diff --git a/swh/web/admin/deposit.py b/swh/web/admin/deposit.py index df66eda9..7cb84f98 100644 --- a/swh/web/admin/deposit.py +++ b/swh/web/admin/deposit.py @@ -1,93 +1,93 @@ # Copyright (C) 2018-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import json import requests from django.core.cache import cache from django.conf import settings from django.contrib.admin.views.decorators import staff_member_required from django.core.paginator import Paginator from django.http import HttpResponse from django.shortcuts import render from requests.auth import HTTPBasicAuth from swh.web.admin.adminurls import admin_route from swh.web.config import get_config config = get_config()['deposit'] @admin_route(r'deposit/', view_name='admin-deposit') -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _admin_origin_save(request): return render(request, 'admin/deposit.html') @admin_route(r'deposit/list/', view_name='admin-deposit-list') -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _admin_deposit_list(request): table_data = {} table_data['draw'] = int(request.GET['draw']) deposits_list_url = config['private_api_url'] + 'deposits' deposits_list_auth = HTTPBasicAuth(config['private_api_user'], config['private_api_password']) try: nb_deposits = requests.get('%s?page_size=1' % deposits_list_url, auth=deposits_list_auth, timeout=30).json()['count'] deposits_data = cache.get('swh-deposit-list') if not deposits_data or deposits_data['count'] != nb_deposits: deposits_data = requests.get('%s?page_size=%s' % (deposits_list_url, nb_deposits), auth=deposits_list_auth, timeout=30).json() cache.set('swh-deposit-list', deposits_data) deposits = deposits_data['results'] search_value = request.GET['search[value]'] if search_value: deposits = \ [d for d in deposits if any(search_value.lower() in val for val in [str(v).lower() for v in d.values()])] column_order = request.GET['order[0][column]'] field_order = request.GET['columns[%s][name]' % column_order] order_dir = request.GET['order[0][dir]'] deposits = sorted(deposits, key=lambda d: d[field_order] or '') if order_dir == 'desc': deposits = list(reversed(deposits)) length = int(request.GET['length']) page = int(request.GET['start']) / length + 1 paginator = Paginator(deposits, length) data = paginator.page(page).object_list table_data['recordsTotal'] = deposits_data['count'] table_data['recordsFiltered'] = len(deposits) table_data['data'] = [{ 'id': d['id'], 'external_id': d['external_id'], 'reception_date': d['reception_date'], 'status': d['status'], 'status_detail': d['status_detail'], 'swh_anchor_id': d['swh_anchor_id'], 'swh_anchor_id_context': d['swh_anchor_id_context'], 'swh_id': d['swh_id'], 'swh_id_context': d['swh_id_context'] } for d in data] except Exception: table_data['error'] = ('An error occurred while retrieving ' 'the list of deposits !') return HttpResponse(json.dumps(table_data), content_type='application/json') diff --git a/swh/web/admin/origin_save.py b/swh/web/admin/origin_save.py index 965a4600..7317e7c0 100644 --- a/swh/web/admin/origin_save.py +++ b/swh/web/admin/origin_save.py @@ -1,206 +1,206 @@ # Copyright (C) 2018-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import json from django.conf import settings from django.contrib.admin.views.decorators import staff_member_required from django.core.exceptions import ObjectDoesNotExist from django.core.paginator import Paginator from django.http import HttpResponse from django.shortcuts import render from django.views.decorators.http import require_POST from swh.web.admin.adminurls import admin_route from swh.web.common.models import ( SaveAuthorizedOrigin, SaveUnauthorizedOrigin, SaveOriginRequest ) from swh.web.common.origin_save import ( create_save_origin_request, get_save_origin_task_info, SAVE_REQUEST_PENDING, SAVE_REQUEST_REJECTED ) @admin_route(r'origin/save/', view_name='admin-origin-save') -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _admin_origin_save(request): return render(request, 'admin/origin-save.html') def _datatables_origin_urls_response(request, urls_query_set): search_value = request.GET['search[value]'] if search_value: urls_query_set = urls_query_set.filter(url__icontains=search_value) column_order = request.GET['order[0][column]'] field_order = request.GET['columns[%s][name]' % column_order] order_dir = request.GET['order[0][dir]'] if order_dir == 'desc': field_order = '-' + field_order urls_query_set = urls_query_set.order_by(field_order) table_data = {} table_data['draw'] = int(request.GET['draw']) table_data['recordsTotal'] = urls_query_set.count() table_data['recordsFiltered'] = urls_query_set.count() length = int(request.GET['length']) page = int(request.GET['start']) / length + 1 paginator = Paginator(urls_query_set, length) urls_query_set = paginator.page(page).object_list table_data['data'] = [{'url': u.url} for u in urls_query_set] table_data_json = json.dumps(table_data, separators=(',', ': ')) return HttpResponse(table_data_json, content_type='application/json') @admin_route(r'origin/save/authorized_urls/list/', view_name='admin-origin-save-authorized-urls-list') @staff_member_required def _admin_origin_save_authorized_urls_list(request): authorized_urls = SaveAuthorizedOrigin.objects.all() return _datatables_origin_urls_response(request, authorized_urls) @admin_route(r'origin/save/authorized_urls/add/(?P.+)/', view_name='admin-origin-save-add-authorized-url') @require_POST -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _admin_origin_save_add_authorized_url(request, origin_url): try: SaveAuthorizedOrigin.objects.get(url=origin_url) except ObjectDoesNotExist: # add the new authorized url SaveAuthorizedOrigin.objects.create(url=origin_url) # check if pending save requests with that url prefix exist pending_save_requests = \ SaveOriginRequest.objects.filter(origin_url__startswith=origin_url, status=SAVE_REQUEST_PENDING) # create origin save tasks for previously pending requests for psr in pending_save_requests: create_save_origin_request(psr.visit_type, psr.origin_url) status_code = 200 else: status_code = 400 return HttpResponse(status=status_code) @admin_route(r'origin/save/authorized_urls/remove/(?P.+)/', view_name='admin-origin-save-remove-authorized-url') @require_POST -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _admin_origin_save_remove_authorized_url(request, origin_url): try: entry = SaveAuthorizedOrigin.objects.get(url=origin_url) except ObjectDoesNotExist: status_code = 404 else: entry.delete() status_code = 200 return HttpResponse(status=status_code) @admin_route(r'origin/save/unauthorized_urls/list/', view_name='admin-origin-save-unauthorized-urls-list') -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _admin_origin_save_unauthorized_urls_list(request): unauthorized_urls = SaveUnauthorizedOrigin.objects.all() return _datatables_origin_urls_response(request, unauthorized_urls) @admin_route(r'origin/save/unauthorized_urls/add/(?P.+)/', view_name='admin-origin-save-add-unauthorized-url') @require_POST -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _admin_origin_save_add_unauthorized_url(request, origin_url): try: SaveUnauthorizedOrigin.objects.get(url=origin_url) except ObjectDoesNotExist: SaveUnauthorizedOrigin.objects.create(url=origin_url) # check if pending save requests with that url prefix exist pending_save_requests = \ SaveOriginRequest.objects.filter(origin_url__startswith=origin_url, status=SAVE_REQUEST_PENDING) # mark pending requests as rejected for psr in pending_save_requests: psr.status = SAVE_REQUEST_REJECTED psr.save() status_code = 200 else: status_code = 400 return HttpResponse(status=status_code) @admin_route(r'origin/save/unauthorized_urls/remove/(?P.+)/', view_name='admin-origin-save-remove-unauthorized-url') @require_POST -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _admin_origin_save_remove_unauthorized_url(request, origin_url): try: entry = SaveUnauthorizedOrigin.objects.get(url=origin_url) except ObjectDoesNotExist: status_code = 404 else: entry.delete() status_code = 200 return HttpResponse(status=status_code) @admin_route(r'origin/save/request/accept/(?P.+)/url/(?P.+)/', # noqa view_name='admin-origin-save-request-accept') @require_POST -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _admin_origin_save_request_accept(request, visit_type, origin_url): try: SaveAuthorizedOrigin.objects.get(url=origin_url) except ObjectDoesNotExist: SaveAuthorizedOrigin.objects.create(url=origin_url) create_save_origin_request(visit_type, origin_url) return HttpResponse(status=200) @admin_route(r'origin/save/request/reject/(?P.+)/url/(?P.+)/', # noqa view_name='admin-origin-save-request-reject') @require_POST -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _admin_origin_save_request_reject(request, visit_type, origin_url): try: SaveUnauthorizedOrigin.objects.get(url=origin_url) except ObjectDoesNotExist: SaveUnauthorizedOrigin.objects.create(url=origin_url) sor = SaveOriginRequest.objects.get(visit_type=visit_type, origin_url=origin_url, status=SAVE_REQUEST_PENDING) sor.status = SAVE_REQUEST_REJECTED sor.save() return HttpResponse(status=200) @admin_route(r'origin/save/request/remove/(?P.+)/', view_name='admin-origin-save-request-remove') @require_POST -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _admin_origin_save_request_remove(request, sor_id): try: entry = SaveOriginRequest.objects.get(id=sor_id) except ObjectDoesNotExist: status_code = 404 else: entry.delete() status_code = 200 return HttpResponse(status=status_code) @admin_route(r'origin/save/task/info/(?P.+)/', view_name='admin-origin-save-task-info') -@staff_member_required(login_url=settings.LOGIN_URL) +@staff_member_required(view_func=None, login_url=settings.LOGIN_URL) def _save_origin_task_info(request, save_request_id): request_info = get_save_origin_task_info(save_request_id) for date_field in ('scheduled', 'started', 'ended'): if date_field in request_info and request_info[date_field] is not None: request_info[date_field] = request_info[date_field].isoformat() return HttpResponse(json.dumps(request_info), content_type='application/json') diff --git a/swh/web/api/apiurls.py b/swh/web/api/apiurls.py index c356c9cd..8694115f 100644 --- a/swh/web/api/apiurls.py +++ b/swh/web/api/apiurls.py @@ -1,85 +1,86 @@ # Copyright (C) 2017-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import functools +from typing import Dict + from rest_framework.decorators import api_view from swh.web.common.urlsindex import UrlsIndex from swh.web.common import throttling class APIUrls(UrlsIndex): """ Class to manage API documentation URLs. - Indexes all routes documented using apidoc's decorators. - Tracks endpoint/request processing method relationships for use in generating related urls in API documentation """ - _apidoc_routes = {} - _method_endpoints = {} + _apidoc_routes = {} # type: Dict[str, Dict[str, str]] scope = 'api' @classmethod def get_app_endpoints(cls): return cls._apidoc_routes @classmethod def add_route(cls, route, docstring, **kwargs): """ Add a route to the self-documenting API reference """ route_view_name = 'api-1-%s' % route[1:-1].replace('/', '-') if route not in cls._apidoc_routes: d = {'docstring': docstring, 'route_view_name': route_view_name} for k, v in kwargs.items(): d[k] = v cls._apidoc_routes[route] = d def api_route(url_pattern=None, view_name=None, methods=['GET', 'HEAD', 'OPTIONS'], throttle_scope='swh_api', api_version='1', checksum_args=None): """ Decorator to ease the registration of an API endpoint using the Django REST Framework. Args: url_pattern: the url pattern used by DRF to identify the API route view_name: the name of the API view associated to the route used to reverse the url methods: array of HTTP methods supported by the API route """ url_pattern = '^' + api_version + url_pattern + '$' def decorator(f): # create a DRF view from the wrapped function @api_view(methods) @throttling.throttle_scope(throttle_scope) @functools.wraps(f) def api_view_f(*args, **kwargs): return f(*args, **kwargs) # small hacks for correctly generating API endpoints index doc api_view_f.__name__ = f.__name__ api_view_f.http_method_names = methods # register the route and its view in the endpoints index APIUrls.add_url_pattern(url_pattern, api_view_f, view_name) if checksum_args: APIUrls.add_redirect_for_checksum_args(view_name, [url_pattern], checksum_args) return f return decorator diff --git a/swh/web/browse/utils.py b/swh/web/browse/utils.py index 038cfed4..a57b9a1d 100644 --- a/swh/web/browse/utils.py +++ b/swh/web/browse/utils.py @@ -1,1098 +1,1100 @@ # Copyright (C) 2017-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import base64 import magic import pypandoc import stat import textwrap from collections import defaultdict from threading import Lock from django.core.cache import cache from django.utils.safestring import mark_safe from django.utils.html import escape from swh.model.identifiers import persistent_identifier from swh.web.common import highlightjs, service from swh.web.common.exc import NotFoundExc, http_status_code_message from swh.web.common.origin_visits import get_origin_visit from swh.web.common.utils import ( reverse, format_utc_iso_date, get_swh_persistent_id, swh_object_icons ) from swh.web.config import get_config def get_directory_entries(sha1_git): """Function that retrieves the content of a directory from the archive. The directories entries are first sorted in lexicographical order. Sub-directories and regular files are then extracted. Args: sha1_git: sha1_git identifier of the directory Returns: A tuple whose first member corresponds to the sub-directories list and second member the regular files list Raises: NotFoundExc if the directory is not found """ cache_entry_id = 'directory_entries_%s' % sha1_git cache_entry = cache.get(cache_entry_id) if cache_entry: return cache_entry entries = list(service.lookup_directory(sha1_git)) for e in entries: e['perms'] = stat.filemode(e['perms']) if e['type'] == 'rev': # modify dir entry name to explicitly show it points # to a revision e['name'] = '%s @ %s' % (e['name'], e['target'][:7]) dirs = [e for e in entries if e['type'] in ('dir', 'rev')] files = [e for e in entries if e['type'] == 'file'] dirs = sorted(dirs, key=lambda d: d['name']) files = sorted(files, key=lambda f: f['name']) cache.set(cache_entry_id, (dirs, files)) return dirs, files _lock = Lock() def get_mimetype_and_encoding_for_content(content): """Function that returns the mime type and the encoding associated to a content buffer using the magic module under the hood. Args: content (bytes): a content buffer Returns: A tuple (mimetype, encoding), for instance ('text/plain', 'us-ascii'), associated to the provided content. """ # https://pypi.org/project/python-magic/ # packaged as python3-magic in debian buster if hasattr(magic, 'from_buffer'): m = magic.Magic(mime=True, mime_encoding=True) mime_encoding = m.from_buffer(content) mime_type, encoding = mime_encoding.split(';') encoding = encoding.replace(' charset=', '') # https://pypi.org/project/file-magic/ # packaged as python3-magic in debian stretch else: # TODO: Remove that code when production environment is upgraded # to debian buster # calls to the file-magic API are not thread-safe so they must # be protected with a Lock to guarantee they will succeed _lock.acquire() magic_result = magic.detect_from_content(content) _lock.release() mime_type = magic_result.mime_type encoding = magic_result.encoding return mime_type, encoding # maximum authorized content size in bytes for HTML display # with code highlighting content_display_max_size = get_config()['content_display_max_size'] snapshot_content_max_size = get_config()['snapshot_content_max_size'] def _re_encode_content(mimetype, encoding, content_data): # encode textual content to utf-8 if needed if mimetype.startswith('text/'): # probably a malformed UTF-8 content, re-encode it # by replacing invalid chars with a substitution one if encoding == 'unknown-8bit': content_data = content_data.decode('utf-8', 'replace')\ .encode('utf-8') elif encoding not in ['utf-8', 'binary']: content_data = content_data.decode(encoding, 'replace')\ .encode('utf-8') elif mimetype.startswith('application/octet-stream'): # file may detect a text content as binary # so try to decode it for display encodings = ['us-ascii'] encodings += ['iso-8859-%s' % i for i in range(1, 17)] for encoding in encodings: try: content_data = content_data.decode(encoding)\ .encode('utf-8') except Exception: pass else: # ensure display in content view mimetype = 'text/plain' break return mimetype, content_data def request_content(query_string, max_size=content_display_max_size, raise_if_unavailable=True, re_encode=True): """Function that retrieves a content from the archive. Raw bytes content is first retrieved, then the content mime type. If the mime type is not stored in the archive, it will be computed using Python magic module. Args: query_string: a string of the form "[ALGO_HASH:]HASH" where optional ALGO_HASH can be either ``sha1``, ``sha1_git``, ``sha256``, or ``blake2s256`` (default to ``sha1``) and HASH the hexadecimal representation of the hash value max_size: the maximum size for a content to retrieve (default to 1MB, no size limit if None) Returns: A tuple whose first member corresponds to the content raw bytes and second member the content mime type Raises: NotFoundExc if the content is not found """ content_data = service.lookup_content(query_string) filetype = None language = None license = None # requests to the indexer db may fail so properly handle # those cases in order to avoid content display errors try: filetype = service.lookup_content_filetype(query_string) language = service.lookup_content_language(query_string) license = service.lookup_content_license(query_string) except Exception: pass mimetype = 'unknown' encoding = 'unknown' if filetype: mimetype = filetype['mimetype'] encoding = filetype['encoding'] # workaround when encountering corrupted data due to implicit # conversion from bytea to text in the indexer db (see T818) # TODO: Remove that code when all data have been correctly converted if mimetype.startswith('\\'): filetype = None content_data['error_code'] = 200 content_data['error_message'] = '' content_data['error_description'] = '' if not max_size or content_data['length'] < max_size: try: content_raw = service.lookup_content_raw(query_string) except Exception as e: if raise_if_unavailable: raise e else: content_data['raw_data'] = None content_data['error_code'] = 404 content_data['error_description'] = \ 'The bytes of the content are currently not available in the archive.' # noqa content_data['error_message'] = \ http_status_code_message[content_data['error_code']] else: content_data['raw_data'] = content_raw['data'] if not filetype: mimetype, encoding = \ get_mimetype_and_encoding_for_content(content_data['raw_data']) # noqa if re_encode: mimetype, raw_data = _re_encode_content( mimetype, encoding, content_data['raw_data']) content_data['raw_data'] = raw_data else: content_data['raw_data'] = None content_data['mimetype'] = mimetype content_data['encoding'] = encoding if language: content_data['language'] = language['lang'] else: content_data['language'] = 'not detected' if license: content_data['licenses'] = ', '.join(license['facts'][0]['licenses']) else: content_data['licenses'] = 'not detected' return content_data _browsers_supported_image_mimes = set(['image/gif', 'image/png', 'image/jpeg', 'image/bmp', 'image/webp', 'image/svg', 'image/svg+xml']) def prepare_content_for_display(content_data, mime_type, path): """Function that prepares a content for HTML display. The function tries to associate a programming language to a content in order to perform syntax highlighting client-side using highlightjs. The language is determined using either the content filename or its mime type. If the mime type corresponds to an image format supported by web browsers, the content will be encoded in base64 for displaying the image. Args: content_data (bytes): raw bytes of the content mime_type (string): mime type of the content path (string): path of the content including filename Returns: A dict containing the content bytes (possibly different from the one provided as parameter if it is an image) under the key 'content_data and the corresponding highlightjs language class under the key 'language'. """ language = highlightjs.get_hljs_language_from_filename(path) if not language: language = highlightjs.get_hljs_language_from_mime_type(mime_type) if not language: language = 'nohighlight' elif mime_type.startswith('application/'): mime_type = mime_type.replace('application/', 'text/') if mime_type.startswith('image/'): if mime_type in _browsers_supported_image_mimes: content_data = base64.b64encode(content_data) content_data = content_data.decode('utf-8') else: content_data = None if mime_type.startswith('image/svg'): mime_type = 'image/svg+xml' return {'content_data': content_data, 'language': language, 'mimetype': mime_type} def process_snapshot_branches(snapshot): """ Process a dictionary describing snapshot branches: extract those targeting revisions and releases, put them in two different lists, then sort those lists in lexicographical order of the branches' names. Args: snapshot_branches (dict): A dict describing the branches of a snapshot as returned for instance by :func:`swh.web.common.service.lookup_snapshot` Returns: tuple: A tuple whose first member is the sorted list of branches targeting revisions and second member the sorted list of branches targeting releases """ snapshot_branches = snapshot['branches'] branches = {} branch_aliases = {} releases = {} revision_to_branch = defaultdict(set) revision_to_release = defaultdict(set) release_to_branch = defaultdict(set) for branch_name, target in snapshot_branches.items(): if not target: # FIXME: display branches with an unknown target anyway continue target_id = target['target'] target_type = target['target_type'] if target_type == 'revision': branches[branch_name] = { 'name': branch_name, 'revision': target_id, } revision_to_branch[target_id].add(branch_name) elif target_type == 'release': release_to_branch[target_id].add(branch_name) elif target_type == 'alias': branch_aliases[branch_name] = target_id # FIXME: handle pointers to other object types def _enrich_release_branch(branch, release): releases[branch] = { 'name': release['name'], 'branch_name': branch, 'date': format_utc_iso_date(release['date']), 'id': release['id'], 'message': release['message'], 'target_type': release['target_type'], 'target': release['target'], } def _enrich_revision_branch(branch, revision): branches[branch].update({ 'revision': revision['id'], 'directory': revision['directory'], 'date': format_utc_iso_date(revision['date']), 'message': revision['message'] }) releases_info = service.lookup_release_multiple( release_to_branch.keys() ) for release in releases_info: branches_to_update = release_to_branch[release['id']] for branch in branches_to_update: _enrich_release_branch(branch, release) if release['target_type'] == 'revision': revision_to_release[release['target']].update( branches_to_update ) revisions = service.lookup_revision_multiple( set(revision_to_branch.keys()) | set(revision_to_release.keys()) ) for revision in revisions: if not revision: continue for branch in revision_to_branch[revision['id']]: _enrich_revision_branch(branch, revision) for release in revision_to_release[revision['id']]: releases[release]['directory'] = revision['directory'] for branch_alias, branch_target in branch_aliases.items(): if branch_target in branches: branches[branch_alias] = dict(branches[branch_target]) else: snp = service.lookup_snapshot(snapshot['id'], branches_from=branch_target, branches_count=1) if snp and branch_target in snp['branches']: if snp['branches'][branch_target] is None: continue target_type = snp['branches'][branch_target]['target_type'] target = snp['branches'][branch_target]['target'] if target_type == 'revision': branches[branch_alias] = snp['branches'][branch_target] revision = service.lookup_revision(target) _enrich_revision_branch(branch_alias, revision) elif target_type == 'release': release = service.lookup_release(target) _enrich_release_branch(branch_alias, release) if branch_alias in branches: branches[branch_alias]['name'] = branch_alias ret_branches = list(sorted(branches.values(), key=lambda b: b['name'])) ret_releases = list(sorted(releases.values(), key=lambda b: b['name'])) return ret_branches, ret_releases def get_snapshot_content(snapshot_id): """Returns the lists of branches and releases associated to a swh snapshot. That list is put in cache in order to speedup the navigation in the swh-web/browse ui. .. warning:: At most 1000 branches contained in the snapshot will be returned for performance reasons. Args: snapshot_id (str): hexadecimal representation of the snapshot identifier Returns: A tuple with two members. The first one is a list of dict describing the snapshot branches. The second one is a list of dict describing the snapshot releases. Raises: NotFoundExc if the snapshot does not exist """ cache_entry_id = 'swh_snapshot_%s' % snapshot_id cache_entry = cache.get(cache_entry_id) if cache_entry: return cache_entry['branches'], cache_entry['releases'] branches = [] releases = [] if snapshot_id: snapshot = service.lookup_snapshot( snapshot_id, branches_count=snapshot_content_max_size) branches, releases = process_snapshot_branches(snapshot) cache.set(cache_entry_id, { 'branches': branches, 'releases': releases, }) return branches, releases def get_origin_visit_snapshot(origin_info, visit_ts=None, visit_id=None, snapshot_id=None): """Returns the lists of branches and releases associated to a swh origin for a given visit. The visit is expressed by a timestamp. In the latter case, the closest visit from the provided timestamp will be used. If no visit parameter is provided, it returns the list of branches found for the latest visit. That list is put in cache in order to speedup the navigation in the swh-web/browse ui. .. warning:: At most 1000 branches contained in the snapshot will be returned for performance reasons. Args: origin_info (dict): a dict filled with origin information (id, url, type) visit_ts (int or str): an ISO date string or Unix timestamp to parse visit_id (int): optional visit id for disambiguation in case several visits have the same timestamp Returns: A tuple with two members. The first one is a list of dict describing the origin branches for the given visit. The second one is a list of dict describing the origin releases for the given visit. Raises: NotFoundExc if the origin or its visit are not found """ visit_info = get_origin_visit(origin_info, visit_ts, visit_id, snapshot_id) return get_snapshot_content(visit_info['snapshot']) def gen_link(url, link_text=None, link_attrs=None): """ Utility function for generating an HTML link to insert in Django templates. Args: url (str): an url link_text (str): optional text for the produced link, if not provided the url will be used link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ attrs = ' ' if link_attrs: for k, v in link_attrs.items(): attrs += '%s="%s" ' % (k, v) if not link_text: link_text = url link = '%s' \ % (attrs, escape(url), escape(link_text)) return mark_safe(link) def _snapshot_context_query_params(snapshot_context): query_params = None if snapshot_context and snapshot_context['origin_info']: origin_info = snapshot_context['origin_info'] query_params = {'origin': origin_info['url']} if 'timestamp' in snapshot_context['url_args']: query_params['timestamp'] = \ snapshot_context['url_args']['timestamp'] if 'visit_id' in snapshot_context['query_params']: query_params['visit_id'] = \ snapshot_context['query_params']['visit_id'] elif snapshot_context: query_params = {'snapshot_id': snapshot_context['snapshot_id']} return query_params def gen_revision_url(revision_id, snapshot_context=None): """ Utility function for generating an url to a revision. Args: revision_id (str): a revision id snapshot_context (dict): if provided, generate snapshot-dependent browsing url Returns: str: The url to browse the revision """ query_params = _snapshot_context_query_params(snapshot_context) return reverse('browse-revision', url_args={'sha1_git': revision_id}, query_params=query_params) def gen_revision_link(revision_id, shorten_id=False, snapshot_context=None, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}): """ Utility function for generating a link to a revision HTML view to insert in Django templates. Args: revision_id (str): a revision id shorten_id (boolean): whether to shorten the revision id to 7 characters for the link text snapshot_context (dict): if provided, generate snapshot-dependent browsing link link_text (str): optional text for the generated link (the revision id will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: str: An HTML link in the form 'revision_id' """ if not revision_id: return None revision_url = gen_revision_url(revision_id, snapshot_context) if shorten_id: return gen_link(revision_url, revision_id[:7], link_attrs) else: if not link_text: link_text = revision_id return gen_link(revision_url, link_text, link_attrs) def gen_directory_link(sha1_git, snapshot_context=None, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}): """ Utility function for generating a link to a directory HTML view to insert in Django templates. Args: sha1_git (str): directory identifier link_text (str): optional text for the generated link (the directory id will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ if not sha1_git: return None query_params = _snapshot_context_query_params(snapshot_context) directory_url = reverse('browse-directory', url_args={'sha1_git': sha1_git}, query_params=query_params) if not link_text: link_text = sha1_git return gen_link(directory_url, link_text, link_attrs) def gen_snapshot_link(snapshot_id, snapshot_context=None, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}): """ Utility function for generating a link to a snapshot HTML view to insert in Django templates. Args: snapshot_id (str): snapshot identifier link_text (str): optional text for the generated link (the snapshot id will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ query_params = _snapshot_context_query_params(snapshot_context) snapshot_url = reverse('browse-snapshot', url_args={'snapshot_id': snapshot_id}, query_params=query_params) if not link_text: link_text = snapshot_id return gen_link(snapshot_url, link_text, link_attrs) def gen_content_link(sha1_git, snapshot_context=None, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}): """ Utility function for generating a link to a content HTML view to insert in Django templates. Args: sha1_git (str): content identifier link_text (str): optional text for the generated link (the content sha1_git will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ if not sha1_git: return None query_params = _snapshot_context_query_params(snapshot_context) content_url = reverse('browse-content', url_args={'query_string': 'sha1_git:' + sha1_git}, query_params=query_params) if not link_text: link_text = sha1_git return gen_link(content_url, link_text, link_attrs) def get_revision_log_url(revision_id, snapshot_context=None): """ Utility function for getting the URL for a revision log HTML view (possibly in the context of an origin). Args: revision_id (str): revision identifier the history heads to snapshot_context (dict): if provided, generate snapshot-dependent browsing link Returns: The revision log view URL """ query_params = {'revision': revision_id} if snapshot_context and snapshot_context['origin_info']: origin_info = snapshot_context['origin_info'] url_args = {'origin_url': origin_info['url']} if 'timestamp' in snapshot_context['url_args']: url_args['timestamp'] = \ snapshot_context['url_args']['timestamp'] if 'visit_id' in snapshot_context['query_params']: query_params['visit_id'] = \ snapshot_context['query_params']['visit_id'] revision_log_url = reverse('browse-origin-log', url_args=url_args, query_params=query_params) elif snapshot_context: url_args = {'snapshot_id': snapshot_context['snapshot_id']} revision_log_url = reverse('browse-snapshot-log', url_args=url_args, query_params=query_params) else: revision_log_url = reverse('browse-revision-log', url_args={'sha1_git': revision_id}) return revision_log_url def gen_revision_log_link(revision_id, snapshot_context=None, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}): """ Utility function for generating a link to a revision log HTML view (possibly in the context of an origin) to insert in Django templates. Args: revision_id (str): revision identifier the history heads to snapshot_context (dict): if provided, generate snapshot-dependent browsing link link_text (str): optional text to use for the generated link (the revision id will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ if not revision_id: return None revision_log_url = get_revision_log_url(revision_id, snapshot_context) if not link_text: link_text = revision_id return gen_link(revision_log_url, link_text, link_attrs) def gen_person_mail_link(person, link_text=None): """ Utility function for generating a mail link to a person to insert in Django templates. Args: person (dict): dictionary containing person data (*name*, *email*, *fullname*) link_text (str): optional text to use for the generated mail link (the person name will be used by default) Returns: str: A mail link to the person or the person name if no email is present in person data """ person_name = person['name'] or person['fullname'] or 'None' if link_text is None: link_text = person_name person_email = person['email'] if person['email'] else None if person_email is None and '@' in person_name and ' ' not in person_name: person_email = person_name if person_email: return gen_link(url='mailto:%s' % person_email, link_text=link_text) else: return person_name def gen_release_link(sha1_git, snapshot_context=None, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}): """ Utility function for generating a link to a release HTML view to insert in Django templates. Args: sha1_git (str): release identifier link_text (str): optional text for the generated link (the release id will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ query_params = _snapshot_context_query_params(snapshot_context) release_url = reverse('browse-release', url_args={'sha1_git': sha1_git}, query_params=query_params) if not link_text: link_text = sha1_git return gen_link(release_url, link_text, link_attrs) def format_log_entries(revision_log, per_page, snapshot_context=None): """ Utility functions that process raw revision log data for HTML display. Its purpose is to: * add links to relevant browse views * format date in human readable format * truncate the message log Args: revision_log (list): raw revision log as returned by the swh-web api per_page (int): number of log entries per page snapshot_context (dict): if provided, generate snapshot-dependent browsing link """ revision_log_data = [] for i, rev in enumerate(revision_log): if i == per_page: break author_name = 'None' author_fullname = 'None' committer_fullname = 'None' if rev['author']: author_name = gen_person_mail_link(rev['author']) author_fullname = rev['author']['fullname'] if rev['committer']: committer_fullname = rev['committer']['fullname'] author_date = format_utc_iso_date(rev['date']) committer_date = format_utc_iso_date(rev['committer_date']) tooltip = 'revision %s\n' % rev['id'] tooltip += 'author: %s\n' % author_fullname tooltip += 'author date: %s\n' % author_date tooltip += 'committer: %s\n' % committer_fullname tooltip += 'committer date: %s\n\n' % committer_date if rev['message']: tooltip += textwrap.indent(rev['message'], ' '*4) revision_log_data.append({ 'author': author_name, 'id': rev['id'][:7], 'message': rev['message'], 'date': author_date, 'commit_date': committer_date, 'url': gen_revision_url(rev['id'], snapshot_context), 'tooltip': tooltip }) return revision_log_data def get_snapshot_context(snapshot_id=None, origin_url=None, timestamp=None, visit_id=None): """ Utility function to compute relevant information when navigating the archive in a snapshot context. The snapshot is either referenced by its id or it will be retrieved from an origin visit. Args: snapshot_id (str): hexadecimal representation of a snapshot identifier, all other parameters will be ignored if it is provided origin_url (str): the origin_url (e.g. https://github.com/(user)/(repo)/) timestamp (str): a datetime string for retrieving the closest visit of the origin visit_id (int): optional visit id for disambiguation in case of several visits with the same timestamp Returns: A dict with the following entries: * origin_info: dict containing origin information * visit_info: dict containing visit information * branches: the list of branches for the origin found during the visit * releases: the list of releases for the origin found during the visit * origin_browse_url: the url to browse the origin * origin_branches_url: the url to browse the origin branches * origin_releases_url': the url to browse the origin releases * origin_visit_url: the url to browse the snapshot of the origin found during the visit * url_args: dict containing url arguments to use when browsing in the context of the origin and its visit Raises: NotFoundExc: if no snapshot is found for the visit of an origin. """ origin_info = None visit_info = None url_args = None query_params = {} branches = [] releases = [] browse_url = None visit_url = None branches_url = None releases_url = None swh_type = 'snapshot' if origin_url: swh_type = 'origin' origin_info = service.lookup_origin({'url': origin_url}) visit_info = get_origin_visit(origin_info, timestamp, visit_id, snapshot_id) fmt_date = format_utc_iso_date(visit_info['date']) visit_info['fmt_date'] = fmt_date snapshot_id = visit_info['snapshot'] if not snapshot_id: raise NotFoundExc('No snapshot associated to the visit of origin ' '%s on %s' % (escape(origin_url), fmt_date)) # provided timestamp is not necessarily equals to the one # of the retrieved visit, so get the exact one in order # use it in the urls generated below if timestamp: timestamp = visit_info['date'] branches, releases = \ get_origin_visit_snapshot(origin_info, timestamp, visit_id, snapshot_id) url_args = {'origin_url': origin_info['url']} query_params = {'visit_id': visit_id} browse_url = reverse('browse-origin-visits', url_args=url_args) if timestamp: url_args['timestamp'] = format_utc_iso_date(timestamp, '%Y-%m-%dT%H:%M:%S') visit_url = reverse('browse-origin-directory', url_args=url_args, query_params=query_params) visit_info['url'] = visit_url branches_url = reverse('browse-origin-branches', url_args=url_args, query_params=query_params) releases_url = reverse('browse-origin-releases', url_args=url_args, query_params=query_params) elif snapshot_id: branches, releases = get_snapshot_content(snapshot_id) url_args = {'snapshot_id': snapshot_id} browse_url = reverse('browse-snapshot', url_args=url_args) branches_url = reverse('browse-snapshot-branches', url_args=url_args) releases_url = reverse('browse-snapshot-releases', url_args=url_args) releases = list(reversed(releases)) snapshot_size = service.lookup_snapshot_size(snapshot_id) is_empty = sum(snapshot_size.values()) == 0 swh_snp_id = persistent_identifier('snapshot', snapshot_id) return { 'swh_type': swh_type, 'swh_object_id': swh_snp_id, 'snapshot_id': snapshot_id, 'snapshot_size': snapshot_size, 'is_empty': is_empty, 'origin_info': origin_info, 'visit_info': visit_info, 'branches': branches, 'releases': releases, 'branch': None, 'release': None, 'browse_url': browse_url, 'branches_url': branches_url, 'releases_url': releases_url, 'url_args': url_args, 'query_params': query_params } # list of common readme names ordered by preference # (lower indices have higher priority) _common_readme_names = [ "readme.markdown", "readme.md", "readme.rst", "readme.txt", "readme" ] def get_readme_to_display(readmes): """ Process a list of readme files found in a directory in order to find the adequate one to display. Args: readmes: a list of dict where keys are readme file names and values are readme sha1s Returns: A tuple (readme_name, readme_sha1) """ readme_name = None readme_url = None readme_sha1 = None readme_html = None lc_readmes = {k.lower(): {'orig_name': k, 'sha1': v} for k, v in readmes.items()} # look for readme names according to the preference order # defined by the _common_readme_names list for common_readme_name in _common_readme_names: if common_readme_name in lc_readmes: readme_name = lc_readmes[common_readme_name]['orig_name'] readme_sha1 = lc_readmes[common_readme_name]['sha1'] readme_url = reverse('browse-content-raw', url_args={'query_string': readme_sha1}, query_params={'re_encode': 'true'}) break # otherwise pick the first readme like file if any if not readme_name and len(readmes.items()) > 0: readme_name = next(iter(readmes)) readme_sha1 = readmes[readme_name] readme_url = reverse('browse-content-raw', url_args={'query_string': readme_sha1}, query_params={'re_encode': 'true'}) # convert rst README to html server side as there is # no viable solution to perform that task client side if readme_name and readme_name.endswith('.rst'): cache_entry_id = 'readme_%s' % readme_sha1 cache_entry = cache.get(cache_entry_id) if cache_entry: readme_html = cache_entry else: try: rst_doc = request_content(readme_sha1) readme_html = pypandoc.convert_text(rst_doc['raw_data'], 'html', format='rst') cache.set(cache_entry_id, readme_html) except Exception: readme_html = 'Readme bytes are not available' return readme_name, readme_url, readme_html def get_swh_persistent_ids(swh_objects, snapshot_context=None): """ Returns a list of dict containing info related to persistent identifiers of swh objects. Args: swh_objects (list): a list of dict with the following keys: + * type: swh object type (content/directory/release/revision/snapshot) * id: swh object id + snapshot_context (dict): optional parameter describing the snapshot in which the object has been found Returns: list: a list of dict with the following keys: * object_type: the swh object type (content/directory/release/revision/snapshot) * object_icon: the swh object icon to use in HTML views * swh_id: the computed swh object persistent identifier * swh_id_url: the url resolving the persistent identifier * show_options: boolean indicating if the persistent id options must be displayed in persistent ids HTML view """ swh_ids = [] for swh_object in swh_objects: if not swh_object['id']: continue swh_id = get_swh_persistent_id(swh_object['type'], swh_object['id']) show_options = swh_object['type'] == 'content' or \ (snapshot_context and snapshot_context['origin_info'] is not None) object_icon = swh_object_icons[swh_object['type']] swh_ids.append({ 'object_type': swh_object['type'], 'object_icon': object_icon, 'swh_id': swh_id, 'swh_id_url': reverse('browse-swh-id', url_args={'swh_id': swh_id}), 'show_options': show_options }) return swh_ids diff --git a/swh/web/common/highlightjs.py b/swh/web/common/highlightjs.py index 6c57a556..ad458149 100644 --- a/swh/web/common/highlightjs.py +++ b/swh/web/common/highlightjs.py @@ -1,359 +1,361 @@ -# Copyright (C) 2017-2018 The Software Heritage developers +# Copyright (C) 2017-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import functools +from typing import Dict + from pygments.lexers import ( get_all_lexers, get_lexer_for_filename ) # set of languages ids that can be highlighted # by highlight.js library _hljs_languages = set([ '1c', 'abnf', 'accesslog', 'actionscript', 'ada', 'angelscript', 'apache', 'applescript', 'arcade', 'arduino', 'armasm', 'asciidoc', 'aspectj', 'autohotkey', 'autoit', 'avrasm', 'awk', 'axapta', 'bash', 'basic', 'bnf', 'brainfuck', 'cal', 'capnproto', 'ceylon', 'clean', 'clojure', 'clojure-repl', 'cmake', 'coffeescript', 'coq', 'cos', 'cpp', 'crmsh', 'crystal', 'cs', 'csp', 'css', 'd', 'dart', 'delphi', 'diff', 'django', 'dns', 'dockerfile', 'dos', 'dsconfig', 'dts', 'dust', 'ebnf', 'elixir', 'elm', 'erb', 'erlang', 'erlang-repl', 'excel', 'fix', 'flix', 'fortran', 'fsharp', 'gams', 'gauss', 'gcode', 'gherkin', 'glsl', 'gml', 'go', 'golo', 'gradle', 'groovy', 'haml', 'handlebars', 'haskell', 'haxe', 'hsp', 'htmlbars', 'http', 'hy', 'inform7', 'ini', 'irpf90', 'isbl', 'java', 'javascript', 'jboss-cli', 'json', 'julia', 'julia-repl', 'kotlin', 'lasso', 'ldif', 'leaf', 'less', 'lisp', 'livecodeserver', 'livescript', 'llvm', 'lsl', 'lua', 'makefile', 'markdown', 'mathematica', 'matlab', 'maxima', 'mel', 'mercury', 'mipsasm', 'mizar', 'mojolicious', 'monkey', 'moonscript', 'n1ql', 'nginx', 'nimrod', 'nix', 'nsis', 'objectivec', 'ocaml', 'openscad', 'oxygene', 'parser3', 'perl', 'pf', 'pgsql', 'php', 'plaintext', 'pony', 'powershell', 'processing', 'profile', 'prolog', 'properties', 'protobuf', 'puppet', 'purebasic', 'python', 'q', 'qml', 'r', 'reasonml', 'rib', 'roboconf', 'routeros', 'rsl', 'ruby', 'ruleslanguage', 'rust', 'sas', 'scala', 'scheme', 'scilab', 'scss', 'shell', 'smali', 'smalltalk', 'sml', 'sqf', 'sql', 'stan', 'stata', 'step21', 'stylus', 'subunit', 'swift', 'taggerscript', 'tap', 'tcl', 'tex', 'thrift', 'tp', 'twig', 'typescript', 'vala', 'vbnet', 'vbscript', 'vbscript-html', 'verilog', 'vhdl', 'vim', 'x86asm', 'xl', 'xml', 'xquery', 'yaml', 'zephir' ]) # languages aliases defined in highlight.js _hljs_languages_aliases = { 'ado': 'stata', 'adoc': 'asciidoc', 'ahk': 'autohotkey', 'aj': 'aspectj', 'apacheconf': 'apache', 'arm': 'armasm', 'as': 'actionscript', 'asc': 'asciidoc', 'atom': 'xml', 'bas': 'basic', 'bat': 'dos', 'bf': 'brainfuck', 'bind': 'dns', 'bsl': '1c', 'c-al': 'cal', 'c': 'cpp', 'c++': 'cpp', 'capnp': 'capnproto', 'cc': 'cpp', 'clj': 'clojure', 'cls': 'cos', 'cmake.in': 'cmake', 'cmd': 'dos', 'coffee': 'coffeescript', 'console': 'shell', 'cr': 'crystal', 'craftcms': 'twig', 'crm': 'crmsh', 'csharp': 'cs', 'cson': 'coffeescript', 'dcl': 'clean', 'dfm': 'delphi', 'do': 'stata', 'docker': 'dockerfile', 'dpr': 'delphi', 'dst': 'dust', 'dtsi': 'dts', 'ep': 'mojolicious', 'erl': 'erlang', 'ex': 'elixir', 'exs': 'elixir', 'f90': 'fortran', 'f95': 'fortran', 'feature': 'gherkin', 'freepascal': 'delphi', 'fs': 'fsharp', 'fsx': 'fsharp', 'gemspec': 'ruby', 'GML': 'gml', 'gms': 'gams', 'golang': 'go', 'graph': 'roboconf', 'gss': 'gauss', 'gyp': 'python', 'h': 'cpp', 'h++': 'cpp', 'hbs': 'handlebars', 'hpp': 'cpp', 'hs': 'haskell', 'html': 'xml', 'html.handlebars': 'handlebars', 'html.hbs': 'handlebars', 'https': 'http', 'hx': 'haxe', 'hylang': 'hy', 'i7': 'inform7', 'i7x': 'inform7', 'iced': 'coffeescript', 'icl': 'clean', 'ino': 'arduino', 'instances': 'roboconf', 'ipynb': 'json', 'irb': 'ruby', 'jinja': 'django', 'js': 'javascript', 'jsp': 'java', 'jsx': 'javascript', 'k': 'q', 'kdb': 'q', 'kt': 'kotlin', 'lassoscript': 'lasso', 'lazarus': 'delphi', 'lc': 'livecode', 'lfm': 'delphi', 'll': 'llvm', 'lpr': 'delphi', 'ls': 'livescript', 'm': 'matlab', 'mak': 'makefile', 'md': 'markdown', 'mikrotik': 'routeros', 'mips': 'mipsasm', 'mk': 'monkey', 'mkd': 'markdown', 'mkdown': 'markdown', 'ml': 'ocaml', 'mli': 'ocaml', 'mm': 'objectivec', 'mma': 'mathematica', 'moo': 'mercury', 'moon': 'moonscript', 'nav': 'cal', 'nb': 'mathematica', 'nc': 'gcode', 'nginxconf': 'nginx', 'ni': 'inform7', 'nim': 'nimrod', 'nixos': 'nix', 'nsi': 'nsis', 'obj-c': 'objectivec', 'objc': 'objectivec', 'osascript': 'applescript', 'osl': 'rsl', 'p': 'parser3', 'p21': 'step21', 'pas': 'delphi', 'pascal': 'delphi', 'patch': 'diff', 'pb': 'purebasic', 'pbi': 'purebasic', 'pcmk': 'crmsh', 'pde': 'processing', 'pf.conf': 'pf', 'php3': 'php', 'php4': 'php', 'php5': 'php', 'php6': 'php', 'php7': 'php', 'pl': 'perl', 'plist': 'xml', 'pm': 'perl', 'podspec': 'ruby', 'postgres': 'pgsql', 'postgresql': 'pgsql', 'pp': 'puppet', 'proto': 'protobuf', 'ps': 'powershell', 'ps1': 'powershell', 'psd1': 'powershell', 'psm1': 'powershell', 'py': 'python', 'qt': 'qml', 'rb': 'ruby', 're': 'reasonml', 'rei': 'reasonml', 'rs': 'rust', 'rsc': 'routeros', 'rss': 'xml', 'rst': 'nohighlight', 's': 'armasm', 'SAS': 'sas', 'scad': 'openscad', 'sci': 'scilab', 'scm': 'scheme', 'sh': 'bash', 'sig': 'sml', 'sl': 'rsl', 'st': 'smalltalk', 'step': 'step21', 'stp': 'step21', 'styl': 'stylus', 'sv': 'verilog', 'svh': 'verilog', 'tao': 'xl', 'thor': 'ruby', 'tk': 'tcl', 'toml': 'ini', 'ts': 'typescript', 'txt': 'nohighlight', 'v': 'coq', 'vb': 'vbnet', 'vbs': 'vbscript', 'vhd': 'vhdl', 'wildfly-cli': 'jboss-cli', 'wl': 'mathematica', 'wls': 'mathematica', 'xhtml': 'xml', 'xjb': 'xml', 'xls': 'excel', 'xlsx': 'excel', 'xpath': 'xquery', 'xpo': 'axapta', 'xpp': 'axapta', 'xq': 'xquery', 'xqy': 'xquery', 'xsd': 'xml', 'xsl': 'xml', 'YAML': 'yaml', 'yml': 'yaml', 'zep': 'zephir', 'zone': 'dns', 'zsh': 'bash' } # dictionary mapping pygment lexers to hljs languages -_pygments_lexer_to_hljs_language = {} +_pygments_lexer_to_hljs_language = {} # type: Dict[str, str] # dictionary mapping mime types to hljs languages _mime_type_to_hljs_language = { 'text/x-c': 'cpp', 'text/x-c++': 'cpp', 'text/x-msdos-batch': 'dos', 'text/x-lisp': 'lisp', 'text/x-shellscript': 'bash', } # dictionary mapping filenames to hljs languages _filename_to_hljs_language = { 'cmakelists.txt': 'cmake', '.htaccess': 'apache', 'httpd.conf': 'apache', 'access.log': 'accesslog', 'nginx.log': 'accesslog', 'resolv.conf': 'dns', 'dockerfile': 'docker', 'nginx.conf': 'nginx', 'pf.conf': 'pf' } # function to fill the above dictionaries def _init_pygments_to_hljs_map(): if len(_pygments_lexer_to_hljs_language) == 0: for lexer in get_all_lexers(): lexer_name = lexer[0] lang_aliases = lexer[1] lang_mime_types = lexer[3] lang = None for lang_alias in lang_aliases: if lang_alias in _hljs_languages: lang = lang_alias _pygments_lexer_to_hljs_language[lexer_name] = lang_alias break if lang: for lang_mime_type in lang_mime_types: _mime_type_to_hljs_language[lang_mime_type] = lang def get_hljs_language_from_filename(filename): """Function that tries to associate a language supported by highlight.js from a filename. Args: filename: input filename Returns: highlight.js language id or None if no correspondence has been found """ _init_pygments_to_hljs_map() if filename: filename_lower = filename.lower() if filename_lower in _filename_to_hljs_language: return _filename_to_hljs_language[filename_lower] if filename_lower in _hljs_languages: return filename_lower exts = filename_lower.split('.') # check if file extension matches an hljs language # also handle .ext.in cases for ext in reversed(exts[-2:]): if ext in _hljs_languages: return ext if ext in _hljs_languages_aliases: return _hljs_languages_aliases[ext] # otherwise use Pygments language database lexer = None # try to find a Pygment lexer try: lexer = get_lexer_for_filename(filename) except Exception: pass # if there is a correspondence between the lexer and an hljs # language, return it if lexer and lexer.name in _pygments_lexer_to_hljs_language: return _pygments_lexer_to_hljs_language[lexer.name] # otherwise, try to find a match between the file extensions # associated to the lexer and the hljs language aliases if lexer: exts = [ext.replace('*.', '') for ext in lexer.filenames] for ext in exts: if ext in _hljs_languages_aliases: return _hljs_languages_aliases[ext] return None def get_hljs_language_from_mime_type(mime_type): """Function that tries to associate a language supported by highlight.js from a mime type. Args: mime_type: input mime type Returns: highlight.js language id or None if no correspondence has been found """ _init_pygments_to_hljs_map() if mime_type and mime_type in _mime_type_to_hljs_language: return _mime_type_to_hljs_language[mime_type] return None @functools.lru_cache() def get_supported_languages(): """ Return the list of programming languages that can be highlighted using the highlight.js library. Returns: List[str]: the list of supported languages """ return sorted(list(_hljs_languages)) diff --git a/swh/web/common/migrations/0001_initial.py b/swh/web/common/migrations/0001_initial.py index a112f750..b81f050f 100644 --- a/swh/web/common/migrations/0001_initial.py +++ b/swh/web/common/migrations/0001_initial.py @@ -1,75 +1,72 @@ # Copyright (C) 2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information # flake8: noqa from __future__ import unicode_literals from django.db import migrations, models _authorized_origins = [ 'https://github.com/', 'https://gitlab.com/', 'https://bitbucket.org/', 'https://git.code.sf.net/', 'http://git.code.sf.net/', 'https://hg.code.sf.net/', 'http://hg.code.sf.net/', 'https://svn.code.sf.net/', 'http://svn.code.sf.net/' ] def _populate_save_authorized_origins(apps, schema_editor): SaveAuthorizedOrigin = apps.get_model('swh.web.common', 'SaveAuthorizedOrigin') for origin_url in _authorized_origins: SaveAuthorizedOrigin.objects.create(url=origin_url) class Migration(migrations.Migration): initial = True - dependencies = [ - ] - operations = [ migrations.CreateModel( name='SaveAuthorizedOrigin', fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), ('url', models.CharField(max_length=200)), ], options={ 'db_table': 'save_authorized_origin', }, ), migrations.CreateModel( name='SaveOriginRequest', fields=[ ('id', models.BigAutoField(primary_key=True, serialize=False)), ('request_date', models.DateTimeField(auto_now_add=True)), ('origin_type', models.CharField(max_length=200)), ('origin_url', models.CharField(max_length=200)), ('status', models.TextField(choices=[('accepted', 'accepted'), ('rejected', 'rejected'), ('pending', 'pending')], default='pending')), ('loading_task_id', models.IntegerField(default=-1)), ], options={ 'db_table': 'save_origin_request', 'ordering': ['-id'], }, ), migrations.CreateModel( name='SaveUnauthorizedOrigin', fields=[ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), ('url', models.CharField(max_length=200)), ], options={ 'db_table': 'save_unauthorized_origin', }, ), migrations.RunPython(_populate_save_authorized_origins) ] diff --git a/swh/web/common/origin_save.py b/swh/web/common/origin_save.py index 28eaee57..e5231393 100644 --- a/swh/web/common/origin_save.py +++ b/swh/web/common/origin_save.py @@ -1,532 +1,535 @@ # Copyright (C) 2018-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import json import logging from bisect import bisect_right from datetime import datetime, timezone, timedelta import requests from django.core.exceptions import ObjectDoesNotExist from django.core.exceptions import ValidationError from django.core.validators import URLValidator from django.utils.html import escape from swh.web import config from swh.web.common import service from swh.web.common.exc import BadInputExc, ForbiddenExc, NotFoundExc from swh.web.common.models import ( SaveUnauthorizedOrigin, SaveAuthorizedOrigin, SaveOriginRequest, SAVE_REQUEST_ACCEPTED, SAVE_REQUEST_REJECTED, SAVE_REQUEST_PENDING, SAVE_TASK_NOT_YET_SCHEDULED, SAVE_TASK_SCHEDULED, SAVE_TASK_SUCCEED, SAVE_TASK_FAILED, SAVE_TASK_RUNNING ) from swh.web.common.origin_visits import get_origin_visits from swh.web.common.utils import parse_timestamp from swh.scheduler.utils import create_oneshot_task_dict scheduler = config.scheduler() logger = logging.getLogger(__name__) def get_origin_save_authorized_urls(): """ Get the list of origin url prefixes authorized to be immediately loaded into the archive (whitelist). Returns: list: The list of authorized origin url prefix """ return [origin.url for origin in SaveAuthorizedOrigin.objects.all()] def get_origin_save_unauthorized_urls(): """ Get the list of origin url prefixes forbidden to be loaded into the archive (blacklist). Returns: list: the list of unauthorized origin url prefix """ return [origin.url for origin in SaveUnauthorizedOrigin.objects.all()] def can_save_origin(origin_url): """ Check if a software origin can be saved into the archive. Based on the origin url, the save request will be either: * immediately accepted if the url is whitelisted * rejected if the url is blacklisted * put in pending state for manual review otherwise Args: origin_url (str): the software origin url to check Returns: str: the origin save request status, either **accepted**, **rejected** or **pending** """ # origin url may be blacklisted for url_prefix in get_origin_save_unauthorized_urls(): if origin_url.startswith(url_prefix): return SAVE_REQUEST_REJECTED # if the origin url is in the white list, it can be immediately saved for url_prefix in get_origin_save_authorized_urls(): if origin_url.startswith(url_prefix): return SAVE_REQUEST_ACCEPTED # otherwise, the origin url needs to be manually verified return SAVE_REQUEST_PENDING # map visit type to scheduler task # TODO: do not hardcode the task name here (T1157) _visit_type_task = { 'git': 'load-git', 'hg': 'load-hg', 'svn': 'load-svn' } # map scheduler task status to origin save status _save_task_status = { 'next_run_not_scheduled': SAVE_TASK_NOT_YET_SCHEDULED, 'next_run_scheduled': SAVE_TASK_SCHEDULED, 'completed': SAVE_TASK_SUCCEED, 'disabled': SAVE_TASK_FAILED } def get_savable_visit_types(): return sorted(list(_visit_type_task.keys())) def _check_visit_type_savable(visit_type): """ Get the list of visit types that can be performed through a save request. Returns: list: the list of saveable visit types """ allowed_visit_types = ', '.join(get_savable_visit_types()) if visit_type not in _visit_type_task: raise BadInputExc('Visit of type %s can not be saved! ' 'Allowed types are the following: %s' % (visit_type, allowed_visit_types)) _validate_url = URLValidator(schemes=['http', 'https', 'svn', 'git']) def _check_origin_url_valid(origin_url): try: _validate_url(origin_url) except ValidationError: raise BadInputExc('The provided origin url (%s) is not valid!' % escape(origin_url)) def _get_visit_info_for_save_request(save_request): visit_date = None visit_status = None try: origin = {'url': save_request.origin_url} origin_info = service.lookup_origin(origin) origin_visits = get_origin_visits(origin_info) visit_dates = [parse_timestamp(v['date']) for v in origin_visits] i = bisect_right(visit_dates, save_request.request_date) if i != len(visit_dates): visit_date = visit_dates[i] visit_status = origin_visits[i]['status'] if origin_visits[i]['status'] == 'ongoing': visit_date = None except Exception: pass return visit_date, visit_status def _check_visit_update_status(save_request, save_task_status): visit_date, visit_status = _get_visit_info_for_save_request(save_request) save_request.visit_date = visit_date # visit has been performed, mark the saving task as succeed if visit_date and visit_status is not None: save_task_status = SAVE_TASK_SUCCEED elif visit_status == 'ongoing': save_task_status = SAVE_TASK_RUNNING else: time_now = datetime.now(tz=timezone.utc) time_delta = time_now - save_request.request_date # consider the task as failed if it is still in scheduled state # 30 days after its submission if time_delta.days > 30: save_task_status = SAVE_TASK_FAILED return visit_date, save_task_status def _save_request_dict(save_request, task=None): must_save = False visit_date = save_request.visit_date # save task still in scheduler db if task: save_task_status = _save_task_status[task['status']] # Consider request from which a visit date has already been found # as succeeded to avoid retrieving it again if save_task_status == SAVE_TASK_SCHEDULED and visit_date: save_task_status = SAVE_TASK_SUCCEED if save_task_status in (SAVE_TASK_FAILED, SAVE_TASK_SUCCEED) \ and not visit_date: visit_date, _ = _get_visit_info_for_save_request(save_request) save_request.visit_date = visit_date must_save = True # Check tasks still marked as scheduled / not yet scheduled if save_task_status in (SAVE_TASK_SCHEDULED, SAVE_TASK_NOT_YET_SCHEDULED): visit_date, save_task_status = _check_visit_update_status( save_request, save_task_status) # save task may have been archived else: save_task_status = save_request.loading_task_status if save_task_status in (SAVE_TASK_SCHEDULED, SAVE_TASK_NOT_YET_SCHEDULED): visit_date, save_task_status = _check_visit_update_status( save_request, save_task_status) else: save_task_status = save_request.loading_task_status if save_request.loading_task_status != save_task_status: save_request.loading_task_status = save_task_status must_save = True if must_save: save_request.save() return {'id': save_request.id, 'visit_type': save_request.visit_type, 'origin_url': save_request.origin_url, 'save_request_date': save_request.request_date.isoformat(), 'save_request_status': save_request.status, 'save_task_status': save_task_status, 'visit_date': visit_date.isoformat() if visit_date else None} def create_save_origin_request(visit_type, origin_url): """ Create a loading task to save a software origin into the archive. This function aims to create a software origin loading task trough the use of the swh-scheduler component. First, some checks are performed to see if the visit type and origin url are valid but also if the the save request can be accepted. If those checks passed, the loading task is then created. Otherwise, the save request is put in pending or rejected state. All the submitted save requests are logged into the swh-web database to keep track of them. Args: visit_type (str): the type of visit to perform (currently only ``git`` but ``svn`` and ``hg`` will soon be available) origin_url (str): the url of the origin to save Raises: BadInputExc: the visit type or origin url is invalid ForbiddenExc: the provided origin url is blacklisted Returns: dict: A dict describing the save request with the following keys: * **visit_type**: the type of visit to perform * **origin_url**: the url of the origin * **save_request_date**: the date the request was submitted * **save_request_status**: the request status, either **accepted**, **rejected** or **pending** * **save_task_status**: the origin loading task status, either **not created**, **not yet scheduled**, **scheduled**, **succeed** or **failed** """ _check_visit_type_savable(visit_type) _check_origin_url_valid(origin_url) save_request_status = can_save_origin(origin_url) task = None # if the origin save request is accepted, create a scheduler # task to load it into the archive if save_request_status == SAVE_REQUEST_ACCEPTED: # create a task with high priority kwargs = {'priority': 'high'} # set task parameters according to the visit type if visit_type == 'git': kwargs['repo_url'] = origin_url elif visit_type == 'hg': kwargs['origin_url'] = origin_url elif visit_type == 'svn': kwargs['origin_url'] = origin_url kwargs['svn_url'] = origin_url sor = None # get list of previously sumitted save requests current_sors = \ list(SaveOriginRequest.objects.filter(visit_type=visit_type, origin_url=origin_url)) can_create_task = False # if no save requests previously submitted, create the scheduler task if not current_sors: can_create_task = True else: # get the latest submitted save request sor = current_sors[0] # if it was in pending state, we need to create the scheduler task # and update the save request info in the database if sor.status == SAVE_REQUEST_PENDING: can_create_task = True # a task has already been created to load the origin elif sor.loading_task_id != -1: # get the scheduler task and its status tasks = scheduler.get_tasks([sor.loading_task_id]) task = tasks[0] if tasks else None task_status = _save_request_dict(sor, task)['save_task_status'] # create a new scheduler task only if the previous one has been # already executed if task_status == SAVE_TASK_FAILED or \ task_status == SAVE_TASK_SUCCEED: can_create_task = True sor = None else: can_create_task = False if can_create_task: # effectively create the scheduler task task_dict = create_oneshot_task_dict( _visit_type_task[visit_type], **kwargs) task = scheduler.create_tasks([task_dict])[0] # pending save request has been accepted if sor: sor.status = SAVE_REQUEST_ACCEPTED sor.loading_task_id = task['id'] sor.save() else: sor = SaveOriginRequest.objects.create(visit_type=visit_type, origin_url=origin_url, status=save_request_status, # noqa loading_task_id=task['id']) # noqa # save request must be manually reviewed for acceptation elif save_request_status == SAVE_REQUEST_PENDING: # check if there is already such a save request already submitted, # no need to add it to the database in that case try: sor = SaveOriginRequest.objects.get(visit_type=visit_type, origin_url=origin_url, status=save_request_status) # if not add it to the database except ObjectDoesNotExist: sor = SaveOriginRequest.objects.create(visit_type=visit_type, origin_url=origin_url, status=save_request_status) # origin can not be saved as its url is blacklisted, # log the request to the database anyway else: sor = SaveOriginRequest.objects.create(visit_type=visit_type, origin_url=origin_url, status=save_request_status) if save_request_status == SAVE_REQUEST_REJECTED: raise ForbiddenExc('The origin url is blacklisted and will not be ' 'loaded into the archive.') return _save_request_dict(sor, task) def get_save_origin_requests_from_queryset(requests_queryset): """ Get all save requests from a SaveOriginRequest queryset. Args: requests_queryset (django.db.models.QuerySet): input SaveOriginRequest queryset Returns: list: A list of save origin requests dict as described in :func:`swh.web.common.origin_save.create_save_origin_request` """ task_ids = [] for sor in requests_queryset: task_ids.append(sor.loading_task_id) save_requests = [] if task_ids: tasks = scheduler.get_tasks(task_ids) tasks = {task['id']: task for task in tasks} for sor in requests_queryset: sr_dict = _save_request_dict(sor, tasks.get(sor.loading_task_id)) save_requests.append(sr_dict) return save_requests def get_save_origin_requests(visit_type, origin_url): """ Get all save requests for a given software origin. Args: visit_type (str): the type of visit origin_url (str): the url of the origin Raises: BadInputExc: the visit type or origin url is invalid NotFoundExc: no save requests can be found for the given origin Returns: list: A list of save origin requests dict as described in :func:`swh.web.common.origin_save.create_save_origin_request` """ _check_visit_type_savable(visit_type) _check_origin_url_valid(origin_url) sors = SaveOriginRequest.objects.filter(visit_type=visit_type, origin_url=origin_url) if sors.count() == 0: raise NotFoundExc(('No save requests found for visit of type ' '%s on origin with url %s.') % (visit_type, origin_url)) return get_save_origin_requests_from_queryset(sors) def get_save_origin_task_info(save_request_id): """ Get detailed information about an accepted save origin request and its associated loading task. If the associated loading task info is archived and removed from the scheduler database, returns an empty dictionary. Args: save_request_id (int): identifier of a save origin request Returns: dict: A dictionary with the following keys: + - **type**: loading task type - **arguments**: loading task arguments - **id**: loading task database identifier - **backend_id**: loading task celery identifier - **scheduled**: loading task scheduling date - **ended**: loading task termination date - **status**: loading task execution status + Depending on the availability of the task logs in the elasticsearch cluster of Software Heritage, the returned dictionary may also contain the following keys: + - **name**: associated celery task name - **message**: relevant log message from task execution - **duration**: task execution time (only if it succeeded) - **worker**: name of the worker that executed the task """ try: save_request = SaveOriginRequest.objects.get(id=save_request_id) except ObjectDoesNotExist: return {} task = scheduler.get_tasks([save_request.loading_task_id]) task = task[0] if task else None if task is None: return {} task_run = scheduler.get_task_runs([task['id']]) task_run = task_run[0] if task_run else None if task_run is None: return {} task_run['type'] = task['type'] task_run['arguments'] = task['arguments'] task_run['id'] = task_run['task'] del task_run['task'] del task_run['metadata'] del task_run['started'] es_workers_index_url = config.get_config()['es_workers_index_url'] if not es_workers_index_url: return task_run es_workers_index_url += '/_search' if save_request.visit_date: min_ts = save_request.visit_date max_ts = min_ts + timedelta(days=7) else: min_ts = save_request.request_date max_ts = min_ts + timedelta(days=30) min_ts = int(min_ts.timestamp()) * 1000 max_ts = int(max_ts.timestamp()) * 1000 save_task_status = _save_task_status[task['status']] priority = '3' if save_task_status == SAVE_TASK_FAILED else '6' query = { 'bool': { 'must': [ { 'match_phrase': { 'priority': { 'query': priority } } }, { 'match_phrase': { 'swh_task_id': { 'query': task_run['backend_id'] } } }, { 'range': { '@timestamp': { 'gte': min_ts, 'lte': max_ts, 'format': 'epoch_millis' } } } ] } } try: response = requests.post(es_workers_index_url, json={'query': query, 'sort': ['@timestamp']}, timeout=30) results = json.loads(response.text) if results['hits']['total'] >= 1: task_run_info = results['hits']['hits'][-1]['_source'] if 'swh_logging_args_runtime' in task_run_info: duration = task_run_info['swh_logging_args_runtime'] task_run['duration'] = duration if 'message' in task_run_info: task_run['message'] = task_run_info['message'] if 'swh_logging_args_name' in task_run_info: task_run['name'] = task_run_info['swh_logging_args_name'] elif 'swh_task_name' in task_run_info: task_run['name'] = task_run_info['swh_task_name'] if 'hostname' in task_run_info: task_run['worker'] = task_run_info['hostname'] elif 'host' in task_run_info: task_run['worker'] = task_run_info['host'] except Exception as e: logger.warning('Request to Elasticsearch failed\n%s' % str(e)) pass return task_run diff --git a/swh/web/common/service.py b/swh/web/common/service.py index b6eea0d0..5ae34eab 100644 --- a/swh/web/common/service.py +++ b/swh/web/common/service.py @@ -1,1113 +1,1108 @@ # Copyright (C) 2015-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import os from collections import defaultdict from swh.model import hashutil from swh.storage.algos import diff, revisions_walker from swh.web.common import converters from swh.web.common import query from swh.web.common.exc import NotFoundExc from swh.web.common.origin_visits import get_origin_visit from swh.web import config storage = config.storage() vault = config.vault() idx_storage = config.indexer_storage() MAX_LIMIT = 50 # Top limit the users can ask for def _first_element(l): """Returns the first element in the provided list or None if it is empty or None""" return next(iter(l or []), None) def lookup_multiple_hashes(hashes): """Lookup the passed hashes in a single DB connection, using batch processing. Args: An array of {filename: X, sha1: Y}, string X, hex sha1 string Y. Returns: The same array with elements updated with elem['found'] = true if the hash is present in storage, elem['found'] = false if not. """ hashlist = [hashutil.hash_to_bytes(elem['sha1']) for elem in hashes] content_missing = storage.content_missing_per_sha1(hashlist) missing = [hashutil.hash_to_hex(x) for x in content_missing] for x in hashes: x.update({'found': True}) for h in hashes: if h['sha1'] in missing: h['found'] = False return hashes def lookup_expression(expression, last_sha1, per_page): """Lookup expression in raw content. Args: expression (str): An expression to lookup through raw indexed content last_sha1 (str): Last sha1 seen per_page (int): Number of results per page Yields: ctags whose content match the expression """ limit = min(per_page, MAX_LIMIT) ctags = idx_storage.content_ctags_search(expression, last_sha1=last_sha1, limit=limit) for ctag in ctags: ctag = converters.from_swh(ctag, hashess={'id'}) ctag['sha1'] = ctag['id'] ctag.pop('id') yield ctag def lookup_hash(q): """Checks if the storage contains a given content checksum Args: query string of the form Returns: Dict with key found containing the hash info if the hash is present, None if not. """ algo, hash = query.parse_hash(q) found = _first_element(storage.content_find({algo: hash})) return {'found': converters.from_content(found), 'algo': algo} def search_hash(q): """Checks if the storage contains a given content checksum Args: query string of the form Returns: Dict with key found to True or False, according to whether the checksum is present or not """ algo, hash = query.parse_hash(q) found = _first_element(storage.content_find({algo: hash})) return {'found': found is not None} def _lookup_content_sha1(q): """Given a possible input, query for the content's sha1. Args: q: query string of the form Returns: binary sha1 if found or None """ algo, hash = query.parse_hash(q) if algo != 'sha1': hashes = _first_element(storage.content_find({algo: hash})) if not hashes: return None return hashes['sha1'] return hash def lookup_content_ctags(q): """Return ctags information from a specified content. Args: q: query string of the form Yields: ctags information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None ctags = list(idx_storage.content_ctags_get([sha1])) if not ctags: return None for ctag in ctags: yield converters.from_swh(ctag, hashess={'id'}) def lookup_content_filetype(q): """Return filetype information from a specified content. Args: q: query string of the form Yields: filetype information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None filetype = _first_element(list(idx_storage.content_mimetype_get([sha1]))) if not filetype: return None return converters.from_filetype(filetype) def lookup_content_language(q): """Return language information from a specified content. Args: q: query string of the form Yields: language information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None lang = _first_element(list(idx_storage.content_language_get([sha1]))) if not lang: return None return converters.from_swh(lang, hashess={'id'}) def lookup_content_license(q): """Return license information from a specified content. Args: q: query string of the form Yields: license information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None lic = _first_element(idx_storage.content_fossology_license_get([sha1])) if not lic: return None return converters.from_swh({'id': sha1, 'facts': lic[sha1]}, hashess={'id'}) def lookup_origin(origin): """Return information about the origin matching dict origin. Args: origin: origin's dict with 'url' key Returns: origin information as dict. """ origin_info = storage.origin_get(origin) if not origin_info: msg = 'Origin with url %s not found!' % origin['url'] raise NotFoundExc(msg) return converters.from_origin(origin_info) def lookup_origins(origin_from=1, origin_count=100): """Get list of archived software origins in a paginated way. Origins are sorted by id before returning them Args: origin_from (int): The minimum id of the origins to return origin_count (int): The maximum number of origins to return Yields: origins information as dicts """ origins = storage.origin_get_range(origin_from, origin_count) return map(converters.from_origin, origins) def search_origin(url_pattern, offset=0, limit=50, regexp=False, with_visit=False): """Search for origins whose urls contain a provided string pattern or match a provided regular expression. Args: url_pattern: the string pattern to search for in origin urls offset: number of found origins to skip before returning results limit: the maximum number of found origins to return Returns: list of origin information as dict. """ origins = storage.origin_search(url_pattern, offset, limit, regexp, with_visit) return map(converters.from_origin, origins) def search_origin_metadata(fulltext, limit=50): """Search for origins whose metadata match a provided string pattern. Args: fulltext: the string pattern to search for in origin metadata offset: number of found origins to skip before returning results limit: the maximum number of found origins to return Returns: list of origin metadata as dict. """ matches = idx_storage.origin_intrinsic_metadata_search_fulltext( conjunction=[fulltext], limit=limit) results = [] for match in matches: match['from_revision'] = hashutil.hash_to_hex(match['from_revision']) - origin = None - if match['origin_url']: - origin = storage.origin_get({'url': match['origin_url']}) - - del match['origin_url'] - if 'id' in match: - del match['id'] + origin = storage.origin_get({'url': match['id']}) + del match['id'] result = converters.from_origin(origin) if result: result['metadata'] = match results.append(result) return results def lookup_origin_intrinsic_metadata(origin_dict): """Return intrinsic metadata for origin whose origin matches given origin. Args: origin_dict: origin's dict with keys ('type' AND 'url') Returns: origin metadata. """ origin_info = storage.origin_get(origin_dict) if not origin_info: msg = 'Origin with url %s not found!' % origin_dict['url'] raise NotFoundExc(msg) origins = [origin_info['url']] match = _first_element( idx_storage.origin_intrinsic_metadata_get(origins)) result = {} if match: result = match['metadata'] return result def _to_sha1_bin(sha1_hex): _, sha1_git_bin = query.parse_hash_with_algorithms_or_throws( sha1_hex, ['sha1'], # HACK: sha1_git really 'Only sha1_git is supported.') return sha1_git_bin def _check_directory_exists(sha1_git, sha1_git_bin): if len(list(storage.directory_missing([sha1_git_bin]))): raise NotFoundExc('Directory with sha1_git %s not found' % sha1_git) def lookup_directory(sha1_git): """Return information about the directory with id sha1_git. Args: sha1_git as string Returns: directory information as dict. """ empty_dir_sha1 = '4b825dc642cb6eb9a060e54bf8d69288fbee4904' if sha1_git == empty_dir_sha1: return [] sha1_git_bin = _to_sha1_bin(sha1_git) _check_directory_exists(sha1_git, sha1_git_bin) directory_entries = storage.directory_ls(sha1_git_bin) return map(converters.from_directory_entry, directory_entries) def lookup_directory_with_path(sha1_git, path_string): """Return directory information for entry with path path_string w.r.t. root directory pointed by directory_sha1_git Args: - directory_sha1_git: sha1_git corresponding to the directory to which we append paths to (hopefully) find the entry - the relative path to the entry starting from the directory pointed by directory_sha1_git Raises: NotFoundExc if the directory entry is not found """ sha1_git_bin = _to_sha1_bin(sha1_git) _check_directory_exists(sha1_git, sha1_git_bin) paths = path_string.strip(os.path.sep).split(os.path.sep) queried_dir = storage.directory_entry_get_by_path( sha1_git_bin, list(map(lambda p: p.encode('utf-8'), paths))) if not queried_dir: raise NotFoundExc(('Directory entry with path %s from %s not found') % (path_string, sha1_git)) return converters.from_directory_entry(queried_dir) def lookup_release(release_sha1_git): """Return information about the release with sha1 release_sha1_git. Args: release_sha1_git: The release's sha1 as hexadecimal Returns: Release information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. """ sha1_git_bin = _to_sha1_bin(release_sha1_git) release = _first_element(storage.release_get([sha1_git_bin])) if not release: raise NotFoundExc('Release with sha1_git %s not found.' % release_sha1_git) return converters.from_release(release) def lookup_release_multiple(sha1_git_list): """Return information about the revisions identified with their sha1_git identifiers. Args: sha1_git_list: A list of revision sha1_git identifiers Returns: Release information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. """ sha1_bin_list = (_to_sha1_bin(sha1_git) for sha1_git in sha1_git_list) releases = storage.release_get(sha1_bin_list) or [] return (converters.from_release(r) for r in releases) def lookup_revision(rev_sha1_git): """Return information about the revision with sha1 revision_sha1_git. Args: revision_sha1_git: The revision's sha1 as hexadecimal Returns: Revision information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. NotFoundExc if there is no revision with the provided sha1_git. """ sha1_git_bin = _to_sha1_bin(rev_sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision with sha1_git %s not found.' % rev_sha1_git) return converters.from_revision(revision) def lookup_revision_multiple(sha1_git_list): """Return information about the revisions identified with their sha1_git identifiers. Args: sha1_git_list: A list of revision sha1_git identifiers Returns: Generator of revisions information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. """ sha1_bin_list = (_to_sha1_bin(sha1_git) for sha1_git in sha1_git_list) revisions = storage.revision_get(sha1_bin_list) or [] return (converters.from_revision(r) for r in revisions) def lookup_revision_message(rev_sha1_git): """Return the raw message of the revision with sha1 revision_sha1_git. Args: revision_sha1_git: The revision's sha1 as hexadecimal Returns: Decoded revision message as dict {'message': } Raises: ValueError if the identifier provided is not of sha1 nature. NotFoundExc if the revision is not found, or if it has no message """ sha1_git_bin = _to_sha1_bin(rev_sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision with sha1_git %s not found.' % rev_sha1_git) if 'message' not in revision: raise NotFoundExc('No message for revision with sha1_git %s.' % rev_sha1_git) res = {'message': revision['message']} return res def _lookup_revision_id_by(origin, branch_name, timestamp): def _get_snapshot_branch(snapshot, branch_name): snapshot = lookup_snapshot(visit['snapshot'], branches_from=branch_name, branches_count=10) branch = None if branch_name in snapshot['branches']: branch = snapshot['branches'][branch_name] return branch if isinstance(origin, int): origin = {'id': origin} elif isinstance(origin, str): origin = {'url': origin} else: raise TypeError('"origin" must be an int or a string.') visit = get_origin_visit(origin, visit_ts=timestamp) branch = _get_snapshot_branch(visit['snapshot'], branch_name) rev_id = None if branch and branch['target_type'] == 'revision': rev_id = branch['target'] elif branch and branch['target_type'] == 'alias': branch = _get_snapshot_branch(visit['snapshot'], branch['target']) if branch and branch['target_type'] == 'revision': rev_id = branch['target'] if not rev_id: raise NotFoundExc('Revision for origin %s and branch %s not found.' % (origin.get('url'), branch_name)) return rev_id def lookup_revision_by(origin, branch_name='HEAD', timestamp=None): """Lookup revision by origin, snapshot branch name and visit timestamp. If branch_name is not provided, lookup using 'HEAD' as default. If timestamp is not provided, use the most recent. Args: origin (Union[int,str]): origin of the revision branch_name (str): snapshot branch name timestamp (str/int): origin visit time frame Returns: dict: The revision matching the criterions Raises: NotFoundExc if no revision corresponds to the criterion """ rev_id = _lookup_revision_id_by(origin, branch_name, timestamp) return lookup_revision(rev_id) def lookup_revision_log(rev_sha1_git, limit): """Lookup revision log by revision id. Args: rev_sha1_git (str): The revision's sha1 as hexadecimal limit (int): the maximum number of revisions returned Returns: list: Revision log as list of revision dicts Raises: ValueError: if the identifier provided is not of sha1 nature. NotFoundExc: if there is no revision with the provided sha1_git. """ lookup_revision(rev_sha1_git) sha1_git_bin = _to_sha1_bin(rev_sha1_git) revision_entries = storage.revision_log([sha1_git_bin], limit) return map(converters.from_revision, revision_entries) def lookup_revision_log_by(origin, branch_name, timestamp, limit): """Lookup revision by origin, snapshot branch name and visit timestamp. Args: origin (Union[int,str]): origin of the revision branch_name (str): snapshot branch timestamp (str/int): origin visit time frame limit (int): the maximum number of revisions returned Returns: list: Revision log as list of revision dicts Raises: NotFoundExc: if no revision corresponds to the criterion """ rev_id = _lookup_revision_id_by(origin, branch_name, timestamp) return lookup_revision_log(rev_id, limit) def lookup_revision_with_context_by(origin, branch_name, timestamp, sha1_git, limit=100): """Return information about revision sha1_git, limited to the sub-graph of all transitive parents of sha1_git_root. sha1_git_root being resolved through the lookup of a revision by origin, branch_name and ts. In other words, sha1_git is an ancestor of sha1_git_root. Args: - origin: origin of the revision. - branch_name: revision's branch. - timestamp: revision's time frame. - sha1_git: one of sha1_git_root's ancestors. - limit: limit the lookup to 100 revisions back. Returns: Pair of (root_revision, revision). Information on sha1_git if it is an ancestor of sha1_git_root including children leading to sha1_git_root Raises: - BadInputExc in case of unknown algo_hash or bad hash. - NotFoundExc if either revision is not found or if sha1_git is not an ancestor of sha1_git_root. """ rev_root_id = _lookup_revision_id_by(origin, branch_name, timestamp) rev_root_id_bin = hashutil.hash_to_bytes(rev_root_id) rev_root = _first_element(storage.revision_get([rev_root_id_bin])) return (converters.from_revision(rev_root), lookup_revision_with_context(rev_root, sha1_git, limit)) def lookup_revision_with_context(sha1_git_root, sha1_git, limit=100): """Return information about revision sha1_git, limited to the sub-graph of all transitive parents of sha1_git_root. In other words, sha1_git is an ancestor of sha1_git_root. Args: sha1_git_root: latest revision. The type is either a sha1 (as an hex string) or a non converted dict. sha1_git: one of sha1_git_root's ancestors limit: limit the lookup to 100 revisions back Returns: Information on sha1_git if it is an ancestor of sha1_git_root including children leading to sha1_git_root Raises: BadInputExc in case of unknown algo_hash or bad hash NotFoundExc if either revision is not found or if sha1_git is not an ancestor of sha1_git_root """ sha1_git_bin = _to_sha1_bin(sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision %s not found' % sha1_git) if isinstance(sha1_git_root, str): sha1_git_root_bin = _to_sha1_bin(sha1_git_root) revision_root = _first_element(storage.revision_get([sha1_git_root_bin])) # noqa if not revision_root: raise NotFoundExc('Revision root %s not found' % sha1_git_root) else: sha1_git_root_bin = sha1_git_root['id'] revision_log = storage.revision_log([sha1_git_root_bin], limit) parents = {} children = defaultdict(list) for rev in revision_log: rev_id = rev['id'] parents[rev_id] = [] for parent_id in rev['parents']: parents[rev_id].append(parent_id) children[parent_id].append(rev_id) if revision['id'] not in parents: raise NotFoundExc('Revision %s is not an ancestor of %s' % (sha1_git, sha1_git_root)) revision['children'] = children[revision['id']] return converters.from_revision(revision) def lookup_directory_with_revision(sha1_git, dir_path=None, with_data=False): """Return information on directory pointed by revision with sha1_git. If dir_path is not provided, display top level directory. Otherwise, display the directory pointed by dir_path (if it exists). Args: sha1_git: revision's hash. dir_path: optional directory pointed to by that revision. with_data: boolean that indicates to retrieve the raw data if the path resolves to a content. Default to False (for the api) Returns: Information on the directory pointed to by that revision. Raises: BadInputExc in case of unknown algo_hash or bad hash. NotFoundExc either if the revision is not found or the path referenced does not exist. NotImplementedError in case of dir_path exists but do not reference a type 'dir' or 'file'. """ sha1_git_bin = _to_sha1_bin(sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision %s not found' % sha1_git) dir_sha1_git_bin = revision['directory'] if dir_path: paths = dir_path.strip(os.path.sep).split(os.path.sep) entity = storage.directory_entry_get_by_path( dir_sha1_git_bin, list(map(lambda p: p.encode('utf-8'), paths))) if not entity: raise NotFoundExc( "Directory or File '%s' pointed to by revision %s not found" % (dir_path, sha1_git)) else: entity = {'type': 'dir', 'target': dir_sha1_git_bin} if entity['type'] == 'dir': directory_entries = storage.directory_ls(entity['target']) or [] return {'type': 'dir', 'path': '.' if not dir_path else dir_path, 'revision': sha1_git, 'content': list(map(converters.from_directory_entry, directory_entries))} elif entity['type'] == 'file': # content content = _first_element( storage.content_find({'sha1_git': entity['target']})) if not content: raise NotFoundExc('Content not found for revision %s' % sha1_git) if with_data: c = _first_element(storage.content_get([content['sha1']])) content['data'] = c['data'] return {'type': 'file', 'path': '.' if not dir_path else dir_path, 'revision': sha1_git, 'content': converters.from_content(content)} elif entity['type'] == 'rev': # revision revision = next(storage.revision_get([entity['target']])) return {'type': 'rev', 'path': '.' if not dir_path else dir_path, 'revision': sha1_git, 'content': converters.from_revision(revision)} else: raise NotImplementedError('Entity of type %s not implemented.' % entity['type']) def lookup_content(q): """Lookup the content designed by q. Args: q: The release's sha1 as hexadecimal Raises: NotFoundExc if the requested content is not found """ algo, hash = query.parse_hash(q) c = _first_element(storage.content_find({algo: hash})) if not c: raise NotFoundExc('Content with %s checksum equals to %s not found!' % (algo, hashutil.hash_to_hex(hash))) return converters.from_content(c) def lookup_content_raw(q): """Lookup the content defined by q. Args: q: query string of the form Returns: dict with 'sha1' and 'data' keys. data representing its raw data decoded. Raises: NotFoundExc if the requested content is not found or if the content bytes are not available in the storage """ c = lookup_content(q) content_sha1_bytes = hashutil.hash_to_bytes(c['checksums']['sha1']) content = _first_element(storage.content_get([content_sha1_bytes])) if not content: algo, hash = query.parse_hash(q) raise NotFoundExc('Bytes of content with %s checksum equals to %s ' 'are not available!' % (algo, hashutil.hash_to_hex(hash))) return converters.from_content(content) def stat_counters(): """Return the stat counters for Software Heritage Returns: A dict mapping textual labels to integer values. """ return storage.stat_counters() def _lookup_origin_visits(origin_url, last_visit=None, limit=10): """Yields the origin origins' visits. Args: origin_url (str): origin to list visits for last_visit (int): last visit to lookup from limit (int): Number of elements max to display Yields: Dictionaries of origin_visit for that origin """ limit = min(limit, MAX_LIMIT) for visit in storage.origin_visit_get( origin_url, last_visit=last_visit, limit=limit): visit['origin'] = origin_url yield visit def lookup_origin_visits(origin, last_visit=None, per_page=10): """Yields the origin origins' visits. Args: origin: origin to list visits for Yields: Dictionaries of origin_visit for that origin """ visits = _lookup_origin_visits(origin, last_visit=last_visit, limit=per_page) for visit in visits: yield converters.from_origin_visit(visit) def lookup_origin_visit_latest(origin_url, require_snapshot): """Return the origin's latest visit Args: origin_url (str): origin to list visits for require_snapshot (bool): filter out origins without a snapshot Returns: dict: The origin_visit concerned """ visit = storage.origin_visit_get_latest( origin_url, require_snapshot=require_snapshot) if isinstance(visit['origin'], int): # soon-to-be-legacy origin ids visit['origin'] = storage.origin_get({'id': visit['origin']})['url'] return converters.from_origin_visit(visit) def lookup_origin_visit(origin_url, visit_id): """Return information about visit visit_id with origin origin. Args: origin (str): origin concerned by the visit visit_id: the visit identifier to lookup Yields: The dict origin_visit concerned """ visit = storage.origin_visit_get_by(origin_url, visit_id) if not visit: raise NotFoundExc('Origin %s or its visit ' 'with id %s not found!' % (origin_url, visit_id)) visit['origin'] = origin_url return converters.from_origin_visit(visit) def lookup_snapshot_size(snapshot_id): """Count the number of branches in the snapshot with the given id Args: snapshot_id (str): sha1 identifier of the snapshot Returns: dict: A dict whose keys are the target types of branches and values their corresponding amount """ snapshot_id_bin = _to_sha1_bin(snapshot_id) snapshot_size = storage.snapshot_count_branches(snapshot_id_bin) if 'revision' not in snapshot_size: snapshot_size['revision'] = 0 if 'release' not in snapshot_size: snapshot_size['release'] = 0 # adjust revision / release count for display if aliases are defined if 'alias' in snapshot_size: aliases = lookup_snapshot(snapshot_id, branches_count=snapshot_size['alias'], target_types=['alias']) for alias in aliases['branches'].values(): if lookup_snapshot(snapshot_id, branches_from=alias['target'], branches_count=1, target_types=['revision']): snapshot_size['revision'] += 1 else: snapshot_size['release'] += 1 del snapshot_size['alias'] return snapshot_size def lookup_snapshot(snapshot_id, branches_from='', branches_count=1000, target_types=None): """Return information about a snapshot, aka the list of named branches found during a specific visit of an origin. Args: snapshot_id (str): sha1 identifier of the snapshot branches_from (str): optional parameter used to skip branches whose name is lesser than it before returning them branches_count (int): optional parameter used to restrain the amount of returned branches target_types (list): optional parameter used to filter the target types of branch to return (possible values that can be contained in that list are `'content', 'directory', 'revision', 'release', 'snapshot', 'alias'`) Returns: A dict filled with the snapshot content. """ snapshot_id_bin = _to_sha1_bin(snapshot_id) snapshot = storage.snapshot_get_branches(snapshot_id_bin, branches_from.encode(), branches_count, target_types) if not snapshot: raise NotFoundExc('Snapshot with id %s not found!' % snapshot_id) return converters.from_snapshot(snapshot) def lookup_latest_origin_snapshot(origin, allowed_statuses=None): """Return information about the latest snapshot of an origin. .. warning:: At most 1000 branches contained in the snapshot will be returned for performance reasons. Args: origin: URL or integer identifier of the origin allowed_statuses: list of visit statuses considered to find the latest snapshot for the visit. For instance, ``allowed_statuses=['full']`` will only consider visits that have successfully run to completion. Returns: A dict filled with the snapshot content. """ snapshot = storage.snapshot_get_latest(origin, allowed_statuses) return converters.from_snapshot(snapshot) def lookup_revision_through(revision, limit=100): """Retrieve a revision from the criterion stored in revision dictionary. Args: revision: Dictionary of criterion to lookup the revision with. Here are the supported combination of possible values: - origin_url, branch_name, ts, sha1_git - origin_url, branch_name, ts - sha1_git_root, sha1_git - sha1_git Returns: None if the revision is not found or the actual revision. """ if ( 'origin_url' in revision and 'branch_name' in revision and 'ts' in revision and 'sha1_git' in revision): return lookup_revision_with_context_by(revision['origin_url'], revision['branch_name'], revision['ts'], revision['sha1_git'], limit) if ( 'origin_url' in revision and 'branch_name' in revision and 'ts' in revision): return lookup_revision_by(revision['origin_url'], revision['branch_name'], revision['ts']) if ( 'sha1_git_root' in revision and 'sha1_git' in revision): return lookup_revision_with_context(revision['sha1_git_root'], revision['sha1_git'], limit) if 'sha1_git' in revision: return lookup_revision(revision['sha1_git']) # this should not happen raise NotImplementedError('Should not happen!') def lookup_directory_through_revision(revision, path=None, limit=100, with_data=False): """Retrieve the directory information from the revision. Args: revision: dictionary of criterion representing a revision to lookup path: directory's path to lookup. limit: optional query parameter to limit the revisions log (default to 100). For now, note that this limit could impede the transitivity conclusion about sha1_git not being an ancestor of. with_data: indicate to retrieve the content's raw data if path resolves to a content. Returns: The directory pointing to by the revision criterions at path. """ rev = lookup_revision_through(revision, limit) if not rev: raise NotFoundExc('Revision with criterion %s not found!' % revision) return (rev['id'], lookup_directory_with_revision(rev['id'], path, with_data)) def vault_cook(obj_type, obj_id, email=None): """Cook a vault bundle. """ return vault.cook(obj_type, obj_id, email=email) def vault_fetch(obj_type, obj_id): """Fetch a vault bundle. """ return vault.fetch(obj_type, obj_id) def vault_progress(obj_type, obj_id): """Get the current progress of a vault bundle. """ return vault.progress(obj_type, obj_id) def diff_revision(rev_id): """Get the list of file changes (insertion / deletion / modification / renaming) for a particular revision. """ rev_sha1_git_bin = _to_sha1_bin(rev_id) changes = diff.diff_revision(storage, rev_sha1_git_bin, track_renaming=True) for change in changes: change['from'] = converters.from_directory_entry(change['from']) change['to'] = converters.from_directory_entry(change['to']) if change['from_path']: change['from_path'] = change['from_path'].decode('utf-8') if change['to_path']: change['to_path'] = change['to_path'].decode('utf-8') return changes class _RevisionsWalkerProxy(object): """ Proxy class wrapping a revisions walker iterator from swh-storage and performing needed conversions. """ def __init__(self, rev_walker_type, rev_start, *args, **kwargs): rev_start_bin = hashutil.hash_to_bytes(rev_start) self.revisions_walker = \ revisions_walker.get_revisions_walker(rev_walker_type, storage, rev_start_bin, *args, **kwargs) def export_state(self): return self.revisions_walker.export_state() def __next__(self): return converters.from_revision(next(self.revisions_walker)) def __iter__(self): return self def get_revisions_walker(rev_walker_type, rev_start, *args, **kwargs): """ Utility function to instantiate a revisions walker of a given type, see :mod:`swh.storage.algos.revisions_walker`. Args: rev_walker_type (str): the type of revisions walker to return, possible values are: ``committer_date``, ``dfs``, ``dfs_post``, ``bfs`` and ``path`` rev_start (str): hexadecimal representation of a revision identifier args (list): position arguments to pass to the revisions walker constructor kwargs (dict): keyword arguments to pass to the revisions walker constructor """ # first check if the provided revision is valid lookup_revision(rev_start) return _RevisionsWalkerProxy(rev_walker_type, rev_start, *args, **kwargs) diff --git a/swh/web/common/urlsindex.py b/swh/web/common/urlsindex.py index 0c9649b7..08000426 100644 --- a/swh/web/common/urlsindex.py +++ b/swh/web/common/urlsindex.py @@ -1,76 +1,80 @@ # Copyright (C) 2017-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information +from typing import Dict, List + +import django.urls + from django.conf.urls import url from django.shortcuts import redirect class UrlsIndex(object): """ Simple helper class for centralizing url patterns of a Django web application. Derived classes should override the 'scope' class attribute otherwise all declared patterns will be grouped under the default one. """ - _urlpatterns = {} + _urlpatterns = {} # type: Dict[str, List[django.urls.URLPattern]] scope = 'default' @classmethod def add_url_pattern(cls, url_pattern, view, view_name=None): """ Class method that adds an url pattern to the current scope. Args: url_pattern: regex describing a Django url view: function implementing the Django view view_name: name of the view used to reverse the url """ if cls.scope not in cls._urlpatterns: cls._urlpatterns[cls.scope] = [] if view_name: cls._urlpatterns[cls.scope].append(url(url_pattern, view, name=view_name)) else: cls._urlpatterns[cls.scope].append(url(url_pattern, view)) @classmethod def add_redirect_for_checksum_args(cls, view_name, url_patterns, checksum_args): """ Class method that redirects to view with lowercase checksums when upper/mixed case checksums are passed as url arguments. Args: view_name (str): name of the view to redirect requests url_patterns (List[str]): regexps describing the view urls checksum_args (List[str]): url argument names corresponding to checksum values """ new_view_name = view_name+'-uppercase-checksum' for url_pattern in url_patterns: url_pattern_upper = url_pattern.replace('[0-9a-f]', '[0-9a-fA-F]') def view_redirect(request, *args, **kwargs): for checksum_arg in checksum_args: checksum_upper = kwargs[checksum_arg] kwargs[checksum_arg] = checksum_upper.lower() return redirect(view_name, *args, **kwargs) cls.add_url_pattern(url_pattern_upper, view_redirect, new_view_name) @classmethod def get_url_patterns(cls): """ Class method that returns the list of url pattern associated to the current scope. Returns: The list of url patterns associated to the current scope """ return cls._urlpatterns[cls.scope] diff --git a/swh/web/config.py b/swh/web/config.py index 19b30826..c506ef76 100644 --- a/swh/web/config.py +++ b/swh/web/config.py @@ -1,158 +1,160 @@ # Copyright (C) 2017-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import os +from typing import Any, Dict + from swh.core import config from swh.indexer.storage import get_indexer_storage from swh.scheduler import get_scheduler from swh.storage import get_storage from swh.vault import get_vault from swh.web import settings SETTINGS_DIR = os.path.dirname(settings.__file__) DEFAULT_CONFIG = { 'allowed_hosts': ('list', []), 'storage': ('dict', { 'cls': 'remote', 'args': { 'url': 'http://127.0.0.1:5002/', 'timeout': 10, }, }), 'indexer_storage': ('dict', { 'cls': 'remote', 'args': { 'url': 'http://127.0.0.1:5007/', 'timeout': 1, } }), 'log_dir': ('string', '/tmp/swh/log'), 'debug': ('bool', False), 'serve_assets': ('bool', False), 'host': ('string', '127.0.0.1'), 'port': ('int', 5004), 'secret_key': ('string', 'development key'), # do not display code highlighting for content > 1MB 'content_display_max_size': ('int', 5 * 1024 * 1024), 'snapshot_content_max_size': ('int', 1000), 'throttling': ('dict', { 'cache_uri': None, # production: memcached as cache (127.0.0.1:11211) # development: in-memory cache so None 'scopes': { 'swh_api': { 'limiter_rate': { 'default': '120/h' }, 'exempted_networks': ['127.0.0.0/8'] }, 'swh_vault_cooking': { 'limiter_rate': { 'default': '120/h', 'GET': '60/m' }, 'exempted_networks': ['127.0.0.0/8'] }, 'swh_save_origin': { 'limiter_rate': { 'default': '120/h', 'POST': '10/h' }, 'exempted_networks': ['127.0.0.0/8'] }, 'swh_api_origin_visit_latest': { 'limiter_rate': { 'default': '700/m' }, 'exempted_networks': ['127.0.0.0/8'], }, } }), 'vault': ('dict', { 'cls': 'remote', 'args': { 'url': 'http://127.0.0.1:5005/', } }), 'scheduler': ('dict', { 'cls': 'remote', 'args': { 'url': 'http://127.0.0.1:5008/' } }), 'development_db': ('string', os.path.join(SETTINGS_DIR, 'db.sqlite3')), 'test_db': ('string', os.path.join(SETTINGS_DIR, 'testdb.sqlite3')), 'production_db': ('string', '/var/lib/swh/web.sqlite3'), 'deposit': ('dict', { 'private_api_url': 'https://deposit.softwareheritage.org/1/private/', 'private_api_user': 'swhworker', 'private_api_password': '' }), 'coverage_count_origins': ('bool', False), 'e2e_tests_mode': ('bool', False), 'es_workers_index_url': ('string', ''), 'history_counters_url': ('string', 'https://stats.export.softwareheritage.org/history_counters.json'), # noqa } -swhweb_config = {} +swhweb_config = {} # type: Dict[str, Any] def get_config(config_file='web/web'): """Read the configuration file `config_file`. If an environment variable SWH_CONFIG_FILENAME is defined, this takes precedence over the config_file parameter. In any case, update the app with parameters (secret_key, conf) and return the parsed configuration as a dict. If no configuration file is provided, return a default configuration. """ if not swhweb_config: config_filename = os.environ.get('SWH_CONFIG_FILENAME') if config_filename: config_file = config_filename cfg = config.load_named_config(config_file, DEFAULT_CONFIG) swhweb_config.update(cfg) config.prepare_folders(swhweb_config, 'log_dir') swhweb_config['storage'] = get_storage(**swhweb_config['storage']) swhweb_config['vault'] = get_vault(**swhweb_config['vault']) swhweb_config['indexer_storage'] = \ get_indexer_storage(**swhweb_config['indexer_storage']) swhweb_config['scheduler'] = get_scheduler( **swhweb_config['scheduler']) return swhweb_config def storage(): """Return the current application's storage. """ return get_config()['storage'] def vault(): """Return the current application's vault. """ return get_config()['vault'] def indexer_storage(): """Return the current application's indexer storage. """ return get_config()['indexer_storage'] def scheduler(): """Return the current application's scheduler. """ return get_config()['scheduler'] diff --git a/swh/web/py.typed b/swh/web/py.typed new file mode 100644 index 00000000..1242d432 --- /dev/null +++ b/swh/web/py.typed @@ -0,0 +1 @@ +# Marker file for PEP 561. diff --git a/swh/web/settings/tests.py b/swh/web/settings/tests.py index a91d22a9..12d2d4bf 100644 --- a/swh/web/settings/tests.py +++ b/swh/web/settings/tests.py @@ -1,103 +1,103 @@ # Copyright (C) 2017-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information """ Django tests settings for swh-web. """ import sys from swh.web.config import get_config scope1_limiter_rate = 3 scope1_limiter_rate_post = 1 scope2_limiter_rate = 5 scope2_limiter_rate_post = 2 scope3_limiter_rate = 1 scope3_limiter_rate_post = 1 save_origin_rate_post = 10 swh_web_config = get_config() swh_web_config.update({ 'debug': False, 'secret_key': 'test', 'history_counters_url': '', 'throttling': { 'cache_uri': None, 'scopes': { 'swh_api': { 'limiter_rate': { 'default': '60/min' }, 'exempted_networks': ['127.0.0.0/8'] }, 'swh_api_origin_visit_latest': { 'limiter_rate': { 'default': '6000/min' }, 'exempted_networks': ['127.0.0.0/8'] }, 'swh_vault_cooking': { 'limiter_rate': { 'default': '120/h', 'GET': '60/m' }, 'exempted_networks': ['127.0.0.0/8'] }, 'swh_save_origin': { 'limiter_rate': { 'default': '120/h', 'POST': '%s/h' % save_origin_rate_post, } }, 'scope1': { 'limiter_rate': { 'default': '%s/min' % scope1_limiter_rate, 'POST': '%s/min' % scope1_limiter_rate_post, } }, 'scope2': { 'limiter_rate': { 'default': '%s/min' % scope2_limiter_rate, 'POST': '%s/min' % scope2_limiter_rate_post } }, 'scope3': { 'limiter_rate': { 'default': '%s/min' % scope3_limiter_rate, 'POST': '%s/min' % scope3_limiter_rate_post }, 'exempted_networks': ['127.0.0.0/8'] } } } }) from .common import * # noqa from .common import ALLOWED_HOSTS, LOGGING # noqa DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': swh_web_config['test_db'], } } # when not running unit tests, make the webapp fetch data from memory storages if 'pytest' not in sys.argv[0]: swh_web_config.update({ 'debug': True, 'e2e_tests_mode': True }) from swh.web.tests.data import get_tests_data, override_storages # noqa test_data = get_tests_data() override_storages(test_data['storage'], test_data['idx_storage']) else: ALLOWED_HOSTS += ['testserver'] # Silent DEBUG output when running unit tests - LOGGING['handlers']['console']['level'] = 'INFO' + LOGGING['handlers']['console']['level'] = 'INFO' # type: ignore diff --git a/swh/web/tests/api/views/test_origin.py b/swh/web/tests/api/views/test_origin.py index aae3df9a..3bd3d3fb 100644 --- a/swh/web/tests/api/views/test_origin.py +++ b/swh/web/tests/api/views/test_origin.py @@ -1,666 +1,632 @@ # Copyright (C) 2015-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from unittest.mock import patch from hypothesis import given, strategies import pytest from requests.utils import parse_header_links from rest_framework.test import APITestCase from swh.storage.exc import StorageDBError, StorageAPIError from swh.web.common.exc import BadInputExc from swh.web.common.utils import reverse from swh.web.common.origin_visits import get_origin_visits from swh.web.tests.strategies import ( origin, new_origin, visit_dates, new_snapshots ) from swh.web.tests.testcase import WebTestCase from swh.web.tests.data import get_tests_data class OriginApiTestCase(WebTestCase, APITestCase): def _scroll_results(self, url): """Iterates through pages of results, and returns them all.""" results = [] while True: rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') results.extend(rv.data) if 'Link' in rv: for link in parse_header_links(rv['Link']): if link['rel'] == 'next': # Found link to next page of results url = link['url'] break else: # No link with 'rel=next' break else: # No Link header break return results @patch('swh.web.api.views.origin.get_origin_visits') def test_api_lookup_origin_visits_raise_error( self, mock_get_origin_visits, ): err_msg = 'voluntary error to check the bad request middleware.' mock_get_origin_visits.side_effect = BadInputExc(err_msg) url = reverse( 'api-1-origin-visits', url_args={'origin_url': 'http://foo'}) rv = self.client.get(url) self.assertEqual(rv.status_code, 400, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'BadInputExc', 'reason': err_msg}) @patch('swh.web.api.views.origin.get_origin_visits') def test_api_lookup_origin_visits_raise_swh_storage_error_db( self, mock_get_origin_visits): err_msg = 'Storage exploded! Will be back online shortly!' mock_get_origin_visits.side_effect = StorageDBError(err_msg) url = reverse( 'api-1-origin-visits', url_args={'origin_url': 'http://foo'}) rv = self.client.get(url) self.assertEqual(rv.status_code, 503, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'StorageDBError', 'reason': 'An unexpected error occurred in the backend: %s' % err_msg}) @patch('swh.web.api.views.origin.get_origin_visits') def test_api_lookup_origin_visits_raise_swh_storage_error_api( self, mock_get_origin_visits): err_msg = 'Storage API dropped dead! Will resurrect asap!' mock_get_origin_visits.side_effect = StorageAPIError(err_msg) url = reverse( 'api-1-origin-visits', url_args={'origin_url': 'http://foo'}) rv = self.client.get(url) self.assertEqual(rv.status_code, 503, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'StorageAPIError', 'reason': 'An unexpected error occurred in the api backend: %s' % err_msg }) @given(new_origin(), visit_dates(3), new_snapshots(3)) def test_api_lookup_origin_visits(self, new_origin, visit_dates, new_snapshots): self.storage.origin_add_one(new_origin) for i, visit_date in enumerate(visit_dates): origin_visit = self.storage.origin_visit_add( new_origin['url'], visit_date, type='git') self.storage.snapshot_add([new_snapshots[i]]) self.storage.origin_visit_update( new_origin['url'], origin_visit['visit'], snapshot=new_snapshots[i]['id']) all_visits = list(reversed(get_origin_visits(new_origin))) for last_visit, expected_visits in ( (None, all_visits[:2]), (all_visits[1]['visit'], all_visits[2:4])): url = reverse('api-1-origin-visits', url_args={'origin_url': new_origin['url']}, query_params={'per_page': 2, 'last_visit': last_visit}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') for expected_visit in expected_visits: origin_visit_url = reverse( 'api-1-origin-visit', url_args={'origin_url': new_origin['url'], 'visit_id': expected_visit['visit']}) snapshot_url = reverse( 'api-1-snapshot', url_args={'snapshot_id': expected_visit['snapshot']}) expected_visit['origin'] = new_origin['url'] expected_visit['origin_visit_url'] = origin_visit_url expected_visit['snapshot_url'] = snapshot_url self.assertEqual(rv.data, expected_visits) @given(new_origin(), visit_dates(3), new_snapshots(3)) def test_api_lookup_origin_visits_by_id(self, new_origin, visit_dates, new_snapshots): self.storage.origin_add_one(new_origin) for i, visit_date in enumerate(visit_dates): origin_visit = self.storage.origin_visit_add( new_origin['url'], visit_date, type='git') self.storage.snapshot_add([new_snapshots[i]]) self.storage.origin_visit_update( new_origin['url'], origin_visit['visit'], snapshot=new_snapshots[i]['id']) all_visits = list(reversed(get_origin_visits(new_origin))) for last_visit, expected_visits in ( (None, all_visits[:2]), (all_visits[1]['visit'], all_visits[2:4])): url = reverse('api-1-origin-visits', url_args={'origin_url': new_origin['url']}, query_params={'per_page': 2, 'last_visit': last_visit}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') for expected_visit in expected_visits: origin_visit_url = reverse( 'api-1-origin-visit', url_args={'origin_url': new_origin['url'], 'visit_id': expected_visit['visit']}) snapshot_url = reverse( 'api-1-snapshot', url_args={'snapshot_id': expected_visit['snapshot']}) expected_visit['origin'] = new_origin['url'] expected_visit['origin_visit_url'] = origin_visit_url expected_visit['snapshot_url'] = snapshot_url self.assertEqual(rv.data, expected_visits) @given(new_origin(), visit_dates(3), new_snapshots(3)) def test_api_lookup_origin_visit(self, new_origin, visit_dates, new_snapshots): self.storage.origin_add_one(new_origin) for i, visit_date in enumerate(visit_dates): origin_visit = self.storage.origin_visit_add( new_origin['url'], visit_date, type='git') visit_id = origin_visit['visit'] self.storage.snapshot_add([new_snapshots[i]]) self.storage.origin_visit_update( new_origin['url'], origin_visit['visit'], snapshot=new_snapshots[i]['id']) url = reverse('api-1-origin-visit', url_args={'origin_url': new_origin['url'], 'visit_id': visit_id}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') expected_visit = self.origin_visit_get_by( new_origin['url'], visit_id) origin_url = reverse('api-1-origin', url_args={'origin_url': new_origin['url']}) snapshot_url = reverse( 'api-1-snapshot', url_args={'snapshot_id': expected_visit['snapshot']}) expected_visit['origin'] = new_origin['url'] expected_visit['origin_url'] = origin_url expected_visit['snapshot_url'] = snapshot_url self.assertEqual(rv.data, expected_visit) @given(new_origin(), visit_dates(2), new_snapshots(1)) def test_api_lookup_origin_visit_latest( self, new_origin, visit_dates, new_snapshots): self.storage.origin_add_one(new_origin) visit_dates.sort() visit_ids = [] for i, visit_date in enumerate(visit_dates): origin_visit = self.storage.origin_visit_add( new_origin['url'], visit_date, type='git') visit_ids.append(origin_visit['visit']) self.storage.snapshot_add([new_snapshots[0]]) self.storage.origin_visit_update( new_origin['url'], visit_ids[0], snapshot=new_snapshots[0]['id']) url = reverse('api-1-origin-visit-latest', url_args={'origin_url': new_origin['url']}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') expected_visit = self.origin_visit_get_by( new_origin['url'], visit_ids[1]) origin_url = reverse('api-1-origin', url_args={'origin_url': new_origin['url']}) expected_visit['origin'] = new_origin['url'] expected_visit['origin_url'] = origin_url expected_visit['snapshot_url'] = None self.assertEqual(rv.data, expected_visit) @given(new_origin(), visit_dates(2), new_snapshots(1)) def test_api_lookup_origin_visit_latest_with_snapshot( self, new_origin, visit_dates, new_snapshots): self.storage.origin_add_one(new_origin) visit_dates.sort() visit_ids = [] for i, visit_date in enumerate(visit_dates): origin_visit = self.storage.origin_visit_add( new_origin['url'], visit_date, type='git') visit_ids.append(origin_visit['visit']) self.storage.snapshot_add([new_snapshots[0]]) self.storage.origin_visit_update( new_origin['url'], visit_ids[0], snapshot=new_snapshots[0]['id']) url = reverse('api-1-origin-visit-latest', url_args={'origin_url': new_origin['url']}) url += '?require_snapshot=true' rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') expected_visit = self.origin_visit_get_by( new_origin['url'], visit_ids[0]) origin_url = reverse('api-1-origin', url_args={'origin_url': new_origin['url']}) snapshot_url = reverse( 'api-1-snapshot', url_args={'snapshot_id': expected_visit['snapshot']}) expected_visit['origin'] = new_origin['url'] expected_visit['origin_url'] = origin_url expected_visit['snapshot_url'] = snapshot_url self.assertEqual(rv.data, expected_visit) @given(origin()) def test_api_lookup_origin_visit_not_found(self, origin): all_visits = list(reversed(get_origin_visits(origin))) max_visit_id = max([v['visit'] for v in all_visits]) url = reverse('api-1-origin-visit', url_args={'origin_url': origin['url'], 'visit_id': max_visit_id + 1}) rv = self.client.get(url) self.assertEqual(rv.status_code, 404, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'NotFoundExc', 'reason': 'Origin %s or its visit with id %s not found!' % (origin['url'], max_visit_id+1) }) @pytest.mark.origin_id def test_api_origins(self): origins = get_tests_data()['origins'] origin_urls = {origin['url'] for origin in origins} # Get only one url = reverse('api-1-origins', query_params={'origin_count': 1}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(len(rv.data), 1) self.assertLess({origin['url'] for origin in rv.data}, origin_urls) # Get all url = reverse('api-1-origins', query_params={'origin_count': len(origins)}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(len(rv.data), len(origins)) self.assertEqual({origin['url'] for origin in rv.data}, origin_urls) # Get "all + 10" url = reverse('api-1-origins', query_params={'origin_count': len(origins)+10}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(len(rv.data), len(origins)) self.assertEqual({origin['url'] for origin in rv.data}, origin_urls) @pytest.mark.origin_id @given(strategies.integers(min_value=1)) def test_api_origins_scroll(self, origin_count): origins = get_tests_data()['origins'] origin_urls = {origin['url'] for origin in origins} url = reverse('api-1-origins', query_params={'origin_count': origin_count}) results = self._scroll_results(url) self.assertEqual(len(results), len(origins)) self.assertEqual({origin['url'] for origin in results}, origin_urls) @given(origin()) def test_api_origin_by_url(self, origin): url = reverse('api-1-origin', url_args={'origin_url': origin['url']}) rv = self.client.get(url) expected_origin = self.origin_get(origin) origin_visits_url = reverse('api-1-origin-visits', url_args={'origin_url': origin['url']}) expected_origin['origin_visits_url'] = origin_visits_url self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, expected_origin) @given(new_origin()) def test_api_origin_not_found(self, new_origin): url = reverse('api-1-origin', url_args={'origin_url': new_origin['url']}) rv = self.client.get(url) self.assertEqual(rv.status_code, 404, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(rv.data, { 'exception': 'NotFoundExc', 'reason': 'Origin with url %s not found!' % new_origin['url'] }) @pytest.mark.origin_id def test_api_origin_search(self): expected_origins = { 'https://github.com/wcoder/highlightjs-line-numbers.js', 'https://github.com/memononen/libtess2', } # Search for 'github.com', get only one url = reverse('api-1-origin-search', url_args={'url_pattern': 'github.com'}, query_params={'limit': 1}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(len(rv.data), 1) self.assertLess({origin['url'] for origin in rv.data}, expected_origins) # Search for 'github.com', get all url = reverse('api-1-origin-search', url_args={'url_pattern': 'github.com'}, query_params={'limit': 2}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual({origin['url'] for origin in rv.data}, expected_origins) # Search for 'github.com', get more than available url = reverse('api-1-origin-search', url_args={'url_pattern': 'github.com'}, query_params={'limit': 10}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual({origin['url'] for origin in rv.data}, expected_origins) @pytest.mark.origin_id def test_api_origin_search_regexp(self): expected_origins = { 'https://github.com/memononen/libtess2', 'repo_with_submodules' } url = reverse('api-1-origin-search', url_args={'url_pattern': '(repo|libtess)'}, query_params={'limit': 10, 'regexp': True}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual({origin['url'] for origin in rv.data}, expected_origins) @pytest.mark.origin_id @given(strategies.integers(min_value=1)) def test_api_origin_search_scroll(self, limit): expected_origins = { 'https://github.com/wcoder/highlightjs-line-numbers.js', 'https://github.com/memononen/libtess2', } url = reverse('api-1-origin-search', url_args={'url_pattern': 'github.com'}, query_params={'limit': limit}) results = self._scroll_results(url) self.assertEqual({origin['url'] for origin in results}, expected_origins) def test_api_origin_search_limit(self): self.storage.origin_add([ {'url': 'http://foobar/{}'.format(i)} for i in range(2000) ]) url = reverse('api-1-origin-search', url_args={'url_pattern': 'foobar'}, query_params={'limit': 1050}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.data) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(len(rv.data), 1000) @given(origin()) def test_api_origin_metadata_search(self, origin): with patch('swh.web.common.service.idx_storage') as mock_idx_storage: mock_idx_storage.origin_intrinsic_metadata_search_fulltext \ .side_effect = lambda conjunction, limit: [{ 'from_revision': ( b'p&\xb7\xc1\xa2\xafVR\x1e\x95\x1c\x01\xed ' b'\xf2U\xfa\x05B8'), 'metadata': {'author': 'Jane Doe'}, - 'origin_url': origin['url'], + 'id': origin['url'], 'tool': { 'configuration': { 'context': ['NpmMapping', 'CodemetaMapping'], 'type': 'local' }, 'id': 3, 'name': 'swh-metadata-detector', 'version': '0.0.1' } }] url = reverse('api-1-origin-metadata-search', query_params={'fulltext': 'Jane Doe'}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.content) self.assertEqual(rv['Content-Type'], 'application/json') expected_data = [{ 'url': origin['url'], 'metadata': { 'metadata': {'author': 'Jane Doe'}, 'from_revision': ( '7026b7c1a2af56521e951c01ed20f255fa054238'), 'tool': { 'configuration': { 'context': ['NpmMapping', 'CodemetaMapping'], 'type': 'local' }, 'id': 3, 'name': 'swh-metadata-detector', 'version': '0.0.1', } } }] - actual_data = rv.data - for d in actual_data: - if 'id' in d: - del d['id'] self.assertEqual(rv.data, expected_data) mock_idx_storage.origin_intrinsic_metadata_search_fulltext \ .assert_called_with(conjunction=['Jane Doe'], limit=70) @given(origin()) def test_api_origin_metadata_search_limit(self, origin): with patch('swh.web.common.service.idx_storage') as mock_idx_storage: mock_idx_storage.origin_intrinsic_metadata_search_fulltext \ .side_effect = lambda conjunction, limit: [{ 'from_revision': ( b'p&\xb7\xc1\xa2\xafVR\x1e\x95\x1c\x01\xed ' b'\xf2U\xfa\x05B8'), 'metadata': {'author': 'Jane Doe'}, - 'origin_url': origin['url'], + 'id': origin['url'], 'tool': { 'configuration': { 'context': ['NpmMapping', 'CodemetaMapping'], 'type': 'local' }, 'id': 3, 'name': 'swh-metadata-detector', 'version': '0.0.1' } }] url = reverse('api-1-origin-metadata-search', query_params={'fulltext': 'Jane Doe'}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.content) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(len(rv.data), 1) mock_idx_storage.origin_intrinsic_metadata_search_fulltext \ .assert_called_with(conjunction=['Jane Doe'], limit=70) url = reverse('api-1-origin-metadata-search', query_params={'fulltext': 'Jane Doe', 'limit': 10}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.content) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(len(rv.data), 1) mock_idx_storage.origin_intrinsic_metadata_search_fulltext \ .assert_called_with(conjunction=['Jane Doe'], limit=10) url = reverse('api-1-origin-metadata-search', query_params={'fulltext': 'Jane Doe', 'limit': 987}) rv = self.client.get(url) self.assertEqual(rv.status_code, 200, rv.content) self.assertEqual(rv['Content-Type'], 'application/json') self.assertEqual(len(rv.data), 1) mock_idx_storage.origin_intrinsic_metadata_search_fulltext \ .assert_called_with(conjunction=['Jane Doe'], limit=100) @given(origin()) def test_api_origin_intrinsic_metadata(self, origin): with patch('swh.web.common.service.idx_storage') as mock_idx_storage: mock_idx_storage.origin_intrinsic_metadata_get \ .side_effect = lambda origin_urls: [{ 'from_revision': ( b'p&\xb7\xc1\xa2\xafVR\x1e\x95\x1c\x01\xed ' b'\xf2U\xfa\x05B8'), 'metadata': {'author': 'Jane Doe'}, - 'origin_url': origin['url'], + 'id': origin['url'], 'tool': { 'configuration': { 'context': ['NpmMapping', 'CodemetaMapping'], 'type': 'local' }, 'id': 3, 'name': 'swh-metadata-detector', 'version': '0.0.1' } }] url = reverse('api-origin-intrinsic-metadata', url_args={'origin_url': origin['url']}) rv = self.client.get(url) mock_idx_storage.origin_intrinsic_metadata_get \ .assert_called_once_with([origin['url']]) self.assertEqual(rv.status_code, 200, rv.content) self.assertEqual(rv['Content-Type'], 'application/json') expected_data = {'author': 'Jane Doe'} self.assertEqual(rv.data, expected_data) @patch('swh.web.common.service.idx_storage') def test_api_origin_metadata_search_invalid(self, mock_idx_storage): url = reverse('api-1-origin-metadata-search') rv = self.client.get(url) self.assertEqual(rv.status_code, 400, rv.content) mock_idx_storage.assert_not_called() - - def test_api_origin_metadata_search_missing_origin_url(self): - with patch('swh.web.common.service.idx_storage') as mock_idx_storage: - mock_idx_storage.origin_intrinsic_metadata_search_fulltext \ - .side_effect = lambda conjunction, limit: [{ - 'from_revision': ( - b'p&\xb7\xc1\xa2\xafVR\x1e\x95\x1c\x01\xed ' - b'\xf2U\xfa\x05B8'), - 'metadata': {'author': 'Jane Doe'}, - 'origin_url': None, - 'tool': { - 'configuration': { - 'context': ['NpmMapping', 'CodemetaMapping'], - 'type': 'local' - }, - 'id': 3, - 'name': 'swh-metadata-detector', - 'version': '0.0.1' - } - }] - - url = reverse('api-1-origin-metadata-search', - query_params={'fulltext': 'Jane Doe'}) - rv = self.client.get(url) - - self.assertEqual(rv.status_code, 200, rv.content) - self.assertEqual(rv['Content-Type'], 'application/json') - self.assertEqual(len(rv.data), 0) - mock_idx_storage.origin_intrinsic_metadata_search_fulltext \ - .assert_called_with(conjunction=['Jane Doe'], limit=70) diff --git a/swh/web/tests/data.py b/swh/web/tests/data.py index bb651049..8ffb740e 100644 --- a/swh/web/tests/data.py +++ b/swh/web/tests/data.py @@ -1,466 +1,467 @@ # Copyright (C) 2018-2019 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information -from copy import deepcopy import os import random +from copy import deepcopy +from typing import Dict from rest_framework.decorators import api_view from rest_framework.response import Response from swh.indexer.fossology_license import FossologyLicenseIndexer from swh.indexer.mimetype import MimetypeIndexer from swh.indexer.ctags import CtagsIndexer from swh.indexer.storage import get_indexer_storage from swh.model.from_disk import Directory from swh.model.hashutil import hash_to_hex, hash_to_bytes, DEFAULT_ALGORITHMS from swh.model.identifiers import directory_identifier from swh.loader.git.from_disk import GitLoaderFromArchive from swh.storage.algos.dir_iterators import dir_iterator from swh.web import config from swh.web.browse.utils import ( get_mimetype_and_encoding_for_content, prepare_content_for_display ) from swh.web.common import service from swh.web.common.highlightjs import get_hljs_language_from_filename # Module used to initialize data that will be provided as tests input # Configuration for git loader _TEST_LOADER_CONFIG = { 'storage': { 'cls': 'memory', 'args': {} }, 'send_contents': True, 'send_directories': True, 'send_revisions': True, 'send_releases': True, 'send_snapshot': True, 'content_size_limit': 100 * 1024 * 1024, 'content_packet_size': 10, 'content_packet_size_bytes': 100 * 1024 * 1024, 'directory_packet_size': 10, 'revision_packet_size': 10, 'release_packet_size': 10, 'save_data': False, } # Base content indexer configuration _TEST_INDEXER_BASE_CONFIG = { 'storage': { 'cls': 'memory', 'args': {}, }, 'objstorage': { 'cls': 'memory', 'args': {}, }, 'indexer_storage': { 'cls': 'memory', 'args': {}, } } def random_sha1(): return hash_to_hex(bytes(random.randint(0, 255) for _ in range(20))) def random_sha256(): return hash_to_hex(bytes(random.randint(0, 255) for _ in range(32))) def random_blake2s256(): return hash_to_hex(bytes(random.randint(0, 255) for _ in range(32))) def random_content(): return { 'sha1': random_sha1(), 'sha1_git': random_sha1(), 'sha256': random_sha256(), 'blake2s256': random_blake2s256(), } # MimetypeIndexer with custom configuration for tests class _MimetypeIndexer(MimetypeIndexer): def parse_config_file(self, *args, **kwargs): return { **_TEST_INDEXER_BASE_CONFIG, 'tools': { 'name': 'file', 'version': '1:5.30-1+deb9u1', 'configuration': { "type": "library", "debian-package": "python3-magic" } } } # FossologyLicenseIndexer with custom configuration for tests class _FossologyLicenseIndexer(FossologyLicenseIndexer): def parse_config_file(self, *args, **kwargs): return { **_TEST_INDEXER_BASE_CONFIG, 'workdir': '/tmp/swh/indexer.fossology.license', 'tools': { 'name': 'nomos', 'version': '3.1.0rc2-31-ga2cbb8c', 'configuration': { 'command_line': 'nomossa ', }, } } # CtagsIndexer with custom configuration for tests class _CtagsIndexer(CtagsIndexer): def parse_config_file(self, *args, **kwargs): return { **_TEST_INDEXER_BASE_CONFIG, 'workdir': '/tmp/swh/indexer.ctags', 'languages': {'c': 'c'}, 'tools': { 'name': 'universal-ctags', 'version': '~git7859817b', 'configuration': { 'command_line': '''ctags --fields=+lnz --sort=no --links=no ''' # noqa '''--output-format=json ''' }, } } # Lightweight git repositories that will be loaded to generate # input data for tests _TEST_ORIGINS = [ { 'type': 'git', 'url': 'https://github.com/wcoder/highlightjs-line-numbers.js', 'archives': ['highlightjs-line-numbers.js.zip', 'highlightjs-line-numbers.js_visit2.zip'], 'visit_date': ['Dec 1 2018, 01:00 UTC', 'Jan 20 2019, 15:00 UTC'] }, { 'type': 'git', 'url': 'https://github.com/memononen/libtess2', 'archives': ['libtess2.zip'], 'visit_date': ['May 25 2018, 01:00 UTC'] }, { 'type': 'git', 'url': 'repo_with_submodules', 'archives': ['repo_with_submodules.tgz'], 'visit_date': ['Jan 1 2019, 01:00 UTC'] } ] _contents = {} # Tests data initialization def _init_tests_data(): # Load git repositories from archives loader = GitLoaderFromArchive(config=_TEST_LOADER_CONFIG) # Get reference to the memory storage storage = loader.storage for origin in _TEST_ORIGINS: for i, archive in enumerate(origin['archives']): origin_repo_archive = \ os.path.join(os.path.dirname(__file__), 'resources/repos/%s' % archive) loader.load(origin['url'], origin_repo_archive, origin['visit_date'][i]) origin.update(storage.origin_get(origin)) # add an 'id' key if enabled contents = set() directories = set() revisions = set() releases = set() snapshots = set() content_path = {} # Get all objects loaded into the test archive for origin in _TEST_ORIGINS: snp = storage.snapshot_get_latest(origin['url']) snapshots.add(hash_to_hex(snp['id'])) for branch_name, branch_data in snp['branches'].items(): if branch_data['target_type'] == 'revision': revisions.add(branch_data['target']) elif branch_data['target_type'] == 'release': release = next(storage.release_get([branch_data['target']])) revisions.add(release['target']) releases.add(hash_to_hex(branch_data['target'])) for rev_log in storage.revision_shortlog(set(revisions)): rev_id = rev_log[0] revisions.add(rev_id) for rev in storage.revision_get(revisions): dir_id = rev['directory'] directories.add(hash_to_hex(dir_id)) for entry in dir_iterator(storage, dir_id): content_path[entry['sha1']] = '/'.join( [hash_to_hex(dir_id), entry['path'].decode('utf-8')]) if entry['type'] == 'file': contents.add(entry['sha1']) elif entry['type'] == 'dir': directories.add(hash_to_hex(entry['target'])) # Get all checksums for each content contents_metadata = storage.content_get_metadata(contents) contents = [] for content_metadata in contents_metadata: contents.append({ algo: hash_to_hex(content_metadata[algo]) for algo in DEFAULT_ALGORITHMS }) path = content_path[content_metadata['sha1']] cnt = next(storage.content_get([content_metadata['sha1']])) mimetype, encoding = get_mimetype_and_encoding_for_content(cnt['data']) content_display_data = prepare_content_for_display( cnt['data'], mimetype, path) contents[-1]['path'] = path contents[-1]['mimetype'] = mimetype contents[-1]['encoding'] = encoding contents[-1]['hljs_language'] = content_display_data['language'] contents[-1]['data'] = content_display_data['content_data'] _contents[contents[-1]['sha1']] = contents[-1] # Create indexer storage instance that will be shared by indexers idx_storage = get_indexer_storage('memory', {}) # Add the empty directory to the test archive empty_dir_id = directory_identifier({'entries': []}) empty_dir_id_bin = hash_to_bytes(empty_dir_id) storage.directory_add([{'id': empty_dir_id_bin, 'entries': []}]) # Return tests data return { 'storage': storage, 'idx_storage': idx_storage, 'origins': _TEST_ORIGINS, 'contents': contents, 'directories': list(directories), 'releases': list(releases), 'revisions': list(map(hash_to_hex, revisions)), 'snapshots': list(snapshots), 'generated_checksums': set(), } def _init_indexers(tests_data): # Instantiate content indexers that will be used in tests # and force them to use the memory storages indexers = {} for idx_name, idx_class in (('mimetype_indexer', _MimetypeIndexer), ('license_indexer', _FossologyLicenseIndexer), ('ctags_indexer', _CtagsIndexer)): idx = idx_class() idx.storage = tests_data['storage'] idx.objstorage = tests_data['storage'].objstorage idx.idx_storage = tests_data['idx_storage'] idx.register_tools(idx.config['tools']) indexers[idx_name] = idx return indexers def get_content(content_sha1): return _contents.get(content_sha1) _tests_data = None _current_tests_data = None _indexer_loggers = {} def get_tests_data(reset=False): """ Initialize tests data and return them in a dict. """ global _tests_data, _current_tests_data if _tests_data is None: _tests_data = _init_tests_data() indexers = _init_indexers(_tests_data) for (name, idx) in indexers.items(): # pytest makes the loggers use a temporary file; and deepcopy # requires serializability. So we remove them, and add them # back after the copy. _indexer_loggers[name] = idx.log del idx.log _tests_data.update(indexers) if reset or _current_tests_data is None: _current_tests_data = deepcopy(_tests_data) for (name, logger) in _indexer_loggers.items(): _current_tests_data[name].log = logger return _current_tests_data def override_storages(storage, idx_storage): """ Helper function to replace the storages from which archive data are fetched. """ swh_config = config.get_config() swh_config.update({'storage': storage}) service.storage = storage swh_config.update({'indexer_storage': idx_storage}) service.idx_storage = idx_storage # Implement some special endpoints used to provide input tests data # when executing end to end tests with cypress -_content_code_data_exts = {} -_content_code_data_filenames = {} -_content_other_data_exts = {} +_content_code_data_exts = {} # type: Dict[str, Dict[str, str]] +_content_code_data_filenames = {} # type: Dict[str, Dict[str, str]] +_content_other_data_exts = {} # type: Dict[str, Dict[str, str]] def _init_content_tests_data(data_path, data_dict, ext_key): """ Helper function to read the content of a directory, store it into a test archive and add some files metadata (sha1 and/or expected programming language) in a dict. Args: data_path (str): path to a directory relative to the tests folder of swh-web data_dict (dict): the dict that will store files metadata ext_key (bool): whether to use file extensions or filenames as dict keys """ test_contents_dir = os.path.join( os.path.dirname(__file__), data_path).encode('utf-8') directory = Directory.from_disk(path=test_contents_dir, data=True, save_path=True) objects = directory.collect() for c in objects['content'].values(): c['status'] = 'visible' sha1 = hash_to_hex(c['sha1']) if ext_key: key = c['path'].decode('utf-8').split('.')[-1] filename = 'test.' + key else: filename = c['path'].decode('utf-8').split('/')[-1] key = filename language = get_hljs_language_from_filename(filename) data_dict[key] = {'sha1': sha1, 'language': language} del c['path'] del c['perms'] storage = get_tests_data()['storage'] storage.content_add(objects['content'].values()) def _init_content_code_data_exts(): """ Fill a global dictionary which maps source file extension to a code content example. """ global _content_code_data_exts _init_content_tests_data('resources/contents/code/extensions', _content_code_data_exts, True) def _init_content_other_data_exts(): """ Fill a global dictionary which maps a file extension to a content example. """ global _content_other_data_exts _init_content_tests_data('resources/contents/other/extensions', _content_other_data_exts, True) def _init_content_code_data_filenames(): """ Fill a global dictionary which maps a filename to a content example. """ global _content_code_data_filenames _init_content_tests_data('resources/contents/code/filenames', _content_code_data_filenames, False) if config.get_config()['e2e_tests_mode']: _init_content_code_data_exts() _init_content_other_data_exts() _init_content_code_data_filenames() @api_view(['GET']) def get_content_code_data_all_exts(request): """ Endpoint implementation returning a list of all source file extensions to test for highlighting using cypress. """ return Response(sorted(_content_code_data_exts.keys()), status=200, content_type='application/json') @api_view(['GET']) def get_content_code_data_by_ext(request, ext): """ Endpoint implementation returning metadata of a code content example based on the source file extension. """ data = None status = 404 if ext in _content_code_data_exts: data = _content_code_data_exts[ext] status = 200 return Response(data, status=status, content_type='application/json') @api_view(['GET']) def get_content_other_data_by_ext(request, ext): """ Endpoint implementation returning metadata of a content example based on the file extension. """ _init_content_other_data_exts() data = None status = 404 if ext in _content_other_data_exts: data = _content_other_data_exts[ext] status = 200 return Response(data, status=status, content_type='application/json') @api_view(['GET']) def get_content_code_data_all_filenames(request): """ Endpoint implementation returning a list of all source filenames to test for highlighting using cypress. """ return Response(sorted(_content_code_data_filenames.keys()), status=200, content_type='application/json') @api_view(['GET']) def get_content_code_data_by_filename(request, filename): """ Endpoint implementation returning metadata of a code content example based on the source filename. """ data = None status = 404 if filename in _content_code_data_filenames: data = _content_code_data_filenames[filename] status = 200 return Response(data, status=status, content_type='application/json') diff --git a/tox.ini b/tox.ini index a2aec2e1..0c360be4 100644 --- a/tox.ini +++ b/tox.ini @@ -1,26 +1,35 @@ [tox] -envlist=flake8,py3 +envlist=flake8,mypy,py3 [testenv:py3] deps = .[testing] pytest-cov pytest-django commands = pytest --hypothesis-profile=swh-web-fast --cov {envsitepackagesdir}/swh/web --cov-branch {posargs} {envsitepackagesdir}/swh/web [testenv:py3-slow] deps = .[testing] pytest-cov pytest-django commands = pytest --hypothesis-profile=swh-web --cov {envsitepackagesdir}/swh/web --cov-branch {posargs} {envsitepackagesdir}/swh/web [testenv:flake8] skip_install = true deps = flake8 commands = {envpython} -m flake8 \ --exclude=.tox,.git,__pycache__,.eggs,*.egg,node_modules + +[testenv:mypy] +setenv = DJANGO_SETTINGS_MODULE = swh.web.settings.development +skip_install = true +deps = + mypy + .[testing] +commands = + mypy swh diff --git a/version.txt b/version.txt index 151582e3..e9aa4472 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -v0.0.219-0-g076e62a5 \ No newline at end of file +v0.0.220-0-g605e0aea \ No newline at end of file