diff --git a/README-uri-scheme.md b/README-uri-scheme.md index ca439b63..3075fc75 100644 --- a/README-uri-scheme.md +++ b/README-uri-scheme.md @@ -1,481 +1,482 @@ URI scheme ========== User URLs --------- ### Context-independent browsing Context-independent URLs provide information about SWH objects (e.g., revisions, directories, contents, person, ...), independently of the contexts where they have been found (e.g., specific repositories, branches, commits, ...). The following endpoints are the same of the API case (see below), and just render the corresponding information for user consumption. Where hyperlinks are created, they always point to other context-independent user URLs: * /content/[:]/ Information on content * /content/[:]/raw/ Display the content data * /content/[:]/origin/ Display information on content with its origin information (Deactivated) * /directory// Browse directory's files * /origin// Information on origin * /person// Information on person * /release// Information on release * /entity// Information on Entity with hierarchy * /revision// Browse revision * /revision//log/ Revision log from Currently, the above endpoints are mounted below the top-level /browse/ namespace. ### Context-dependent browsing Context-dependent URLs provide information about SWH objects, limited to specific contexts where the objects have been found. For example, users might want to see: - the commits that descend (i.e., are based on and hence more recent) from a given commit but only in a given repository, ignoring "forks" elsewhere - the parent directory of a given one, limited to a specific revision and starting root directory (note indeed that in the general case a given directory might be mounted in multiple places, which might vary across revisions) ### Context: a specific revision (AKA commit) * /revision// Show information about a given revision, pointing to parent revisions only (i.e., no links/info about child revisions as they cannot be limited a priori). Links to parent revisions maintains a reference to , using the /history/ URL scheme (see below). * /revision//history// Show information about revision SHA1_GIT, limited to the sub-graph rooted at . The obtained page show both parent and child revisions of , but exclude all revisions that are *not* transitively reachable (going back in time) from . Links to all revisions SHA1_GIT' reachable from SHA1_GIT are of the form /revision//history//, where SHA1_GIT_ROOT is the same as before. In the degenerate case of browsing back to the revision root, we might end up on the URL /revision//history// where SHA1_GIT_1 == SHA1_GIT_2. That URL is equivalent to /revision// and might be simplified redirecting to it. * /revision//directory/[] * /revision//history//directory/[] Starting from the revision identified as in the previous URLs, navigate the directory associated to that revision. When is absent, show the content of the root directory for the given revision. When is present, treat it as a local path starting at that root directory, resolve it, and show the content of the obtained directory. Links to *sub*-directory/files append new parts to . Links to parent directories remove trailing parts of . Note that this latter operation is well-defined, given that we are looking at a specific revision and navigation starts at the root directory. ### Context: a specific point in spacetime Instead of having to specify a (root) revision by SHA1_GIT, users might want to specify a place and a time. In SWH a "place" is an origin, with an optional branch name; a "time" is a timestamp at which some place has been observed by SWH crawlers. Wherever a revision context is expected in a path (i.e., a "/revision//" path fragment) we can put in its stead a path fragment of the form /origin/[/branch/][/ts//]. Such a fragment is resolved, internally by the SWH archive, to a SHA1_GIT as follows: - [if is absent] look for the most recent crawl of origin - [if is given] look for the most recent crawl of origin whose timestamp is <= - [if is given] look for the branch - [if is absent] look for branch "master" - return the pointed by the chosen branch The already mentioned URLs for revision contexts can therefore be alternatively specified by users as: * /revision/origin/[/branch/][/ts/]/ * /revision/origin/[/branch/][/ts/]/history// * /revision/origin/[/branch/][/ts/]/directory/[] * /revision/origin/[/branch/][/ts/]/history//directory/[] Typing: - s are given as integer identifiers, pointing into the origin table. There will be separate mechanisms for finding origins by other means (e.g., URLs, metadata, etc). Once an origin is found, it can be used by ID into the above URL schemes - names are given as per the corresponding VCS (e.g., Git) and might therefore contains characters that are either invalid in URLs, or that might make the above URL schemes ambiguous (e.g., '/'). All those characters will need to be URL-escaped. (e.g., '/' will become '%2F') - s are given in a format as liberal as possible, to uphold the principle of least surprise. At the very minimum it should be possible to enter timestamps as: - ISO 8601 timestamps (see for instance the output of `date -I`, `date -Is`) - YYYY[MM[DD[HH[MM[SS]]]]] ad-hoc format Implementation proposal: use Python dateutil's parser and be done with it https://dateutil.readthedocs.org/en/latest/parser.html . Note: that dateutil does *not* allow to use classical UNIX timestamps expressed as seconds since the epoch (i.e., `date +%s` output). We will need to single case them. The same escaping considerations given for apply. Notes: - Differently from , s are still specified as SHA1 and cannot be specified a origin/branch/ts triples. This is to preserve some URL sanity. API URLs -------- ### Endpoints The api /api/1 is partially browsable on defined endpoints (/api, /api/1). * /api/ and /api/1/ List endpoint methods as per the client's 'Accept' header request. The following routes are to be anchored at at /api/1 * /revision/: show commit information $ curl -H 'Accept: application/json' http://localhost:6543/api/1/revision/18d8be353ed3480476f032475e7c233eff7371d5 { "author_email": "robot@softwareheritage.org", "author_name": "Software Heritage", "committer_date": "Mon, 17 Jan 2000 10:23:54 GMT", "committer_date_offset": 0, "committer_email": "robot@softwareheritage.org", "committer_name": "Software Heritage", "date": "Mon, 17 Jan 2000 10:23:54 GMT", "date_offset": 0, "directory": "7834ef7e7c357ce2af928115c6c6a42b7e2a44e6", "id": "18d8be353ed3480476f032475e7c233eff7371d5", "message": "synthetic revision message", "metadata": { "original_artifact": [ { "archive_type": "tar", "name": "webbase-5.7.0.tar.gz", "sha1": "147f73f369733d088b7a6fa9c4e0273dcd3c7ccd", "sha1_git": "6a15ea8b881069adedf11feceec35588f2cfe8f1", "sha256": "401d0df797110bea805d358b85bcc1ced29549d3d73f309d36484e7edf7bb912" } ] }, "parents": [ null ], "synthetic": true, "type": "tar" } * /directory/: show directory information (including ls) curl -X GET http://localhost:6543/api/1/directory/3126f46e2f7dc752227131a2a658265e58f53e38 [ { "dir_id": "3126f46e2f7dc752227131a2a658265e58f53e38", "name": "Makefile.am", "perms": 100644, "sha1": "b0283d8126f975e7b4a4348d13b07ddebe2cf8bf", "sha1_git": "e0522786777256d57c5210219bcbe8dacdad273d", "sha256": "897f3189dcfba96281b2190325c54afc74a42e2419c053baadfadc14386935ee", "status": "visible", "target": "e0522786777256d57c5210219bcbe8dacdad273d", "type": "file" }, { "dir_id": "3126f46e2f7dc752227131a2a658265e58f53e38", "name": "Makefile.in", "perms": 100644, "sha1": "81f5757b9451811cfb3ef84612e45a973c70b4e6", "sha1_git": "3b948d966fd8e99f93670025f63a550168d57d71", "sha256": "f5acd84a40f05d997a36b8846c4872a92ee57083abb77c82e05e9763c8edb59a", "status": "visible", "target": "3b948d966fd8e99f93670025f63a550168d57d71", "type": "file" }, ... snip ... { "dir_id": "3126f46e2f7dc752227131a2a658265e58f53e38", "name": "webtools.h", "perms": 100644, "sha1": "4b4c942ddd490ec1e312074ddfac352097886c02", "sha1_git": "e6fb8969d00e23dd152df5e7fb167118eab67342", "sha256": "95ffe6c0108f6ec48ccb0c93e966b54f1494f5cc353b066644c11fa47766620f", "status": "visible", "target": "e6fb8969d00e23dd152df5e7fb167118eab67342", "type": "file" }, { "dir_id": "3126f46e2f7dc752227131a2a658265e58f53e38", "name": "ylwrap", "perms": 100644, "sha1": "9073938df9ae47d585bfdf176bfff45d06f3e13e", "sha1_git": "13fc38d75f2a47bc55e90ad5bf8d8a0184b14878", "sha256": "184eb644e51154c79b42df70c22955b818d057491f84ca0e579e4f9e48a60d7b", "status": "visible", "target": "13fc38d75f2a47bc55e90ad5bf8d8a0184b14878", "type": "file" } ] * /content/[:]: show content information - content is specified by HASH, according to HASH_ALGO, where HASH_ALGO is one of: sha1, sha1_git, sha256. This means that several different URLs (at least one per HASH_ALGO) will point to the same content - HASH_ALGO defaults to "sha1" (?) curl -X GET http://localhost:6543/api/1/content/sha1:486b486d2a4998929c68265fa85ab2326db5528a { "data": "/api/1/content/486b486d2a4998929c68265fa85ab2326db5528a/raw", "sha1": "486b486d2a4998929c68265fa85ab2326db5528a" } curl -X GET http://localhost:6543/api/1/content/sha1:4a1b6d7dd0a923ed90156c4e2f5db030095d8e08/ {"error": "Content with sha1:4a1b6d7dd0a923ed90156c4e2f5db030095d8e08 not found."} * /content/[/raw curl -H 'Accept: text/plain' http://localhost:6543/api/1/content/sha1:486b486d2a4998929c68265fa85ab2326db5528a/raw The GNU cfs-el web homepage is at @uref{http://www.gnu.org/software/cfs-el/cfs-el.html}. You can find the latest distribution of GNU cfs-el at @uref{ftp://ftp.gnu.org/gnu/} or at any of its mirrors. * /release/: show release information Sample: $ curl -X GET http://localhost:6543/api/1/release/4a1b6d7dd0a923ed90156c4e2f5db030095d8e08 { "author_name": "Software Heritage", "author_email": "robot@softwareheritage.org", "comment": "synthetic release message", "date": "Sat, 04 Mar 2000 07:50:35 GMT", "date_offset": 0, "id": "4a1b6d7dd0a923ed90156c4e2f5db030095d8e08", "name": "4.0.6", "revision": "5c7814ce9978d4e16f3858925b5cea611e500eec", "synthetic": true }% * /person/: show person information curl http://localhost:6543/api/1/person/1 { "email": "robot@softwareheritage.org", "id": 1, "name": "Software Heritage" } curl http://localhost:6543/api/1/person/2 {"error": "Person with id 2 not found."} * /origin/: show origin information Sample: $ curl -X GET http://localhost:6543/api/1/origin/1 { "id": 1, "lister": null, "project": null, "type": "ftp", "url": "rsync://ftp.gnu.org/old-gnu/solfege" }% * /browse/ TODO: rename this to something more explicit about the fact we want more information about some content Return content information up to one of its origin if the content is found. curl http://localhost:6543/api/1/browse/sha1:2e98ab73456aad8dfc6cc50d562ee1b80d201753 { "path": "republique.py", "origin_url": "file:///dev/null", "origin_type": "git", "revision": "8f8640a1c024c2ef85fa8e8d9297ea289134472d", "branch": "refs/remotes/origin/master" } * /uploadnsearch/ TODO: remove this? Post a file's content to api. Api computes the sha1 hash and checks in the storage if such sha1 exists. Json answer: {'sha1': hexadecimal sha1, 'found': true or false} Sample: $ curl -X POST -F filename=@/path/to/file http://localhost:6543/api/1/uploadnsearch { "found": false, "sha1": "e95097ad2d607b4c89c1ce7ca1fef2a1e4450558" }% * /revision//log Show all revisions (~git log) starting from . The first element is the given sha1_git. Sample: curl http://localhost:6543/api/1/revision/7026b7c1a2af56521e951c01ed20f255fa054238/log/ [ { "id": "7026b7c1a2af56521e951c01ed20f255fa054238", "parents": [], "type": "git", "committer_date": "Mon, 12 Oct 2015 11:05:53 GMT", "synthetic": false, "committer": { "email": "a3nm@a3nm.net", "name": "Antoine Amarilli" }, "message": "+1 limitation\n", "author": { "email": "a3nm@a3nm.net", "name": "Antoine Amarilli" }, "date": "Mon, 12 Oct 2015 11:05:53 GMT", "metadata": null, "directory": "a33a9acf2419b9a291e8a02302e6347dcffde5a6" }, { "id": "368a48fe15b7db2383775f97c6b247011b3f14f4", "parents": [], "type": "git", "committer_date": "Mon, 12 Oct 2015 10:57:11 GMT", "synthetic": false, "committer": { "email": "a3nm@a3nm.net", "name": "Antoine Amarilli" }, "message": "actually fix bug\n", "author": { "email": "a3nm@a3nm.net", "name": "Antoine Amarilli" }, "date": "Mon, 12 Oct 2015 10:57:11 GMT", "metadata": null, "directory": "1d5188e4991510c74d62272f0301352c5c1b850b" }, ... ] * /project/: show project information * /organization/: show organization information * /directory//path/to/file-or-dir: ditto, but for file or directory pointed by path - note: This is the same as /directory/, where is the sha1_git ID of the directory pointed by path or /content/sha1_git: (for content) ### Global behavior The api routes outputs 'application/json' as default. #### Accept header Also, you can specify the following 'Accept' header in your client query: - application/json - application/yaml - text/html The client can use specific filters and compose them as (s)he sees fit. #### Fields The client can filter the result output by field names when requesting `application/json` or `application/yaml` output. Ex: curl http://localhost:6543/api/1/stat/counters?fields=revision,release,content { "content": 133616, "revision": 1042, "release": 660 } #### JSONP When using the accept header 'application/json', the route can be enhanced by adding a `callback` parameter. This will output the result in a json function whose name is the callback parameter Ex: curl http://localhost:6543/api/1/stat/counters?callback=jsonp&fields=directory_entry_dir,revision,entity jsonp({ "directory_entry_dir": 12478, "revision": 1042, "entity": 0 }) #### Error When an error is raised, the error code response is used: - 400: user's input is not correct regarding the API -- 404: user's input is ok but we did not found what (s)he was looking forbidden +- 404: user's input is ok but we did not found what (s)he was looking for +- 503: temporary internal server error (backend is down for example) And the body of the response should be a dictionary with some more information on the error. Bad request sample: curl http://localhost:6543/api/1/revision/18d8be353ed3480476f032475e7c233eff7371d {"error": "Invalid checksum query string 18d8be353ed3480476f032475e7c233eff7371d"} curl http://localhost:6543/api/1/revision/sha1:18d8be353ed3480476f032475e7c233eff7371d {"error": "Invalid hash 18d8be353ed3480476f032475e7c233eff7371d for algorithm sha1"} Not found sample: curl http://localhost:6543/api/1/revision/sha1:18d8be353ed3480476f032475e7c233eff7371df {"error": "Revision with sha1_git sha1:18d8be353ed3480476f032475e7c233eff7371df not found."} diff --git a/swh/web/ui/tests/test_views.py b/swh/web/ui/tests/test_views.py index dfb0ef78..487f1e29 100644 --- a/swh/web/ui/tests/test_views.py +++ b/swh/web/ui/tests/test_views.py @@ -1,846 +1,863 @@ # Copyright (C) 2015 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from nose.tools import istest from swh.web.ui.tests import test_app from unittest.mock import patch from swh.web.ui.exc import BadInputExc class FileMock(): def __init__(self, filename): self.filename = filename class ViewTestCase(test_app.SWHViewTestCase): render_template = False + @patch('swh.web.ui.views.flask') + @istest + def homepage(self, mock_flask): + # given + mock_flask.flash.return_value = 'something' + + # when + rv = self.client.get('/') + + # then + self.assertEquals(rv.status_code, 200) + self.assert_template_used('home.html') + + mock_flask.flash.assert_called_once_with( + 'This Web app is still work in progress, use at your own risk', + 'warning') + @istest def info(self): # when rv = self.client.get('/about/') self.assertEquals(rv.status_code, 200) self.assert_template_used('about.html') self.assertIn(b'About', rv.data) @istest def search_default(self): # when rv = self.client.get('/search/') self.assertEquals(rv.status_code, 200) self.assertEqual(self.get_context_variable('q'), '') self.assertEqual(self.get_context_variable('messages'), []) self.assertEqual(self.get_context_variable('filename'), None) self.assertEqual(self.get_context_variable('file'), None) self.assert_template_used('upload_and_search.html') @patch('swh.web.ui.views.service') @istest def search_get_query_hash_not_found(self, mock_service): # given mock_service.lookup_hash.return_value = {'found': None} # when rv = self.client.get('/search/?q=sha1:456') self.assertEquals(rv.status_code, 200) self.assertEqual(self.get_context_variable('q'), 'sha1:456') self.assertEqual(self.get_context_variable('messages'), ['Content with hash sha1:456 not found!']) self.assertEqual(self.get_context_variable('filename'), None) self.assertEqual(self.get_context_variable('file'), None) self.assert_template_used('upload_and_search.html') mock_service.lookup_hash.assert_called_once_with('sha1:456') @patch('swh.web.ui.views.service') @istest def search_get_query_hash_bad_input(self, mock_service): # given mock_service.lookup_hash.side_effect = BadInputExc('error msg') # when rv = self.client.get('/search/?q=sha1_git:789') self.assertEquals(rv.status_code, 200) self.assertEqual(self.get_context_variable('q'), 'sha1_git:789') self.assertEqual(self.get_context_variable('messages'), ['error msg']) self.assertEqual(self.get_context_variable('filename'), None) self.assertEqual(self.get_context_variable('file'), None) self.assert_template_used('upload_and_search.html') mock_service.lookup_hash.assert_called_once_with('sha1_git:789') @patch('swh.web.ui.views.service') @istest def search_get_query_hash_found(self, mock_service): # given mock_service.lookup_hash.return_value = {'found': True} # when rv = self.client.get('/search/?q=sha1:123') self.assertEquals(rv.status_code, 200) self.assertEqual(self.get_context_variable('q'), 'sha1:123') self.assertEqual(self.get_context_variable('messages'), ['Content with hash sha1:123 found!']) self.assertEqual(self.get_context_variable('filename'), None) self.assertEqual(self.get_context_variable('file'), None) self.assert_template_used('upload_and_search.html') mock_service.lookup_hash.assert_called_once_with('sha1:123') @patch('swh.web.ui.views.service') @istest def search_post_query_hash_not_found(self, mock_service): # given mock_service.lookup_hash.return_value = {'found': None} # when rv = self.client.get('/search/?q=sha1:456') self.assertEquals(rv.status_code, 200) self.assertEqual(self.get_context_variable('q'), 'sha1:456') self.assertEqual(self.get_context_variable('messages'), ['Content with hash sha1:456 not found!']) self.assertEqual(self.get_context_variable('filename'), None) self.assertEqual(self.get_context_variable('file'), None) self.assert_template_used('upload_and_search.html') mock_service.lookup_hash.assert_called_once_with('sha1:456') @patch('swh.web.ui.views.service') @istest def search_post_query_hash_bad_input(self, mock_service): # given mock_service.lookup_hash.side_effect = BadInputExc('error msg!') # when rv = self.client.post('/search/', data=dict(q='sha1_git:987')) self.assertEquals(rv.status_code, 200) self.assertEqual(self.get_context_variable('q'), 'sha1_git:987') self.assertEqual(self.get_context_variable('messages'), ['error msg!']) self.assertEqual(self.get_context_variable('filename'), None) self.assertEqual(self.get_context_variable('file'), None) self.assert_template_used('upload_and_search.html') mock_service.lookup_hash.assert_called_once_with('sha1_git:987') @patch('swh.web.ui.views.service') @istest def search_post_query_hash_found(self, mock_service): # given mock_service.lookup_hash.return_value = {'found': True} # when rv = self.client.post('/search/', data=dict(q='sha1:321')) self.assertEquals(rv.status_code, 200) self.assertEqual(self.get_context_variable('q'), 'sha1:321') self.assertEqual(self.get_context_variable('messages'), ['Content with hash sha1:321 found!']) self.assertEqual(self.get_context_variable('filename'), None) self.assertEqual(self.get_context_variable('file'), None) self.assert_template_used('upload_and_search.html') mock_service.lookup_hash.assert_called_once_with('sha1:321') @patch('swh.web.ui.views.service') @patch('swh.web.ui.views.request') @istest def search_post_upload_and_hash_bad_input(self, mock_request, mock_service): # given mock_request.data = {} mock_request.method = 'POST' mock_request.files = dict(filename=FileMock('foobar')) mock_service.upload_and_search.side_effect = BadInputExc( 'error bad input') # when (mock_request completes the post request) rv = self.client.post('/search/') # then self.assertEquals(rv.status_code, 200) self.assertEqual(self.get_context_variable('messages'), ['error bad input']) self.assert_template_used('upload_and_search.html') mock_service.upload_and_search.called = True @patch('swh.web.ui.views.service') @patch('swh.web.ui.views.request') @istest def search_post_upload_and_hash_not_found(self, mock_request, mock_service): # given mock_request.data = {} mock_request.method = 'POST' mock_request.files = dict(filename=FileMock('foobar')) mock_service.upload_and_search.return_value = {'filename': 'foobar', 'sha1': 'blahhash', 'found': False} # when (mock_request completes the post request) rv = self.client.post('/search/') # then self.assertEquals(rv.status_code, 200) self.assertEqual(self.get_context_variable('messages'), ["File foobar with hash blahhash not found!"]) self.assertEqual(self.get_context_variable('filename'), 'foobar') self.assertEqual(self.get_context_variable('sha1'), 'blahhash') self.assert_template_used('upload_and_search.html') mock_service.upload_and_search.called = True @patch('swh.web.ui.views.service') @patch('swh.web.ui.views.request') @istest def search_post_upload_and_hash_found(self, mock_request, mock_service): # given mock_request.data = {} mock_request.method = 'POST' mock_request.files = dict(filename=FileMock('foobar')) mock_service.upload_and_search.return_value = {'filename': 'foobar', 'sha1': '123456789', 'found': True} # when (mock_request completes the post request) rv = self.client.post('/search/') # then self.assertEquals(rv.status_code, 200) self.assertEqual(self.get_context_variable('messages'), ["File foobar with hash 123456789 found!"]) self.assertEqual(self.get_context_variable('filename'), 'foobar') self.assertEqual(self.get_context_variable('sha1'), '123456789') self.assert_template_used('upload_and_search.html') mock_service.upload_and_search.called = True @patch('swh.web.ui.views.service') @istest def browse_content_detail_not_found(self, mock_service): # given mock_service.lookup_content.return_value = None # when rv = self.client.get('/browse/content/sha1:sha1-hash/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('content.html') self.assertEqual(self.get_context_variable('message'), 'Content with sha1:sha1-hash not found.') self.assertEqual(self.get_context_variable('content'), None) mock_service.lookup_content.assert_called_once_with( 'sha1:sha1-hash') @patch('swh.web.ui.views.service') @istest def browse_content_detail_bad_input(self, mock_service): # given mock_service.lookup_content.side_effect = BadInputExc('Bad input!') # when rv = self.client.get('/browse/content/sha1:sha1-hash/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('content.html') self.assertEqual(self.get_context_variable('message'), 'Bad input!') self.assertIsNone(self.get_context_variable('content')) mock_service.lookup_content.assert_called_once_with( 'sha1:sha1-hash') @patch('swh.web.ui.views.service') @istest def browse_content_detail(self, mock_service): # given stub_content = {'sha1': 'sha1_hash'} mock_service.lookup_content.return_value = stub_content # when rv = self.client.get('/browse/content/sha1:sha1-hash/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('content.html') self.assertIsNone(self.get_context_variable('message')) self.assertEqual(self.get_context_variable('content'), {'sha1': 'sha1_hash'}) mock_service.lookup_content.assert_called_once_with( 'sha1:sha1-hash') @patch('swh.web.ui.views.service') @istest def browse_content_data(self, mock_service): # given stub_content_raw = { 'sha1': 'sha1-hash', 'data': b'some-data' } mock_service.lookup_content_raw.return_value = stub_content_raw # when rv = self.client.get('/browse/content/sha1:sha1-hash/raw/') self.assertEquals(rv.status_code, 200) self.assert_template_used('content-data.html') self.assertEqual(self.get_context_variable('message'), 'Content sha1-hash') self.assertEqual(self.get_context_variable('content'), stub_content_raw) mock_service.lookup_content_raw.assert_called_once_with( 'sha1:sha1-hash') @patch('swh.web.ui.views.service') @istest def browse_content_data_not_found(self, mock_service): # given mock_service.lookup_content_raw.return_value = None # when rv = self.client.get('/browse/content/sha1:sha1-unknown/raw/') self.assertEquals(rv.status_code, 200) self.assert_template_used('content-data.html') self.assertEqual(self.get_context_variable('message'), 'Content with sha1:sha1-unknown not found.') self.assertEqual(self.get_context_variable('content'), None) mock_service.lookup_content_raw.assert_called_once_with( 'sha1:sha1-unknown') @patch('swh.web.ui.views.service') @istest def browse_content_data_invalid_hash(self, mock_service): # given mock_service.lookup_content_raw.side_effect = BadInputExc( 'Invalid hash') # when rv = self.client.get('/browse/content/sha2:sha1-invalid/raw/') self.assertEquals(rv.status_code, 200) self.assert_template_used('content-data.html') self.assertEqual(self.get_context_variable('message'), 'Invalid hash') self.assertEqual(self.get_context_variable('content'), None) mock_service.lookup_content_raw.assert_called_once_with( 'sha2:sha1-invalid') @patch('swh.web.ui.views.service') @patch('swh.web.ui.utils') @istest def browse_directory_bad_input(self, mock_utils, mock_service): # given mock_service.lookup_directory.side_effect = BadInputExc('Invalid hash') # when rv = self.client.get('/browse/directory/sha2-invalid/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('directory.html') self.assertEqual(self.get_context_variable('message'), 'Invalid hash') self.assertEqual(self.get_context_variable('files'), []) mock_service.lookup_directory.assert_called_once_with( 'sha2-invalid') @patch('swh.web.ui.views.service') @patch('swh.web.ui.utils') @istest def browse_directory_empty_result(self, mock_utils, mock_service): # given mock_service.lookup_directory.return_value = None # when rv = self.client.get('/browse/directory/some-sha1/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('directory.html') self.assertEqual(self.get_context_variable('message'), 'Directory some-sha1 not found.') self.assertEqual(self.get_context_variable('files'), []) mock_service.lookup_directory.assert_called_once_with( 'some-sha1') @patch('swh.web.ui.views.service') @patch('swh.web.ui.views.utils') @istest def browse_directory(self, mock_utils, mock_service): # given stub_directory_ls = [ {'type': 'dir', 'target': '123', 'name': 'some-dir-name'}, {'type': 'file', 'sha1': '654', 'name': 'some-filename'}, {'type': 'dir', 'target': '987', 'name': 'some-other-dirname'} ] mock_service.lookup_directory.return_value = stub_directory_ls stub_directory_map = [ {'link': '/path/to/url/dir/123', 'name': 'some-dir-name'}, {'link': '/path/to/url/file/654', 'name': 'some-filename'}, {'link': '/path/to/url/dir/987', 'name': 'some-other-dirname'} ] mock_utils.prepare_directory_listing.return_value = stub_directory_map # when rv = self.client.get('/browse/directory/some-sha1/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('directory.html') self.assertEqual(self.get_context_variable('message'), 'Listing for directory some-sha1:') self.assertEqual(self.get_context_variable('files'), stub_directory_map) mock_service.lookup_directory.assert_called_once_with( 'some-sha1') mock_utils.prepare_directory_listing.assert_called_once_with( stub_directory_ls) @patch('swh.web.ui.views.service') # @istest def browse_content_with_origin_content_not_found(self, mock_service): # given mock_service.lookup_hash.return_value = {'found': False} # when rv = self.client.get('/browse/content/sha256:some-sha256/origin/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('content-with-origin.html') self.assertEqual(self.get_context_variable('message'), 'Hash sha256:some-sha256 was not found.') mock_service.lookup_hash.assert_called_once_with( 'sha256:some-sha256') mock_service.lookup_hash_origin.called = False @patch('swh.web.ui.views.service') # @istest def browse_content_with_origin_bad_input(self, mock_service): # given mock_service.lookup_hash.side_effect = BadInputExc('Invalid hash') # when rv = self.client.get('/browse/content/sha256:some-sha256/origin/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('content-with-origin.html') self.assertEqual( self.get_context_variable('message'), 'Invalid hash') mock_service.lookup_hash.assert_called_once_with( 'sha256:some-sha256') mock_service.lookup_hash_origin.called = False @patch('swh.web.ui.views.service') # @istest def browse_content_with_origin(self, mock_service): # given mock_service.lookup_hash.return_value = {'found': True} mock_service.lookup_hash_origin.return_value = { 'origin_type': 'ftp', 'origin_url': '/some/url', 'revision': 'revision-hash', 'branch': 'master', 'path': '/path/to', } # when rv = self.client.get('/browse/content/sha256:some-sha256/origin/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('content-with-origin.html') self.assertEqual( self.get_context_variable('message'), "The content with hash sha256:some-sha256 has been seen on " + "origin with type 'ftp'\n" + "at url '/some/url'. The revision was identified at " + "'revision-hash' on branch 'master'.\n" + "The file's path referenced was '/path/to'.") mock_service.lookup_hash.assert_called_once_with( 'sha256:some-sha256') mock_service.lookup_hash_origin.assert_called_once_with( 'sha256:some-sha256') @patch('swh.web.ui.views.service') @istest def browse_origin_not_found(self, mock_service): # given mock_service.lookup_origin.return_value = None # when rv = self.client.get('/browse/origin/1/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('origin.html') self.assertEqual(self.get_context_variable('origin_id'), 1) self.assertEqual( self.get_context_variable('message'), 'Origin 1 not found!') mock_service.lookup_origin.assert_called_once_with(1) @patch('swh.web.ui.views.service') @istest def browse_origin_found(self, mock_service): # given mock_origin = {'type': 'git', 'lister': None, 'project': None, 'url': 'rsync://some/url', 'id': 426} mock_service.lookup_origin.return_value = mock_origin # when rv = self.client.get('/browse/origin/426/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('origin.html') self.assertEqual(self.get_context_variable('origin_id'), 426) self.assertEqual(self.get_context_variable('origin'), mock_origin) mock_service.lookup_origin.assert_called_once_with(426) @patch('swh.web.ui.views.service') @istest def browse_origin_bad_input(self, mock_service): # given mock_service.lookup_origin.side_effect = BadInputExc('wrong input') # when rv = self.client.get('/browse/origin/426/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('origin.html') self.assertEqual(self.get_context_variable('origin_id'), 426) mock_service.lookup_origin.assert_called_once_with(426) @patch('swh.web.ui.views.service') @istest def browse_person_not_found(self, mock_service): # given mock_service.lookup_person.return_value = None # when rv = self.client.get('/browse/person/1/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('person.html') self.assertEqual(self.get_context_variable('person_id'), 1) self.assertEqual( self.get_context_variable('message'), 'Person 1 not found!') mock_service.lookup_person.assert_called_once_with(1) @patch('swh.web.ui.views.service') @istest def browse_person_found(self, mock_service): # given mock_person = {'type': 'git', 'lister': None, 'project': None, 'url': 'rsync://some/url', 'id': 426} mock_service.lookup_person.return_value = mock_person # when rv = self.client.get('/browse/person/426/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('person.html') self.assertEqual(self.get_context_variable('person_id'), 426) self.assertEqual(self.get_context_variable('person'), mock_person) mock_service.lookup_person.assert_called_once_with(426) @patch('swh.web.ui.views.service') @istest def browse_person_bad_input(self, mock_service): # given mock_service.lookup_person.side_effect = BadInputExc('wrong input') # when rv = self.client.get('/browse/person/426/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('person.html') self.assertEqual(self.get_context_variable('person_id'), 426) mock_service.lookup_person.assert_called_once_with(426) @patch('swh.web.ui.views.service') @istest def browse_release_not_found(self, mock_service): # given mock_service.lookup_release.return_value = None # when rv = self.client.get('/browse/release/1/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('release.html') self.assertEqual(self.get_context_variable('sha1_git'), '1') self.assertEqual( self.get_context_variable('message'), 'Release 1 not found!') mock_service.lookup_release.assert_called_once_with('1') @patch('swh.web.ui.views.service') @istest def browse_release_bad_input(self, mock_service): # given mock_service.lookup_release.side_effect = BadInputExc('wrong input') # when rv = self.client.get('/browse/release/426/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('release.html') self.assertEqual(self.get_context_variable('sha1_git'), '426') mock_service.lookup_release.assert_called_once_with('426') @patch('swh.web.ui.views.service') @istest def browse_release(self, mock_service): # given mock_release = { "date": "Sun, 05 Jul 2015 18:02:06 GMT", "id": "1e951912027ea6873da6985b91e50c47f645ae1a", "target": "d770e558e21961ad6cfdf0ff7df0eb5d7d4f0754", "synthetic": False, "target_type": "revision", "author": { "email": "torvalds@linux-foundation.org", "name": "Linus Torvalds" }, "message": "Linux 4.2-rc1\n", "name": "v4.2-rc1" } mock_service.lookup_release.return_value = mock_release expected_release = { "date": "Sun, 05 Jul 2015 18:02:06 GMT", "id": "1e951912027ea6873da6985b91e50c47f645ae1a", "target": '/browse/revision/d770e558e21961ad6cfdf0ff7df0' 'eb5d7d4f0754/', "synthetic": False, "target_type": "revision", "author": "Linus Torvalds ", "message": "Linux 4.2-rc1\n", "name": "v4.2-rc1" } # when rv = self.client.get('/browse/release/426/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('release.html') self.assertEqual(self.get_context_variable('sha1_git'), '426') self.assertEqual(self.get_context_variable('release'), expected_release) self.assertEqual(self.get_context_variable('keys'), [ 'id', 'name', 'date', 'message', 'author', 'target', 'target_type']) mock_service.lookup_release.assert_called_once_with('426') @patch('swh.web.ui.views.service') @istest def browse_revision_not_found(self, mock_service): # given mock_service.lookup_revision.return_value = None # when rv = self.client.get('/browse/revision/1/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('revision.html') self.assertEqual(self.get_context_variable('sha1_git'), '1') self.assertEqual( self.get_context_variable('message'), 'Revision 1 not found!') mock_service.lookup_revision.assert_called_once_with('1') @patch('swh.web.ui.views.service') @istest def browse_revision_bad_input(self, mock_service): # given mock_service.lookup_revision.side_effect = BadInputExc('wrong input') # when rv = self.client.get('/browse/revision/426/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('revision.html') self.assertEqual(self.get_context_variable('sha1_git'), '426') mock_service.lookup_revision.assert_called_once_with('426') @patch('swh.web.ui.views.service') @istest def browse_revision_found(self, mock_service): # given mock_revision = { 'id': 'd770e558e21961ad6cfdf0ff7df0eb5d7d4f0754', 'date': 'Sun, 05 Jul 2015 18:01:52 GMT', 'committer': { 'email': 'torvalds@linux-foundation.org', 'name': 'Linus Torvalds' }, 'committer_date': 'Sun, 05 Jul 2015 18:01:52 GMT', 'metadata': None, 'type': 'git', 'author': { 'email': 'torvalds@linux-foundation.org', 'name': 'Linus Torvalds' }, 'message': 'Linux 4.2-rc1\n', 'synthetic': False, 'directory': '2a1dbabeed4dcf1f4a4c441993b2ffc9d972780b', 'parents': [ 'a585d2b738bfa26326b3f1f40f0f1eda0c067ccf' ], } mock_service.lookup_revision.return_value = mock_revision expected_revision = { 'id': 'd770e558e21961ad6cfdf0ff7df0eb5d7d4f0754', 'date': 'Sun, 05 Jul 2015 18:01:52 GMT', 'committer': 'Linus Torvalds ', 'committer_date': 'Sun, 05 Jul 2015 18:01:52 GMT', 'type': 'git', 'author': 'Linus Torvalds ', 'message': 'Linux 4.2-rc1\n', 'synthetic': False, 'metadata': None, 'parents': [ '/browse/revision/a585d2b738bfa26326b3f1f40f0f1eda0c067ccf/' ], 'directory': '/browse/directory/2a1dbabeed4dcf1f4a4c441993b2f' 'fc9d972780b/', } # when rv = self.client.get('/browse/revision/426/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('revision.html') self.assertEqual(self.get_context_variable('sha1_git'), '426') self.assertEqual(self.get_context_variable('revision'), expected_revision) self.assertEqual(self.get_context_variable('keys'), ['id', 'message', 'date', 'author', 'committer', 'committer_date', 'synthetic']) mock_service.lookup_revision.assert_called_once_with('426') @patch('swh.web.ui.views.service') @istest def browse_entity_not_found(self, mock_service): # given mock_service.lookup_entity_by_uuid.return_value = [] # when rv = self.client.get('/browse/entity/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('entity.html') self.assertEqual(self.get_context_variable('entities'), []) self.assertEqual( self.get_context_variable('message'), "Entity '5f4d4c51-498a-4e28-88b3-b3e4e8396cba' not found!") mock_service.lookup_entity_by_uuid.assert_called_once_with( '5f4d4c51-498a-4e28-88b3-b3e4e8396cba') @patch('swh.web.ui.views.service') @istest def browse_entity_bad_input(self, mock_service): # given mock_service.lookup_entity_by_uuid.side_effect = BadInputExc( 'wrong input') # when rv = self.client.get('/browse/entity/blah-blah-uuid/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('entity.html') self.assertEqual(self.get_context_variable('entities'), []) mock_service.lookup_entity_by_uuid.assert_called_once_with( 'blah-blah-uuid') @patch('swh.web.ui.views.service') @istest def browse_entity(self, mock_service): # given stub_entities = [ {'id': '5f4d4c51-5a9b-4e28-88b3-b3e4e8396cba'}] mock_service.lookup_entity_by_uuid.return_value = stub_entities # when rv = self.client.get('/browse/entity/' '5f4d4c51-5a9b-4e28-88b3-b3e4e8396cba/') # then self.assertEquals(rv.status_code, 200) self.assert_template_used('entity.html') self.assertEqual(self.get_context_variable('entities'), stub_entities) self.assertIsNone(self.get_context_variable('message')) mock_service.lookup_entity_by_uuid.assert_called_once_with( '5f4d4c51-5a9b-4e28-88b3-b3e4e8396cba') diff --git a/swh/web/ui/views.py b/swh/web/ui/views.py index 67348249..46bc47b6 100644 --- a/swh/web/ui/views.py +++ b/swh/web/ui/views.py @@ -1,411 +1,411 @@ # Copyright (C) 2015 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information +import flask -from flask import render_template, flash, request, url_for +from flask import render_template, request, url_for from flask.ext.api.decorators import set_renderers from flask.ext.api.renderers import HTMLRenderer from swh.core.hashutil import ALGORITHMS from swh.web.ui import service, utils from swh.web.ui.exc import BadInputExc from swh.web.ui.main import app hash_filter_keys = ALGORITHMS @app.route('/') @set_renderers(HTMLRenderer) def homepage(): """Home page """ - flash('This Web app is still work in progress, use at your own risk', - 'warning') - # return redirect(url_for('about')) + flask.flash('This Web app is still work in progress, use at your own risk', + 'warning') return render_template('home.html') @app.route('/about/') @set_renderers(HTMLRenderer) def about(): return render_template('about.html') @app.route('/search/', methods=['GET', 'POST']) @set_renderers(HTMLRenderer) def search(): """Search for hashes in swh-storage. One form to submit either: - hash query to look up in swh storage - some file content to upload, compute its hash and look it up in swh storage - both Returns: dict representing data to look for in swh storage. The following keys are returned: - file: File submitted for upload - filename: Filename submitted for upload - q: Query on hash to look for - message: Message detailing if data has been found or not. """ env = {'filename': None, 'q': None, 'file': None} data = None q = env['q'] file = env['file'] if request.method == 'GET': data = request.args elif request.method == 'POST': data = request.data # or hash and search a file file = request.files.get('filename') # could either be a query for sha1 hash q = data.get('q') messages = [] if q: env['q'] = q try: r = service.lookup_hash(q) messages.append('Content with hash %s%sfound!' % ( q, ' ' if r.get('found') else ' not ')) except BadInputExc as e: messages.append(str(e)) if file and file.filename: env['file'] = file try: uploaded_content = service.upload_and_search(file) filename = uploaded_content['filename'] sha1 = uploaded_content['sha1'] found = uploaded_content['found'] messages.append('File %s with hash %s%sfound!' % ( filename, sha1, ' ' if found else ' not ')) env.update({ 'filename': filename, 'sha1': sha1, }) except BadInputExc as e: messages.append(str(e)) env['q'] = q if q else '' env['messages'] = messages return render_template('upload_and_search.html', **env) def _origin_seen(q, data): """Given an origin, compute a message string with the right information. Args: origin: a dictionary with keys: - origin: a dictionary with type and url keys - occurrence: a dictionary with a validity range Returns: Message as a string """ origin_type = data['origin_type'] origin_url = data['origin_url'] revision = data['revision'] branch = data['branch'] path = data['path'] return """The content with hash %s has been seen on origin with type '%s' at url '%s'. The revision was identified at '%s' on branch '%s'. The file's path referenced was '%s'.""" % (q, origin_type, origin_url, revision, branch, path) @app.route('/browse/content/') @app.route('/browse/content//') @set_renderers(HTMLRenderer) def browse_content_detail(q='5d448a06f02d9de748b6b0b9620cba1bed8480da'): """Given a hash and a checksum, display the content's meta-data. Args: q is of the form algo_hash:hash with algo_hash in (sha1, sha1_git, sha256) Returns: Information on one possible origin for such content. Raises: BadInputExc in case of unknown algo_hash or bad hash NotFoundExc if the content is not found. """ env = {} message = None try: content = service.lookup_content(q) if not content: message = 'Content with %s not found.' % q except BadInputExc as e: message = str(e) content = None env['message'] = message env['content'] = content return render_template('content.html', **env) @app.route('/browse/content//raw/') @set_renderers(HTMLRenderer) def browse_content_data(q): """Given a hash and a checksum, display the content's raw data. Args: q is of the form algo_hash:hash with algo_hash in (sha1, sha1_git, sha256) Returns: Information on one possible origin for such content. Raises: BadInputExc in case of unknown algo_hash or bad hash NotFoundExc if the content is not found. """ env = {} try: content = service.lookup_content_raw(q) if content: # FIXME: will break if not utf-8 content['data'] = content['data'].decode('utf-8') message = 'Content %s' % content['sha1'] else: message = 'Content with %s not found.' % q except BadInputExc as e: message = str(e) content = None env['message'] = message env['content'] = content return render_template('content-data.html', **env) # @app.route('/browse/content//origin/') @set_renderers(HTMLRenderer) def browse_content_with_origin( q='sha1:4320781056e5a735a39de0b8c229aea224590052'): """Show content information. Args: - q: query string of the form with `algo_hash` in sha1, sha1_git, sha256. This means that several different URLs (at least one per HASH_ALGO) will point to the same content sha: the sha with 'hash' format Returns: The content's information at for a given checksum. """ env = {'q': q} try: content = service.lookup_hash(q) if not content.get('found'): message = "Hash %s was not found." % q else: origin = service.lookup_hash_origin(q) message = _origin_seen(q, origin) except BadInputExc as e: # do not like it but do not duplicate code message = str(e) env['message'] = message return render_template('content-with-origin.html', **env) @app.route('/browse/directory/') @app.route('/browse/directory//') @set_renderers(HTMLRenderer) def browse_directory(sha1_git='dcf3289b576b1c8697f2a2d46909d36104208ba3'): """Show directory information. Args: - sha1_git: the directory's sha1 git identifier. Returns: The content's information at sha1_git """ env = {'sha1_git': sha1_git} try: directory_files = service.lookup_directory(sha1_git) if directory_files: message = "Listing for directory %s:" % sha1_git files = utils.prepare_directory_listing(directory_files) else: message = "Directory %s not found." % sha1_git files = [] except BadInputExc as e: # do not like it but do not duplicate code message = str(e) files = [] env['message'] = message env['files'] = files return render_template('directory.html', **env) @app.route('/browse/origin/') @app.route('/browse/origin//') @set_renderers(HTMLRenderer) def browse_origin(origin_id=1): """Browse origin with id id. """ env = {'origin_id': origin_id, 'origin': None} try: ori = service.lookup_origin(origin_id) if ori: env.update({'origin': ori}) else: env.update({'message': 'Origin %s not found!' % origin_id}) except BadInputExc as e: env.update({'message': str(e)}) return render_template('origin.html', **env) @app.route('/browse/person/') @app.route('/browse/person//') @set_renderers(HTMLRenderer) def browse_person(person_id=1): """Browse person with id id. """ env = {'person_id': person_id, 'person': None} try: ori = service.lookup_person(person_id) if ori: env.update({'person': ori}) else: env.update({'message': 'Person %s not found!' % person_id}) except BadInputExc as e: env.update({'message': str(e)}) return render_template('person.html', **env) @app.route('/browse/release/') @app.route('/browse/release//') @set_renderers(HTMLRenderer) def browse_release(sha1_git='1e951912027ea6873da6985b91e50c47f645ae1a'): """Browse release with sha1_git. """ env = {'sha1_git': sha1_git, 'release': None} try: rel = service.lookup_release(sha1_git) if rel: author = rel.get('author') if author: rel['author'] = utils.person_to_string(author) target_type = rel.get('target_type') if target_type == 'revision': rel['target'] = url_for('browse_revision', sha1_git=rel['target']) env.update({'release': rel, 'keys': ['id', 'name', 'date', 'message', 'author', 'target', 'target_type']}) else: env.update({'message': 'Release %s not found!' % sha1_git}) except BadInputExc as e: env.update({'message': str(e)}) return render_template('release.html', **env) @app.route('/browse/revision/') @app.route('/browse/revision//') @set_renderers(HTMLRenderer) def browse_revision(sha1_git='d770e558e21961ad6cfdf0ff7df0eb5d7d4f0754'): """Browse revision with sha1_git. """ env = {'sha1_git': sha1_git, 'revision': None} try: rev = service.lookup_revision(sha1_git) if rev: author = rev.get('author') if author: rev['author'] = utils.person_to_string(author) committer = rev.get('committer') if committer: rev['committer'] = utils.person_to_string(committer) parent_links = [] for parent in rev.get('parents', []): parent_links.append(url_for('browse_revision', sha1_git=parent)) rev['parents'] = parent_links directory = rev.get('directory') if directory: rev['directory'] = url_for('browse_directory', sha1_git=rev['directory']) env.update({'revision': rev, 'keys': ['id', 'message', 'date', 'author', 'committer', 'committer_date', 'synthetic']}) else: env.update({'message': 'Revision %s not found!' % sha1_git}) except BadInputExc as e: env.update({'message': str(e)}) return render_template('revision.html', **env) @app.route('/browse/entity/') @app.route('/browse/entity//') @set_renderers(HTMLRenderer) def browse_entity(uuid='5f4d4c51-498a-4e28-88b3-b3e4e8396cba'): env = {'entities': [], 'message': None} entities = env['entities'] try: entities = service.lookup_entity_by_uuid(uuid) if not entities: env['message'] = "Entity '%s' not found!" % uuid except BadInputExc as e: env.update({'message': str(e)}) env['entities'] = entities return render_template('entity.html', **env)