diff --git a/debian/control b/debian/control index a6653888..fa3f1f75 100644 --- a/debian/control +++ b/debian/control @@ -1,46 +1,46 @@ Source: swh-web Maintainer: Software Heritage developers Section: python Priority: optional Build-Depends: curl, debhelper (>= 9), dh-python (>= 2), python3-all, python3-bs4, python3-django (>= 1.10.7~), python3-djangorestframework (>= 3.4.0~), python3-django-webpack-loader, python3-django-js-reverse, python3-docutils, python3-htmlmin, python3-magic (>= 0.3.0~), python3-lxml, python3-pytest, python3-pytest-django, python3-pygments, python3-pypandoc, python3-requests, python3-setuptools, python3-sphinx, python3-sphinxcontrib.httpdomain, python3-yaml, python3-swh.core (>= 0.0.40~), python3-swh.model (>= 0.0.25~), - python3-swh.storage (>= 0.0.107~), + python3-swh.storage (>= 0.0.109~), python3-swh.indexer.storage (>= 0.0.52~), python3-swh.vault (>= 0.0.20~), python3-swh.scheduler (>= 0.0.31~) Standards-Version: 3.9.6 Homepage: https://forge.softwareheritage.org/diffusion/DWUI/ Package: python3-swh.web Architecture: all Depends: python3-swh.core (>= 0.0.40~), python3-swh.model (>= 0.0.25~), - python3-swh.storage (>= 0.0.107~), + python3-swh.storage (>= 0.0.109~), python3-swh.indexer.storage (>= 0.0.52~), python3-swh.vault (>= 0.0.20~), python3-swh.scheduler (>= 0.0.31~), ${misc:Depends}, ${python3:Depends} Description: Software Heritage Web Applications diff --git a/docs/uri-scheme-browse-origin.rst b/docs/uri-scheme-browse-origin.rst index 7948172d..6001405b 100644 --- a/docs/uri-scheme-browse-origin.rst +++ b/docs/uri-scheme-browse-origin.rst @@ -1,489 +1,489 @@ Origin ^^^^^^ This describes the URI scheme when one wants to browse the Software Heritage archive in the context of an origin (for instance, a repository crawled from GitHub or a Debian source package). All the views pointed by that scheme offer quick links to browse objects as found during the associated crawls performed by Software Heritage: * the root directory of the origin * the list of branches of the origin * the list of releases of the origin Origin visits """"""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visits/ HTML view that displays a visits reporting for a SWH origin identified by its type and url. :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :statuscode 200: no error :statuscode 404: requested origin can not be found in the SWH archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/visits/` :swh_web_browse:`origin/git/url/https://github.com/python/cpython/visits/` :swh_web_browse:`origin/deb://Debian-Security/packages/mediawiki/visits/` :swh_web_browse:`origin/https://gitorious.org/qt/qtbase.git/visits/` Origin directory """""""""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/directory/[(path)/] HTML view for browsing the content of a directory reachable from the root directory (including itself) associated to the latest full visit of a SWH origin. The content of the directory is first sorted in lexicographical order and the sub-directories are displayed before the regular files. The view enables to navigate from the requested directory to directories reachable from it in a recursive way but also up to the origin root directory. A breadcrumb located in the top part of the view allows to keep track of the paths navigated so far. The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the directory content can also be specified by using the branch query parameter. :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string path: optional parameter used to specify the path of a directory reachable from the origin root one :query string branch: specify the origin branch name from which to retrieve the root directory :query string release: specify the origin release name from which to retrieve the root directory :query string revision: specify the origin revision, identified by the hexadecimal representation of its *sha1_git* value, from which to retrieve the root directory :query int visit_id: specify a SWH visit id to retrieve the directory from instead of using the latest full visit by default :statuscode 200: no error :statuscode 404: requested origin can not be found in the SWH archive or the provided path does not exist from the origin root directory **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/directory/` :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/directory/net/ethernet/` :swh_web_browse:`origin/https://github.com/python/cpython/directory/` :swh_web_browse:`origin/https://github.com/python/cpython/directory/Python/` :swh_web_browse:`origin/https://github.com/python/cpython/directory/?branch=refs/heads/2.7` .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/directory/[(path)/] HTML view for browsing the content of a directory reachable from the root directory (including itself) associated to a visit of a SWH origin closest to a provided timestamp. The content of the directory is first sorted in lexicographical order and the sub-directories are displayed before the regular files. The view enables to navigate from the requested directory to directories reachable from it in a recursive way but also up to the origin root directory. A breadcrumb located in the top part of the view allows to keep track of the paths navigated so far. The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the directory content can also be specified by using the branch query parameter. :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string timestamp: a date string (any format parsable by `dateutil.parser.parse`_) or Unix timestamp to parse in order to find the closest SWH visit. :param path: optional parameter used to specify the path of a directory reachable from the origin root one :type path: string :query string branch: specify the origin branch name from which to retrieve the root directory :query string release: specify the origin release name from which to retrieve the root directory :query string revision: specify the origin revision, identified by the hexadecimal representation of its *sha1_git* value, from which to retrieve the directory :query int visit_id: specify a SWH visit id to retrieve the directory from instead of using the provided timestamp :statuscode 200: no error :statuscode 404: requested origin can not be found in the SWH archive, requested visit timestamp does not exist or the provided path does not exist from the origin root directory **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/visit/1493926809/directory/` :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/visit/2016-09-14T10:36:21/directory/net/ethernet/` :swh_web_browse:`origin/git/url/https://github.com/python/cpython/visit/1474620651/directory/` :swh_web_browse:`origin/git/url/https://github.com/python/cpython/visit/2017-05-05/directory/Python/` :swh_web_browse:`origin/git/url/https://github.com/python/cpython/visit/2015-08/directory/?branch=refs/heads/2.7` Origin content """""""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/content/(path)/ HTML view that produces a display of a SWH content associated to the latest full visit of a SWH origin. If the content to display is textual, it will be highlighted client-side if possible using highlightjs_. The procedure to perform that task is described in :http:get:`/browse/content/[(algo_hash):](hash)/`. It is also possible to highlight specific lines of a textual content (not in terms of syntax highlighting but to emphasize some relevant content part) by either: * clicking on line numbers (holding shift to highlight a lines range) * using an url fragment in the form '#Ln' or '#Lm-Ln' The view displays a breadcrumb on top of the rendered content in order to easily navigate up to the origin root directory. The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the content can also be specified by using the branch query parameter. :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string path: path of a content reachable from the origin root directory :query string branch: specify the origin branch name from which to retrieve the content :query string release: specify the origin release name from which to retrieve the content :query string revision: specify the origin revision, identified by the hexadecimal representation of its *sha1_git* value, from which to retrieve the content :query int visit_id: specify a SWH visit id to retrieve the content from instead of using the latest full visit by default :statuscode 200: no error :statuscode 404: requested origin can not be found in the SWH archive, or the provided content path does not exist from the origin root directory **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/git/git/content/git.c/` :swh_web_browse:`origin/git/url/https://github.com/git/git/content/git.c/` :swh_web_browse:`origin/https://github.com/mozilla/gecko-dev/content/js/src/json.cpp/` :swh_web_browse:`origin/https://github.com/git/git/content/git.c/?branch=refs/heads/next` .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/content/(path)/ HTML view that produces a display of a SWH content associated to a visit of a SWH origin closest to a provided timestamp. If the content to display is textual, it will be highlighted client-side if possible using highlightjs_. The procedure to perform that task is described in :http:get:`/browse/content/[(algo_hash):](hash)/`. It is also possible to highlight specific lines of a textual content (not in terms of syntax highlighting but to emphasize some relevant content part) by either: * clicking on line numbers (holding shift to highlight a lines range) * using an url fragment in the form '#Ln' or '#Lm-Ln' The view displays a breadcrumb on top of the rendered content in order to easily navigate up to the origin root directory. The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the content can also be specified by using the branch query parameter. :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string timestamp: a date string (any format parsable by `dateutil.parser.parse`_) or Unix timestamp to parse in order to find the closest SWH visit. :param string path: path of a content reachable from the origin root directory :query string branch: specify the origin branch name from which to retrieve the content :query string release: specify the origin release name from which to retrieve the content :query string revision: specify the origin revision, identified by the hexadecimal representation of its *sha1_git* value, from which to retrieve the content :query int visit_id: specify a SWH visit id to retrieve the content from instead of using the provided timestamp :statuscode 200: no error :statuscode 404: requested origin can not be found in the SWH archive, requested visit timestamp does not exist or the provided content path does not exist from the origin root directory **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/git/git/visit/1473933564/content/git.c/` :swh_web_browse:`origin/git/url/https://github.com/git/git/visit/2016-05-05T00:0:00+00:00/content/git.c/` :swh_web_browse:`origin/https://github.com/mozilla/gecko-dev/visit/1490126182/content/js/src/json.cpp/` :swh_web_browse:`origin/https://github.com/mozilla/gecko-dev/visit/2017-03-21/content/js/src/json.cpp/#L904-L931` :swh_web_browse:`origin/https://github.com/git/git/visit/2017-09-15/content/git.c/?branch=refs/heads/next` Origin history """""""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/log/ HTML view that produces a display of revisions history heading to the last revision found during the latest visit of a SWH origin. In other words, it shows the commit log associated to the latest full visit of a SWH origin. The following data are displayed for each log entry: + * link to browse the associated revision in the origin context * author of the revision - * link to the revision metadata - * message associated the revision * date of the revision - * link to browse the associated source tree in the origin context + * message associated the revision + * commit date of the revision - N log entries are displayed per page (default is 20). In order to navigate + By default, the revisions are ordered in reverse chronological order of + their commit date. + + N log entries are displayed per page (default is 100). In order to navigate in a large history, two buttons are present at the bottom of the view: * *Newer*: fetch and display if available the N more recent log entries than the ones currently displayed * *Older*: fetch and display if available the N older log entries than the ones currently displayed The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the content can also be specified by using the branch query parameter. :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) - :query string revs_breadcrumb: used internally to store - the navigation breadcrumbs (i.e. the list of descendant revisions - visited so far). It must be a string in the form - "(rev_1)[/(rev_2)/.../(rev_n)]" where rev_i corresponds to a - revision *sha1_git*. :query int per_page: the number of log entries to display per page - (default is 20, max is 50) + :query int offset: the number of revisions to skip before returning those to display + :query str revs_ordering: specify the revisions ordering, possible values are *committer_date*, + *dfs*, *dfs_post* and *bfs* :query string branch: specify the origin branch name from which to retrieve the commit log :query string release: specify the origin release name from which to retrieve the commit log :query string revision: specify the origin revision, identified by the hexadecimal representation of its *sha1_git* value, from which to retrieve the commit log :query int visit_id: specify a SWH visit id to retrieve the history log from instead of using the latest visit by default :statuscode 200: no error :statuscode 404: requested origin can not be found in the SWH archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/videolan/vlc/log/` :swh_web_browse:`origin/https://github.com/Kitware/CMake/log/` :swh_web_browse:`origin/https://github.com/Kitware/CMake/log/?branch=refs/heads/release` .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/log/ HTML view that produces a display of revisions history heading to the last revision found during a visit of a SWH origin closest to the provided timestamp. In other words, it shows the commit log associated to a visit of a SWH origin closest to a provided timestamp. The following data are displayed for each log entry: * author of the revision * link to the revision metadata * message associated the revision * date of the revision * link to browse the associated source tree in the origin context N log entries are displayed per page (default is 20). In order to navigate in a large history, two buttons are present at the bottom of the view: * *Newer*: fetch and display if available the N more recent log entries than the ones currently displayed * *Older*: fetch and display if available the N older log entries than the ones currently displayed The view also enables to easily switch between the origin branches and releases through a dropdown menu. The origin branch (default to master) from which to retrieve the content can also be specified by using the branch query parameter. :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string timestamp: a date string (any format parsable by `dateutil.parser.parse`_) or Unix timestamp to parse in order to find the closest SWH visit. :query string revs_breadcrumb: used internally to store the navigation breadcrumbs (i.e. the list of descendant revisions visited so far). It must be a string in the form "(rev_1)[/(rev_2)/.../(rev_n)]" where rev_i corresponds to a revision *sha1_git*. :query int per_page: the number of log entries to display per page (default is 20, max is 50) :query string branch: specify the origin branch name from which to retrieve the commit log :query string release: specify the origin release name from which to retrieve the commit log :query string revision: specify the origin revision, identified by the hexadecimal representation of its *sha1_git* value, from which to retrieve the commit log :query int visit_id: specify a SWH visit id to retrieve the history log from instead of using the provided timestamp :statuscode 200: no error :statuscode 404: requested origin can not be found in the SWH archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/videolan/vlc/visit/1459651262/log/` :swh_web_browse:`origin/git/url/https://github.com/Kitware/CMake/visit/2016-04-01/log/` :swh_web_browse:`origin/https://github.com/Kitware/CMake/visit/1438116814/log/?branch=refs/heads/release` :swh_web_browse:`origin/https://github.com/Kitware/CMake/visit/2017-05-05T03:14:23/log/?branch=refs/heads/release` Origin branches """"""""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/branches/ HTML view that produces a display of the list of branches found during the latest full visit of a SWH origin. The following data are displayed for each branch: * its name * a link to browse the associated directory * a link to browse the associated revision * last commit message * last commit date That list of branches is paginated, each page displaying a maximum of 100 branches. :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :statuscode 200: no error :statuscode 404: requested origin can not be found in the SWH archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/deb/url/deb://Debian/packages/linux/branches/` :swh_web_browse:`origin/https://github.com/webpack/webpack/branches/` .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/branches/ HTML view that produces a display of the list of branches found during a visit of a SWH origin closest to the provided timestamp. The following data are displayed for each branch: * its name * a link to browse the associated directory * a link to browse the associated revision * last commit message * last commit date That list of branches is paginated, each page displaying a maximum of 100 branches. :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string timestamp: a date string (any format parsable by `dateutil.parser.parse`_) or Unix timestamp to parse in order to find the closest SWH visit. :statuscode 200: no error :statuscode 404: requested origin can not be found in the SWH archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/kripken/emscripten/visit/2017-05-05T12:02:03/branches/` :swh_web_browse:`origin/deb://Debian/packages/apache2-mod-xforward/visit/2017-11-15T05:15:09/branches/` Origin releases """"""""""""""" .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/releases/ HTML view that produces a display of the list of releases found during the latest full visit of a SWH origin. The following data are displayed for each release: * its name * a link to browse the release details * its target type (revision, directory, content or release) * its associated message * its date That list of releases is paginated, each page displaying a maximum of 100 releases. :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :statuscode 200: no error :statuscode 404: requested origin can not be found in the SWH archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/git/git/releases/` :swh_web_browse:`origin/https://github.com/webpack/webpack/releases/` .. http:get:: /browse/origin/[(origin_type)/url/](origin_url)/visit/(timestamp)/releases/ HTML view that produces a display of the list of releases found during a visit of a SWH origin closest to the provided timestamp. The following data are displayed for each release: * its name * a link to browse the release details * its target type (revision, directory, content or release) * its associated message * its date That list of releases is paginated, each page displaying a maximum of 100 releases. :param string origin_type: the type of the SWH origin (*git*, *svn*, *deb* ...) :param string origin_url: the url of the origin (e.g. https://github.com/(user)/(repo)/) :param string timestamp: a date string (any format parsable by `dateutil.parser.parse`_) or Unix timestamp to parse in order to find the closest SWH visit. :statuscode 200: no error :statuscode 404: requested origin can not be found in the SWH archive **Examples:** .. parsed-literal:: :swh_web_browse:`origin/git/url/https://github.com/torvalds/linux/visit/2017-11-21T19:37:42/releases/` :swh_web_browse:`origin/https://github.com/Kitware/CMake/visit/2016-09-23T14:06:35/releases/` .. _highlightjs: https://highlightjs.org/ .. _dateutil.parser.parse: http://dateutil.readthedocs.io/en/stable/parser.html diff --git a/docs/uri-scheme-browse-revision.rst b/docs/uri-scheme-browse-revision.rst index 823a5947..88c3215e 100644 --- a/docs/uri-scheme-browse-revision.rst +++ b/docs/uri-scheme-browse-revision.rst @@ -1,81 +1,81 @@ Revision ^^^^^^^^ .. http:get:: /browse/revision/(sha1_git)/ HTML view to browse a SWH revision. It notably shows the revision date and message but also offers links to get more details on: * its author * its parent revisions * the history log reachable from it The view also enables to navigate in the source tree associated to the revision and browse its content. Last but not least, the view displays the list of file changes introduced in the revision but also the diffs of each changed files. :param string sha1_git: hexadecimal representation for the *sha1_git* identifier of a SWH revision :query string origin_type: used internally to associate a SWH origin type (*git*, *svn*, *deb* ...) to the revision :query string origin_url: used internally to associate an origin url (e.g. https://github.com/user/repo) to the revision :query string timestamp: used internally to associate an origin visit to the revision, must be a date string (any format parsable by `dateutil.parser.parse`_) or Unix timestamp to parse in order to find the closest SWH visit. :query int visit_id: used internally to specify a SWH visit id instead of using the provided timestamp :query string path: used internally when navigating in the source tree associated to the revision :statuscode 200: no error :statuscode 404: requested revision can not be found in the SWH archive **Examples:** .. parsed-literal:: :swh_web_browse:`revision/f1b94134a4b879bc55c3dacdb496690c8ebdc03f/` :swh_web_browse:`revision/d1aa2b3f607b35dc5dbf613b2334b6d243ec2bda/` .. _dateutil.parser.parse: http://dateutil.readthedocs.io/en/stable/parser.html .. http:get:: /browse/revision/(sha1_git)/log/ HTML view that displays the list of revisions heading to a given one. In other words, it shows a commit log. The following data are displayed for each log entry: - * author of the revision * link to browse the revision - * message associated to the revision + * author of the revision * date of the revision - * link to browse the associated source tree + * message associated to the revision + * commit date of the revision + + By default, the revisions are ordered in reverse chronological order of + their commit date. - N log entries are displayed per page (default is 20). In order to navigate + N log entries are displayed per page (default is 100). In order to navigate in a large history, two buttons are present at the bottom of the view: * *Newer*: fetch and display if available the N more recent log entries than the ones currently displayed * *Older*: fetch and display if available the N older log entries than the ones currently displayed :param string sha1_git: hexadecimal representation for the *sha1_git* identifier of a SWH revision - :query string revs_breadcrumb: used internally to store - the navigation breadcrumbs (i.e. the list of descendant revisions - visited so far). It must be a string in the form - "[//.../]" where rev_i corresponds to a - revision sha1_git. :query int per_page: the number of log entries to display per page - (default is 20, max is 50) + :query int offset: the number of revisions to skip before returning those to display + :query str revs_ordering: specify the revisions ordering, possible values are *committer_date*, + *dfs*, *dfs_post* and *bfs* :statuscode 200: no error :statuscode 404: requested revision can not be found in the SWH archive **Examples:** .. parsed-literal:: :swh_web_browse:`revision/f1b94134a4b879bc55c3dacdb496690c8ebdc03f/log/` :swh_web_browse:`revision/d1aa2b3f607b35dc5dbf613b2334b6d243ec2bda/log/` diff --git a/requirements-swh.txt b/requirements-swh.txt index 9f97f76c..b0e2fd13 100644 --- a/requirements-swh.txt +++ b/requirements-swh.txt @@ -1,6 +1,6 @@ swh.core >= 0.0.40 swh.model >= 0.0.25 -swh.storage >= 0.0.107 +swh.storage >= 0.0.109 swh.vault >= 0.0.20 swh.indexer >= 0.0.52 swh.scheduler >= 0.0.31 diff --git a/requirements.txt b/requirements.txt index a2a031b4..0a6e87aa 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,26 +1,25 @@ # Add here external Python modules dependencies, one per line. Module names # should match https://pypi.python.org/pypi names. For the full spec or # dependency lines, see https://pip.readthedocs.org/en/1.1/requirements.html # Runtime dependencies beautifulsoup4 Django >= 1.10.7, < 2.0 djangorestframework >= 3.4.0 django_webpack_loader django_js_reverse docutils file_magic >= 0.3.0 htmlmin lxml pygments pypandoc python-dateutil pyyaml requests -#Doc dependencies +# Doc dependencies sphinx sphinxcontrib-httpdomain -# Test dependencies diff --git a/swh/web/assets/src/bundles/browse/browse.css b/swh/web/assets/src/bundles/browse/browse.css index affcc1b0..7210eb91 100644 --- a/swh/web/assets/src/bundles/browse/browse.css +++ b/swh/web/assets/src/bundles/browse/browse.css @@ -1,119 +1,119 @@ /** * Copyright (C) 2018 The Software Heritage developers * See the AUTHORS file at the top-level directory of this distribution * License: GNU Affero General Public License version 3, or any later version * See top-level LICENSE file for more information */ .swh-browse-nav li a { border-radius: 4px; } .scrollable-menu { max-height: 180px; overflow-x: hidden; } .swh-corner-ribbon { width: 200px; background: #fecd1b; color: #e20026; position: absolute; text-align: center; line-height: 50px; letter-spacing: 1px; box-shadow: 0 0 3px rgba(0, 0, 0, 0.3); top: 55px; right: -50px; left: auto; transform: rotate(45deg); z-index: 2000; } .swh-loading { display: none; text-align: center; margin-top: 10px; } .swh-loading.show { display: block; } .swh-metadata-table-row { border-top: 1px solid #ddd !important; } .swh-metadata-table-key { min-width: 200px; max-width: 200px; width: 200px; } .swh-metadata-table-value pre { white-space: pre-wrap; } .swh-table-cell-text-overflow { white-space: nowrap; overflow: hidden; text-overflow: ellipsis; } .swh-directory-table { margin-bottom: 0; } .swh-directory-table td { border-top: 1px solid #ddd !important; } .swh-title-color { color: #e20026; } .swh-log-entry-message { - min-width: 460px; - max-width: 460px; - width: 460px; + min-width: 440px; + max-width: 440px; + width: 440px; } .swh-popover { max-height: 50vh; max-width: 80vw; overflow-y: auto; overflow-x: auto; padding: 0; padding-right: 1.4em; } .swh-search-pagination { margin-top: 5px; } .ui-slideouttab-panel { z-index: 30000; } #swh-identifiers { width: 70vw; top: 0; border: 1px solid #e20026; } #swh-identifiers .handle { background-color: #e20026; border: 1px solid #e20026; color: white; padding-top: 0.1em; padding-bottom: 0.1em; } #swh-identifiers-content { height: 100%; overflow: auto; } .swh-empty-snapshot { white-space: pre-line; } diff --git a/swh/web/assets/src/bundles/revision/index.js b/swh/web/assets/src/bundles/revision/index.js index 7f6c1ba7..119a8b95 100644 --- a/swh/web/assets/src/bundles/revision/index.js +++ b/swh/web/assets/src/bundles/revision/index.js @@ -1,11 +1,12 @@ /** * Copyright (C) 2018 The Software Heritage developers * See the AUTHORS file at the top-level directory of this distribution * License: GNU Affero General Public License version 3, or any later version * See top-level LICENSE file for more information */ // bundle for browse/revision view import './revision.css'; export * from './diff-utils'; +export * from './log-utils'; diff --git a/swh/web/assets/src/bundles/revision/log-utils.js b/swh/web/assets/src/bundles/revision/log-utils.js new file mode 100644 index 00000000..ef81a764 --- /dev/null +++ b/swh/web/assets/src/bundles/revision/log-utils.js @@ -0,0 +1,26 @@ +export function revsOrderingTypeClicked(event) { + let urlParams = new URLSearchParams(window.location.search); + let orderingType = $(event.target).val(); + if (orderingType) { + urlParams.set('revs_ordering', $(event.target).val()); + } else if (urlParams.has('revs_ordering')) { + urlParams.delete('revs_ordering'); + } + window.location.search = urlParams.toString(); +} + +export function initRevisionsLog() { + $(document).ready(() => { + let urlParams = new URLSearchParams(window.location.search); + let revsOrderingType = urlParams.get('revs_ordering'); + if (revsOrderingType) { + $(`:input[value="${revsOrderingType}"]`).prop('checked', true); + } + + $('tr.swh-revision-log-entry').on('click', function() { + window.location = $(this).data('href'); + return false; + }); + $('td > a').on('click', e => e.stopPropagation()); + }); +} diff --git a/swh/web/assets/src/bundles/revision/revision.css b/swh/web/assets/src/bundles/revision/revision.css index ee729ea6..c4ec5c7b 100644 --- a/swh/web/assets/src/bundles/revision/revision.css +++ b/swh/web/assets/src/bundles/revision/revision.css @@ -1,50 +1,68 @@ /** * Copyright (C) 2018 The Software Heritage developers * See the AUTHORS file at the top-level directory of this distribution * License: GNU Affero General Public License version 3, or any later version * See top-level LICENSE file for more information */ .swh-diff-lines-info { background-color: rgba(0, 0, 255, 0.1) !important; } .swh-diff-added-line { background-color: rgba(0, 255, 0, 0.1) !important; } .swh-diff-removed-line { background-color: rgba(255, 0, 0, 0.1) !important; } span.no-nl-marker { position: relative; color: #cb2431; vertical-align: middle; } span.no-nl-marker svg { vertical-align: text-bottom; } span.no-nl-marker svg path { fill: currentColor; } +tr.swh-revision-log-entry { + cursor: pointer; +} + +tr.swh-revision-log-entry:hover td { + background: #fecd1b; +} + +tr.swh-revision-log-entry td a { + text-decoration: none; +} + .swh-revision-log-entry-id { min-width: 110px; max-width: 110px; width: 110px; } .swh-revision-log-entry-author { min-width: 160px; max-width: 160px; width: 160px; } .swh-revision-log-entry-date { - min-width: 230px; - max-width: 230px; - width: 230px; + min-width: 200px; + max-width: 200px; + width: 200px; +} + +.swh-revision-log-entry-commit-date { + min-width: 200px; + max-width: 200px; + width: 200px; } diff --git a/swh/web/browse/utils.py b/swh/web/browse/utils.py index 0047d4c2..ef0a1ee9 100644 --- a/swh/web/browse/utils.py +++ b/swh/web/browse/utils.py @@ -1,1197 +1,1173 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import base64 from collections import defaultdict import magic import math import pypandoc import stat +import textwrap from django.core.cache import cache from django.utils.safestring import mark_safe from importlib import reload from swh.model.identifiers import persistent_identifier from swh.web.common import highlightjs, service from swh.web.common.exc import NotFoundExc, http_status_code_message from swh.web.common.utils import ( reverse, format_utc_iso_date, parse_timestamp, get_origin_visits, get_swh_persistent_id, swh_object_icons ) from swh.web.config import get_config def get_directory_entries(sha1_git): """Function that retrieves the content of a SWH directory from the SWH archive. The directories entries are first sorted in lexicographical order. Sub-directories and regular files are then extracted. Args: sha1_git: sha1_git identifier of the directory Returns: A tuple whose first member corresponds to the sub-directories list and second member the regular files list Raises: NotFoundExc if the directory is not found """ cache_entry_id = 'directory_entries_%s' % sha1_git cache_entry = cache.get(cache_entry_id) if cache_entry: return cache_entry entries = list(service.lookup_directory(sha1_git)) for e in entries: e['perms'] = stat.filemode(e['perms']) if e['type'] == 'rev': # modify dir entry name to explicitly show it points # to a revision e['name'] = '%s @ %s' % (e['name'], e['target'][:7]) dirs = [e for e in entries if e['type'] in ('dir', 'rev')] files = [e for e in entries if e['type'] == 'file'] dirs = sorted(dirs, key=lambda d: d['name']) files = sorted(files, key=lambda f: f['name']) cache.set(cache_entry_id, (dirs, files)) return dirs, files def get_mimetype_and_encoding_for_content(content): """Function that returns the mime type and the encoding associated to a content buffer using the magic module under the hood. Args: content (bytes): a content buffer Returns: A tuple (mimetype, encoding), for instance ('text/plain', 'us-ascii'), associated to the provided content. """ while True: try: magic_result = magic.detect_from_content(content) mime_type = magic_result.mime_type encoding = magic_result.encoding break except Exception: # workaround an issue with the magic module who can fail # if detect_from_content is called multiple times in # a short amount of time reload(magic) return mime_type, encoding # maximum authorized content size in bytes for HTML display # with code highlighting content_display_max_size = get_config()['content_display_max_size'] snapshot_content_max_size = get_config()['snapshot_content_max_size'] def request_content(query_string, max_size=content_display_max_size, raise_if_unavailable=True, reencode=True): """Function that retrieves a SWH content from the SWH archive. Raw bytes content is first retrieved, then the content mime type. If the mime type is not stored in the archive, it will be computed using Python magic module. Args: query_string: a string of the form "[ALGO_HASH:]HASH" where optional ALGO_HASH can be either *sha1*, *sha1_git*, *sha256*, or *blake2s256* (default to *sha1*) and HASH the hexadecimal representation of the hash value max_size: the maximum size for a content to retrieve (default to 1MB, no size limit if None) Returns: A tuple whose first member corresponds to the content raw bytes and second member the content mime type Raises: NotFoundExc if the content is not found """ content_data = service.lookup_content(query_string) filetype = None language = None license = None # requests to the indexer db may fail so properly handle # those cases in order to avoid content display errors try: filetype = service.lookup_content_filetype(query_string) language = service.lookup_content_language(query_string) license = service.lookup_content_license(query_string) except Exception: pass mimetype = 'unknown' encoding = 'unknown' if filetype: mimetype = filetype['mimetype'] encoding = filetype['encoding'] content_data['error_code'] = 200 content_data['error_message'] = '' content_data['error_description'] = '' if not max_size or content_data['length'] < max_size: try: content_raw = service.lookup_content_raw(query_string) except Exception as e: if raise_if_unavailable: raise e else: content_data['raw_data'] = None content_data['error_code'] = 404 content_data['error_description'] = \ 'The bytes of the content are currently not available in the archive.' # noqa content_data['error_message'] = \ http_status_code_message[content_data['error_code']] else: content_data['raw_data'] = content_raw['data'] if not filetype: mimetype, encoding = \ get_mimetype_and_encoding_for_content(content_data['raw_data']) # noqa # encode textual content to utf-8 if needed if reencode and mimetype.startswith('text/'): # probably a malformed UTF-8 content, re-encode it # by replacing invalid chars with a substitution one if encoding == 'unknown-8bit': content_data['raw_data'] = \ content_data['raw_data'].decode('utf-8', 'replace')\ .encode('utf-8') elif 'ascii' not in encoding and encoding not in ['utf-8', 'binary']: # noqa content_data['raw_data'] = \ content_data['raw_data'].decode(encoding, 'replace')\ .encode('utf-8') elif reencode and mimetype.startswith('application/octet-stream'): # file may detect a text content as binary # so try to decode it for display encodings = ['us-ascii'] encodings += ['iso-8859-%s' % i for i in range(1, 17)] for encoding in encodings: try: content_data['raw_data'] = \ content_data['raw_data'].decode(encoding)\ .encode('utf-8') except Exception: pass else: # ensure display in content view mimetype = 'text/plain' break else: content_data['raw_data'] = None content_data['mimetype'] = mimetype content_data['encoding'] = encoding if language: content_data['language'] = language['lang'] else: content_data['language'] = 'not detected' if license: content_data['licenses'] = ', '.join(license['facts'][0]['licenses']) else: content_data['licenses'] = 'not detected' return content_data _browsers_supported_image_mimes = set(['image/gif', 'image/png', 'image/jpeg', 'image/bmp', 'image/webp', 'image/svg', 'image/svg+xml']) def prepare_content_for_display(content_data, mime_type, path): """Function that prepares a content for HTML display. The function tries to associate a programming language to a content in order to perform syntax highlighting client-side using highlightjs. The language is determined using either the content filename or its mime type. If the mime type corresponds to an image format supported by web browsers, the content will be encoded in base64 for displaying the image. Args: content_data (bytes): raw bytes of the content mime_type (string): mime type of the content path (string): path of the content including filename Returns: A dict containing the content bytes (possibly different from the one provided as parameter if it is an image) under the key 'content_data and the corresponding highlightjs language class under the key 'language'. """ language = highlightjs.get_hljs_language_from_filename(path) if not language: language = highlightjs.get_hljs_language_from_mime_type(mime_type) if not language: language = 'nohighlight' elif mime_type.startswith('application/'): mime_type = mime_type.replace('application/', 'text/') if mime_type.startswith('image/'): if mime_type in _browsers_supported_image_mimes: content_data = base64.b64encode(content_data) else: content_data = None if mime_type.startswith('image/svg'): mime_type = 'image/svg+xml' return {'content_data': content_data, 'language': language, 'mimetype': mime_type} def get_origin_visit(origin_info, visit_ts=None, visit_id=None, snapshot_id=None): """Function that returns information about a SWH visit for a given origin. The visit is retrieved from a provided timestamp. The closest visit from that timestamp is selected. Args: origin_info (dict): a dict filled with origin information (id, url, type) visit_ts (int or str): an ISO date string or Unix timestamp to parse Returns: A dict containing the visit info as described below:: {'origin': 2, 'date': '2017-10-08T11:54:25.582463+00:00', 'metadata': {}, 'visit': 25, 'status': 'full'} """ visits = get_origin_visits(origin_info) if not visits: raise NotFoundExc('No SWH visit associated to origin with' ' type %s and url %s!' % (origin_info['type'], origin_info['url'])) if snapshot_id: visit = [v for v in visits if v['snapshot'] == snapshot_id] if len(visit) == 0: raise NotFoundExc( 'Visit for snapshot with id %s for origin with type %s' ' and url %s not found!' % (snapshot_id, origin_info['type'], origin_info['url'])) return visit[0] if visit_id: visit = [v for v in visits if v['visit'] == int(visit_id)] if len(visit) == 0: raise NotFoundExc( 'Visit with id %s for origin with type %s' ' and url %s not found!' % (visit_id, origin_info['type'], origin_info['url'])) return visit[0] if not visit_ts: # returns the latest full visit when no timestamp is provided for v in reversed(visits): if v['status'] == 'full': return v return visits[-1] parsed_visit_ts = math.floor(parse_timestamp(visit_ts).timestamp()) visit_idx = None for i, visit in enumerate(visits): ts = math.floor(parse_timestamp(visit['date']).timestamp()) if i == 0 and parsed_visit_ts <= ts: return visit elif i == len(visits) - 1: if parsed_visit_ts >= ts: return visit else: next_ts = math.floor( parse_timestamp(visits[i+1]['date']).timestamp()) if parsed_visit_ts >= ts and parsed_visit_ts < next_ts: if (parsed_visit_ts - ts) < (next_ts - parsed_visit_ts): visit_idx = i break else: visit_idx = i+1 break if visit_idx is not None: visit = visits[visit_idx] while visit_idx < len(visits) - 1 and \ visit['date'] == visits[visit_idx+1]['date']: visit_idx = visit_idx + 1 visit = visits[visit_idx] return visit else: raise NotFoundExc( 'Visit with timestamp %s for origin with type %s and url %s not found!' % # noqa (visit_ts, origin_info['type'], origin_info['url'])) def process_snapshot_branches(snapshot_branches): """ Process a dictionary describing snapshot branches: extract those targeting revisions and releases, put them in two different lists, then sort those lists in lexicographical order of the branches' names. Args: snapshot_branches (dict): A dict describing the branches of a snapshot as returned for instance by :func:`swh.web.common.service.lookup_snapshot` Returns: tuple: A tuple whose first member is the sorted list of branches targeting revisions and second member the sorted list of branches targeting releases """ # noqa branches = {} releases = {} revision_to_branch = defaultdict(set) revision_to_release = defaultdict(set) release_to_branch = defaultdict(set) for branch_name, target in snapshot_branches.items(): if not target: # FIXME: display branches with an unknown target anyway continue target_id = target['target'] target_type = target['target_type'] if target_type == 'revision': branches[branch_name] = { 'name': branch_name, 'revision': target_id, } revision_to_branch[target_id].add(branch_name) elif target_type == 'release': release_to_branch[target_id].add(branch_name) # FIXME: handle pointers to other object types # FIXME: handle branch aliases releases_info = service.lookup_release_multiple( release_to_branch.keys() ) for release in releases_info: branches_to_update = release_to_branch[release['id']] for branch in branches_to_update: releases[branch] = { 'name': release['name'], 'branch_name': branch, 'date': format_utc_iso_date(release['date']), 'id': release['id'], 'message': release['message'], 'target_type': release['target_type'], 'target': release['target'], } if release['target_type'] == 'revision': revision_to_release[release['target']].update( branches_to_update ) revisions = service.lookup_revision_multiple( set(revision_to_branch.keys()) | set(revision_to_release.keys()) ) for revision in revisions: if not revision: continue revision_data = { 'directory': revision['directory'], 'date': format_utc_iso_date(revision['date']), 'message': revision['message'], } for branch in revision_to_branch[revision['id']]: branches[branch].update(revision_data) for release in revision_to_release[revision['id']]: releases[release]['directory'] = revision['directory'] ret_branches = list(sorted(branches.values(), key=lambda b: b['name'])) ret_releases = list(sorted(releases.values(), key=lambda b: b['name'])) return ret_branches, ret_releases def get_snapshot_content(snapshot_id): """Returns the lists of branches and releases associated to a swh snapshot. That list is put in cache in order to speedup the navigation in the swh-web/browse ui. .. warning:: At most 1000 branches contained in the snapshot will be returned for performance reasons. Args: snapshot_id (str): hexadecimal representation of the snapshot identifier Returns: A tuple with two members. The first one is a list of dict describing the snapshot branches. The second one is a list of dict describing the snapshot releases. Raises: NotFoundExc if the snapshot does not exist """ cache_entry_id = 'swh_snapshot_%s' % snapshot_id cache_entry = cache.get(cache_entry_id) if cache_entry: return cache_entry['branches'], cache_entry['releases'] branches = [] releases = [] if snapshot_id: snapshot = service.lookup_snapshot( snapshot_id, branches_count=snapshot_content_max_size) branches, releases = process_snapshot_branches(snapshot['branches']) cache.set(cache_entry_id, { 'branches': branches, 'releases': releases, }) return branches, releases def get_origin_visit_snapshot(origin_info, visit_ts=None, visit_id=None, snapshot_id=None): """Returns the lists of branches and releases associated to a swh origin for a given visit. The visit is expressed by a timestamp. In the latter case, the closest visit from the provided timestamp will be used. If no visit parameter is provided, it returns the list of branches found for the latest visit. That list is put in cache in order to speedup the navigation in the swh-web/browse ui. .. warning:: At most 1000 branches contained in the snapshot will be returned for performance reasons. Args: origin_info (dict): a dict filled with origin information (id, url, type) visit_ts (int or str): an ISO date string or Unix timestamp to parse visit_id (int): optional visit id for disambiguation in case several visits have the same timestamp Returns: A tuple with two members. The first one is a list of dict describing the origin branches for the given visit. The second one is a list of dict describing the origin releases for the given visit. Raises: NotFoundExc if the origin or its visit are not found """ visit_info = get_origin_visit(origin_info, visit_ts, visit_id, snapshot_id) return get_snapshot_content(visit_info['snapshot']) def gen_link(url, link_text=None, link_attrs={}): """ Utility function for generating an HTML link to insert in Django templates. Args: url (str): an url link_text (str): optional text for the produced link, if not provided the url will be used link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ attrs = ' ' for k, v in link_attrs.items(): attrs += '%s="%s" ' % (k, v) if not link_text: link_text = url link = '%s' % (attrs, url, link_text) return mark_safe(link) def gen_person_link(person_id, person_name, snapshot_context=None, link_attrs={}): """ Utility function for generating a link to a SWH person HTML view to insert in Django templates. Args: person_id (int): a SWH person id person_name (str): the associated person name link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'person_name' """ query_params = None if snapshot_context and snapshot_context['origin_info']: origin_info = snapshot_context['origin_info'] query_params = {'origin_type': origin_info['type'], 'origin': origin_info['url']} if 'timestamp' in snapshot_context['url_args']: query_params['timestamp'] = \ snapshot_context['url_args']['timestamp'] if 'visit_id' in snapshot_context['query_params']: query_params['visit_id'] = \ snapshot_context['query_params']['visit_id'] elif snapshot_context: query_params = {'snapshot_id': snapshot_context['snapshot_id']} person_url = reverse('browse-person', url_args={'person_id': person_id}, query_params=query_params) return gen_link(person_url, person_name or 'None', link_attrs) def gen_revision_url(revision_id, snapshot_context=None): """ Utility function for generating an url to a SWH revision. Args: revision_id (str): a SWH revision id snapshot_context (dict): if provided, generate snapshot-dependent browsing url Returns: str: The url to browse the revision """ query_params = None if snapshot_context and snapshot_context['origin_info']: origin_info = snapshot_context['origin_info'] origin_type = snapshot_context['origin_type'] query_params = {'origin_type': origin_type, 'origin': origin_info['url']} if 'timestamp' in snapshot_context['url_args']: query_params['timestamp'] = \ snapshot_context['url_args']['timestamp'] if 'visit_id' in snapshot_context['query_params']: query_params['visit_id'] = \ snapshot_context['query_params']['visit_id'] elif snapshot_context: query_params = {'snapshot_id': snapshot_context['snapshot_id']} return reverse('browse-revision', url_args={'sha1_git': revision_id}, query_params=query_params) def gen_revision_link(revision_id, shorten_id=False, snapshot_context=None, link_text=None, link_attrs={}): """ Utility function for generating a link to a SWH revision HTML view to insert in Django templates. Args: revision_id (str): a SWH revision id shorten_id (boolean): whether to shorten the revision id to 7 characters for the link text snapshot_context (dict): if provided, generate snapshot-dependent browsing link link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: str: An HTML link in the form 'revision_id' """ if not revision_id: return None revision_url = gen_revision_url(revision_id, snapshot_context) if shorten_id: return gen_link(revision_url, revision_id[:7], link_attrs) else: if not link_text: link_text = revision_id return gen_link(revision_url, link_text, link_attrs) def gen_origin_link(origin_info, link_attrs={}): """ Utility function for generating a link to a SWH origin HTML view to insert in Django templates. Args: origin_info (dict): a dict filled with origin information (id, type, url) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'Origin: origin_url' """ # noqa origin_browse_url = reverse('browse-origin', url_args={'origin_type': origin_info['type'], 'origin_url': origin_info['url']}) return gen_link(origin_browse_url, 'Origin: ' + origin_info['url'], link_attrs) def gen_directory_link(sha1_git, link_text=None, link_attrs={}): """ Utility function for generating a link to a SWH directory HTML view to insert in Django templates. Args: sha1_git (str): directory identifier link_text (str): optional text for the generated link (the generated url will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ if not sha1_git: return None directory_url = reverse('browse-directory', url_args={'sha1_git': sha1_git}) if not link_text: link_text = directory_url return gen_link(directory_url, link_text, link_attrs) def gen_snapshot_link(snapshot_id, link_text=None, link_attrs={}): """ Utility function for generating a link to a SWH snapshot HTML view to insert in Django templates. Args: snapshot_id (str): snapshot identifier link_text (str): optional text for the generated link (the generated url will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ snapshot_url = reverse('browse-snapshot', url_args={'snapshot_id': snapshot_id}) if not link_text: link_text = snapshot_url return gen_link(snapshot_url, link_text, link_attrs) def gen_snapshot_directory_link(snapshot_context, revision_id=None, link_text=None, link_attrs={}): """ Utility function for generating a link to a SWH directory HTML view in the context of a snapshot to insert in Django templates. Args: snapshot_context (dict): the snapshot information revision_id (str): optional revision identifier in order to use the associated directory link_text (str): optional text to use for the generated link link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'origin_directory_view_url' """ query_params = {'revision': revision_id} if snapshot_context['origin_info']: origin_info = snapshot_context['origin_info'] - url_args = {'origin_type': origin_info['type'], - 'origin_url': origin_info['url']} + url_args = {'origin_url': origin_info['url']} if 'timestamp' in snapshot_context['url_args']: url_args['timestamp'] = \ snapshot_context['url_args']['timestamp'] if 'visit_id' in snapshot_context['query_params']: query_params['visit_id'] = \ snapshot_context['query_params']['visit_id'] directory_url = reverse('browse-origin-directory', url_args=url_args, query_params=query_params) else: url_args = {'snapshot_id': snapshot_context['snapshot_id']} directory_url = reverse('browse-snapshot-directory', url_args=url_args, query_params=query_params) if not link_text: link_text = directory_url return gen_link(directory_url, link_text, link_attrs) def gen_content_link(sha1_git, link_text=None, link_attrs={}): """ Utility function for generating a link to a SWH content HTML view to insert in Django templates. Args: sha1_git (str): content identifier link_text (str): optional text for the generated link (the generated url will be used by default) link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ if not sha1_git: return None content_url = reverse('browse-content', url_args={'query_string': 'sha1_git:' + sha1_git}) if not link_text: link_text = content_url return gen_link(content_url, link_text, link_attrs) def get_revision_log_url(revision_id, snapshot_context=None): """ Utility function for getting the URL for a SWH revision log HTML view (possibly in the context of an origin). Args: revision_id (str): revision identifier the history heads to snapshot_context (dict): if provided, generate snapshot-dependent browsing link Returns: The SWH revision log view URL """ query_params = {'revision': revision_id} if snapshot_context and snapshot_context['origin_info']: origin_info = snapshot_context['origin_info'] - url_args = {'origin_type': origin_info['type'], - 'origin_url': origin_info['url']} + url_args = {'origin_url': origin_info['url']} if 'timestamp' in snapshot_context['url_args']: url_args['timestamp'] = \ snapshot_context['url_args']['timestamp'] if 'visit_id' in snapshot_context['query_params']: query_params['visit_id'] = \ snapshot_context['query_params']['visit_id'] revision_log_url = reverse('browse-origin-log', url_args=url_args, query_params=query_params) elif snapshot_context: url_args = {'snapshot_id': snapshot_context['snapshot_id']} revision_log_url = reverse('browse-snapshot-log', url_args=url_args, query_params=query_params) else: revision_log_url = reverse('browse-revision-log', url_args={'sha1_git': revision_id}) return revision_log_url def gen_revision_log_link(revision_id, snapshot_context=None, link_text=None, link_attrs={}): """ Utility function for generating a link to a SWH revision log HTML view (possibly in the context of an origin) to insert in Django templates. Args: revision_id (str): revision identifier the history heads to snapshot_context (dict): if provided, generate snapshot-dependent browsing link link_text (str): optional text to use for the generated link link_attrs (dict): optional attributes (e.g. class) to add to the link Returns: An HTML link in the form 'link_text' """ if not revision_id: return None revision_log_url = get_revision_log_url(revision_id, snapshot_context) if not link_text: link_text = revision_log_url return gen_link(revision_log_url, link_text, link_attrs) -def _format_log_entries(revision_log, per_page, snapshot_context=None): - revision_log_data = [] - for i, log in enumerate(revision_log): - if i == per_page: - break - author_name = 'None' - author_link = 'None' - if log['author']: - author_name = log['author']['name'] or log['author']['fullname'] - author_link = gen_person_link(log['author']['id'], author_name, - snapshot_context) - revision_log_data.append( - {'author': author_link, - 'revision': gen_revision_link(log['id'], True, snapshot_context), - 'message': log['message'], - 'date': format_utc_iso_date(log['date']), - 'directory': log['directory']}) - return revision_log_data - - -def prepare_revision_log_for_display(revision_log, per_page, revs_breadcrumb, - snapshot_context=None): +def format_log_entries(revision_log, per_page, snapshot_context=None): """ Utility functions that process raw revision log data for HTML display. Its purpose is to: * add links to relevant SWH browse views * format date in human readable format * truncate the message log - It also computes the data needed to generate the links for navigating back - and forth in the history log. - Args: revision_log (list): raw revision log as returned by the SWH web api per_page (int): number of log entries per page - revs_breadcrumb (str): breadcrumbs of revisions navigated so far, - in the form 'rev1[/rev2/../revN]'. Each revision corresponds to - the first one displayed in the HTML view for history log. snapshot_context (dict): if provided, generate snapshot-dependent browsing link """ - current_rev = revision_log[0]['id'] - next_rev = None - prev_rev = None - next_revs_breadcrumb = None - prev_revs_breadcrumb = None - if len(revision_log) == per_page + 1: - prev_rev = revision_log[-1]['id'] - - prev_rev_bc = current_rev - if snapshot_context: - prev_rev_bc = prev_rev - - if revs_breadcrumb: - revs = revs_breadcrumb.split('/') - next_rev = revs[-1] - if len(revs) > 1: - next_revs_breadcrumb = '/'.join(revs[:-1]) - if len(revision_log) == per_page + 1: - prev_revs_breadcrumb = revs_breadcrumb + '/' + prev_rev_bc - else: - prev_revs_breadcrumb = prev_rev_bc - - return {'revision_log_data': _format_log_entries(revision_log, per_page, - snapshot_context), - 'prev_rev': prev_rev, - 'prev_revs_breadcrumb': prev_revs_breadcrumb, - 'next_rev': next_rev, - 'next_revs_breadcrumb': next_revs_breadcrumb} + revision_log_data = [] + for i, rev in enumerate(revision_log): + if i == per_page: + break + author_name = 'None' + author_fullname = 'None' + committer_fullname = 'None' + if rev['author']: + author_name = rev['author']['name'] or rev['author']['fullname'] + author_fullname = rev['author']['fullname'] + if rev['committer']: + committer_fullname = rev['committer']['fullname'] + author_date = format_utc_iso_date(rev['date']) + committer_date = format_utc_iso_date(rev['committer_date']) + + tooltip = 'revision %s\n' % rev['id'] + tooltip += 'author: %s\n' % author_fullname + tooltip += 'author date: %s\n' % author_date + tooltip += 'committer: %s\n' % committer_fullname + tooltip += 'committer date: %s\n\n' % committer_date + tooltip += textwrap.indent(rev['message'], ' '*4) + + revision_log_data.append({ + 'author': author_name, + 'id': rev['id'][:7], + 'message': rev['message'], + 'date': author_date, + 'commit_date': committer_date, + 'url': gen_revision_url(rev['id'], snapshot_context), + 'tooltip': tooltip + }) + return revision_log_data # list of origin types that can be found in the swh archive # TODO: retrieve it dynamically in an efficient way instead # of hardcoding it _swh_origin_types = ['git', 'svn', 'deb', 'hg', 'ftp', 'deposit', 'pypi'] def get_origin_info(origin_url, origin_type=None): """ Get info about a SWH origin. Its main purpose is to automatically find an origin type when it is not provided as parameter. Args: origin_url (str): complete url of a SWH origin origin_type (str): optional origin type Returns: A dict with the following entries: * type: the origin type * url: the origin url * id: the SWH internal id of the origin """ if origin_type: return service.lookup_origin({'type': origin_type, 'url': origin_url}) else: for origin_type in _swh_origin_types: try: origin_info = service.lookup_origin({'type': origin_type, 'url': origin_url}) return origin_info except Exception: pass raise NotFoundExc('Origin with url %s not found!' % origin_url) def get_snapshot_context(snapshot_id=None, origin_type=None, origin_url=None, timestamp=None, visit_id=None): """ Utility function to compute relevant information when navigating the SWH archive in a snapshot context. The snapshot is either referenced by its id or it will be retrieved from an origin visit. Args: snapshot_id (str): hexadecimal representation of a snapshot identifier, all other parameters will be ignored if it is provided origin_type (str): the origin type (git, svn, deposit, ...) origin_url (str): the origin_url (e.g. https://github.com/(user)/(repo)/) timestamp (str): a datetime string for retrieving the closest SWH visit of the origin visit_id (int): optional visit id for disambiguation in case of several visits with the same timestamp Returns: A dict with the following entries: * origin_info: dict containing origin information * visit_info: dict containing SWH visit information * branches: the list of branches for the origin found during the visit * releases: the list of releases for the origin found during the visit * origin_browse_url: the url to browse the origin * origin_branches_url: the url to browse the origin branches * origin_releases_url': the url to browse the origin releases * origin_visit_url: the url to browse the snapshot of the origin found during the visit * url_args: dict containing url arguments to use when browsing in the context of the origin and its visit Raises: NotFoundExc: if no snapshot is found for the visit of an origin. """ # noqa origin_info = None visit_info = None url_args = None query_params = {} branches = [] releases = [] browse_url = None visit_url = None branches_url = None releases_url = None swh_type = 'snapshot' if origin_url: swh_type = 'origin' origin_info = get_origin_info(origin_url, origin_type) visit_info = get_origin_visit(origin_info, timestamp, visit_id, snapshot_id) fmt_date = format_utc_iso_date(visit_info['date']) visit_info['fmt_date'] = fmt_date snapshot_id = visit_info['snapshot'] if not snapshot_id: raise NotFoundExc('No snapshot associated to the visit of origin ' '%s on %s' % (origin_url, fmt_date)) # provided timestamp is not necessarily equals to the one # of the retrieved visit, so get the exact one in order # use it in the urls generated below if timestamp: timestamp = visit_info['date'] branches, releases = \ get_origin_visit_snapshot(origin_info, timestamp, visit_id, snapshot_id) url_args = {'origin_type': origin_type, 'origin_url': origin_info['url']} query_params = {'visit_id': visit_id} browse_url = reverse('browse-origin-visits', url_args=url_args) if timestamp: url_args['timestamp'] = format_utc_iso_date(timestamp, '%Y-%m-%dT%H:%M:%S') visit_url = reverse('browse-origin-directory', url_args=url_args, query_params=query_params) visit_info['url'] = visit_url branches_url = reverse('browse-origin-branches', url_args=url_args, query_params=query_params) releases_url = reverse('browse-origin-releases', url_args=url_args, query_params=query_params) elif snapshot_id: branches, releases = get_snapshot_content(snapshot_id) url_args = {'snapshot_id': snapshot_id} browse_url = reverse('browse-snapshot', url_args=url_args) branches_url = reverse('browse-snapshot-branches', url_args=url_args) releases_url = reverse('browse-snapshot-releases', url_args=url_args) releases = list(reversed(releases)) snapshot_size = service.lookup_snapshot_size(snapshot_id) is_empty = sum(snapshot_size.values()) == 0 swh_snp_id = persistent_identifier('snapshot', snapshot_id) return { 'swh_type': swh_type, 'swh_object_id': swh_snp_id, 'snapshot_id': snapshot_id, 'snapshot_size': snapshot_size, 'is_empty': is_empty, 'origin_info': origin_info, # keep track if the origin type was provided as url argument 'origin_type': origin_type, 'visit_info': visit_info, 'branches': branches, 'releases': releases, 'branch': None, 'release': None, 'browse_url': browse_url, 'branches_url': branches_url, 'releases_url': releases_url, 'url_args': url_args, 'query_params': query_params } # list of common readme names ordered by preference # (lower indices have higher priority) _common_readme_names = [ "readme.markdown", "readme.md", "readme.rst", "readme.txt", "readme" ] def get_readme_to_display(readmes): """ Process a list of readme files found in a directory in order to find the adequate one to display. Args: readmes: a list of dict where keys are readme file names and values are readme sha1s Returns: A tuple (readme_name, readme_sha1) """ readme_name = None readme_url = None readme_sha1 = None readme_html = None lc_readmes = {k.lower(): {'orig_name': k, 'sha1': v} for k, v in readmes.items()} # look for readme names according to the preference order # defined by the _common_readme_names list for common_readme_name in _common_readme_names: if common_readme_name in lc_readmes: readme_name = lc_readmes[common_readme_name]['orig_name'] readme_sha1 = lc_readmes[common_readme_name]['sha1'] readme_url = reverse('browse-content-raw', url_args={'query_string': readme_sha1}) break # otherwise pick the first readme like file if any if not readme_name and len(readmes.items()) > 0: readme_name = next(iter(readmes)) readme_sha1 = readmes[readme_name] readme_url = reverse('browse-content-raw', url_args={'query_string': readme_sha1}) # convert rst README to html server side as there is # no viable solution to perform that task client side if readme_name and readme_name.endswith('.rst'): cache_entry_id = 'readme_%s' % readme_sha1 cache_entry = cache.get(cache_entry_id) if cache_entry: readme_html = cache_entry else: try: rst_doc = request_content(readme_sha1) readme_html = pypandoc.convert_text(rst_doc['raw_data'], 'html', format='rst') cache.set(cache_entry_id, readme_html) except Exception: readme_html = 'Readme bytes are not available' return readme_name, readme_url, readme_html def get_swh_persistent_ids(swh_objects, snapshot_context=None): """ Returns a list of dict containing info related to persistent identifiers of swh objects. Args: swh_objects (list): a list of dict with the following keys: * type: swh object type (content/directory/release/revision/snapshot) * id: swh object id snapshot_context (dict): optional parameter describing the snapshot in which the object has been found Returns: list: a list of dict with the following keys: * object_type: the swh object type (content/directory/release/revision/snapshot) * object_icon: the swh object icon to use in HTML views * swh_id: the computed swh object persistent identifier * swh_id_url: the url resolving the persistent identifier * show_options: boolean indicating if the persistent id options must be displayed in persistent ids HTML view """ # noqa swh_ids = [] for swh_object in swh_objects: if not swh_object['id']: continue swh_id = get_swh_persistent_id(swh_object['type'], swh_object['id']) show_options = swh_object['type'] == 'content' or \ (snapshot_context and snapshot_context['origin_info'] is not None) object_icon = swh_object_icons[swh_object['type']] swh_ids.append({ 'object_type': swh_object['type'], 'object_icon': object_icon, 'swh_id': swh_id, 'swh_id_url': reverse('browse-swh-id', url_args={'swh_id': swh_id}), 'show_options': show_options }) return swh_ids diff --git a/swh/web/browse/views/revision.py b/swh/web/browse/views/revision.py index 92d0aebe..38ac23bf 100644 --- a/swh/web/browse/views/revision.py +++ b/swh/web/browse/views/revision.py @@ -1,544 +1,551 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import hashlib import json import textwrap from django.http import HttpResponse from django.shortcuts import render, redirect from django.template.defaultfilters import filesizeformat from django.utils.safestring import mark_safe from swh.model.identifiers import persistent_identifier from swh.web.common import service from swh.web.common.utils import ( reverse, format_utc_iso_date, gen_path_info, swh_object_icons ) from swh.web.common.exc import NotFoundExc, handle_view_exception from swh.web.browse.browseurls import browse_route from swh.web.browse.utils import ( gen_link, gen_person_link, gen_revision_link, gen_revision_url, - prepare_revision_log_for_display, get_snapshot_context, gen_snapshot_directory_link, get_revision_log_url, get_directory_entries, gen_directory_link, request_content, prepare_content_for_display, content_display_max_size, gen_snapshot_link, get_readme_to_display, - get_swh_persistent_ids + get_swh_persistent_ids, format_log_entries ) def _gen_content_url(revision, query_string, path, snapshot_context): if snapshot_context: url_args = snapshot_context['url_args'] url_args['path'] = path query_params = snapshot_context['query_params'] query_params['revision'] = revision['id'] content_url = reverse('browse-origin-content', url_args=url_args, query_params=query_params) else: content_path = '%s/%s' % (revision['directory'], path) content_url = reverse('browse-content', url_args={'query_string': query_string}, query_params={'path': content_path}) return content_url def _gen_diff_link(idx, diff_anchor, link_text): if idx < _max_displayed_file_diffs: return gen_link(diff_anchor, link_text) else: return link_text # TODO: put in conf _max_displayed_file_diffs = 1000 def _gen_revision_changes_list(revision, changes, snapshot_context): """ Returns a HTML string describing the file changes introduced in a revision. As this string will be displayed in the browse revision view, links to adequate file diffs are also generated. Args: revision (str): hexadecimal representation of a revision identifier changes (list): list of file changes in the revision snapshot_context (dict): optional origin context used to reverse the content urls Returns: A string to insert in a revision HTML view. """ changes_msg = [] for i, change in enumerate(changes): hasher = hashlib.sha1() from_query_string = '' to_query_string = '' diff_id = 'diff-' if change['from']: from_query_string = 'sha1_git:' + change['from']['target'] diff_id += change['from']['target'] + '-' + change['from_path'] diff_id += '-' if change['to']: to_query_string = 'sha1_git:' + change['to']['target'] diff_id += change['to']['target'] + change['to_path'] change['path'] = change['to_path'] or change['from_path'] url_args = {'from_query_string': from_query_string, 'to_query_string': to_query_string} query_params = {'path': change['path']} change['diff_url'] = reverse('diff-contents', url_args=url_args, query_params=query_params) hasher.update(diff_id.encode('utf-8')) diff_id = hasher.hexdigest() change['id'] = diff_id panel_diff_link = '#panel_' + diff_id if change['type'] == 'modify': change['content_url'] = \ _gen_content_url(revision, to_query_string, change['to_path'], snapshot_context) changes_msg.append('modified: %s' % _gen_diff_link(i, panel_diff_link, change['to_path'])) elif change['type'] == 'insert': change['content_url'] = \ _gen_content_url(revision, to_query_string, change['to_path'], snapshot_context) changes_msg.append('new file: %s' % _gen_diff_link(i, panel_diff_link, change['to_path'])) elif change['type'] == 'delete': parent = service.lookup_revision(revision['parents'][0]) change['content_url'] = \ _gen_content_url(parent, from_query_string, change['from_path'], snapshot_context) changes_msg.append('deleted: %s' % _gen_diff_link(i, panel_diff_link, change['from_path'])) elif change['type'] == 'rename': change['content_url'] = \ _gen_content_url(revision, to_query_string, change['to_path'], snapshot_context) link_text = change['from_path'] + ' → ' + change['to_path'] changes_msg.append('renamed: %s' % _gen_diff_link(i, panel_diff_link, link_text)) if not changes: changes_msg.append('No changes') return mark_safe('\n'.join(changes_msg)) @browse_route(r'revision/(?P[0-9a-f]+)/diff/', view_name='diff-revision') def _revision_diff(request, sha1_git): """ Browse internal endpoint to compute revision diff """ try: revision = service.lookup_revision(sha1_git) snapshot_context = None origin_type = request.GET.get('origin_type', None) origin_url = request.GET.get('origin_url', None) if not origin_url: origin_url = request.GET.get('origin', None) timestamp = request.GET.get('timestamp', None) visit_id = request.GET.get('visit_id', None) if origin_url: snapshot_context = get_snapshot_context(None, origin_type, origin_url, timestamp, visit_id) except Exception as exc: return handle_view_exception(request, exc) changes = service.diff_revision(sha1_git) changes_msg = _gen_revision_changes_list(revision, changes, snapshot_context) diff_data = { 'total_nb_changes': len(changes), 'changes': changes[:_max_displayed_file_diffs], 'changes_msg': changes_msg } diff_data_json = json.dumps(diff_data, separators=(',', ': ')) return HttpResponse(diff_data_json, content_type='application/json') -NB_LOG_ENTRIES = 20 +NB_LOG_ENTRIES = 100 @browse_route(r'revision/(?P[0-9a-f]+)/log/', view_name='browse-revision-log') def revision_log_browse(request, sha1_git): """ Django view that produces an HTML display of the history log for a SWH revision identified by its id. - The url that points to it is :http:get:`/browse/revision/(sha1_git)/log/`. + The url that points to it is :http:get:`/browse/revision/(sha1_git)/log/` """ # noqa try: per_page = int(request.GET.get('per_page', NB_LOG_ENTRIES)) - revision_log = service.lookup_revision_log(sha1_git, - limit=per_page+1) - revision_log = list(revision_log) + offset = int(request.GET.get('offset', 0)) + revs_ordering = request.GET.get('revs_ordering', 'committer_date') + session_key = 'rev_%s_log_ordering_%s' % (sha1_git, revs_ordering) + rev_log_session = request.session.get(session_key, None) + rev_log = [] + revs_walker_state = None + if rev_log_session: + rev_log = rev_log_session['rev_log'] + revs_walker_state = rev_log_session['revs_walker_state'] + + if len(rev_log) < offset+per_page: + revs_walker = \ + service.get_revisions_walker(revs_ordering, sha1_git, + max_revs=offset+per_page+1, + state=revs_walker_state) + + rev_log += list(revs_walker) + revs_walker_state = revs_walker.export_state() + + revision_log = rev_log[offset:offset+per_page] + + request.session[session_key] = { + 'rev_log': rev_log, + 'revs_walker_state': revs_walker_state + } except Exception as exc: return handle_view_exception(request, exc) - revs_breadcrumb = request.GET.get('revs_breadcrumb', None) - - revision_log_display_data = prepare_revision_log_for_display( - revision_log, per_page, revs_breadcrumb) + revs_ordering = request.GET.get('revs_ordering', '') - prev_rev = revision_log_display_data['prev_rev'] - prev_revs_breadcrumb = revision_log_display_data['prev_revs_breadcrumb'] prev_log_url = None - if prev_rev: - prev_log_url = \ - reverse('browse-revision-log', - url_args={'sha1_git': prev_rev}, - query_params={'revs_breadcrumb': prev_revs_breadcrumb, - 'per_page': per_page}) - - next_rev = revision_log_display_data['next_rev'] - next_revs_breadcrumb = revision_log_display_data['next_revs_breadcrumb'] + if len(rev_log) > offset + per_page: + prev_log_url = reverse('browse-revision-log', + url_args={'sha1_git': sha1_git}, + query_params={'per_page': per_page, + 'offset': offset + per_page, + 'revs_ordering': revs_ordering}) + next_log_url = None - if next_rev: - next_log_url = \ - reverse('browse-revision-log', - url_args={'sha1_git': next_rev}, - query_params={'revs_breadcrumb': next_revs_breadcrumb, - 'per_page': per_page}) - - revision_log_data = revision_log_display_data['revision_log_data'] - - for log in revision_log_data: - log['directory'] = gen_directory_link( - log['directory'], - link_text='Browse files', - link_attrs={'class': 'btn btn-default btn-sm', - 'role': 'button'}) + if offset != 0: + next_log_url = reverse('browse-revision-log', + url_args={'sha1_git': sha1_git}, + query_params={'per_page': per_page, + 'offset': offset - per_page, + 'revs_ordering': revs_ordering}) + + revision_log_data = format_log_entries(revision_log, per_page) swh_rev_id = persistent_identifier('revision', sha1_git) return render(request, 'browse/revision-log.html', {'heading': 'Revision history', 'swh_object_id': swh_rev_id, 'swh_object_name': 'Revisions history', 'swh_object_metadata': None, 'revision_log': revision_log_data, + 'revs_ordering': revs_ordering, 'next_log_url': next_log_url, 'prev_log_url': prev_log_url, 'breadcrumbs': None, 'top_right_link': None, 'snapshot_context': None, 'vault_cooking': None, 'show_actions_menu': True, 'swh_ids': None}) @browse_route(r'revision/(?P[0-9a-f]+)/', r'revision/(?P[0-9a-f]+)/(?P.+)/', view_name='browse-revision') def revision_browse(request, sha1_git, extra_path=None): """ Django view that produces an HTML display of a SWH revision identified by its id. The url that points to it is :http:get:`/browse/revision/(sha1_git)/`. """ try: revision = service.lookup_revision(sha1_git) # some readme files can reference assets reachable from the # browsed directory, handle that special case in order to # correctly displayed them if extra_path: dir_info = \ service.lookup_directory_with_path(revision['directory'], extra_path) if dir_info and dir_info['type'] == 'file': file_raw_url = reverse( 'browse-content-raw', url_args={'query_string': dir_info['checksums']['sha1']}) return redirect(file_raw_url) origin_info = None snapshot_context = None origin_type = request.GET.get('origin_type', None) origin_url = request.GET.get('origin_url', None) if not origin_url: origin_url = request.GET.get('origin', None) timestamp = request.GET.get('timestamp', None) visit_id = request.GET.get('visit_id', None) snapshot_id = request.GET.get('snapshot_id', None) path = request.GET.get('path', None) dir_id = None dirs, files = None, None content_data = None if origin_url: try: snapshot_context = get_snapshot_context(None, origin_type, origin_url, timestamp, visit_id) except Exception: raw_rev_url = reverse('browse-revision', url_args={'sha1_git': sha1_git}) error_message = \ ('The Software Heritage archive has a revision ' 'with the hash you provided but the origin ' 'mentioned in your request appears broken: %s. ' 'Please check the URL and try again.\n\n' 'Nevertheless, you can still browse the revision ' 'without origin information: %s' % (gen_link(origin_url), gen_link(raw_rev_url))) raise NotFoundExc(error_message) origin_info = snapshot_context['origin_info'] snapshot_id = snapshot_context['snapshot_id'] elif snapshot_id: snapshot_context = get_snapshot_context(snapshot_id) if path: file_info = \ service.lookup_directory_with_path(revision['directory'], path) if file_info['type'] == 'dir': dir_id = file_info['target'] else: query_string = 'sha1_git:' + file_info['target'] content_data = request_content(query_string, raise_if_unavailable=False) else: dir_id = revision['directory'] if dir_id: path = '' if path is None else (path + '/') dirs, files = get_directory_entries(dir_id) except Exception as exc: return handle_view_exception(request, exc) revision_data = {} author_name = 'None' revision_data['author'] = 'None' if revision['author']: author_name = revision['author']['name'] or \ revision['author']['fullname'] revision_data['author'] = \ gen_person_link(revision['author']['id'], author_name, snapshot_context) revision_data['committer'] = 'None' if revision['committer']: revision_data['committer'] = \ gen_person_link(revision['committer']['id'], revision['committer']['name'], snapshot_context) - revision_data['committer date'] = format_utc_iso_date( - revision['committer_date']) + revision_data['committer date'] = \ + format_utc_iso_date(revision['committer_date']) revision_data['date'] = format_utc_iso_date(revision['date']) if snapshot_context: revision_data['snapshot id'] = snapshot_id revision_data['directory'] = \ gen_snapshot_directory_link(snapshot_context, sha1_git, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', # noqa 'role': 'button'}) else: revision_data['directory'] = \ gen_directory_link(revision['directory'], link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) revision_data['id'] = sha1_git revision_data['merge'] = revision['merge'] revision_data['metadata'] = json.dumps(revision['metadata'], sort_keys=True, indent=4, separators=(',', ': ')) if origin_info: revision_data['context-independent revision'] = \ gen_revision_link(sha1_git, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) revision_data['origin id'] = origin_info['id'] revision_data['origin type'] = origin_info['type'] revision_data['origin url'] = gen_link(origin_info['url'], origin_info['url']) browse_snapshot_link = \ gen_snapshot_link(snapshot_id, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) revision_data['snapshot'] = browse_snapshot_link parents = '' for p in revision['parents']: parent_link = gen_revision_link(p, snapshot_context=snapshot_context) parents += parent_link + '
' revision_data['parents'] = mark_safe(parents) revision_data['synthetic'] = revision['synthetic'] revision_data['type'] = revision['type'] message_lines = ['None'] if revision['message']: message_lines = revision['message'].split('\n') parents = [] for p in revision['parents']: parent_url = gen_revision_url(p, snapshot_context) parents.append({'id': p, 'url': parent_url}) path_info = gen_path_info(path) query_params = {'snapshot_id': snapshot_id, 'origin_type': origin_type, 'origin': origin_url, 'timestamp': timestamp, 'visit_id': visit_id} breadcrumbs = [] breadcrumbs.append({'name': revision['directory'][:7], 'url': reverse('browse-revision', url_args={'sha1_git': sha1_git}, query_params=query_params)}) for pi in path_info: query_params['path'] = pi['path'] breadcrumbs.append({'name': pi['name'], 'url': reverse('browse-revision', url_args={'sha1_git': sha1_git}, query_params=query_params)}) vault_cooking = { 'directory_context': False, 'directory_id': None, 'revision_context': True, 'revision_id': sha1_git } swh_objects = [{'type': 'revision', 'id': sha1_git}] content = None content_size = None mimetype = None language = None readme_name = None readme_url = None readme_html = None readmes = {} error_code = 200 error_message = '' error_description = '' if content_data: breadcrumbs[-1]['url'] = None content_size = content_data['length'] mimetype = content_data['mimetype'] if content_data['raw_data']: content_display_data = prepare_content_for_display( content_data['raw_data'], content_data['mimetype'], path) content = content_display_data['content_data'] language = content_display_data['language'] query_params = {} if path: query_params['filename'] = path_info[-1]['name'] top_right_link = { 'url': reverse('browse-content-raw', url_args={'query_string': query_string}, query_params=query_params), 'icon': swh_object_icons['content'], 'text': 'Raw File' } swh_objects.append({'type': 'content', 'id': file_info['target']}) error_code = content_data['error_code'] error_message = content_data['error_message'] error_description = content_data['error_description'] else: for d in dirs: if d['type'] == 'rev': d['url'] = reverse('browse-revision', url_args={'sha1_git': d['target']}) else: query_params['path'] = path + d['name'] d['url'] = reverse('browse-revision', url_args={'sha1_git': sha1_git}, query_params=query_params) for f in files: query_params['path'] = path + f['name'] f['url'] = reverse('browse-revision', url_args={'sha1_git': sha1_git}, query_params=query_params) if f['length'] is not None: f['length'] = filesizeformat(f['length']) if f['name'].lower().startswith('readme'): readmes[f['name']] = f['checksums']['sha1'] readme_name, readme_url, readme_html = get_readme_to_display(readmes) top_right_link = { 'url': get_revision_log_url(sha1_git, snapshot_context), 'icon': swh_object_icons['revisions history'], 'text': 'History' } vault_cooking['directory_context'] = True vault_cooking['directory_id'] = dir_id swh_objects.append({'type': 'directory', 'id': dir_id}) diff_revision_url = reverse('diff-revision', url_args={'sha1_git': sha1_git}, query_params={'origin_type': origin_type, 'origin': origin_url, 'timestamp': timestamp, 'visit_id': visit_id}) if snapshot_id: swh_objects.append({'type': 'snapshot', 'id': snapshot_id}) swh_ids = get_swh_persistent_ids(swh_objects, snapshot_context) heading = 'Revision - %s - %s' %\ (sha1_git[:7], textwrap.shorten(message_lines[0], width=70)) if snapshot_context: context_found = 'snapshot: %s' % snapshot_context['snapshot_id'] if origin_info: context_found = 'origin: %s' % origin_info['url'] heading += ' - %s' % context_found return render(request, 'browse/revision.html', {'heading': heading, 'swh_object_id': swh_ids[0]['swh_id'], 'swh_object_name': 'Revision', 'swh_object_metadata': revision_data, 'message_header': message_lines[0], 'message_body': '\n'.join(message_lines[1:]), 'parents': parents, 'snapshot_context': snapshot_context, 'dirs': dirs, 'files': files, 'content': content, 'content_size': content_size, 'max_content_size': content_display_max_size, 'mimetype': mimetype, 'language': language, 'readme_name': readme_name, 'readme_url': readme_url, 'readme_html': readme_html, 'breadcrumbs': breadcrumbs, 'top_right_link': top_right_link, 'vault_cooking': vault_cooking, 'diff_revision_url': diff_revision_url, 'show_actions_menu': True, 'swh_ids': swh_ids, 'error_code': error_code, 'error_message': error_message, 'error_description': error_description}, status=error_code) diff --git a/swh/web/browse/views/utils/snapshot_context.py b/swh/web/browse/views/utils/snapshot_context.py index 365d8f86..ffbf512d 100644 --- a/swh/web/browse/views/utils/snapshot_context.py +++ b/swh/web/browse/views/utils/snapshot_context.py @@ -1,951 +1,945 @@ # Copyright (C) 2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information # Utility module implementing Django views for browsing the SWH archive # in a snapshot context. # Its purpose is to factorize code for the views reachable from the # /origin/.* and /snapshot/.* endpoints. from django.shortcuts import render, redirect from django.template.defaultfilters import filesizeformat from swh.model.identifiers import snapshot_identifier from swh.web.browse.utils import ( get_snapshot_context, get_directory_entries, gen_directory_link, gen_revision_link, request_content, gen_content_link, prepare_content_for_display, content_display_max_size, - prepare_revision_log_for_display, gen_snapshot_directory_link, - gen_revision_log_link, gen_link, get_readme_to_display, - get_swh_persistent_ids, process_snapshot_branches + format_log_entries, gen_revision_log_link, gen_link, + get_readme_to_display, get_swh_persistent_ids, process_snapshot_branches ) from swh.web.common import service from swh.web.common.exc import ( handle_view_exception, NotFoundExc ) from swh.web.common.utils import ( reverse, gen_path_info, format_utc_iso_date, swh_object_icons ) _empty_snapshot_id = snapshot_identifier({'branches': {}}) def _get_branch(branches, branch_name, snapshot_id): """ Utility function to get a specific branch from a branches list. Its purpose is to get the default HEAD branch as some SWH origin (e.g those with svn type) does not have it. In that latter case, check if there is a master branch instead and returns it. """ filtered_branches = \ [b for b in branches if b['name'].endswith(branch_name)] if len(filtered_branches) > 0: return filtered_branches[0] elif branch_name == 'HEAD': filtered_branches = \ [b for b in branches if b['name'].endswith('master')] if len(filtered_branches) > 0: return filtered_branches[0] elif len(branches) > 0: return branches[0] else: # case where a large branches list has been truncated snp_branch = service.lookup_snapshot(snapshot_id, branches_from=branch_name, branches_count=1, target_types=['revision']) snp_branch, _ = process_snapshot_branches(snp_branch['branches']) if snp_branch: branches.append(snp_branch[0]) return snp_branch[0] return None def _get_release(releases, release_name): """ Utility function to get a specific release from a releases list. Returns None if the release can not be found in the list. """ filtered_releases = \ [r for r in releases if r['name'] == release_name] if len(filtered_releases) > 0: return filtered_releases[0] else: return None def _branch_not_found(branch_type, branch, branches, snapshot_id=None, origin_info=None, timestamp=None, visit_id=None): """ Utility function to raise an exception when a specified branch/release can not be found. """ if branch_type == 'branch': branch_type = 'Branch' branch_type_plural = 'branches' else: branch_type = 'Release' branch_type_plural = 'releases' if snapshot_id and len(branches) == 0: msg = 'Snapshot with id %s has an empty list' \ ' of %s!' % (snapshot_id, branch_type_plural) elif snapshot_id: msg = '%s %s for snapshot with id %s' \ ' not found!' % (branch_type, branch, snapshot_id) elif visit_id and len(branches) == 0: msg = 'Origin with type %s and url %s' \ ' for visit with id %s has an empty list' \ ' of %s!' % (origin_info['type'], origin_info['url'], visit_id, branch_type_plural) elif visit_id: msg = '%s %s associated to visit with' \ ' id %s for origin with type %s and url %s' \ ' not found!' % (branch_type, branch, visit_id, origin_info['type'], origin_info['url']) elif len(branches) == 0: msg = 'Origin with type %s and url %s' \ ' for visit with timestamp %s has an empty list' \ ' of %s!' % (origin_info['type'], origin_info['url'], timestamp, branch_type_plural) else: msg = '%s %s associated to visit with' \ ' timestamp %s for origin with type %s' \ ' and url %s not found!' % (branch_type, branch, timestamp, origin_info['type'], origin_info['url']) raise NotFoundExc(msg) def _process_snapshot_request(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None, path=None, browse_context='directory'): """ Utility function to perform common input request processing for snapshot context views. """ visit_id = request.GET.get('visit_id', None) snapshot_context = get_snapshot_context(snapshot_id, origin_type, origin_url, timestamp, visit_id) swh_type = snapshot_context['swh_type'] origin_info = snapshot_context['origin_info'] branches = snapshot_context['branches'] releases = snapshot_context['releases'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] if snapshot_context['visit_info']: timestamp = format_utc_iso_date(snapshot_context['visit_info']['date'], '%Y-%m-%dT%H:%M:%SZ') snapshot_context['timestamp'] = \ format_utc_iso_date(snapshot_context['visit_info']['date']) browse_view_name = 'browse-' + swh_type + '-' + browse_context root_sha1_git = None revision_id = request.GET.get('revision', None) release_name = request.GET.get('release', None) release_id = None branch_name = None snapshot_total_size = sum(snapshot_context['snapshot_size'].values()) if snapshot_total_size and revision_id: revision = service.lookup_revision(revision_id) root_sha1_git = revision['directory'] branches.append({'name': revision_id, 'revision': revision_id, 'directory': root_sha1_git, 'url': None}) branch_name = revision_id query_params['revision'] = revision_id elif snapshot_total_size and release_name: release = _get_release(releases, release_name) try: root_sha1_git = release['directory'] revision_id = release['target'] release_id = release['id'] query_params['release'] = release_name except Exception: _branch_not_found("release", release_name, releases, snapshot_id, origin_info, timestamp, visit_id) elif snapshot_total_size: branch_name = request.GET.get('branch', None) if branch_name: query_params['branch'] = branch_name branch = _get_branch(branches, branch_name or 'HEAD', snapshot_context['snapshot_id']) try: branch_name = branch['name'] revision_id = branch['revision'] root_sha1_git = branch['directory'] except Exception: _branch_not_found("branch", branch_name, branches, snapshot_id, origin_info, timestamp, visit_id) for b in branches: branch_url_args = dict(url_args) branch_query_params = dict(query_params) if 'release' in branch_query_params: del branch_query_params['release'] branch_query_params['branch'] = b['name'] if path: b['path'] = path branch_url_args['path'] = path b['url'] = reverse(browse_view_name, url_args=branch_url_args, query_params=branch_query_params) for r in releases: release_url_args = dict(url_args) release_query_params = dict(query_params) if 'branch' in release_query_params: del release_query_params['branch'] release_query_params['release'] = r['name'] if path: r['path'] = path release_url_args['path'] = path r['url'] = reverse(browse_view_name, url_args=release_url_args, query_params=release_query_params) snapshot_context['query_params'] = query_params snapshot_context['root_sha1_git'] = root_sha1_git snapshot_context['revision_id'] = revision_id snapshot_context['branch'] = branch_name snapshot_context['release'] = release_name snapshot_context['release_id'] = release_id return snapshot_context def browse_snapshot_directory(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None, path=None): """ Django view implementation for browsing a directory in a snapshot context. """ try: snapshot_context = _process_snapshot_request(request, snapshot_id, origin_type, origin_url, timestamp, path, browse_context='directory') # noqa root_sha1_git = snapshot_context['root_sha1_git'] sha1_git = root_sha1_git if root_sha1_git and path: dir_info = service.lookup_directory_with_path(root_sha1_git, path) # some readme files can reference assets reachable from the # browsed directory, handle that special case in order to # correctly displayed them if dir_info and dir_info['type'] == 'file': file_raw_url = reverse( 'browse-content-raw', url_args={'query_string': dir_info['checksums']['sha1']}) return redirect(file_raw_url) sha1_git = dir_info['target'] dirs = [] files = [] if sha1_git: dirs, files = get_directory_entries(sha1_git) except Exception as exc: return handle_view_exception(request, exc) swh_type = snapshot_context['swh_type'] origin_info = snapshot_context['origin_info'] visit_info = snapshot_context['visit_info'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] revision_id = snapshot_context['revision_id'] snapshot_id = snapshot_context['snapshot_id'] path_info = gen_path_info(path) browse_view_name = 'browse-' + swh_type + '-directory' breadcrumbs = [] if root_sha1_git: breadcrumbs.append({'name': root_sha1_git[:7], 'url': reverse(browse_view_name, url_args=url_args, query_params=query_params)}) for pi in path_info: bc_url_args = dict(url_args) bc_url_args['path'] = pi['path'] breadcrumbs.append({'name': pi['name'], 'url': reverse(browse_view_name, url_args=bc_url_args, query_params=query_params)}) path = '' if path is None else (path + '/') for d in dirs: if d['type'] == 'rev': d['url'] = reverse('browse-revision', url_args={'sha1_git': d['target']}) else: bc_url_args = dict(url_args) bc_url_args['path'] = path + d['name'] d['url'] = reverse(browse_view_name, url_args=bc_url_args, query_params=query_params) sum_file_sizes = 0 readmes = {} browse_view_name = 'browse-' + swh_type + '-content' for f in files: bc_url_args = dict(url_args) bc_url_args['path'] = path + f['name'] f['url'] = reverse(browse_view_name, url_args=bc_url_args, query_params=query_params) if f['length'] is not None: sum_file_sizes += f['length'] f['length'] = filesizeformat(f['length']) if f['name'].lower().startswith('readme'): readmes[f['name']] = f['checksums']['sha1'] readme_name, readme_url, readme_html = get_readme_to_display(readmes) browse_view_name = 'browse-' + swh_type + '-log' history_url = None if snapshot_id != _empty_snapshot_id: history_url = reverse(browse_view_name, url_args=url_args, query_params=query_params) nb_files = None nb_dirs = None sum_file_sizes = None dir_path = None if root_sha1_git: nb_files = len(files) nb_dirs = len(dirs) sum_file_sizes = filesizeformat(sum_file_sizes) dir_path = '/' + path browse_dir_link = \ gen_directory_link(sha1_git, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) browse_rev_link = \ gen_revision_link(revision_id, snapshot_context=snapshot_context, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) dir_metadata = {'id': sha1_git, 'context-independent directory': browse_dir_link, 'number of regular files': nb_files, 'number of subdirectories': nb_dirs, 'sum of regular file sizes': sum_file_sizes, 'path': dir_path, 'revision id': revision_id, 'revision': browse_rev_link, 'snapshot id': snapshot_id} if origin_info: dir_metadata['origin id'] = origin_info['id'] dir_metadata['origin type'] = origin_info['type'] dir_metadata['origin url'] = origin_info['url'] dir_metadata['origin visit date'] = format_utc_iso_date(visit_info['date']) # noqa dir_metadata['origin visit id'] = visit_info['visit'] snapshot_context_url = reverse('browse-snapshot-directory', url_args={'snapshot_id': snapshot_id}, query_params=request.GET) browse_snapshot_link = \ gen_link(snapshot_context_url, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) dir_metadata['snapshot context'] = browse_snapshot_link vault_cooking = { 'directory_context': True, 'directory_id': sha1_git, 'revision_context': True, 'revision_id': revision_id } swh_objects = [{'type': 'directory', 'id': sha1_git}, {'type': 'revision', 'id': revision_id}, {'type': 'snapshot', 'id': snapshot_id}] release_id = snapshot_context['release_id'] if release_id: swh_objects.append({'type': 'release', 'id': release_id}) swh_ids = get_swh_persistent_ids(swh_objects, snapshot_context) dir_path = '/'.join([bc['name'] for bc in breadcrumbs]) + '/' context_found = 'snapshot: %s' % snapshot_context['snapshot_id'] if origin_info: context_found = 'origin: %s' % origin_info['url'] heading = 'Directory - %s - %s - %s' %\ (dir_path, snapshot_context['branch'], context_found) return render(request, 'browse/directory.html', {'heading': heading, 'swh_object_name': 'Directory', 'swh_object_metadata': dir_metadata, 'dirs': dirs, 'files': files, 'breadcrumbs': breadcrumbs if root_sha1_git else [], 'top_right_link': { 'url': history_url, 'icon': swh_object_icons['revisions history'], 'text': 'History' }, 'readme_name': readme_name, 'readme_url': readme_url, 'readme_html': readme_html, 'snapshot_context': snapshot_context, 'vault_cooking': vault_cooking, 'show_actions_menu': True, 'swh_ids': swh_ids}) def browse_snapshot_content(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None, path=None): """ Django view implementation for browsing a content in a snapshot context. """ try: snapshot_context = _process_snapshot_request(request, snapshot_id, origin_type, origin_url, timestamp, path, browse_context='content') root_sha1_git = snapshot_context['root_sha1_git'] sha1_git = None query_string = None content_data = None if root_sha1_git: content_info = service.lookup_directory_with_path(root_sha1_git, path) sha1_git = content_info['target'] query_string = 'sha1_git:' + sha1_git content_data = request_content(query_string, raise_if_unavailable=False) except Exception as exc: return handle_view_exception(request, exc) swh_type = snapshot_context['swh_type'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] revision_id = snapshot_context['revision_id'] origin_info = snapshot_context['origin_info'] visit_info = snapshot_context['visit_info'] snapshot_id = snapshot_context['snapshot_id'] content = None language = None mimetype = None if content_data and content_data['raw_data'] is not None: content_display_data = prepare_content_for_display( content_data['raw_data'], content_data['mimetype'], path) content = content_display_data['content_data'] language = content_display_data['language'] mimetype = content_display_data['mimetype'] filename = None path_info = None browse_view_name = 'browse-' + swh_type + '-directory' breadcrumbs = [] split_path = path.split('/') filename = split_path[-1] path_info = gen_path_info(path[:-len(filename)]) if root_sha1_git: breadcrumbs.append({'name': root_sha1_git[:7], 'url': reverse(browse_view_name, url_args=url_args, query_params=query_params)}) for pi in path_info: bc_url_args = dict(url_args) bc_url_args['path'] = pi['path'] breadcrumbs.append({'name': pi['name'], 'url': reverse(browse_view_name, url_args=bc_url_args, query_params=query_params)}) breadcrumbs.append({'name': filename, 'url': None}) browse_content_link = \ gen_content_link(sha1_git, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) content_raw_url = None if query_string: content_raw_url = reverse('browse-content-raw', url_args={'query_string': query_string}, query_params={'filename': filename}) browse_rev_link = \ gen_revision_link(revision_id, snapshot_context=snapshot_context, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) content_metadata = { 'context-independent content': browse_content_link, 'path': None, 'filename': None, 'revision id': revision_id, 'revision': browse_rev_link, 'snapshot id': snapshot_id } cnt_sha1_git = None content_size = None error_code = 200 error_description = '' error_message = '' if content_data: content_metadata['sha1 checksum'] = \ content_data['checksums']['sha1'] content_metadata['sha1_git checksum'] = \ content_data['checksums']['sha1_git'] content_metadata['sha256 checksum'] = \ content_data['checksums']['sha256'] content_metadata['blake2s256 checksum'] = \ content_data['checksums']['blake2s256'] content_metadata['mime type'] = content_data['mimetype'] content_metadata['encoding'] = content_data['encoding'] content_metadata['size'] = filesizeformat(content_data['length']) content_metadata['language'] = content_data['language'] content_metadata['licenses'] = content_data['licenses'] content_metadata['path'] = '/' + path[:-len(filename)] content_metadata['filename'] = filename cnt_sha1_git = content_data['checksums']['sha1_git'] content_size = content_data['length'] error_code = content_data['error_code'] error_message = content_data['error_message'] error_description = content_data['error_description'] if origin_info: content_metadata['origin id'] = origin_info['id'] content_metadata['origin type'] = origin_info['type'] content_metadata['origin url'] = origin_info['url'] content_metadata['origin visit date'] = format_utc_iso_date(visit_info['date']) # noqa content_metadata['origin visit id'] = visit_info['visit'] browse_snapshot_url = reverse('browse-snapshot-content', url_args={'snapshot_id': snapshot_id, 'path': path}, query_params=request.GET) browse_snapshot_link = \ gen_link(browse_snapshot_url, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) content_metadata['snapshot context'] = browse_snapshot_link swh_objects = [{'type': 'content', 'id': cnt_sha1_git}, {'type': 'revision', 'id': revision_id}, {'type': 'snapshot', 'id': snapshot_id}] release_id = snapshot_context['release_id'] if release_id: swh_objects.append({'type': 'release', 'id': release_id}) swh_ids = get_swh_persistent_ids(swh_objects, snapshot_context) content_path = '/'.join([bc['name'] for bc in breadcrumbs]) context_found = 'snapshot: %s' % snapshot_context['snapshot_id'] if origin_info: context_found = 'origin: %s' % origin_info['url'] heading = 'Content - %s - %s - %s' %\ (content_path, snapshot_context['branch'], context_found) return render(request, 'browse/content.html', {'heading': heading, 'swh_object_name': 'Content', 'swh_object_metadata': content_metadata, 'content': content, 'content_size': content_size, 'max_content_size': content_display_max_size, 'mimetype': mimetype, 'language': language, 'breadcrumbs': breadcrumbs if root_sha1_git else [], 'top_right_link': { 'url': content_raw_url, 'icon': swh_object_icons['content'], 'text': 'Raw File' }, 'snapshot_context': snapshot_context, 'vault_cooking': None, 'show_actions_menu': True, 'swh_ids': swh_ids, 'error_code': error_code, 'error_message': error_message, 'error_description': error_description}, status=error_code) PER_PAGE = 100 def browse_snapshot_log(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None): """ Django view implementation for browsing a revision history in a snapshot context. """ try: snapshot_context = _process_snapshot_request(request, snapshot_id, origin_type, origin_url, timestamp, browse_context='log') # noqa revision_id = snapshot_context['revision_id'] - current_rev = revision_id - per_page = int(request.GET.get('per_page', PER_PAGE)) - revs_breadcrumb = request.GET.get('revs_breadcrumb', None) - if revs_breadcrumb: - current_rev = revs_breadcrumb.split('/')[-1] - revision_log = [] - if current_rev: - revision_log = list(service.lookup_revision_log(current_rev, - limit=per_page+1)) + per_page = int(request.GET.get('per_page', PER_PAGE)) + offset = int(request.GET.get('offset', 0)) + revs_ordering = request.GET.get('revs_ordering', 'committer_date') + session_key = 'rev_%s_log_ordering_%s' % (revision_id, revs_ordering) + rev_log_session = request.session.get(session_key, None) + rev_log = [] + revs_walker_state = None + if rev_log_session: + rev_log = rev_log_session['rev_log'] + revs_walker_state = rev_log_session['revs_walker_state'] + + if len(rev_log) < offset+per_page: + revs_walker = \ + service.get_revisions_walker(revs_ordering, + revision_id, + max_revs=offset+per_page+1, + state=revs_walker_state) + rev_log += list(revs_walker) + revs_walker_state = revs_walker.export_state() + + revision_log = rev_log[offset:offset+per_page] + + request.session[session_key] = { + 'rev_log': rev_log, + 'revs_walker_state': revs_walker_state + } except Exception as exc: return handle_view_exception(request, exc) swh_type = snapshot_context['swh_type'] origin_info = snapshot_context['origin_info'] visit_info = snapshot_context['visit_info'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] snapshot_id = snapshot_context['snapshot_id'] query_params['per_page'] = per_page + revs_ordering = request.GET.get('revs_ordering', '') + query_params['revs_ordering'] = revs_ordering - revision_log_data = [] - next_log_url = '' - prev_log_url = '' - if revision_log: - revision_log_display_data = prepare_revision_log_for_display( - revision_log, per_page, revs_breadcrumb, snapshot_context) - - browse_view_name = 'browse-' + swh_type + '-log' - - prev_rev = revision_log_display_data['prev_rev'] - prev_revs_breadcrumb = revision_log_display_data['prev_revs_breadcrumb'] # noqa - prev_log_url = None - query_params['revs_breadcrumb'] = prev_revs_breadcrumb - if prev_rev: - prev_log_url = \ - reverse(browse_view_name, - url_args=url_args, - query_params=query_params) - - next_rev = revision_log_display_data['next_rev'] - next_revs_breadcrumb = revision_log_display_data['next_revs_breadcrumb'] # noqa - next_log_url = None - query_params['revs_breadcrumb'] = next_revs_breadcrumb - if next_rev: - next_log_url = \ - reverse(browse_view_name, - url_args=url_args, - query_params=query_params) - - revision_log_data = revision_log_display_data['revision_log_data'] - - for i, log in enumerate(revision_log_data): - params = { - 'revision': revision_log[i]['id'], - } - if 'visit_id' in query_params: - params['visit_id'] = query_params['visit_id'] - log['directory'] = gen_snapshot_directory_link( - snapshot_context, revision_log[i]['id'], - link_text='Browse files', - link_attrs={'class': 'btn btn-default btn-sm', - 'role': 'button'}) + browse_view_name = 'browse-' + swh_type + '-log' + + prev_log_url = None + if len(rev_log) > offset + per_page: + query_params['offset'] = offset + per_page + prev_log_url = reverse(browse_view_name, + url_args=url_args, + query_params=query_params) + + next_log_url = None + if offset != 0: + query_params['offset'] = offset - per_page + next_log_url = reverse(browse_view_name, + url_args=url_args, + query_params=query_params) + + revision_log_data = format_log_entries(revision_log, per_page, + snapshot_context) browse_log_link = \ gen_revision_log_link(revision_id, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) revision_metadata = { 'context-independent revision history': browse_log_link, 'snapshot id': snapshot_id } if origin_info: revision_metadata['origin id'] = origin_info['id'] revision_metadata['origin type'] = origin_info['type'] revision_metadata['origin url'] = origin_info['url'] revision_metadata['origin visit date'] = format_utc_iso_date(visit_info['date']) # noqa revision_metadata['origin visit id'] = visit_info['visit'] browse_snapshot_url = reverse('browse-snapshot-log', url_args={'snapshot_id': snapshot_id}, query_params=request.GET) browse_snapshot_link = \ gen_link(browse_snapshot_url, link_text='Browse', link_attrs={'class': 'btn btn-default btn-sm', 'role': 'button'}) revision_metadata['snapshot context'] = browse_snapshot_link swh_objects = [{'type': 'revision', 'id': revision_id}, {'type': 'snapshot', 'id': snapshot_id}] release_id = snapshot_context['release_id'] if release_id: swh_objects.append({'type': 'release', 'id': release_id}) swh_ids = get_swh_persistent_ids(swh_objects, snapshot_context) context_found = 'snapshot: %s' % snapshot_context['snapshot_id'] if origin_info: context_found = 'origin: %s' % origin_info['url'] heading = 'Revision history - %s - %s' %\ (snapshot_context['branch'], context_found) return render(request, 'browse/revision-log.html', {'heading': heading, 'swh_object_name': 'Revisions history', 'swh_object_metadata': revision_metadata, 'revision_log': revision_log_data, + 'revs_ordering': revs_ordering, 'next_log_url': next_log_url, 'prev_log_url': prev_log_url, 'breadcrumbs': None, 'top_right_link': None, 'snapshot_context': snapshot_context, 'vault_cooking': None, 'show_actions_menu': True, 'swh_ids': swh_ids}) def browse_snapshot_branches(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None): """ Django view implementation for browsing a list of branches in a snapshot context. """ try: snapshot_context = _process_snapshot_request(request, snapshot_id, origin_type, origin_url, timestamp) branches_bc = request.GET.get('branches_breadcrumbs', '') branches_bc = \ branches_bc.split(',') if branches_bc else [] branches_from = branches_bc[-1] if branches_bc else '' swh_type = snapshot_context['swh_type'] origin_info = snapshot_context['origin_info'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] browse_view_name = 'browse-' + swh_type + '-directory' displayed_branches = \ service.lookup_snapshot(snapshot_context['snapshot_id'], branches_from, PER_PAGE+1, target_types=['revision'])['branches'] except Exception as exc: return handle_view_exception(request, exc) displayed_branches, _ = process_snapshot_branches(displayed_branches) for branch in displayed_branches: if snapshot_id: revision_url = reverse('browse-revision', url_args={'sha1_git': branch['revision']}, query_params={'snapshot_id': snapshot_id}) else: revision_url = reverse('browse-revision', url_args={'sha1_git': branch['revision']}, query_params={'origin_type': origin_type, 'origin': origin_info['url']}) query_params['branch'] = branch['name'] directory_url = reverse(browse_view_name, url_args=url_args, query_params=query_params) del query_params['branch'] branch['revision_url'] = revision_url branch['directory_url'] = directory_url browse_view_name = 'browse-' + swh_type + '-branches' prev_branches_url = None next_branches_url = None if branches_bc: query_params_prev = dict(query_params) query_params_prev['branches_breadcrumbs'] = \ ','.join(branches_bc[:-1]) prev_branches_url = reverse(browse_view_name, url_args=url_args, query_params=query_params_prev) elif branches_from: prev_branches_url = reverse(browse_view_name, url_args=url_args, query_params=query_params) if len(displayed_branches) > PER_PAGE: query_params_next = dict(query_params) next_branch = displayed_branches[-1]['name'] del displayed_branches[-1] branches_bc.append(next_branch) query_params_next['branches_breadcrumbs'] = \ ','.join(branches_bc) next_branches_url = reverse(browse_view_name, url_args=url_args, query_params=query_params_next) heading = 'Branches - ' if origin_info: heading += 'origin: %s' % origin_info['url'] else: heading += 'snapshot: %s' % snapshot_id return render(request, 'browse/branches.html', {'heading': heading, 'swh_object_name': 'Branches', 'swh_object_metadata': {}, 'top_right_link': None, 'displayed_branches': displayed_branches, 'prev_branches_url': prev_branches_url, 'next_branches_url': next_branches_url, 'snapshot_context': snapshot_context}) def browse_snapshot_releases(request, snapshot_id=None, origin_type=None, origin_url=None, timestamp=None): """ Django view implementation for browsing a list of releases in a snapshot context. """ try: snapshot_context = _process_snapshot_request(request, snapshot_id, origin_type, origin_url, timestamp) rel_bc = request.GET.get('releases_breadcrumbs', '') rel_bc = \ rel_bc.split(',') if rel_bc else [] rel_from = rel_bc[-1] if rel_bc else '' swh_type = snapshot_context['swh_type'] origin_info = snapshot_context['origin_info'] url_args = snapshot_context['url_args'] query_params = snapshot_context['query_params'] displayed_releases = \ service.lookup_snapshot(snapshot_context['snapshot_id'], rel_from, PER_PAGE+1, target_types=['release'])['branches'] except Exception as exc: return handle_view_exception(request, exc) _, displayed_releases = process_snapshot_branches(displayed_releases) for release in displayed_releases: if snapshot_id: query_params_tgt = {'snapshot_id': snapshot_id} else: query_params_tgt = {'origin': origin_info['url']} release_url = reverse('browse-release', url_args={'sha1_git': release['id']}, query_params=query_params_tgt) target_url = '' if release['target_type'] == 'revision': target_url = reverse('browse-revision', url_args={'sha1_git': release['target']}, query_params=query_params_tgt) elif release['target_type'] == 'directory': target_url = reverse('browse-directory', url_args={'sha1_git': release['target']}, query_params=query_params_tgt) elif release['target_type'] == 'content': target_url = reverse('browse-content', url_args={'sha1_git': release['target']}, query_params=query_params_tgt) elif release['target_type'] == 'release': target_url = reverse('browse-release', url_args={'sha1_git': release['target']}, query_params=query_params_tgt) release['release_url'] = release_url release['target_url'] = target_url browse_view_name = 'browse-' + swh_type + '-releases' prev_releases_url = None next_releases_url = None if rel_bc: query_params_prev = dict(query_params) query_params_prev['releases_breadcrumbs'] = \ ','.join(rel_bc[:-1]) prev_releases_url = reverse(browse_view_name, url_args=url_args, query_params=query_params_prev) elif rel_from: prev_releases_url = reverse(browse_view_name, url_args=url_args, query_params=query_params) if len(displayed_releases) > PER_PAGE: query_params_next = dict(query_params) next_rel = displayed_releases[-1]['branch_name'] del displayed_releases[-1] rel_bc.append(next_rel) query_params_next['releases_breadcrumbs'] = \ ','.join(rel_bc) next_releases_url = reverse(browse_view_name, url_args=url_args, query_params=query_params_next) heading = 'Releases - ' if origin_info: heading += 'origin: %s' % origin_info['url'] else: heading += 'snapshot: %s' % snapshot_id return render(request, 'browse/releases.html', {'heading': heading, 'top_panel_visible': False, 'top_panel_collapsible': False, 'swh_object_name': 'Releases', 'swh_object_metadata': {}, 'top_right_link': None, 'displayed_releases': displayed_releases, 'prev_releases_url': prev_releases_url, 'next_releases_url': next_releases_url, 'snapshot_context': snapshot_context, 'vault_cooking': None, 'show_actions_menu': False}) diff --git a/swh/web/common/service.py b/swh/web/common/service.py index 9ec735ee..ddcc7818 100644 --- a/swh/web/common/service.py +++ b/swh/web/common/service.py @@ -1,983 +1,1029 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import os from collections import defaultdict from swh.model import hashutil +from swh.storage.algos import revisions_walker + from swh.web.common import converters from swh.web.common import query from swh.web.common.exc import NotFoundExc from swh.web import config storage = config.storage() vault = config.vault() idx_storage = config.indexer_storage() MAX_LIMIT = 50 # Top limit the users can ask for def _first_element(l): """Returns the first element in the provided list or None if it is empty or None""" return next(iter(l or []), None) def lookup_multiple_hashes(hashes): """Lookup the passed hashes in a single DB connection, using batch processing. Args: An array of {filename: X, sha1: Y}, string X, hex sha1 string Y. Returns: The same array with elements updated with elem['found'] = true if the hash is present in storage, elem['found'] = false if not. """ hashlist = [hashutil.hash_to_bytes(elem['sha1']) for elem in hashes] content_missing = storage.content_missing_per_sha1(hashlist) missing = [hashutil.hash_to_hex(x) for x in content_missing] for x in hashes: x.update({'found': True}) for h in hashes: if h['sha1'] in missing: h['found'] = False return hashes def lookup_expression(expression, last_sha1, per_page): """Lookup expression in raw content. Args: expression (str): An expression to lookup through raw indexed content last_sha1 (str): Last sha1 seen per_page (int): Number of results per page Returns: List of ctags whose content match the expression """ limit = min(per_page, MAX_LIMIT) ctags = idx_storage.content_ctags_search(expression, last_sha1=last_sha1, limit=limit) for ctag in ctags: ctag = converters.from_swh(ctag, hashess={'id'}) ctag['sha1'] = ctag['id'] ctag.pop('id') yield ctag def lookup_hash(q): """Checks if the storage contains a given content checksum Args: query string of the form Returns: Dict with key found containing the hash info if the hash is present, None if not. """ algo, hash = query.parse_hash(q) found = storage.content_find({algo: hash}) return {'found': found, 'algo': algo} def search_hash(q): """Checks if the storage contains a given content checksum Args: query string of the form Returns: Dict with key found to True or False, according to whether the checksum is present or not """ algo, hash = query.parse_hash(q) found = storage.content_find({algo: hash}) return {'found': found is not None} def lookup_content_provenance(q): """Return provenance information from a specified content. Args: q: query string of the form Yields: provenance information (dict) list if the content is found. """ algo, hash = query.parse_hash(q) provenances = storage.content_find_provenance({algo: hash}) if not provenances: return None return (converters.from_provenance(p) for p in provenances) def _lookup_content_sha1(q): """Given a possible input, query for the content's sha1. Args: q: query string of the form Returns: binary sha1 if found or None """ algo, hash = query.parse_hash(q) if algo != 'sha1': hashes = storage.content_find({algo: hash}) if not hashes: return None return hashes['sha1'] return hash def lookup_content_ctags(q): """Return ctags information from a specified content. Args: q: query string of the form Yields: ctags information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None ctags = list(idx_storage.content_ctags_get([sha1])) if not ctags: return None for ctag in ctags: yield converters.from_swh(ctag, hashess={'id'}) def lookup_content_filetype(q): """Return filetype information from a specified content. Args: q: query string of the form Yields: filetype information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None filetype = _first_element(list(idx_storage.content_mimetype_get([sha1]))) if not filetype: return None return converters.from_filetype(filetype) def lookup_content_language(q): """Return language information from a specified content. Args: q: query string of the form Yields: language information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None lang = _first_element(list(idx_storage.content_language_get([sha1]))) if not lang: return None return converters.from_swh(lang, hashess={'id'}) def lookup_content_license(q): """Return license information from a specified content. Args: q: query string of the form Yields: license information (dict) list if the content is found. """ sha1 = _lookup_content_sha1(q) if not sha1: return None lic = _first_element(idx_storage.content_fossology_license_get([sha1])) if not lic: return None return converters.from_swh({'id': sha1, 'facts': lic[sha1]}, hashess={'id'}) def lookup_origin(origin): """Return information about the origin matching dict origin. Args: origin: origin's dict with keys either 'id' or ('type' AND 'url') Returns: origin information as dict. """ origin_info = storage.origin_get(origin) if not origin_info: if 'id' in origin and origin['id']: msg = 'Origin with id %s not found!' % origin['id'] else: msg = 'Origin with type %s and url %s not found!' % \ (origin['type'], origin['url']) raise NotFoundExc(msg) return converters.from_origin(origin_info) def search_origin(url_pattern, offset=0, limit=50, regexp=False, with_visit=False): """Search for origins whose urls contain a provided string pattern or match a provided regular expression. Args: url_pattern: the string pattern to search for in origin urls offset: number of found origins to skip before returning results limit: the maximum number of found origins to return Returns: list of origin information as dict. """ origins = storage.origin_search(url_pattern, offset, limit, regexp, with_visit) return map(converters.from_origin, origins) def lookup_person(person_id): """Return information about the person with id person_id. Args: person_id as string Returns: person information as dict. Raises: NotFoundExc if there is no person with the provided id. """ person = _first_element(storage.person_get([person_id])) if not person: raise NotFoundExc('Person with id %s not found' % person_id) return converters.from_person(person) def _to_sha1_bin(sha1_hex): _, sha1_git_bin = query.parse_hash_with_algorithms_or_throws( sha1_hex, ['sha1'], # HACK: sha1_git really 'Only sha1_git is supported.') return sha1_git_bin def lookup_directory(sha1_git): """Return information about the directory with id sha1_git. Args: sha1_git as string Returns: directory information as dict. """ empty_dir_sha1 = '4b825dc642cb6eb9a060e54bf8d69288fbee4904' if sha1_git == empty_dir_sha1: return [] sha1_git_bin = _to_sha1_bin(sha1_git) directory_entries = storage.directory_ls(sha1_git_bin) if directory_entries: return map(converters.from_directory_entry, directory_entries) else: raise NotFoundExc('Directory with sha1_git %s not found' % sha1_git) def lookup_directory_with_path(directory_sha1_git, path_string): """Return directory information for entry with path path_string w.r.t. root directory pointed by directory_sha1_git Args: - directory_sha1_git: sha1_git corresponding to the directory to which we append paths to (hopefully) find the entry - the relative path to the entry starting from the directory pointed by directory_sha1_git Raises: NotFoundExc if the directory entry is not found """ sha1_git_bin = _to_sha1_bin(directory_sha1_git) paths = path_string.strip(os.path.sep).split(os.path.sep) queried_dir = storage.directory_entry_get_by_path( sha1_git_bin, list(map(lambda p: p.encode('utf-8'), paths))) if not queried_dir: raise NotFoundExc(('Directory entry with path %s from %s not found') % (path_string, directory_sha1_git)) return converters.from_directory_entry(queried_dir) def lookup_release(release_sha1_git): """Return information about the release with sha1 release_sha1_git. Args: release_sha1_git: The release's sha1 as hexadecimal Returns: Release information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. """ sha1_git_bin = _to_sha1_bin(release_sha1_git) release = _first_element(storage.release_get([sha1_git_bin])) if not release: raise NotFoundExc('Release with sha1_git %s not found.' % release_sha1_git) return converters.from_release(release) def lookup_release_multiple(sha1_git_list): """Return information about the revisions identified with their sha1_git identifiers. Args: sha1_git_list: A list of revision sha1_git identifiers Returns: Release information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. """ sha1_bin_list = (_to_sha1_bin(sha1_git) for sha1_git in sha1_git_list) releases = storage.release_get(sha1_bin_list) or [] return (converters.from_release(r) for r in releases) def lookup_revision(rev_sha1_git): """Return information about the revision with sha1 revision_sha1_git. Args: revision_sha1_git: The revision's sha1 as hexadecimal Returns: Revision information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. NotFoundExc if there is no revision with the provided sha1_git. """ sha1_git_bin = _to_sha1_bin(rev_sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision with sha1_git %s not found.' % rev_sha1_git) return converters.from_revision(revision) def lookup_revision_multiple(sha1_git_list): """Return information about the revisions identified with their sha1_git identifiers. Args: sha1_git_list: A list of revision sha1_git identifiers Returns: Generator of revisions information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. """ sha1_bin_list = (_to_sha1_bin(sha1_git) for sha1_git in sha1_git_list) revisions = storage.revision_get(sha1_bin_list) or [] return (converters.from_revision(r) for r in revisions) def lookup_revision_message(rev_sha1_git): """Return the raw message of the revision with sha1 revision_sha1_git. Args: revision_sha1_git: The revision's sha1 as hexadecimal Returns: Decoded revision message as dict {'message': } Raises: ValueError if the identifier provided is not of sha1 nature. NotFoundExc if the revision is not found, or if it has no message """ sha1_git_bin = _to_sha1_bin(rev_sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision with sha1_git %s not found.' % rev_sha1_git) if 'message' not in revision: raise NotFoundExc('No message for revision with sha1_git %s.' % rev_sha1_git) res = {'message': revision['message']} return res def lookup_revision_by(origin_id, branch_name="refs/heads/master", timestamp=None): """Lookup revisions by origin_id, branch_name and timestamp. If: - branch_name is not provided, lookup using 'refs/heads/master' as default. - ts is not provided, use the most recent Args: - origin_id: origin of the revision. - branch_name: revision's branch. - timestamp: revision's time frame. Yields: The revisions matching the criterions. Raises: NotFoundExc if no revision corresponds to the criterion """ res = _first_element(storage.revision_get_by(origin_id, branch_name, timestamp=timestamp, limit=1)) if not res: raise NotFoundExc('Revision for origin %s and branch %s not found.' % (origin_id, branch_name)) return converters.from_revision(res) def lookup_revision_log(rev_sha1_git, limit): """Return information about the revision with sha1 revision_sha1_git. Args: revision_sha1_git: The revision's sha1 as hexadecimal limit: the maximum number of revisions returned Returns: Revision information as dict. Raises: ValueError if the identifier provided is not of sha1 nature. NotFoundExc if there is no revision with the provided sha1_git. """ sha1_git_bin = _to_sha1_bin(rev_sha1_git) revision_entries = storage.revision_log([sha1_git_bin], limit) if not revision_entries: raise NotFoundExc('Revision with sha1_git %s not found.' % rev_sha1_git) return map(converters.from_revision, revision_entries) def lookup_revision_log_by(origin_id, branch_name, timestamp, limit): """Return information about the revision with sha1 revision_sha1_git. Args: origin_id: origin of the revision branch_name: revision's branch timestamp: revision's time frame limit: the maximum number of revisions returned Returns: Revision information as dict. Raises: NotFoundExc if no revision corresponds to the criterion """ revision_entries = storage.revision_log_by(origin_id, branch_name, timestamp, limit=limit) if not revision_entries: return None return map(converters.from_revision, revision_entries) def lookup_revision_with_context_by(origin_id, branch_name, ts, sha1_git, limit=100): """Return information about revision sha1_git, limited to the sub-graph of all transitive parents of sha1_git_root. sha1_git_root being resolved through the lookup of a revision by origin_id, branch_name and ts. In other words, sha1_git is an ancestor of sha1_git_root. Args: - origin_id: origin of the revision. - branch_name: revision's branch. - timestamp: revision's time frame. - sha1_git: one of sha1_git_root's ancestors. - limit: limit the lookup to 100 revisions back. Returns: Pair of (root_revision, revision). Information on sha1_git if it is an ancestor of sha1_git_root including children leading to sha1_git_root Raises: - BadInputExc in case of unknown algo_hash or bad hash. - NotFoundExc if either revision is not found or if sha1_git is not an ancestor of sha1_git_root. """ rev_root = _first_element(storage.revision_get_by(origin_id, branch_name, timestamp=ts, limit=1)) if not rev_root: raise NotFoundExc('Revision with (origin_id: %s, branch_name: %s' ', ts: %s) not found.' % (origin_id, branch_name, ts)) return (converters.from_revision(rev_root), lookup_revision_with_context(rev_root, sha1_git, limit)) def lookup_revision_with_context(sha1_git_root, sha1_git, limit=100): """Return information about revision sha1_git, limited to the sub-graph of all transitive parents of sha1_git_root. In other words, sha1_git is an ancestor of sha1_git_root. Args: sha1_git_root: latest revision. The type is either a sha1 (as an hex string) or a non converted dict. sha1_git: one of sha1_git_root's ancestors limit: limit the lookup to 100 revisions back Returns: Information on sha1_git if it is an ancestor of sha1_git_root including children leading to sha1_git_root Raises: BadInputExc in case of unknown algo_hash or bad hash NotFoundExc if either revision is not found or if sha1_git is not an ancestor of sha1_git_root """ sha1_git_bin = _to_sha1_bin(sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision %s not found' % sha1_git) if isinstance(sha1_git_root, str): sha1_git_root_bin = _to_sha1_bin(sha1_git_root) revision_root = _first_element(storage.revision_get([sha1_git_root_bin])) # noqa if not revision_root: raise NotFoundExc('Revision root %s not found' % sha1_git_root) else: sha1_git_root_bin = sha1_git_root['id'] revision_log = storage.revision_log([sha1_git_root_bin], limit) parents = {} children = defaultdict(list) for rev in revision_log: rev_id = rev['id'] parents[rev_id] = [] for parent_id in rev['parents']: parents[rev_id].append(parent_id) children[parent_id].append(rev_id) if revision['id'] not in parents: raise NotFoundExc('Revision %s is not an ancestor of %s' % (sha1_git, sha1_git_root)) revision['children'] = children[revision['id']] return converters.from_revision(revision) def lookup_directory_with_revision(sha1_git, dir_path=None, with_data=False): """Return information on directory pointed by revision with sha1_git. If dir_path is not provided, display top level directory. Otherwise, display the directory pointed by dir_path (if it exists). Args: sha1_git: revision's hash. dir_path: optional directory pointed to by that revision. with_data: boolean that indicates to retrieve the raw data if the path resolves to a content. Default to False (for the api) Returns: Information on the directory pointed to by that revision. Raises: BadInputExc in case of unknown algo_hash or bad hash. NotFoundExc either if the revision is not found or the path referenced does not exist. NotImplementedError in case of dir_path exists but do not reference a type 'dir' or 'file'. """ sha1_git_bin = _to_sha1_bin(sha1_git) revision = _first_element(storage.revision_get([sha1_git_bin])) if not revision: raise NotFoundExc('Revision %s not found' % sha1_git) dir_sha1_git_bin = revision['directory'] if dir_path: paths = dir_path.strip(os.path.sep).split(os.path.sep) entity = storage.directory_entry_get_by_path( dir_sha1_git_bin, list(map(lambda p: p.encode('utf-8'), paths))) if not entity: raise NotFoundExc( "Directory or File '%s' pointed to by revision %s not found" % (dir_path, sha1_git)) else: entity = {'type': 'dir', 'target': dir_sha1_git_bin} if entity['type'] == 'dir': directory_entries = storage.directory_ls(entity['target']) or [] return {'type': 'dir', 'path': '.' if not dir_path else dir_path, 'revision': sha1_git, 'content': map(converters.from_directory_entry, directory_entries)} elif entity['type'] == 'file': # content content = storage.content_find({'sha1_git': entity['target']}) if with_data: c = _first_element(storage.content_get([content['sha1']])) content['data'] = c['data'] return {'type': 'file', 'path': '.' if not dir_path else dir_path, 'revision': sha1_git, 'content': converters.from_content(content)} else: raise NotImplementedError('Entity of type %s not implemented.' % entity['type']) def lookup_content(q): """Lookup the content designed by q. Args: q: The release's sha1 as hexadecimal Raises: NotFoundExc if the requested content is not found """ algo, hash = query.parse_hash(q) c = storage.content_find({algo: hash}) if not c: raise NotFoundExc('Content with %s checksum equals to %s not found!' % (algo, hashutil.hash_to_hex(hash))) return converters.from_content(c) def lookup_content_raw(q): """Lookup the content defined by q. Args: q: query string of the form Returns: dict with 'sha1' and 'data' keys. data representing its raw data decoded. Raises: NotFoundExc if the requested content is not found or if the content bytes are not available in the storage """ c = lookup_content(q) content = _first_element(storage.content_get([c['checksums']['sha1']])) if not content: algo, hash = query.parse_hash(q) raise NotFoundExc('Bytes of content with %s checksum equals to %s ' 'are not available!' % (algo, hashutil.hash_to_hex(hash))) return converters.from_content(content) def stat_counters(): """Return the stat counters for Software Heritage Returns: A dict mapping textual labels to integer values. """ return storage.stat_counters() def _lookup_origin_visits(origin_id, last_visit=None, limit=10): """Yields the origin origin_ids' visits. Args: origin_id (int): origin to list visits for last_visit (int): last visit to lookup from limit (int): Number of elements max to display Yields: Dictionaries of origin_visit for that origin """ limit = min(limit, MAX_LIMIT) yield from storage.origin_visit_get( origin_id, last_visit=last_visit, limit=limit) def lookup_origin_visits(origin_id, last_visit=None, per_page=10): """Yields the origin origin_ids' visits. Args: origin_id: origin to list visits for Yields: Dictionaries of origin_visit for that origin """ visits = _lookup_origin_visits(origin_id, last_visit=last_visit, limit=per_page) for visit in visits: yield converters.from_origin_visit(visit) def lookup_origin_visit(origin_id, visit_id): """Return information about visit visit_id with origin origin_id. Args: origin_id: origin concerned by the visit visit_id: the visit identifier to lookup Yields: The dict origin_visit concerned """ visit = storage.origin_visit_get_by(origin_id, visit_id) if not visit: raise NotFoundExc('Origin with id %s or its visit ' 'with id %s not found!' % (origin_id, visit_id)) return converters.from_origin_visit(visit) def lookup_snapshot_size(snapshot_id): """Count the number of branches in the snapshot with the given id Args: snapshot_id (str): sha1 identifier of the snapshot Returns: dict: A dict whose keys are the target types of branches and values their corresponding amount """ snapshot_id_bin = _to_sha1_bin(snapshot_id) snapshot_size = storage.snapshot_count_branches(snapshot_id_bin) if 'revision' not in snapshot_size: snapshot_size['revision'] = 0 if 'release' not in snapshot_size: snapshot_size['release'] = 0 return snapshot_size def lookup_snapshot(snapshot_id, branches_from='', branches_count=None, target_types=None): """Return information about a snapshot, aka the list of named branches found during a specific visit of an origin. Args: snapshot_id (str): sha1 identifier of the snapshot branches_from (str): optional parameter used to skip branches whose name is lesser than it before returning them branches_count (int): optional parameter used to restrain the amount of returned branches target_types (list): optional parameter used to filter the target types of branch to return (possible values that can be contained in that list are `'content', 'directory', 'revision', 'release', 'snapshot', 'alias'`) Returns: A dict filled with the snapshot content. """ snapshot_id_bin = _to_sha1_bin(snapshot_id) snapshot = storage.snapshot_get_branches(snapshot_id_bin, branches_from.encode(), branches_count, target_types) if not snapshot: raise NotFoundExc('Snapshot with id %s not found!' % snapshot_id) return converters.from_snapshot(snapshot) def lookup_latest_origin_snapshot(origin_id, allowed_statuses=None): """Return information about the latest snapshot of an origin. .. warning:: At most 1000 branches contained in the snapshot will be returned for performance reasons. Args: origin_id: integer identifier of the origin allowed_statuses: list of visit statuses considered to find the latest snapshot for the visit. For instance, ``allowed_statuses=['full']`` will only consider visits that have successfully run to completion. Returns: A dict filled with the snapshot content. """ snapshot = storage.snapshot_get_latest(origin_id, allowed_statuses) return converters.from_snapshot(snapshot) def lookup_entity_by_uuid(uuid): """Return the entity's hierarchy from its uuid. Args: uuid: entity's identifier. Returns: List of hierarchy entities from the entity with uuid. """ uuid = query.parse_uuid4(uuid) for entity in storage.entity_get(uuid): entity = converters.from_swh(entity, convert={'last_seen', 'uuid'}, convert_fn=lambda x: str(x)) yield entity def lookup_revision_through(revision, limit=100): """Retrieve a revision from the criterion stored in revision dictionary. Args: revision: Dictionary of criterion to lookup the revision with. Here are the supported combination of possible values: - origin_id, branch_name, ts, sha1_git - origin_id, branch_name, ts - sha1_git_root, sha1_git - sha1_git Returns: None if the revision is not found or the actual revision. """ if 'origin_id' in revision and \ 'branch_name' in revision and \ 'ts' in revision and \ 'sha1_git' in revision: return lookup_revision_with_context_by(revision['origin_id'], revision['branch_name'], revision['ts'], revision['sha1_git'], limit) if 'origin_id' in revision and \ 'branch_name' in revision and \ 'ts' in revision: return lookup_revision_by(revision['origin_id'], revision['branch_name'], revision['ts']) if 'sha1_git_root' in revision and \ 'sha1_git' in revision: return lookup_revision_with_context(revision['sha1_git_root'], revision['sha1_git'], limit) if 'sha1_git' in revision: return lookup_revision(revision['sha1_git']) # this should not happen raise NotImplementedError('Should not happen!') def lookup_directory_through_revision(revision, path=None, limit=100, with_data=False): """Retrieve the directory information from the revision. Args: revision: dictionary of criterion representing a revision to lookup path: directory's path to lookup. limit: optional query parameter to limit the revisions log (default to 100). For now, note that this limit could impede the transitivity conclusion about sha1_git not being an ancestor of. with_data: indicate to retrieve the content's raw data if path resolves to a content. Returns: The directory pointing to by the revision criterions at path. """ rev = lookup_revision_through(revision, limit) if not rev: raise NotFoundExc('Revision with criterion %s not found!' % revision) return (rev['id'], lookup_directory_with_revision(rev['id'], path, with_data)) def vault_cook(obj_type, obj_id, email=None): """Cook a vault bundle. """ return vault.cook(obj_type, obj_id, email=email) def vault_fetch(obj_type, obj_id): """Fetch a vault bundle. """ return vault.fetch(obj_type, obj_id) def vault_progress(obj_type, obj_id): """Get the current progress of a vault bundle. """ return vault.progress(obj_type, obj_id) def diff_revision(rev_id): """Get the list of file changes (insertion / deletion / modification / renaming) for a particular revision. """ rev_sha1_git_bin = _to_sha1_bin(rev_id) changes = storage.diff_revision(rev_sha1_git_bin, track_renaming=True) for change in changes: change['from'] = converters.from_directory_entry(change['from']) change['to'] = converters.from_directory_entry(change['to']) if change['from_path']: change['from_path'] = change['from_path'].decode('utf-8') if change['to_path']: change['to_path'] = change['to_path'].decode('utf-8') return changes + + +class _RevisionsWalkerProxy(object): + """ + Proxy class wrapping a revisions walker iterator from + swh-storage and performing needed conversions. + """ + def __init__(self, rev_walker_type, rev_start, *args, **kwargs): + rev_start_bin = hashutil.hash_to_bytes(rev_start) + self.revisions_walker = \ + revisions_walker.get_revisions_walker(rev_walker_type, + storage, + rev_start_bin, + *args, **kwargs) + + def export_state(self): + return self.revisions_walker.export_state() + + def __next__(self): + return converters.from_revision(next(self.revisions_walker)) + + def __iter__(self): + return self + + +def get_revisions_walker(rev_walker_type, rev_start, *args, **kwargs): + """ + Utility function to instantiate a revisions walker of a given type, + see :mod:`swh.storage.algos.revisions_walker`. + + Args: + rev_walker_type (str): the type of revisions walker to return, + possible values are: *committer_date*, *dfs*, *dfs_post*, + *bfs* and *path* + rev_start (str): hexadecimal representation of a revision identifier + args (list): position arguments to pass to the revisions walker + constructor + kwargs (dict): keyword arguments to pass to the revisions walker + constructor + + """ + # first check if the provided revision is valid + lookup_revision(rev_start) + return _RevisionsWalkerProxy(rev_walker_type, rev_start, *args, **kwargs) diff --git a/swh/web/settings/common.py b/swh/web/settings/common.py index b941fbe8..b62e388f 100644 --- a/swh/web/settings/common.py +++ b/swh/web/settings/common.py @@ -1,234 +1,235 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information """ Django common settings for swh-web. """ import os from swh.web.config import get_config swh_web_config = get_config() # Build paths inside the project like this: os.path.join(BASE_DIR, ...) PROJECT_DIR = os.path.dirname(os.path.abspath(__file__)) # Quick-start development settings - unsuitable for production # See https://docs.djangoproject.com/en/1.11/howto/deployment/checklist/ # SECURITY WARNING: keep the secret key used in production secret! SECRET_KEY = swh_web_config['secret_key'] # SECURITY WARNING: don't run with debug turned on in production! DEBUG = swh_web_config['debug'] DEBUG_PROPAGATE_EXCEPTIONS = swh_web_config['debug'] ALLOWED_HOSTS = ['127.0.0.1', 'localhost'] + swh_web_config['allowed_hosts'] # Application definition INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'rest_framework', 'swh.web.common', 'swh.web.api', 'swh.web.browse', 'webpack_loader', 'django_js_reverse' ] MIDDLEWARE = [ 'django.middleware.security.SecurityMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'swh.web.common.middlewares.ThrottlingHeadersMiddleware' ] # Compress all assets (static ones and dynamically generated html) # served by django in a local development environment context. # In a production environment, assets compression will be directly # handled by web servers like apache or nginx. if swh_web_config['serve_assets']: MIDDLEWARE.insert(0, 'django.middleware.gzip.GZipMiddleware') ROOT_URLCONF = 'swh.web.urls' TEMPLATES = [ { 'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [os.path.join(PROJECT_DIR, "../templates")], 'APP_DIRS': True, 'OPTIONS': { 'context_processors': [ 'django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', 'swh.web.common.utils.context_processor' ], 'libraries': { 'swh_templatetags': 'swh.web.common.swh_templatetags', }, }, }, ] WSGI_APPLICATION = 'swh.web.wsgi.application' - DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': os.path.join(PROJECT_DIR, 'db.sqlite3'), } } # Password validation # https://docs.djangoproject.com/en/1.11/ref/settings/#auth-password-validators AUTH_PASSWORD_VALIDATORS = [ { 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', # noqa }, { 'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator', # noqa }, { 'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator', # noqa }, { 'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator', # noqa }, ] # Internationalization # https://docs.djangoproject.com/en/1.11/topics/i18n/ LANGUAGE_CODE = 'en-us' TIME_ZONE = 'UTC' USE_I18N = True USE_L10N = True USE_TZ = True # Static files (CSS, JavaScript, Images) # https://docs.djangoproject.com/en/1.11/howto/static-files/ STATIC_URL = '/static/' STATICFILES_DIRS = [ os.path.join(PROJECT_DIR, "../static") ] INTERNAL_IPS = ['127.0.0.1'] throttle_rates = {} http_requests = ['GET', 'HEAD', 'POST', 'PUT', 'DELETE', 'OPTIONS', 'PATCH'] throttling = swh_web_config['throttling'] for limiter_scope, limiter_conf in throttling['scopes'].items(): if 'default' in limiter_conf['limiter_rate']: throttle_rates[limiter_scope] = limiter_conf['limiter_rate']['default'] # for backward compatibility else: throttle_rates[limiter_scope] = limiter_conf['limiter_rate'] # register sub scopes specific for HTTP request types for http_request in http_requests: if http_request in limiter_conf['limiter_rate']: throttle_rates[limiter_scope + '_' + http_request.lower()] = \ limiter_conf['limiter_rate'][http_request] REST_FRAMEWORK = { 'DEFAULT_RENDERER_CLASSES': ( 'rest_framework.renderers.JSONRenderer', 'swh.web.api.renderers.YAMLRenderer', 'rest_framework.renderers.TemplateHTMLRenderer' ), 'DEFAULT_THROTTLE_CLASSES': ( 'swh.web.common.throttling.SwhWebRateThrottle', ), 'DEFAULT_THROTTLE_RATES': throttle_rates } LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'filters': { 'require_debug_false': { '()': 'django.utils.log.RequireDebugFalse', }, 'require_debug_true': { '()': 'django.utils.log.RequireDebugTrue', }, }, 'formatters': { 'verbose': { 'format': '[%(asctime)s] [%(levelname)s] %(request)s %(status_code)s', # noqa 'datefmt': "%d/%b/%Y %H:%M:%S" }, }, 'handlers': { 'console': { 'level': 'DEBUG', 'filters': ['require_debug_true'], 'class': 'logging.StreamHandler', }, 'file': { 'level': 'INFO', 'filters': ['require_debug_false'], 'class': 'logging.FileHandler', 'filename': os.path.join(swh_web_config['log_dir'], 'swh-web.log'), 'formatter': 'verbose' }, 'null': { 'class': 'logging.NullHandler', }, }, 'loggers': { 'django': { 'handlers': ['console', 'file'], 'level': 'DEBUG' if DEBUG else 'INFO', 'propagate': True, }, 'django.request': { 'handlers': ['file'], 'level': 'DEBUG' if DEBUG else 'INFO', 'propagate': False, }, 'django.db.backends': { 'handlers': ['null'], 'propagate': False } }, } WEBPACK_LOADER = { # noqa 'DEFAULT': { 'CACHE': False, 'BUNDLE_DIR_NAME': './', 'STATS_FILE': os.path.join(PROJECT_DIR, '../static/webpack-stats.json'), # noqa 'POLL_INTERVAL': 0.1, 'TIMEOUT': None, 'IGNORE': ['.+\.hot-update.js', '.+\.map'] } } LOGIN_URL = '/admin/login/' LOGIN_REDIRECT_URL = 'admin' + +SESSION_ENGINE = 'django.contrib.sessions.backends.cache' diff --git a/swh/web/templates/browse/revision-log.html b/swh/web/templates/browse/revision-log.html index cf7cbd40..4499265f 100644 --- a/swh/web/templates/browse/revision-log.html +++ b/swh/web/templates/browse/revision-log.html @@ -1,68 +1,126 @@ {% extends "./browse.html" %} {% comment %} Copyright (C) 2017-2018 The Software Heritage developers See the AUTHORS file at the top-level directory of this distribution License: GNU Affero General Public License version 3, or any later version See top-level LICENSE file for more information {% endcomment %} {% load render_bundle from webpack_loader %} {% load swh_templatetags %} {% block header %} {{ block.super }} {% render_bundle 'revision' %} {% endblock %} {% block swh-browse-content %} {% if snapshot_context %} {% include "includes/top-navigation.html" %} {% endif %} {% if snapshot_context and snapshot_context.is_empty %} {% include "includes/empty-snapshot.html" %} {% else %} + +
+ +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+
- - + + - {% for log in revision_log %} - - - - - - + {% for rev in revision_log %} + + + + + + {% endfor %}
Revision AuthorMessage DateMessageCommit Date
{{ log.revision }}{{ log.author }}{{ log.message }}{{ log.date }}{{ log.directory }}
+ + {{ rev.id }} + + + + {{ rev.author }} + + + + {{ rev.date }} + + + + {{ rev.message }} + + + + {{ rev.commit_date }} + +
+ + {% endif %} {% endblock %} {% block swh-browse-after-content %} {% if not snapshot_context or not snapshot_context.is_empty %} {% endif %} {% endblock %} diff --git a/swh/web/templates/browse/revision.html b/swh/web/templates/browse/revision.html index dc6cb004..ef856f5e 100644 --- a/swh/web/templates/browse/revision.html +++ b/swh/web/templates/browse/revision.html @@ -1,110 +1,111 @@ {% extends "./browse.html" %} {% comment %} Copyright (C) 2017-2018 The Software Heritage developers See the AUTHORS file at the top-level directory of this distribution License: GNU Affero General Public License version 3, or any later version See top-level LICENSE file for more information {% endcomment %} {% load static %} {% load swh_templatetags %} {% load render_bundle from webpack_loader %} {% block header %} {{ block.super }} {% render_bundle 'revision' %} {% endblock %} {% block swh-browse-content %}
- Revision {{ swh_object_metadata.id }} - authored by {{ swh_object_metadata.author }} on {{ swh_object_metadata.date }} + Revision {{ swh_object_metadata.id }} + authored by {{ swh_object_metadata.author }} on {{ swh_object_metadata.date }}, + committed by {{ swh_object_metadata.committer }} on {{ swh_object_metadata|key_value:'committer date' }} 
{{ message_header }}
{% if message_body %} 
{{ message_body }}
{% endif %}
{{ parents|length }} parent{% if parents|length > 1 %}s{% endif %} {% for parent in parents %} {{ parent.id|slice:":7" }} {% if not forloop.last %} + {% endif %} {% endfor %}
{% include "includes/top-navigation.html" %} {% if content_size %} {% include "includes/content-display.html" %} {% else %} {% include "includes/directory-display.html" %} {% endif %}
{% endblock %} {% block swh-browse-after-content %} {% include "includes/readme-display.html" %} {% endblock %} diff --git a/swh/web/tests/browse/test_utils.py b/swh/web/tests/browse/test_utils.py index 7696e659..a03b4182 100644 --- a/swh/web/tests/browse/test_utils.py +++ b/swh/web/tests/browse/test_utils.py @@ -1,418 +1,244 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information # flake8: noqa from unittest.mock import patch from swh.web.browse import utils from swh.web.common.exc import NotFoundExc from swh.web.common.utils import reverse from swh.web.tests.testcase import SWHWebTestCase from .views.data.revision_test_data import revision_history_log_test class SwhBrowseUtilsTestCase(SWHWebTestCase): def test_get_mimetype_and_encoding_for_content(self): text = b'Hello world!' self.assertEqual(utils.get_mimetype_and_encoding_for_content(text), ('text/plain', 'us-ascii')) @patch('swh.web.browse.utils.get_origin_visits') def test_get_origin_visit(self, mock_origin_visits): origin_info = { 'id': 2, 'type': 'git', 'url': 'https://github.com/foo/bar', } visits = \ [{'status': 'full', 'date': '2015-07-09T21:09:24+00:00', 'visit': 1, 'origin': origin_info['id'] }, {'status': 'full', 'date': '2016-02-23T18:05:23.312045+00:00', 'visit': 2, 'origin': origin_info['id'] }, {'status': 'full', 'date': '2016-03-28T01:35:06.554111+00:00', 'visit': 3, 'origin': origin_info['id'] }, {'status': 'full', 'date': '2016-06-18T01:22:24.808485+00:00', 'visit': 4, 'origin': origin_info['id'] }, {'status': 'full', 'date': '2016-08-14T12:10:00.536702+00:00', 'visit': 5, 'origin': origin_info['id'] }] mock_origin_visits.return_value = visits visit_id = 12 with self.assertRaises(NotFoundExc) as cm: visit = utils.get_origin_visit(origin_info, visit_id=visit_id) exception_text = cm.exception.args[0] self.assertIn('Visit with id %s' % visit_id, exception_text) self.assertIn('type %s' % origin_info['type'], exception_text) self.assertIn('url %s' % origin_info['url'], exception_text) visit = utils.get_origin_visit(origin_info, visit_id=2) self.assertEqual(visit, visits[1]) visit = utils.get_origin_visit( origin_info, visit_ts='2016-02-23T18:05:23.312045+00:00') self.assertEqual(visit, visits[1]) visit = utils.get_origin_visit( origin_info, visit_ts='2016-02-20') self.assertEqual(visit, visits[1]) visit = utils.get_origin_visit( origin_info, visit_ts='2016-06-18T01:22') self.assertEqual(visit, visits[3]) visit = utils.get_origin_visit( origin_info, visit_ts='2016-06-18 01:22') self.assertEqual(visit, visits[3]) visit = utils.get_origin_visit( origin_info, visit_ts=1466208000) self.assertEqual(visit, visits[3]) visit = utils.get_origin_visit( origin_info, visit_ts='2014-01-01') self.assertEqual(visit, visits[0]) visit = utils.get_origin_visit( origin_info, visit_ts='2018-01-01') self.assertEqual(visit, visits[-1]) @patch('swh.web.browse.utils.service') @patch('swh.web.browse.utils.get_origin_visit') def test_get_origin_visit_snapshot(self, mock_get_origin_visit, mock_service): mock_get_origin_visit.return_value = \ {'status': 'full', 'date': '2015-08-04T22:26:14.804009+00:00', 'visit': 1, 'origin': 1, 'snapshot': '584b2fe3ce6218a96892e73bd76c2966bbc2a797'} mock_service.lookup_snapshot.return_value = \ {'branches': { 'HEAD': { 'target': '9fbd21adbac36be869514e82e2e98505dc47219c', 'target_type': 'revision', 'target_url': '/api/1/revision/9fbd21adbac36be869514e82e2e98505dc47219c/' }, 'refs/heads/master': { 'target': '9fbd21adbac36be869514e82e2e98505dc47219c', 'target_type': 'revision', 'target_url': '/api/1/revision/9fbd21adbac36be869514e82e2e98505dc47219c/' }, 'refs/tags/0.10.0': { 'target': '7045404f3d1c54e6473c71bbb716529fbad4be24', 'target_type': 'release', 'target_url': '/api/1/release/7045404f3d1c54e6473c71bbb716529fbad4be24/' }, 'refs/tags/0.10.1': { 'target': 'c893f4549c367e68288b0eb74595050410aa0de7', 'target_type': 'release', 'target_url': '/api/1/release/c893f4549c367e68288b0eb74595050410aa0de7/' } }, 'id': '584b2fe3ce6218a96892e73bd76c2966bbc2a797'} mock_service.lookup_release_multiple.return_value = \ [{'name': '0.10.0', 'message': '0.10: The "Oh fuck it\'s PyCon" release\n', 'id': '7045404f3d1c54e6473c71bbb716529fbad4be24', 'date': '2014-04-10T23:01:28-04:00', 'target_type': 'revision', 'target': '6072557b6c10cd9a21145781e26ad1f978ed14b9'}, {'name': '0.10.1', 'message': 'Tagging 0.10.1\n', 'id': 'c893f4549c367e68288b0eb74595050410aa0de7', 'date': '2014-10-10T09:45:52-04:00', 'target_type': 'revision', 'target': 'ecc003b43433e5b46511157598e4857a761007bf'}] mock_service.lookup_revision_multiple.return_value = \ [{'date': '2015-08-04T13:16:54+03:00', 'directory': '828da2b80e41aa958b2c98526f4a1d2cc7d298b7', 'id': '9fbd21adbac36be869514e82e2e98505dc47219c', 'message': 'Merge pull request #678 from algernon'}, {'date': '2014-04-10T23:01:11-04:00', 'directory': '2df4cd84ecc65b50b1d5318d3727e02a39b8a4cf', 'id': '6072557b6c10cd9a21145781e26ad1f978ed14b9', 'message': '0.10: The "Oh fuck it\'s PyCon" release\n'}, {'date': '2014-10-10T09:45:23-04:00', 'directory': '28ba64f97ef709e54838ae482c2da2619a74a0bd', 'id': 'ecc003b43433e5b46511157598e4857a761007bf', 'message': '0.10.1\n'}] expected_result = ( [{'name': 'HEAD', 'message': 'Merge pull request #678 from algernon', 'date': '04 August 2015, 10:16 UTC', 'revision': '9fbd21adbac36be869514e82e2e98505dc47219c', 'directory': '828da2b80e41aa958b2c98526f4a1d2cc7d298b7'}, {'name': 'refs/heads/master', 'message': 'Merge pull request #678 from algernon', 'date': '04 August 2015, 10:16 UTC', 'revision': '9fbd21adbac36be869514e82e2e98505dc47219c', 'directory': '828da2b80e41aa958b2c98526f4a1d2cc7d298b7'}], [{'name': '0.10.0', 'branch_name': 'refs/tags/0.10.0', 'id': '7045404f3d1c54e6473c71bbb716529fbad4be24', 'message': '0.10: The "Oh fuck it\'s PyCon" release\n', 'date': '11 April 2014, 03:01 UTC', 'target_type': 'revision', 'target': '6072557b6c10cd9a21145781e26ad1f978ed14b9', 'directory': '2df4cd84ecc65b50b1d5318d3727e02a39b8a4cf'}, {'name': '0.10.1', 'branch_name': 'refs/tags/0.10.1', 'id': 'c893f4549c367e68288b0eb74595050410aa0de7', 'message': 'Tagging 0.10.1\n', 'date': '10 October 2014, 13:45 UTC', 'target_type': 'revision', 'target': 'ecc003b43433e5b46511157598e4857a761007bf', 'directory': '28ba64f97ef709e54838ae482c2da2619a74a0bd'}] ) origin_info = { 'id': 1, 'type': 'git', 'url': 'https://github.com/hylang/hy' } origin_visit_branches = \ utils.get_origin_visit_snapshot(origin_info, visit_id=1) lookup_release_calls = mock_service.lookup_release_multiple.call_args_list self.assertEqual(len(lookup_release_calls), 1) # Check that we looked up the two expected releases self.assertCountEqual(set(lookup_release_calls[0][0][0]), { '7045404f3d1c54e6473c71bbb716529fbad4be24', 'c893f4549c367e68288b0eb74595050410aa0de7', }) lookup_revision_calls = mock_service.lookup_revision_multiple.call_args_list self.assertEqual(len(lookup_revision_calls), 1) # Check that we looked up the three expected revisions self.assertCountEqual(set(lookup_revision_calls[0][0][0]), { '9fbd21adbac36be869514e82e2e98505dc47219c', '6072557b6c10cd9a21145781e26ad1f978ed14b9', 'ecc003b43433e5b46511157598e4857a761007bf', }) self.assertEqual(origin_visit_branches, expected_result) def test_gen_link(self): self.assertEqual(utils.gen_link('https://www.softwareheritage.org/', 'SWH'), 'SWH') def test_gen_person_link(self): person_id = 8221896 person_name = 'Antoine Lambert' person_url = reverse('browse-person', url_args={'person_id': person_id}) self.assertEqual(utils.gen_person_link(person_id, person_name), '%s' % (person_url, person_name)) def test_gen_revision_link(self): revision_id = '28a0bc4120d38a394499382ba21d6965a67a3703' revision_url = reverse('browse-revision', url_args={'sha1_git': revision_id}) self.assertEqual(utils.gen_revision_link(revision_id), '%s' % (revision_url, revision_id)) self.assertEqual(utils.gen_revision_link(revision_id, shorten_id=True), '%s' % (revision_url, revision_id[:7])) - - def test_prepare_revision_log_for_display_no_contex(self): - per_page = 10 - first_page_logs_data = revision_history_log_test[:per_page+1] - second_page_logs_data = revision_history_log_test[per_page:2*per_page+1] - third_page_logs_data = revision_history_log_test[2*per_page:3*per_page+1] - last_page_logs_data = revision_history_log_test[3*per_page:3*per_page+5] - - revision_log_display_data = utils.prepare_revision_log_for_display( - first_page_logs_data, per_page, None) - - self.assertEqual(revision_log_display_data['revision_log_data'], - utils._format_log_entries(first_page_logs_data, - per_page)) - - self.assertEqual(revision_log_display_data['prev_rev'], - first_page_logs_data[-1]['id']) - - self.assertEqual(revision_log_display_data['prev_revs_breadcrumb'], - first_page_logs_data[0]['id']) - - self.assertEqual(revision_log_display_data['next_rev'], None) - self.assertEqual(revision_log_display_data['next_revs_breadcrumb'], - None) - - old_prev_revs_bc = str(revision_log_display_data['prev_revs_breadcrumb']) - - revision_log_display_data = utils.prepare_revision_log_for_display( - second_page_logs_data, per_page, old_prev_revs_bc) - - self.assertEqual(revision_log_display_data['revision_log_data'], - utils._format_log_entries(second_page_logs_data, - per_page)) - - self.assertEqual(revision_log_display_data['prev_rev'], - second_page_logs_data[-1]['id']) - - self.assertEqual(revision_log_display_data['prev_revs_breadcrumb'], - old_prev_revs_bc + '/' + second_page_logs_data[0]['id']) - - self.assertEqual(revision_log_display_data['next_rev'], - old_prev_revs_bc) - self.assertEqual(revision_log_display_data['next_revs_breadcrumb'], - None) - - old_prev_revs_bc = str(revision_log_display_data['prev_revs_breadcrumb']) - - revision_log_display_data = utils.prepare_revision_log_for_display( - third_page_logs_data, per_page, old_prev_revs_bc) - - self.assertEqual(revision_log_display_data['revision_log_data'], - utils._format_log_entries(third_page_logs_data, per_page)) - - self.assertEqual(revision_log_display_data['prev_rev'], - third_page_logs_data[-1]['id']) - - self.assertEqual(revision_log_display_data['prev_revs_breadcrumb'], - old_prev_revs_bc + '/' + third_page_logs_data[0]['id']) - - self.assertEqual(revision_log_display_data['next_rev'], - old_prev_revs_bc.split('/')[-1]) - - self.assertEqual(revision_log_display_data['next_revs_breadcrumb'], - '/'.join(old_prev_revs_bc.split('/')[:-1])) - - old_prev_revs_bc = str(revision_log_display_data['prev_revs_breadcrumb']) - - revision_log_display_data = utils.prepare_revision_log_for_display( - last_page_logs_data, per_page, old_prev_revs_bc) - - self.assertEqual(revision_log_display_data['revision_log_data'], - utils._format_log_entries(last_page_logs_data, per_page)) - - self.assertEqual(revision_log_display_data['prev_rev'], - None) - - self.assertEqual(revision_log_display_data['prev_revs_breadcrumb'], - None) - - self.assertEqual(revision_log_display_data['next_rev'], old_prev_revs_bc.split('/')[-1]) - self.assertEqual(revision_log_display_data['next_revs_breadcrumb'], - '/'.join(old_prev_revs_bc.split('/')[:-1])) - - def test_prepare_revision_log_for_display_snapshot_context(self): - per_page = 10 - first_page_logs_data = revision_history_log_test[:per_page+1] - second_page_logs_data = revision_history_log_test[per_page:2*per_page+1] - third_page_logs_data = revision_history_log_test[2*per_page:3*per_page+1] - last_page_logs_data = revision_history_log_test[3*per_page:3*per_page+5] - - snapshot_context = { - 'origin_info': {'type': 'git', - 'url': 'https://github.com/git/git'}, - 'origin_type': 'git', - 'url_args': {}, - 'query_params': {} - } - - revision_log_display_data = utils.prepare_revision_log_for_display( - first_page_logs_data, per_page, None, snapshot_context=snapshot_context) - - self.assertEqual(revision_log_display_data['revision_log_data'], - utils._format_log_entries(first_page_logs_data, - per_page, snapshot_context=snapshot_context)) - - self.assertEqual(revision_log_display_data['prev_rev'], - first_page_logs_data[-1]['id']) - - self.assertEqual(revision_log_display_data['prev_revs_breadcrumb'], - first_page_logs_data[-1]['id']) - - self.assertEqual(revision_log_display_data['next_rev'], None) - self.assertEqual(revision_log_display_data['next_revs_breadcrumb'], - None) - - old_prev_revs_bc = str(revision_log_display_data['prev_revs_breadcrumb']) - - revision_log_display_data = utils.prepare_revision_log_for_display( - second_page_logs_data, per_page, old_prev_revs_bc, snapshot_context=snapshot_context) - - self.assertEqual(revision_log_display_data['revision_log_data'], - utils._format_log_entries(second_page_logs_data, - per_page, snapshot_context=snapshot_context)) - - self.assertEqual(revision_log_display_data['prev_rev'], - second_page_logs_data[-1]['id']) - - self.assertEqual(revision_log_display_data['prev_revs_breadcrumb'], - old_prev_revs_bc + '/' + second_page_logs_data[-1]['id']) - - self.assertEqual(revision_log_display_data['next_rev'], - old_prev_revs_bc) - self.assertEqual(revision_log_display_data['next_revs_breadcrumb'], - None) - - old_prev_revs_bc = str(revision_log_display_data['prev_revs_breadcrumb']) - - revision_log_display_data = utils.prepare_revision_log_for_display( - third_page_logs_data, per_page, old_prev_revs_bc, snapshot_context=snapshot_context) - - self.assertEqual(revision_log_display_data['revision_log_data'], - utils._format_log_entries(third_page_logs_data, per_page, - snapshot_context=snapshot_context)) - - self.assertEqual(revision_log_display_data['prev_rev'], - third_page_logs_data[-1]['id']) - - self.assertEqual(revision_log_display_data['prev_revs_breadcrumb'], - old_prev_revs_bc + '/' + third_page_logs_data[-1]['id']) - - self.assertEqual(revision_log_display_data['next_rev'], - old_prev_revs_bc.split('/')[-1]) - - self.assertEqual(revision_log_display_data['next_revs_breadcrumb'], - '/'.join(old_prev_revs_bc.split('/')[:-1])) - - old_prev_revs_bc = str(revision_log_display_data['prev_revs_breadcrumb']) - - revision_log_display_data = utils.prepare_revision_log_for_display( - last_page_logs_data, per_page, old_prev_revs_bc, snapshot_context=snapshot_context) - - self.assertEqual(revision_log_display_data['revision_log_data'], - utils._format_log_entries(last_page_logs_data, per_page, - snapshot_context=snapshot_context)) - - self.assertEqual(revision_log_display_data['prev_rev'], - None) - - self.assertEqual(revision_log_display_data['prev_revs_breadcrumb'], - None) - - self.assertEqual(revision_log_display_data['next_rev'], old_prev_revs_bc.split('/')[-1]) - self.assertEqual(revision_log_display_data['next_revs_breadcrumb'], - '/'.join(old_prev_revs_bc.split('/')[:-1])) diff --git a/swh/web/tests/browse/views/data/revision_test_data.py b/swh/web/tests/browse/views/data/revision_test_data.py index a34b2a5f..243d425e 100644 --- a/swh/web/tests/browse/views/data/revision_test_data.py +++ b/swh/web/tests/browse/views/data/revision_test_data.py @@ -1,859 +1,905 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information # flake8: noqa revision_id_test = '7bc08e1aa0b08cb23e18715a32aa38517ad34672' revision_metadata_test = \ {'id': '7bc08e1aa0b08cb23e18715a32aa38517ad34672', 'type': 'git', 'parents': ['bf3652b16b65c27db5243aa0d674e2de4a8ccde9', 'a952bb99a6830804d06c5b8e04b75c66100fbae9'], 'metadata': {}, 'committer': {'name': 'GitHub', 'fullname': 'GitHub ', 'id': 10932771, 'email': 'noreply@github.com'}, 'directory': 'ae59ceecf46367e8e4ad800e231fc76adc3afffb', 'date': '2017-05-04T13:27:13+02:00', 'merge': True, 'committer_date': '2017-05-04T13:27:13+02:00', 'author': {'name': 'Tobias Koppers', 'fullname': 'Tobias Koppers ', 'id': 141959, 'email': 'tobias.koppers@googlemail.com'}, 'message': 'Merge pull request #4816 from webpack/bugfix/hoist-immutable-export\n\nhoist exports', 'synthetic': False } revision_history_log_test = \ [{'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-05-04T13:27:13+02:00', 'date': '2017-05-04T13:27:13+02:00', 'directory': 'ae59ceecf46367e8e4ad800e231fc76adc3afffb', 'id': '7bc08e1aa0b08cb23e18715a32aa38517ad34672', 'merge': True, 'message': 'Merge pull request #4816 from ' 'webpack/bugfix/hoist-immutable-export\n' '\n' 'hoist exports', 'metadata': {}, 'parents': ['bf3652b16b65c27db5243aa0d674e2de4a8ccde9', 'a952bb99a6830804d06c5b8e04b75c66100fbae9'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer_date': '2017-05-04T12:17:13+02:00', 'date': '2017-05-04T12:17:13+02:00', 'directory': '344f6b38f021fa0bbd60ca06fe1cbc61164e7abe', 'id': 'bf3652b16b65c27db5243aa0d674e2de4a8ccde9', 'merge': False, 'message': '2.5.0\n', 'metadata': {}, 'parents': ['cd1cd29fba46bd0133db0ca89acbe6c6c0240323'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer_date': '2017-05-04T13:00:52+02:00', 'date': '2017-05-04T12:56:31+02:00', 'directory': 'ae59ceecf46367e8e4ad800e231fc76adc3afffb', 'id': 'a952bb99a6830804d06c5b8e04b75c66100fbae9', 'merge': False, 'message': 'change some magic numbers to hoist exports\n\nfixes #4753\n', 'metadata': {}, 'parents': ['bf3652b16b65c27db5243aa0d674e2de4a8ccde9'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-05-04T11:41:43+02:00', 'date': '2017-05-04T11:41:43+02:00', 'directory': 'fbc01dab452f80bf49d554cc920979a66839707f', 'id': 'cd1cd29fba46bd0133db0ca89acbe6c6c0240323', 'merge': True, 'message': 'Merge pull request #4815 from ' 'webpack/bugfix/extract-async-initial\n' '\n' "CommonsChunkPlugin in async mode doesn't select initial chunks", 'metadata': {}, 'parents': ['8bab88c50fcb87c749c244b9ab28a1fb7e173bea', 'b45588bc1197abbc309eb3705a4bf89960b001ae'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-05-04T10:50:03+02:00', 'date': '2017-05-04T10:50:03+02:00', 'directory': '74ed598c6b831fe7f68697f6ed761660a842b973', 'id': '8bab88c50fcb87c749c244b9ab28a1fb7e173bea', 'merge': True, 'message': 'Merge pull request #4814 from webpack/test/move-entry\n' '\n' 'add testcase for moving entry modules into the commons chunk', 'metadata': {}, 'parents': ['85dc98f17aa39d5d3337e3791bf25634a1f7e445', 'a244879a07e04e6b5951520ca3cd80c3ef160f8e'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer_date': '2017-05-04T10:50:46+02:00', 'date': '2017-05-04T10:50:46+02:00', 'directory': '735b9f6d54d62470944c4743d52931b308a9072c', 'id': 'b45588bc1197abbc309eb3705a4bf89960b001ae', 'merge': False, 'message': "CommonsChunkPlugin in async mode doesn't select initial chunks\n" '\n' 'fixes #4795\n', 'metadata': {}, 'parents': ['c91ba4949503de5ca9d98c98188c2654b095f2cb'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-05-04T08:04:41+02:00', 'date': '2017-05-04T08:04:41+02:00', 'directory': 'c3c24aeafabe1d441c37a2769e9b65ae10075925', 'id': '85dc98f17aa39d5d3337e3791bf25634a1f7e445', 'merge': True, 'message': 'Merge pull request #4813 from JLHwung/perf/date-now\n' '\n' 'Perf/use Date.now() instead of +new Date()/new Date().getTime()', 'metadata': {}, 'parents': ['c91ba4949503de5ca9d98c98188c2654b095f2cb', '6afc397b99aef338a9d66add4488ce03ea3f7a43'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer_date': '2017-05-04T10:01:04+02:00', 'date': '2017-05-04T10:01:04+02:00', 'directory': '4de09be1628ed81def03f78d8d832c93efdf0af4', 'id': 'a244879a07e04e6b5951520ca3cd80c3ef160f8e', 'merge': False, 'message': 'add testcase for moving entry modules into the commons chunk\n' '#4795\n', 'metadata': {}, 'parents': ['c91ba4949503de5ca9d98c98188c2654b095f2cb'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-30T09:31:17+02:00', 'date': '2017-04-30T09:31:17+02:00', 'directory': '16c9d449871efccef7a0f53b29bd6218cc769e30', 'id': 'c91ba4949503de5ca9d98c98188c2654b095f2cb', 'merge': True, 'message': 'Merge pull request #4791 from deificx/master\n' '\n' 'add option to lib/Stats.js to disable stack trace on errors and ' 'warnings', 'metadata': {}, 'parents': ['94ba75f7940836390c041846f2c334929ee14332', '84ea1ffd3d0892f3356ac0494947bbc2a0e39d51'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'i@jhuang.me', 'fullname': 'J Huang ', 'id': 12072398, 'name': 'J Huang'}, 'committer': {'email': 'i@jhuang.me', 'fullname': 'J Huang ', 'id': 12072398, 'name': 'J Huang'}, 'committer_date': '2017-05-04T10:19:54+08:00', 'date': '2017-05-04T10:19:54+08:00', 'directory': 'c3c24aeafabe1d441c37a2769e9b65ae10075925', 'id': '6afc397b99aef338a9d66add4488ce03ea3f7a43', 'merge': False, 'message': 'perf: use Date.now() instead of new Date().getTime()\n' '\n' 'new Date().getTime() is 2x slower than Date.now(), see ' 'https://jsperf.com/new-date-vs-date-now-vs-performance-now/6\n', 'metadata': {}, 'parents': ['94d0641ba40d65d5fcbd64414b6ea9f8a2589538'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-30T09:30:03+02:00', 'date': '2017-04-30T09:30:03+02:00', 'directory': 'f78fe6939706d761f0bc60f9d566bd414ef5c113', 'id': '94ba75f7940836390c041846f2c334929ee14332', 'merge': True, 'message': 'Merge pull request #4794 from ' 'ndresx/disable-manifest-json-pretty-print\n' '\n' 'Disable manifest.json pretty print', 'metadata': {}, 'parents': ['24ef6ea1b56b358caeb4b07476a909f4f86c2d8a', 'de87f93c1b050db59ffbeff3aed4ac3b3eb57da3'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'even.alander@videxio.com', 'fullname': 'Even Alander ', 'id': 3407016, 'name': 'Even Alander'}, 'committer': {'email': 'even.alander@videxio.com', 'fullname': 'Even Alander ', 'id': 3407016, 'name': 'Even Alander'}, 'committer_date': '2017-04-29T20:53:42+02:00', 'date': '2017-04-29T20:52:43+02:00', 'directory': '68fcb2d3c60aa32b708ce9ca9fa5c4b1f7a2532d', 'id': '84ea1ffd3d0892f3356ac0494947bbc2a0e39d51', 'merge': False, 'message': 'added error to stats.moduleTrace test name to trigger test cases ' 'corretly\n', 'metadata': {}, 'parents': ['8ad4386bdf9cd607310a4255dff8cb9ccd12afad'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'i@jhuang.me', 'fullname': 'J Huang ', 'id': 12072398, 'name': 'J Huang'}, 'committer': {'email': 'i@jhuang.me', 'fullname': 'J Huang ', 'id': 12072398, 'name': 'J Huang'}, 'committer_date': '2017-05-04T10:19:10+08:00', 'date': '2017-05-04T10:17:23+08:00', 'directory': '41965bffa3f73c8928beb2a6580c253adb9bd5cf', 'id': '94d0641ba40d65d5fcbd64414b6ea9f8a2589538', 'merge': False, 'message': 'perf: use Date.now() instead of +new Date()\n' '\n' '+new Date() is 2x slower than Date.now(), see ' 'https://jsperf.com/new-date-vs-date-now-vs-performance-now/6\n', 'metadata': {}, 'parents': ['c91ba4949503de5ca9d98c98188c2654b095f2cb'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-26T09:05:56+02:00', 'date': '2017-04-26T09:05:56+02:00', 'directory': 'aa78629e4aefc836b5a967db57cac29f79fea074', 'id': '24ef6ea1b56b358caeb4b07476a909f4f86c2d8a', 'merge': True, 'message': 'Merge pull request #4779 from ndresx/ignoreplugin-typo-fix\n' '\n' 'Fix typo in IgnorePlugin', 'metadata': {}, 'parents': ['34315db3ffe20b3d7e9a50885ea623fc90ac0861', 'aa172f0a359ab3c0b0987101c64c8b41b07d581f'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'ndresx@gmail.com', 'fullname': 'Martin Veith ', 'id': 14855385, 'name': 'Martin Veith'}, 'committer': {'email': 'ndresx@gmail.com', 'fullname': 'Martin Veith ', 'id': 14855385, 'name': 'Martin Veith'}, 'committer_date': '2017-04-28T21:34:38+02:00', 'date': '2017-04-28T21:34:38+02:00', 'directory': 'f78fe6939706d761f0bc60f9d566bd414ef5c113', 'id': 'de87f93c1b050db59ffbeff3aed4ac3b3eb57da3', 'merge': False, 'message': 'Disable manifest.json pretty print\n', 'metadata': {}, 'parents': ['24ef6ea1b56b358caeb4b07476a909f4f86c2d8a'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'even.alander@videxio.com', 'fullname': 'Even Alander ', 'id': 3407016, 'name': 'Even Alander'}, 'committer': {'email': 'even.alander@videxio.com', 'fullname': 'Even Alander ', 'id': 3407016, 'name': 'Even Alander'}, 'committer_date': '2017-04-29T19:43:45+02:00', 'date': '2017-04-29T19:43:45+02:00', 'directory': '893a65a9ad785d719c44a3e764d150bb47cba6c1', 'id': '8ad4386bdf9cd607310a4255dff8cb9ccd12afad', 'merge': False, 'message': 'test cases for stats.moduleTrace option\n', 'metadata': {}, 'parents': ['958156ae4201e75f8cb04e306d2a38b94d2a8a1d'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-23T14:25:42+02:00', 'date': '2017-04-23T14:25:42+02:00', 'directory': 'f511a1febf3d6b9b788080f3eb69ff3cb963758f', 'id': '34315db3ffe20b3d7e9a50885ea623fc90ac0861', 'merge': True, 'message': 'Merge pull request #3875 from webpack/test/circleci\n' '\n' 'Update circle.yml', 'metadata': {}, 'parents': ['6a26e9ba7f7f1be7e76054f219bf1e094f2c3264', '0f91f949e2d41ef5cb92493bcc6c1fa7578ac27d'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'ndresx@gmail.com', 'fullname': 'Martin Veith ', 'id': 14855385, 'name': 'Martin Veith'}, 'committer': {'email': 'ndresx@gmail.com', 'fullname': 'Martin Veith ', 'id': 14855385, 'name': 'Martin Veith'}, 'committer_date': '2017-04-25T22:07:43+02:00', 'date': '2017-04-25T22:07:43+02:00', 'directory': 'aa78629e4aefc836b5a967db57cac29f79fea074', 'id': 'aa172f0a359ab3c0b0987101c64c8b41b07d581f', 'merge': False, 'message': 'Fix typo in IgnorePlugin\n', 'metadata': {}, 'parents': ['34315db3ffe20b3d7e9a50885ea623fc90ac0861'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'even.alander@videxio.com', 'fullname': 'Even Alander ', 'id': 3407016, 'name': 'Even Alander'}, 'committer': {'email': 'even.alander@videxio.com', 'fullname': 'Even Alander ', 'id': 3407016, 'name': 'Even Alander'}, 'committer_date': '2017-04-29T19:43:15+02:00', 'date': '2017-04-29T19:43:15+02:00', 'directory': '9d7347c42e29afda2849996ca6b61cdc7ea1b5a1', 'id': '958156ae4201e75f8cb04e306d2a38b94d2a8a1d', 'merge': False, 'message': 'moduleTrace added to webpackOptionsSchema.json\n', 'metadata': {}, 'parents': ['41310135bb4e30a0a6f71eddd3d74419ec8512a7'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-23T12:57:02+02:00', 'date': '2017-04-23T12:57:02+02:00', 'directory': '71848e66952829c98ec4e57e2338be986390b850', 'id': '6a26e9ba7f7f1be7e76054f219bf1e094f2c3264', 'merge': True, 'message': 'Merge pull request #4693 from ' 'Travmatth/fix-4252-BannerPlugin-placeholder\n' '\n' 'Fix 4252 banner plugin placeholder', 'metadata': {}, 'parents': ['53bb15b1ed64f8636036f773100d502909bd1e6b', '08eca2fb4517bb2a14e21fa46e3291878c79be0e'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer_date': '2017-04-23T14:02:06+02:00', 'date': '2017-04-23T12:50:01+02:00', 'directory': '29cf488a00126c0e45c78b92f97cdbc733f4d07e', 'id': '0f91f949e2d41ef5cb92493bcc6c1fa7578ac27d', 'merge': False, 'message': 'improve circleci build\n', 'metadata': {}, 'parents': ['53bb15b1ed64f8636036f773100d502909bd1e6b'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'even.alander@videxio.com', 'fullname': 'Even Alander ', 'id': 3407016, 'name': 'Even Alander'}, 'committer': {'email': 'even.alander@videxio.com', 'fullname': 'Even Alander ', 'id': 3407016, 'name': 'Even Alander'}, 'committer_date': '2017-04-28T09:20:14+02:00', 'date': '2017-04-28T09:20:14+02:00', 'directory': 'f8b679bd62a33880d7f39f70233c3c14aead7c22', 'id': '41310135bb4e30a0a6f71eddd3d74419ec8512a7', 'merge': False, 'message': 'rename stats.stackTrace to stats.moduleTrace\n', 'metadata': {}, 'parents': ['7e4310a1759bd35834290e4c4af207ce9b05ffd9'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-23T12:10:20+02:00', 'date': '2017-04-23T12:10:20+02:00', 'directory': '1887ba14662a7e45567fa6daa39ab72975b429aa', 'id': '53bb15b1ed64f8636036f773100d502909bd1e6b', 'merge': True, 'message': 'Merge pull request #3934 from ' 'timse/refactor-watching-in-compiler\n' '\n' 'Refactor _done of Watching in compiler', 'metadata': {}, 'parents': ['b67d61abd6dd878f7a9aec4d10689c8b1678e28a', 'ab30c6b1c0100634ad46bcaf47ff12963e22aac9'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-23T12:00:53+02:00', 'date': '2017-04-23T12:00:53+02:00', 'directory': '4da5fda41a4b5cdbcf1fbb2604e9696c67106271', 'id': '08eca2fb4517bb2a14e21fa46e3291878c79be0e', 'merge': False, 'message': 'spacing', 'metadata': {}, 'parents': ['7061c2c76b984e9c7ab893bf4b14b4b102b5fa0d'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'even.alander@videxio.com', 'fullname': 'Even Alander ', 'id': 3407016, 'name': 'Even Alander'}, 'committer': {'email': 'even.alander@videxio.com', 'fullname': 'Even Alander ', 'id': 3407016, 'name': 'Even Alander'}, 'committer_date': '2017-04-27T10:22:20+02:00', 'date': '2017-04-27T10:22:20+02:00', 'directory': '21f3f99258fad8513cbdd40f549cb067362add27', 'id': '7e4310a1759bd35834290e4c4af207ce9b05ffd9', 'merge': False, 'message': 'add option to lib/Stats.js to disable stack trace on errors and ' 'warnings\n', 'metadata': {}, 'parents': ['24ef6ea1b56b358caeb4b07476a909f4f86c2d8a'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-23T12:04:28+02:00', 'date': '2017-04-23T12:04:28+02:00', 'directory': 'effe11da8d44198a760cab867351f66d4b0da7d1', 'id': 'b67d61abd6dd878f7a9aec4d10689c8b1678e28a', 'merge': True, 'message': 'Merge pull request #4755 from aretecode/lint-update\n' '\n' 'update node linting', 'metadata': {}, 'parents': ['a2ec4c8cea281715d3043a6dcb4198be039cfb54', '5c423d6feb6521c9b5274ba342b7621c453f35eb'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tim.sebastian@gmail.com', 'fullname': 'Tim Sebastian ', 'id': 3654888, 'name': 'Tim Sebastian'}, 'committer': {'email': 'tim.sebastian@gmail.com', 'fullname': 'Tim Sebastian ', 'id': 3654888, 'name': 'Tim Sebastian'}, 'committer_date': '2017-04-08T09:13:38+10:00', 'date': '2017-04-08T09:13:38+10:00', 'directory': '20805e5ff5776062b9108138c4dfce8124185f5a', 'id': 'ab30c6b1c0100634ad46bcaf47ff12963e22aac9', 'merge': False, 'message': 'pass stats to error handler again, to prevent breaking change\n', 'metadata': {}, 'parents': ['b863851ce969317688799b754131e3546206c7ad'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'trav.matth@gmail.com', 'fullname': 'Travis Matthews ', 'id': 439443, 'name': 'Travis Matthews'}, 'committer': {'email': 'trav.matth@gmail.com', 'fullname': 'Travis Matthews ', 'id': 439443, 'name': 'Travis Matthews'}, 'committer_date': '2017-04-19T09:46:34-04:00', 'date': '2017-04-19T09:46:34-04:00', 'directory': '3c9ea2ac7fec857634b7b46dc08b52a5670106c5', 'id': '7061c2c76b984e9c7ab893bf4b14b4b102b5fa0d', 'merge': False, 'message': 'added file test, remove unused [basename]\n', 'metadata': {}, 'parents': ['527f9434f74e63ae76ecbf18a4710589667dc731'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-23T12:02:55+02:00', 'date': '2017-04-23T12:02:55+02:00', 'directory': '5185e53216f3a836598f946f7302155f5bb50d02', 'id': 'a2ec4c8cea281715d3043a6dcb4198be039cfb54', 'merge': True, 'message': 'Merge pull request #4722 from ' 'willmendesneto/refactor-format-location\n' '\n' 'refactor(formatLocation): upgrade to ES6', 'metadata': {}, 'parents': ['1c5f3bf59f8a1e06bbe83b7459362f69bb5b7460', 'c826edde1be8af20a02166f792f79bbf2a231c4e'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-22T19:02:01+02:00', 'date': '2017-04-22T19:02:01+02:00', 'directory': '461c61e52405649d272d7e79b8a1abe27e5b33b2', 'id': '5c423d6feb6521c9b5274ba342b7621c453f35eb', 'merge': False, 'message': 'change supported node.js version to 4', 'metadata': {}, 'parents': ['219a95dddca951737d4b947fc2d59e3a2b5cbb23'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tim.sebastian@gmail.com', 'fullname': 'Tim Sebastian ', 'id': 3654888, 'name': 'Tim Sebastian'}, 'committer': {'email': 'tim.sebastian@gmail.com', 'fullname': 'Tim Sebastian ', 'id': 3654888, 'name': 'Tim Sebastian'}, 'committer_date': '2017-04-06T21:52:06+10:00', 'date': '2017-04-06T21:52:06+10:00', 'directory': '803b83492f4be6b472720dfd5dd0f45675213ac2', 'id': 'b863851ce969317688799b754131e3546206c7ad', 'merge': False, 'message': 'add getStats again - remove breaking change\n', 'metadata': {}, 'parents': ['2a1bcff9e6999d471747ab3575d82c33a0e5e082'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'trav.matth@gmail.com', 'fullname': 'Travis Matthews ', 'id': 439443, 'name': 'Travis Matthews'}, 'committer': {'email': 'trav.matth@gmail.com', 'fullname': 'Travis Matthews ', 'id': 439443, 'name': 'Travis Matthews'}, 'committer_date': '2017-04-19T09:28:25-04:00', 'date': '2017-04-19T09:28:25-04:00', 'directory': 'aafcbc664c51bf17dcfde6b7112efe95331c6fcd', 'id': '527f9434f74e63ae76ecbf18a4710589667dc731', 'merge': False, 'message': 'removed single quotes\n', 'metadata': {}, 'parents': ['c68bd16b265b6d5f8478e5728ea077ead90e8a37'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-23T11:49:28+02:00', 'date': '2017-04-23T11:49:28+02:00', 'directory': 'cf61c993ed0335a88d7975aba07ef743eaa7b845', 'id': '1c5f3bf59f8a1e06bbe83b7459362f69bb5b7460', 'merge': True, 'message': 'Merge pull request #4748 from ' 'aretecode/examples-dll-readme-update2\n' '\n' 'examples/dll readme updates', 'metadata': {}, 'parents': ['cbd493904f222abac1d33918b4686b0bc4de3a42', 'f223aecd05b47b9cfff34609f730c81d03034514'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'willmendesneto@gmail.com', 'fullname': 'Will Mendes ', 'id': 93385, 'name': 'Will Mendes'}, 'committer': {'email': 'willmendesneto@gmail.com', 'fullname': 'Will Mendes ', 'id': 93385, 'name': 'Will Mendes'}, 'committer_date': '2017-04-17T22:00:47+10:00', 'date': '2017-04-17T22:00:47+10:00', 'directory': '817b5de0d36aae8f3243b472e19b2fae1adf9173', 'id': 'c826edde1be8af20a02166f792f79bbf2a231c4e', 'merge': False, 'message': 'refactor(formatLocation): upgrade to ES6\n', 'metadata': {}, 'parents': ['82ddd16080663e3773bdac2a62082991acb8637e'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'aretecode@gmail.com', 'fullname': 'James ', 'id': 11020868, 'name': 'James'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-19T23:53:44-07:00', 'date': '2017-04-19T23:53:44-07:00', 'directory': 'fa17b78edd5fd19a589663489e91b14e2c6337c0', 'id': '219a95dddca951737d4b947fc2d59e3a2b5cbb23', 'merge': False, 'message': 'lint autofix the eslint file', 'metadata': {}, 'parents': ['cf666188298dfe75cc343018bf35b1c753fb6636'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tim.sebastian@gmail.com', 'fullname': 'Tim Sebastian ', 'id': 3654888, 'name': 'Tim Sebastian'}, 'committer': {'email': 'tim.sebastian@gmail.com', 'fullname': 'Tim Sebastian ', 'id': 3654888, 'name': 'Tim Sebastian'}, 'committer_date': '2017-04-06T21:46:07+10:00', 'date': '2017-01-12T22:33:03+11:00', 'directory': '7c6b85c086b56f11cf29bc3d13836145dc225407', 'id': '2a1bcff9e6999d471747ab3575d82c33a0e5e082', 'merge': False, 'message': 'refactor the _done method of Watching\n' '\n' '- we can safely ignore the "else" cases of not having an error ' 'as _done() is only called without arguments if "this.invalid is ' 'true"\n' '- if we get passed the point of `this.invalid` either `err` or ' '`compilation` are !!always!! set. therefore later checks can ' 'again be ignored\n' '- early return in error case\n' '- ignore `this.error` if we make it passed the error as it will ' 'be unset at this point.\n' '- remove the setting of `this.error` or `this.stats` as the only ' 'use is inside this method and only allow weird behaviour if ' 'someone set them from outside\n', 'metadata': {}, 'parents': ['8165164d3e50fef3252d7c274cef1c0b595c6992'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'trav.matth@gmail.com', 'fullname': 'Travis Matthews ', 'id': 439443, 'name': 'Travis Matthews'}, 'committer': {'email': 'trav.matth@gmail.com', 'fullname': 'Travis Matthews ', 'id': 439443, 'name': 'Travis Matthews'}, 'committer_date': '2017-04-19T08:41:00-04:00', 'date': '2017-04-19T08:41:00-04:00', 'directory': '34bfecaa341626e1c60d1d2a1f697fb1f74f8bcd', 'id': 'c68bd16b265b6d5f8478e5728ea077ead90e8a37', 'merge': False, 'message': 'remove comment\n', 'metadata': {}, 'parents': ['f2e30693ac0e3a9a0fe6a01a89fe6072dee7f405'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-22T12:45:04+02:00', 'date': '2017-04-22T12:45:04+02:00', 'directory': '412c6f5c00ce7d02efcdd6f23160a36eb4e5a1f4', 'id': 'cbd493904f222abac1d33918b4686b0bc4de3a42', 'merge': True, 'message': 'Merge pull request #4768 from xizhao/patch-1\n' '\n' 'Add license scan report and status', 'metadata': {}, 'parents': ['d7f30392ddda27613e0ea05cb60ec985b4f75e5c', 'ff211108d888908d41470cea6187133ccdb56e87'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'aretecode@gmail.com', 'fullname': 'Arete Code ', 'id': 8227899, 'name': 'Arete Code'}, 'committer': {'email': 'aretecode@gmail.com', 'fullname': 'Arete Code ', 'id': 8227899, 'name': 'Arete Code'}, 'committer_date': '2017-04-21T15:48:06-07:00', 'date': '2017-04-21T15:48:06-07:00', 'directory': '9d69f66cf3d2fa5119910bfd6a16f16e996795cc', 'id': 'f223aecd05b47b9cfff34609f730c81d03034514', 'merge': False, 'message': 'rebase for semicolon\n', 'metadata': {}, 'parents': ['914fe2c923533d43f5edff54b1704bbd3dd407bc'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'tobias.koppers@googlemail.com', 'fullname': 'Tobias Koppers ', 'id': 141959, 'name': 'Tobias Koppers'}, 'committer': {'email': 'noreply@github.com', 'fullname': 'GitHub ', 'id': 10932771, 'name': 'GitHub'}, 'committer_date': '2017-04-15T19:26:54+02:00', 'date': '2017-04-15T19:26:54+02:00', 'directory': 'c2da553b359190d3dc4266d77e311bd002d7ccd2', 'id': '82ddd16080663e3773bdac2a62082991acb8637e', 'merge': True, 'message': 'Merge pull request #4717 from STRML/fix/hashSalt\n' '\n' 'fix missing `hashSalt` from options schema', 'metadata': {}, 'parents': ['bd753567da1248624beaaea14af31d6dbe303411', '805c9fadf05c00f3df16d1faf4d3f608b78e3b59'], 'synthetic': False, 'type': 'git'}, {'author': {'email': 'aretecode@gmail.com', 'fullname': 'Arete Code ', 'id': 8227899, 'name': 'Arete Code'}, 'committer': {'email': 'aretecode@gmail.com', 'fullname': 'Arete Code ', 'id': 8227899, 'name': 'Arete Code'}, 'committer_date': '2017-04-19T19:05:28-07:00', 'date': '2017-04-19T19:05:28-07:00', 'directory': 'dd7dc17a8a28b6bb415b1f6980de6fcd3715847c', 'id': 'cf666188298dfe75cc343018bf35b1c753fb6636', 'merge': False, 'message': '👕 update node linting\n' '\n' 'update linting for destructuring and latest node\n' '\n' 'uses .js rather than .eslintrc file\n' '\n' 'pulls in rules from\n' 'https://github.com/webpack/webpack-cli/pull/46/files#diff-df39304d828831c44a2b9f38cd45289cR40\n' '\n' 'adds spacing for this\n' 'screen shot 2017-04-19 at 7 03 16 pm\n', 'metadata': {}, 'parents': ['8aa8a7b63fed3c7909a6d4f15159c036a0561d64'], 'synthetic': False, - 'type': 'git'} + 'type': 'git'}, + {'author': { + 'email': 'tobias.koppers@googlemail.com', + 'fullname': 'Tobias Koppers ', + 'id': 141959, + 'name': 'Tobias Koppers' + }, + 'committer': { + 'email': 'noreply@github.com', + 'fullname': 'GitHub ', + 'id': 10932771, + 'name': 'GitHub' + }, + 'committer_date': '2017-04-21T19:00:50+02:00', + 'committer_url': '/api/1/person/10932771/', + 'date': '2017-04-21T19:00:50+02:00', + 'directory': '52cf6f28b1dbfe98f485ea78ae03942f55cd8fa0', + 'id': 'd7f30392ddda27613e0ea05cb60ec985b4f75e5c', + 'merge': True, + 'message': 'Merge pull request #4729 from simon04/provide-plugin-es2015\n\nProvidePlugin: add test case for ES2015 modules', + 'metadata': {}, + 'parents': ['88f37348e7de240d794713c0c38170f17a0a8c0e', + 'd0bbf967fb51c031e16c5dfe040afce9a4113b5b'], + 'synthetic': False, + 'type': 'git'}, + {'author': { + 'email': 'kevin@fossa.io', + 'fullname': 'Kevin Wang ', + 'id': 14855611, + 'name': 'Kevin Wang' + }, + 'committer': { + 'email': 'noreply@github.com', + 'fullname': 'GitHub ', + 'id': 10932771, + 'name': 'GitHub' + }, + 'committer_date': '2017-04-21T18:28:41-07:00', + 'date': '2017-04-21T18:28:41-07:00', + 'directory': '412c6f5c00ce7d02efcdd6f23160a36eb4e5a1f4', + 'id': 'ff211108d888908d41470cea6187133ccdb56e87', + 'merge': False, + 'message': 'Add license scan report and status', + 'metadata': {}, + 'parents': ['d7f30392ddda27613e0ea05cb60ec985b4f75e5c'], + 'synthetic': False, + 'type': 'git'}, ] \ No newline at end of file diff --git a/swh/web/tests/browse/views/test_revision.py b/swh/web/tests/browse/views/test_revision.py index 2fb69797..4535fcce 100644 --- a/swh/web/tests/browse/views/test_revision.py +++ b/swh/web/tests/browse/views/test_revision.py @@ -1,279 +1,255 @@ # Copyright (C) 2017-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information # flake8: noqa -from unittest.mock import patch +from unittest.mock import patch, MagicMock from django.utils.html import escape from swh.web.common.exc import NotFoundExc from swh.web.common.utils import ( - reverse, format_utc_iso_date, get_swh_persistent_id + reverse, format_utc_iso_date, get_swh_persistent_id, + parse_timestamp ) from swh.web.tests.testcase import SWHWebTestCase from .data.revision_test_data import ( revision_id_test, revision_metadata_test, revision_history_log_test ) from .data.origin_test_data import stub_origin_visits, stub_origin_snapshot class SwhBrowseRevisionTest(SWHWebTestCase): @patch('swh.web.browse.utils.get_origin_visit_snapshot') @patch('swh.web.browse.views.revision.service') @patch('swh.web.browse.utils.service') @patch('swh.web.common.utils.service') def test_revision_browse(self, mock_service_common, mock_service_utils, mock_service, mock_get_origin_visit_snapshot): mock_service.lookup_revision.return_value = revision_metadata_test url = reverse('browse-revision', url_args={'sha1_git': revision_id_test}) author_id = revision_metadata_test['author']['id'] author_name = revision_metadata_test['author']['name'] committer_id = revision_metadata_test['committer']['id'] committer_name = revision_metadata_test['committer']['name'] dir_id = revision_metadata_test['directory'] author_url = reverse('browse-person', url_args={'person_id': author_id}) committer_url = reverse('browse-person', url_args={'person_id': committer_id}) directory_url = reverse('browse-directory', url_args={'sha1_git': dir_id}) history_url = reverse('browse-revision-log', url_args={'sha1_git': revision_id_test}) resp = self.client.get(url) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('browse/revision.html') self.assertContains(resp, '%s' % (author_url, author_name)) self.assertContains(resp, '%s' % (committer_url, committer_name)) self.assertContains(resp, directory_url) self.assertContains(resp, history_url) for parent in revision_metadata_test['parents']: parent_url = reverse('browse-revision', url_args={'sha1_git': parent}) self.assertContains(resp, '%s' % (parent_url, parent)) author_date = revision_metadata_test['date'] committer_date = revision_metadata_test['committer_date'] message_lines = revision_metadata_test['message'].split('\n') self.assertContains(resp, format_utc_iso_date(author_date)) self.assertContains(resp, format_utc_iso_date(committer_date)) self.assertContains(resp, message_lines[0]) self.assertContains(resp, '\n'.join(message_lines[1:])) origin_info = { 'id': '7416001', 'type': 'git', 'url': 'https://github.com/webpack/webpack' } mock_service_utils.lookup_origin.return_value = origin_info mock_service_common.lookup_origin_visits.return_value = stub_origin_visits mock_get_origin_visit_snapshot.return_value = stub_origin_snapshot mock_service_utils.lookup_snapshot_size.return_value = { 'revision': len(stub_origin_snapshot[0]), 'release': len(stub_origin_snapshot[1]) } mock_service_common.MAX_LIMIT = 20 origin_directory_url = reverse('browse-origin-directory', - url_args={'origin_type': origin_info['type'], - 'origin_url': origin_info['url']}, + url_args={'origin_url': origin_info['url']}, query_params={'revision': revision_id_test}) origin_revision_log_url = reverse('browse-origin-log', - url_args={'origin_type': origin_info['type'], - 'origin_url': origin_info['url']}, + url_args={'origin_url': origin_info['url']}, query_params={'revision': revision_id_test}) url = reverse('browse-revision', url_args={'sha1_git': revision_id_test}, - query_params={'origin_type': origin_info['type'], - 'origin': origin_info['url']}) + query_params={'origin': origin_info['url']}) resp = self.client.get(url) self.assertContains(resp, origin_directory_url) self.assertContains(resp, origin_revision_log_url) for parent in revision_metadata_test['parents']: parent_url = reverse('browse-revision', url_args={'sha1_git': parent}, - query_params={'origin_type': origin_info['type'], - 'origin': origin_info['url']}) + query_params={'origin': origin_info['url']}) self.assertContains(resp, '%s' % (parent_url, parent)) self.assertContains(resp, 'vault-cook-directory') self.assertContains(resp, 'vault-cook-revision') swh_rev_id = get_swh_persistent_id('revision', revision_id_test) swh_rev_id_url = reverse('browse-swh-id', url_args={'swh_id': swh_rev_id}) self.assertContains(resp, swh_rev_id) self.assertContains(resp, swh_rev_id_url) swh_dir_id = get_swh_persistent_id('directory', dir_id) swh_dir_id_url = reverse('browse-swh-id', url_args={'swh_id': swh_dir_id}) self.assertContains(resp, swh_dir_id) self.assertContains(resp, swh_dir_id_url) @patch('swh.web.browse.views.revision.service') def test_revision_log_browse(self, mock_service): per_page = 10 - mock_service.lookup_revision_log.return_value = \ - revision_history_log_test[:per_page+1] + revision_history_log_test_sorted = \ + sorted(revision_history_log_test, + key=lambda rev: -parse_timestamp(rev['committer_date']).timestamp()) + + mock_revs_walker = MagicMock() + mock_revs_walker.__iter__.return_value = revision_history_log_test_sorted + mock_revs_walker.export_state.return_value = {} + + mock_service.get_revisions_walker.return_value = mock_revs_walker url = reverse('browse-revision-log', url_args={'sha1_git': revision_id_test}, query_params={'per_page': per_page}) resp = self.client.get(url) - prev_rev = revision_history_log_test[per_page]['id'] next_page_url = reverse('browse-revision-log', - url_args={'sha1_git': prev_rev}, - query_params={'revs_breadcrumb': revision_id_test, + url_args={'sha1_git': revision_id_test}, + query_params={'offset': per_page, 'per_page': per_page}) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('browse/revision-log.html') - self.assertContains(resp, '', + self.assertContains(resp, 'Newer') - self.assertContains(resp, '
  • Older
    • ' % + self.assertContains(resp, 'Newer') + self.assertContains(resp, 'Older' % escape(next_page_url)) - for log in revision_history_log_test[:per_page]: + for log in revision_history_log_test_sorted[:per_page]: author_url = reverse('browse-person', url_args={'person_id': log['author']['id']}) revision_url = reverse('browse-revision', url_args={'sha1_git': log['id']}) - directory_url = reverse('browse-directory', - url_args={'sha1_git': log['directory']}) - self.assertContains(resp, '%s' % - (author_url, log['author']['name'])) - self.assertContains(resp, '%s' % - (revision_url, log['id'][:7])) - self.assertContains(resp, directory_url) - - mock_service.lookup_revision_log.return_value = \ - revision_history_log_test[per_page:2*per_page+1] + self.assertContains(resp, log['id'][:7]) + self.assertContains(resp, log['author']['name']) + self.assertContains(resp, format_utc_iso_date(log['date'])) + self.assertContains(resp, escape(log['message'])) + self.assertContains(resp, format_utc_iso_date(log['committer_date'])) + self.assertContains(resp, revision_url) resp = self.client.get(next_page_url) - prev_prev_rev = revision_history_log_test[2*per_page]['id'] prev_page_url = reverse('browse-revision-log', url_args={'sha1_git': revision_id_test}, query_params={'per_page': per_page}) next_page_url = reverse('browse-revision-log', - url_args={'sha1_git': prev_prev_rev}, - query_params={'revs_breadcrumb': revision_id_test + '/' + prev_rev, + url_args={'sha1_git': revision_id_test}, + query_params={'offset': 2 * per_page, 'per_page': per_page}) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('browse/revision-log.html') - self.assertContains(resp, '', + self.assertContains(resp, 'Newer' % + self.assertContains(resp, 'Newer' % escape(prev_page_url)) - self.assertContains(resp, '
  • Older
    • ' % + self.assertContains(resp, 'Older' % escape(next_page_url)) - mock_service.lookup_revision_log.return_value = \ - revision_history_log_test[2*per_page:3*per_page+1] - resp = self.client.get(next_page_url) - prev_prev_prev_rev = revision_history_log_test[3*per_page]['id'] prev_page_url = reverse('browse-revision-log', - url_args={'sha1_git': prev_rev}, - query_params={'revs_breadcrumb': revision_id_test, + url_args={'sha1_git': revision_id_test}, + query_params={'offset': per_page, 'per_page': per_page}) next_page_url = reverse('browse-revision-log', - url_args={'sha1_git': prev_prev_prev_rev}, - query_params={'revs_breadcrumb': revision_id_test + '/' + prev_rev + '/' + prev_prev_rev, + url_args={'sha1_git': revision_id_test}, + query_params={'offset': 3 * per_page, 'per_page': per_page}) self.assertEqual(resp.status_code, 200) self.assertTemplateUsed('browse/revision-log.html') - self.assertContains(resp, '', + self.assertContains(resp, 'Newer' % + self.assertContains(resp, 'Newer' % escape(prev_page_url)) - self.assertContains(resp, '
  • Older
    • ' % + self.assertContains(resp, 'Older' % escape(next_page_url)) - mock_service.lookup_revision_log.return_value = \ - revision_history_log_test[3*per_page:3*per_page+per_page//2] - - resp = self.client.get(next_page_url) - - prev_page_url = reverse('browse-revision-log', - url_args={'sha1_git': prev_prev_rev}, - query_params={'revs_breadcrumb': revision_id_test + '/' + prev_rev, - 'per_page': per_page}) - - self.assertEqual(resp.status_code, 200) - self.assertTemplateUsed('browse/revision-log.html') - self.assertContains(resp, '', - count=per_page//2) - self.assertContains(resp, '
  • Older
    • ') - self.assertContains(resp, '
  • Newer
    • ' % - escape(prev_page_url)) - @patch('swh.web.browse.utils.service') @patch('swh.web.browse.views.revision.service') def test_revision_request_errors(self, mock_service, mock_utils_service): mock_service.lookup_revision.side_effect = \ NotFoundExc('Revision not found') url = reverse('browse-revision', url_args={'sha1_git': revision_id_test}) resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') self.assertContains(resp, 'Revision not found', status_code=404) - mock_service.lookup_revision_log.side_effect = \ + mock_service.get_revisions_walker.side_effect = \ NotFoundExc('Revision not found') url = reverse('browse-revision-log', url_args={'sha1_git': revision_id_test}) resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') self.assertContains(resp, 'Revision not found', status_code=404) url = reverse('browse-revision', url_args={'sha1_git': revision_id_test}, query_params={'origin_type': 'git', 'origin': 'https://github.com/foo/bar'}) mock_service.lookup_revision.side_effect = None mock_utils_service.lookup_origin.side_effect = \ NotFoundExc('Origin not found') resp = self.client.get(url) self.assertEqual(resp.status_code, 404) self.assertTemplateUsed('error.html') self.assertContains(resp, 'Origin not found', status_code=404)