You must be logged in to submit an add forge request. Please log in + href="{% url login_url %}?next={% url 'forge-add-create' %}"> + log in +
+ {% else %}Once an add-forge-request is submitted, its status can be viewed in the submitted requests list. This process involves a moderator approval and might take a few days to handle (it primarily depends on the response time from the forge).
{% endif %}You can jump directly to the endpoint index , which lists all available API functionalities, or read on for more general information about the API.
The Software Heritage project harvests publicly available source code by tracking software distribution channels such as version control systems, tarball releases, and distribution packages.
All retrieved source code and related metadata are stored in the Software Heritage archive, that is conceptually a Merkle DAG. All nodes in the graph are content-addressable, i.e., their node identifiers are computed by hashing their content and, transitively, that of all nodes reachable from them; and no node or edge is ever removed from the graph: the Software Heritage archive is an append-only data structure.
The following types of objects (i.e., graph nodes) can be found in the Software Heritage archive (for more information see the Software Heritage glossary):
The current version of the API isv1.
Warning: this version of the API is not to be considered stable yet. Non-backward compatible changes might happen even without changing the API version number.
API access is over HTTPS.
All API endpoints are rooted at https://archive.softwareheritage.org/api/1/.
Data is sent and received as JSON by default.
Example:
from the command line:
curl -i https://archive.softwareheritage.org/api/1/stat/counters/
The response format can be overridden using the Accept request header. In
particular, Accept: text/html (that web browsers send by default) requests HTML
pretty-printing, whereas Accept: application/yaml requests YAML-encoded responses.
Example:
from the command line:
curl -i -H 'Accept: application/yaml' https://archive.softwareheritage.org/api/1/stat/counters/Some API endpoints can be tweaked by passing optional parameters. For GET requests, optional parameters can be passed as an HTTP query string.
The optional parameter fields is accepted by all endpoints that return dictionaries
and can be used to restrict the list of fields returned by the API, in case you are not
interested in all of them. By default, all available fields are returned.
Example:
from the command line:
curl https://archive.softwareheritage.org/api/1/stat/counters/?fields=content,directory,revisionWhile API endpoints will return different kinds of errors depending on their own semantics, some error patterns are common across all endpoints.
Sending malformed data, including syntactically incorrect object identifiers, will result in a
400 Bad Request HTTP response. Example:
from the command line:
curl -i https://archive.softwareheritage.org/api/1/content/deadbeef/
Requesting non existent resources will result in a 404 Not Found HTTP response.
Example:
from the command line:
curl -i https://archive.softwareheritage.org/api/1/content/04740277a81c5be6c16f6c9da488ca073b770d7f/
Unavailability of the underlying storage backend will result in a 503 Service
Unavailable HTTP response.
While attempting to decode UTF-8 strings from raw bytes stored in the archive, some errors might
happen when generating an API response. In that case, an extra field
decoding_failures will be added to each concerned JSON object (possibly nested). It
will contain the list of its key names where UTF-8 decoding failed.
A string that could not be decoded will have the bytes of its invalid UTF-8 sequences escaped as
\\x<hex value>.
Requests that might potentially return many items will be paginated.
Page size is set to a default (usually: 10 items), but might be overridden with the
per_page query parameter up to a maximum (usually: 50 items). Example:
curl https://archive.softwareheritage.org/api/1/origin/1/visits/?per_page=2
To navigate through paginated results, a Link HTTP response header is available to
link the current result page to the next one. Example:
curl -i https://archive.softwareheritage.org/api/1/origin/1/visits/?per_page=2 | grep ^Link:
Link: </api/1/origin/1/visits/?last_visit=2&per_page=2>; rel="next",
Due to limited resource availability on the back end side, API usage is currently rate limited.
API users can be either anonymous or authenticated. For rate-limiting purposes, anonymous users
are identified by their origin IP address; authenticated users identify themselves via user-specific
credentials, like authentication tokens.
A higher rate-limit quota is available by default for authenticated users.
Three HTTP response fields will inform you about the current state of limits that apply to your current rate limiting bucket:
X-RateLimit-Limit: maximum number of permitted requests per hour
(120 for anonymous users, 1200 for authenticated users)
X-RateLimit-Remaining: number of permitted requests remaining before the next
reset
X-RateLimit-Reset: the time (expressed in Unix
time seconds) at which the current rate limiting will expire, resetting to a fresh
X-RateLimit-Limit
Example:
curl -i https://archive.softwareheritage.org/api/1/stat/counters/ | grep ^X-RateLimit
X-RateLimit-Limit: 120
X-RateLimit-Remaining: 119
X-RateLimit-Reset: 1620639052
- It is possible to perform authenticated requests to the Web API through the use of a bearer token
- sent in HTTP Authorization headers.
-
- To obtain such a token, an account to the
- Software Heritage Authentication service must be created.
-
- To generate and manage bearer tokens, a dedicated interface is available on the
- user profile page once logged in.
-
- The following shows how to perform an authenticated request to the Web API using curl.
-
export TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJhMTMxYTQ1My1hM2IyLTQwMTUtO...
-curl -H "Authorization: Bearer ${TOKEN}" {{ site_base_url }}api/...
-
- - Authenticated requests can be used to lift rate limiting if the user account has the adequate - permission. - If you are in such a need, please contact us - and we will review your request. -
+ {% if oidc_enabled %} +
+ It is possible to perform authenticated requests to the Web API through the use of a bearer token
+ sent in HTTP Authorization headers.
+
+ To obtain such a token, an account to the
+ Software Heritage Authentication service must be created.
+
+ To generate and manage bearer tokens, a dedicated interface is available on the
+ user profile page once logged in.
+
+ The following shows how to perform an authenticated request to the Web API using curl.
+
export TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJhMTMxYTQ1My1hM2IyLTQwMTUtO...
+ curl -H "Authorization: Bearer ${TOKEN}" {{ site_base_url }}api/...
+
+ + Authenticated requests can be used to lift rate limiting if the user account has the adequate + permission. + If you are in such a need, please contact us + and we will review your request. +
+ {% endif %}Your authenticated session expired so you have been automatically logged out.
{% else %}You have been successfully logged out.
{% endif %}-{% if oidc_enabled and 'remote_user' in request.GET %} -{% if 'next_path' in request.GET %} - +{% if 'next' in request.GET %} + {% else %} - -{% endif %} -{% else %} - + {% endif %} Log in again ?
-{% if 'next_path' in request.GET %} +{% if 'next' in request.GET %}Or go back to - + previous page.
{% endif %} {% endblock %} diff --git a/swh/web/auth/urls.py b/swh/web/auth/urls.py index f08e033f..3308836d 100644 --- a/swh/web/auth/urls.py +++ b/swh/web/auth/urls.py @@ -1,52 +1,75 @@ # Copyright (C) 2022 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information from django.contrib.auth.views import LoginView, LogoutView from django.urls import re_path as url from swh.auth.django.views import urlpatterns as auth_urlpatterns from swh.web.auth.views import ( oidc_generate_bearer_token, oidc_generate_bearer_token_complete, oidc_get_bearer_token, oidc_list_bearer_tokens, oidc_profile_view, oidc_revoke_bearer_tokens, ) +from swh.web.config import get_config -urlpatterns = auth_urlpatterns + [ - url( - r"^oidc/generate-bearer-token/$", - oidc_generate_bearer_token, - name="oidc-generate-bearer-token", - ), - url( - r"^oidc/generate-bearer-token-complete/$", - oidc_generate_bearer_token_complete, - name="oidc-generate-bearer-token-complete", - ), - url( - r"^oidc/list-bearer-token/$", - oidc_list_bearer_tokens, - name="oidc-list-bearer-tokens", - ), - url( - r"^oidc/get-bearer-token/$", - oidc_get_bearer_token, - name="oidc-get-bearer-token", - ), - url( - r"^oidc/revoke-bearer-tokens/$", - oidc_revoke_bearer_tokens, - name="oidc-revoke-bearer-tokens", - ), +config = get_config() + +oidc_enabled = bool(config["keycloak"]["server_url"]) + +urlpatterns = [] + +if not oidc_enabled: + urlpatterns = [ + url( + r"^login/$", + LoginView.as_view(template_name="login.html"), + name="login", + ) + ] + +if oidc_enabled or config["e2e_tests_mode"]: + urlpatterns += auth_urlpatterns + [ + url( + r"^oidc/generate-bearer-token/$", + oidc_generate_bearer_token, + name="oidc-generate-bearer-token", + ), + url( + r"^oidc/generate-bearer-token-complete/$", + oidc_generate_bearer_token_complete, + name="oidc-generate-bearer-token-complete", + ), + url( + r"^oidc/list-bearer-token/$", + oidc_list_bearer_tokens, + name="oidc-list-bearer-tokens", + ), + url( + r"^oidc/get-bearer-token/$", + oidc_get_bearer_token, + name="oidc-get-bearer-token", + ), + url( + r"^oidc/revoke-bearer-tokens/$", + oidc_revoke_bearer_tokens, + name="oidc-revoke-bearer-tokens", + ), + url( + r"^oidc/profile/$", + oidc_profile_view, + name="oidc-profile", + ), + ] + +urlpatterns.append( url( - r"^oidc/profile/$", - oidc_profile_view, - name="oidc-profile", - ), - url(r"^login/$", LoginView.as_view(template_name="login.html"), name="login"), - url(r"^logout/$", LogoutView.as_view(template_name="logout.html"), name="logout"), -] + r"^logout/$", + LogoutView.as_view(template_name="logout.html"), + name="logout", + ) +) diff --git a/swh/web/auth/views.py b/swh/web/auth/views.py index 0efd1c99..297edacd 100644 --- a/swh/web/auth/views.py +++ b/swh/web/auth/views.py @@ -1,155 +1,155 @@ # Copyright (C) 2020-2022 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import json from typing import Any, Dict, Union, cast from cryptography.fernet import InvalidToken from django.contrib.auth.decorators import login_required from django.core.paginator import Paginator from django.http import HttpRequest from django.http.response import ( HttpResponse, HttpResponseBadRequest, HttpResponseForbidden, HttpResponseRedirect, JsonResponse, ) from django.shortcuts import render from django.views.decorators.http import require_http_methods from swh.auth.django.models import OIDCUser from swh.auth.django.utils import keycloak_oidc_client from swh.auth.django.views import get_oidc_login_data, oidc_login_view from swh.auth.keycloak import KeycloakError, keycloak_error_message from swh.web.auth.models import OIDCUserOfflineTokens from swh.web.auth.utils import decrypt_data, encrypt_data from swh.web.config import get_config from swh.web.utils import reverse from swh.web.utils.exc import ForbiddenExc def oidc_generate_bearer_token(request: HttpRequest) -> HttpResponse: if not request.user.is_authenticated or not isinstance(request.user, OIDCUser): return HttpResponseForbidden() redirect_uri = reverse("oidc-generate-bearer-token-complete", request=request) return oidc_login_view( request, redirect_uri=redirect_uri, scope="openid offline_access" ) def oidc_generate_bearer_token_complete(request: HttpRequest) -> HttpResponse: if not request.user.is_authenticated or not isinstance(request.user, OIDCUser): raise ForbiddenExc("You are not allowed to generate bearer tokens.") if "error" in request.GET: raise Exception(request.GET["error"]) login_data = get_oidc_login_data(request) oidc_client = keycloak_oidc_client() oidc_profile = oidc_client.authorization_code( code=request.GET["code"], code_verifier=login_data["code_verifier"], redirect_uri=login_data["redirect_uri"], ) user = cast(OIDCUser, request.user) token = oidc_profile["refresh_token"] secret = get_config()["secret_key"].encode() salt = user.sub.encode() encrypted_token = encrypt_data(token.encode(), secret, salt) OIDCUserOfflineTokens.objects.create( user_id=str(user.id), offline_token=encrypted_token ).save() return HttpResponseRedirect(reverse("oidc-profile") + "#tokens") def oidc_list_bearer_tokens(request: HttpRequest) -> HttpResponse: if not request.user.is_authenticated or not isinstance(request.user, OIDCUser): return HttpResponseForbidden() tokens = OIDCUserOfflineTokens.objects.filter(user_id=str(request.user.id)) tokens = tokens.order_by("-creation_date") length = int(request.GET["length"]) page = int(request.GET["start"]) / length + 1 paginator = Paginator(tokens, length) tokens_data = [ {"id": t.id, "creation_date": t.creation_date.isoformat()} for t in paginator.page(int(page)).object_list ] table_data: Dict[str, Any] = {} table_data["recordsTotal"] = len(tokens_data) table_data["draw"] = int(request.GET["draw"]) table_data["data"] = tokens_data table_data["recordsFiltered"] = len(tokens_data) return JsonResponse(table_data) def _encrypted_token_bytes(token: Union[bytes, memoryview]) -> bytes: # token has been retrieved from a PosgreSQL database if isinstance(token, memoryview): return token.tobytes() else: return token @require_http_methods(["POST"]) def oidc_get_bearer_token(request: HttpRequest) -> HttpResponse: if not request.user.is_authenticated or not isinstance(request.user, OIDCUser): return HttpResponseForbidden() try: data = json.loads(request.body.decode("ascii")) user = cast(OIDCUser, request.user) token_data = OIDCUserOfflineTokens.objects.get(id=data["token_id"]) secret = get_config()["secret_key"].encode() salt = user.sub.encode() decrypted_token = decrypt_data( _encrypted_token_bytes(token_data.offline_token), secret, salt ) refresh_token = decrypted_token.decode("ascii") # check token is still valid oidc_client = keycloak_oidc_client() oidc_client.refresh_token(refresh_token) return HttpResponse(refresh_token, content_type="text/plain") except InvalidToken: return HttpResponse(status=401) except KeycloakError as ke: error_msg = keycloak_error_message(ke) if error_msg in ( "invalid_grant: Offline session not active", "invalid_grant: Offline user session not found", ): error_msg = "Bearer token has expired, please generate a new one." return HttpResponseBadRequest(error_msg, content_type="text/plain") @require_http_methods(["POST"]) def oidc_revoke_bearer_tokens(request: HttpRequest) -> HttpResponse: if not request.user.is_authenticated or not isinstance(request.user, OIDCUser): return HttpResponseForbidden() try: data = json.loads(request.body.decode("ascii")) user = cast(OIDCUser, request.user) for token_id in data["token_ids"]: token_data = OIDCUserOfflineTokens.objects.get(id=token_id) secret = get_config()["secret_key"].encode() salt = user.sub.encode() decrypted_token = decrypt_data( _encrypted_token_bytes(token_data.offline_token), secret, salt ) oidc_client = keycloak_oidc_client() oidc_client.logout(decrypted_token.decode("ascii")) token_data.delete() return HttpResponse(status=200) except InvalidToken: return HttpResponse(status=401) -@login_required(login_url="/oidc/login/", redirect_field_name="next_path") +@login_required(login_url="oidc-login") def oidc_profile_view(request: HttpRequest) -> HttpResponse: return render(request, "profile.html") diff --git a/swh/web/deposit/urls.py b/swh/web/deposit/urls.py index 84e747d9..7f82bd9a 100644 --- a/swh/web/deposit/urls.py +++ b/swh/web/deposit/urls.py @@ -1,48 +1,47 @@ # Copyright (C) 2018-2022 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU Affero General Public License version 3, or any later version # See top-level LICENSE file for more information import requests from requests.auth import HTTPBasicAuth -from django.conf import settings from django.contrib.auth.decorators import user_passes_test from django.http import JsonResponse from django.shortcuts import render from django.urls import re_path as url from swh.web.auth.utils import ADMIN_LIST_DEPOSIT_PERMISSION from swh.web.config import get_config def can_list_deposits(user): return user.is_staff or user.has_perm(ADMIN_LIST_DEPOSIT_PERMISSION) -@user_passes_test(can_list_deposits, login_url=settings.LOGIN_URL) +@user_passes_test(can_list_deposits) def admin_deposit(request): return render(request, "deposit-admin.html") -@user_passes_test(can_list_deposits, login_url=settings.LOGIN_URL) +@user_passes_test(can_list_deposits) def admin_deposit_list(request): config = get_config()["deposit"] private_api_url = config["private_api_url"].rstrip("/") + "/" deposits_list_url = private_api_url + "deposits/datatables/" deposits_list_auth = HTTPBasicAuth( config["private_api_user"], config["private_api_password"] ) deposits = requests.get( deposits_list_url, auth=deposits_list_auth, params=request.GET, timeout=30 ).json() return JsonResponse(deposits) urlpatterns = [ url(r"^admin/deposit/$", admin_deposit, name="admin-deposit"), url(r"^admin/deposit/list/$", admin_deposit_list, name="admin-deposit-list"), ] diff --git a/swh/web/save_code_now/templates/origin-save-help.html b/swh/web/save_code_now/templates/origin-save-help.html index 05c9fadb..115e5dcf 100644 --- a/swh/web/save_code_now/templates/origin-save-help.html +++ b/swh/web/save_code_now/templates/origin-save-help.html @@ -1,54 +1,54 @@ {% extends "./origin-save.html" %} {% comment %} Copyright (C) 2018-2021 The Software Heritage developers See the AUTHORS file at the top-level directory of this distribution License: GNU Affero General Public License version 3, or any later version See top-level LICENSE file for more information {% endcomment %} {% block tab_content %}A "Save code now" request takes the following parameters:
git, for origins using Githg, for origins using Mercurialsvn, for origins using Subversioncvs, for origins using CVSbzr, for origins using Bazaargit
origin into the archive, you should check that the command $ git clone <origin_url>Once submitted, your save request can either be:
Once a save request has been accepted, you can follow its current status in the
submitted save requests list.
- If you submitted requests while authenticated, you will be able
+ If you submitted requests while authenticated, you will be able
to only display your own requests.
