Page MenuHomeSoftware Heritage
Paste Active Pastes
  • journal:
    brokers:
    - kafka1.internal.softwareheritage.org:9094
    - kafka2.internal.softwareheritage.org:9094
    - kafka3.internal.softwareheritage.org:9094
    ...
    • Apr 12 2022, 5:27 PM
    • 11 Lines
  • softwareheritage=> SELECT count(*) FROM origin_visit ov INNER JOIN origin o ON o.id = ov.origin WHERE o.url = 'https://subversion.renater.fr/anonscm/svn/panda';
    count
    --------
    132706
    (1 row)
    • Apr 12 2022, 12:02 PM
    • 5 Lines
  • class SupportsObjectType(Protocol):
    object_type: str
    _L: List[SupportsObjectType] = [
    Person,
    ...
    • Apr 11 2022, 10:29 AM
    • 29 Lines
    • Python
  • Traceback (most recent call last):
    File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
    File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
    ...
    • Apr 8 2022, 5:37 PM
    • 26 Lines
  • 13:42 $ swh search -C ~/.config/swh/search.yml rpc-serve
    * Serving Flask app 'swh.search.api.server' (lazy loading)
    * Environment: production
    WARNING: This is a development server. Do not use it in a production deployment.
    Use a production WSGI server instead.
    ...
    • Apr 7 2022, 1:45 PM
    • 35 Lines
  • P1333 query 3
    query test {
    origin(url: "https://github.com/python/cpython") {
    url
    id
    visits(first: 2, after: "Mg==") {
    ...
    • Apr 6 2022, 6:01 PM
    • 50 Lines
  • P1332 query 2
    query rels {
    release(SWHId: "d28343dc3ad53a411ae3685e7d6a7866c8c22d6b") {
    id
    date
    name
    ...
    • Apr 6 2022, 5:58 PM
    • 15 Lines
  • P1331 query1
    query snapshots {
    snapshot(SWHId: "6f1e3067acba64891af07680367a38cc1b9cb57a") {
    branches(first: 10) {
    nodes {
    name
    ...
    • Apr 6 2022, 5:57 PM
    • 31 Lines
  • #!/usr/bin/env bash
    set -xe
    DBPORT=5432
    ...
    • Apr 6 2022, 4:36 PM
    • 15 Lines
  • $ sudo pidof dnsmasq
    240165 240164 240049 240048
    $ ls /var/lib/libvirt
    boot dnsmasq images qemu sanlock
    $ ls /var/lib/libvirt/dnsmasq
    ...
    • Apr 4 2022, 5:08 PM
    • 25 Lines
  • version: '2'
    services:
    ...
    ...
    • Apr 4 2022, 1:30 PM
    • 8 Lines
  • diff --git java/src/main/java/org/softwareheritage/graph/compress/LabelMapBuilder.java java/src/main/java/org/softwareheritage/graph/compress/LabelMapBuilder.java
    index 600870e..0825266 100644
    --- java/src/main/java/org/softwareheritage/graph/compress/LabelMapBuilder.java
    +++ java/src/main/java/org/softwareheritage/graph/compress/LabelMapBuilder.java
    @@ -143,7 +143,7 @@ public class LabelMapBuilder {
    ...
    • Apr 1 2022, 4:45 PM
    • 104 Lines
    • Diff
  • query testquery {
    origin(url: "https://github.com/python/cpython") {
    url
    id
    visits(first: 2, after: "Mg==") {
    ...
    • Apr 1 2022, 3:36 PM
    • 45 Lines
  • (swh) parallels@ubuntu-linux-20-04-desktop:~/Desktop/swh/swh-environment/docker$ sudo docker-compose restart
    Restarting docker_swh-web-cron_1 ...
    Restarting docker_swh-vault-worker_1 ...
    Restarting docker_swh-indexer-journal-client_1 ...
    Restarting docker_swh-lister_1 ...
    ...
    • Apr 1 2022, 10:02 AM
    • 304 Lines
  • diff --git a/swh/web/admin/deposit.py b/swh/web/admin/deposit.py
    index ddbfe304..1d653f36 100644
    --- a/swh/web/admin/deposit.py
    +++ b/swh/web/admin/deposit.py
    @@ -3,21 +3,18 @@
    ...
    • Mar 30 2022, 6:04 PM
    • 132 Lines
  • diff --git a/swh/deposit/api/private/deposit_list.py b/swh/deposit/api/private/deposit_list.py
    index 4bb40e31..e8c07e31 100644
    --- a/swh/deposit/api/private/deposit_list.py
    +++ b/swh/deposit/api/private/deposit_list.py
    @@ -3,10 +3,20 @@
    ...
    • Mar 30 2022, 6:03 PM
    • 162 Lines
  • # swh-core
    diff --git a/swh/core/db/common.py b/swh/core/db/common.py
    index 65e6d75..6f89058 100644
    --- a/swh/core/db/common.py
    +++ b/swh/core/db/common.py
    ...
    • Mar 30 2022, 1:45 PM
    • 76 Lines
  • make test-frontend-ui-cmd
    rm -f swh-web-test.sqlite3
    django-admin migrate --settings=swh.web.settings.tests
    Operations to perform:
    Apply all migrations: admin, auth, contenttypes, sessions, swh_web_add_forge_now, swh_web_auth, swh_web_common
    ...
    • Mar 29 2022, 5:50 PM
    • 49 Lines
  • ERROR: Command errored out with exit status 1:
    command: /home/parallels/.virtualenvs/swh/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0_w3ghla/confluent-kafka/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0_w3ghla/confluent-kafka/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-tx1t5lrk
    cwd: /tmp/pip-install-0_w3ghla/confluent-kafka/
    Complete output (116 lines):
    running bdist_wheel
    ...
    • Mar 28 2022, 10:14 PM
    • 122 Lines
  • codemeta:identifier does not allow propertyvalues (source: https://github.com/codemeta/codemeta/blob/master/codemeta.jsonld#L36 ; `"@type": "@id"` means the value must be a URI, but propertyvalues are not URIs), so we need to replace:
    <codemeta:identifier>
    <codemeta:type>schema:PropertyValue</codemeta:type>
    <schema:propertyID>HAL-ID</schema:propertyID>
    ...
    • Mar 28 2022, 4:26 PM
    • 15 Lines
  • 11:14 <KShivendu> That's great! Now try fixing some easy hacks and write a draft for your proposal and share it with us :)
    12:59 <KShivendu> Hey vsellier can you please tell me about how we have deployed the elasticsearch cluster. for example: what kind of a machine(s) is it, and where is it hosted?
    13:00 <KShivendu> also, did we face any challenge in this because of the scale? if yes, please briefly describe them :)
    13:25 <+ardumont> KShivendu: where services are exposed is described in https://docs.softwareheritage.org/sysadm/network-architecture/service-urls.html
    13:25 <+ardumont> ah he is not here
    ...
    • Mar 28 2022, 1:53 PM
    • 11 Lines
  • describe('Browse requests list tests', function() {
    beforeEach(function() {
    this.addForgeNowUrl = this.Urls.forge_add();
    this.listAddForgeRequestsUrl = this.Urls.add_forge_request_list_datatables();
    });
    ...
    • Mar 25 2022, 3:02 PM
    • 48 Lines
  • softwareheritage=> SELECT DISTINCT ON (ov.visit) ov.visit, ov.type, ov.date, ovs.date as status_date, ovs.status
    FROM origin_visit ov
    INNER JOIN origin o ON o.id = ov.origin
    INNER JOIN origin_visit_status ovs USING (origin, visit)
    WHERE o.url = 'https://github.com/python/cpython'
    ...
    • Mar 25 2022, 2:27 PM
    • 78 Lines
  • diff --git a/CHANGELOG.rst b/CHANGELOG.rst
    index e417bcd..198150f 100644
    --- a/CHANGELOG.rst
    +++ b/CHANGELOG.rst
    @@ -57,7 +57,7 @@ Added
    ...
    • Mar 22 2022, 4:43 PM
    • 148 Lines
    • Diff
  • new file cypress/fixtures/add-forge-now-create-request-fail.json
    @@ -0,0 +1,8 @@
    +{
    + "status": 400,
    + "ok": false,
    ...
    • Mar 18 2022, 3:13 PM
    • 43 Lines
  • make test-frontend-ui-cmd
    rm -f swh-web-test.sqlite3
    django-admin migrate --settings=swh.web.settings.tests -v0 2>/dev/null
    cat swh/web/tests/create_test_admin.py | django-admin shell --settings=swh.web.settings.tests
    cat swh/web/tests/create_test_users.py | django-admin shell --settings=swh.web.settings.tests
    ...
    • Mar 17 2022, 11:08 AM
    • 62 Lines
  • swh/web/tests/create_test_users.py:
    ```
    from swh.web.auth.utils import SWH_AMBASSADOR_PERMISSION, SWH_MODERATOR_PERMISSION
    ...
    • Mar 15 2022, 11:11 AM
    • 106 Lines
  • softwareheritage=> explain select ov.* from origin_visit ov INNER JOIN origin o ON o.id = ov.origin WHERE o.url = 'svn://scm.gforge.inria.fr/svnroot/fpg-ipc/' ORDER BY ov.visit DESC LIMIT 1;
    QUERY PLAN
    ----------------------------------------------------------------------------------------------------------------
    Limit (cost=633.57..633.57 rows=1 width=28)
    -> Sort (cost=633.57..633.59 rows=7 width=28)
    ...
    • Mar 11 2022, 11:53 AM
    • 29 Lines
  • [alias]
    checkout-from-staging = "!f() { git fetch -nf https://forge.softwareheritage.org/source/staging.git phabricator/diff/$1:diff/$1 && git checkout diff/$1; }; f"
    • Mar 8 2022, 2:33 PM
    • 2 Lines
  • query test {
    origin(url: "https://github.com/python/cpython") {
    id
    url
    visits(first: 2) {
    ...
    • Mar 8 2022, 9:16 AM
    • 46 Lines
  • softwareheritage=> \copy (select encode(metadata, 'escape') from raw_extrinsic_metadata where target='swh:1:dir:5d6f2287a9e2e08aaad3f31f833c82d50bcf3c88') to stdout
    <
    ?
    x
    m
    ...
    • Mar 2 2022, 2:48 PM
    • 71 Lines
  • diff --git a/swh/objstorage/backends/azure.py b/swh/objstorage/backends/azure.py
    index be7aeaa..80d8fba 100644
    --- a/swh/objstorage/backends/azure.py
    +++ b/swh/objstorage/backends/azure.py
    @@ -78,12 +78,23 @@ def get_container_url(
    ...
    • Feb 28 2022, 5:06 PM
    • 28 Lines
    • Diff
  • ✘ dev@desktop5  ~/swh-environment/swh-deposit   validate-metadata ●  pip3 show xmlschema
    Name: xmlschema
    Version: 1.9.2
    Summary: An XML Schema validator and decoder
    Home-page: https://github.com/sissaschool/xmlschema
    ...
    • Feb 24 2022, 6:35 PM
    • 20 Lines
  • ### Strucure of answer with short answer
    <details>
    <summary> use this template for dropdown </summary>
    <br>
    ...
    • Feb 23 2022, 5:01 PM
    • 7 Lines
  • class SwhGraphTinkerpopProvider extends WebGraphTinkerpopProvider {
    @Override
    public ImmutableGraph load() {
    var graph = SwhBidirectionalImmutableGraph.loadLabelled(...);
    graph.loadCommitAuthors(LoadMethod.MAPPED);
    ...
    • Feb 23 2022, 4:32 PM
    • 19 Lines
    • Java
  • Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install python3-swh.web' returned 100: Reading package lists...
    Building dependency tree...
    Reading state information...
    Some packages could not be installed. This may mean that you have
    requested an impossible situation or if you are using the unstable
    ...
    • Feb 23 2022, 3:29 PM
    • 24 Lines
  • $ echo "world" > "<string>"
    $ python3 -c "hello"
    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    world
    ...
    • Feb 23 2022, 11:59 AM
    • 6 Lines
  • query myquery {
    origin(url: "https://github.com/python/cpython") {
    url
    id
    visits(first: 2) {
    ...
    • Feb 23 2022, 10:51 AM
    • 27 Lines
  • swh  tony  yavin4  ~  work  …  sysadm-provisioning  azure  terraform   master  2⬆  8⚑  %  terraform apply
    azurerm_resource_group.euwest-vault: Refreshing state... [id=/subscriptions/49b7f681-8efc-4689-8524-870fc0c1db09/resourceGroups/euwest-vault]
    azurerm_resource_group.euwest-cassandra: Refreshing state... [id=/subscriptions/49b7f681-8efc-4689-8524-870fc0c1db09/resourceGroups/euwest-cassandra]
    azurerm_storage_account.vault-storage: Refreshing state... [id=/subscriptions/49b7f681-8efc-4689-8524-870fc0c1db09/resourceGroups/euwest-vault/providers/Microsoft.Storage/storageAccounts/swhvaultstorage]
    azurerm_network_interface.storage-interface["storage01"]: Refreshing state... [id=/subscriptions/49b7f681-8efc-4689-8524-870fc0c1db09/resourceGroups/euwest-servers/providers/Microsoft.Network/networkInterfaces/storage01-interface]
    ...
    • Feb 18 2022, 4:33 PM
    • 1,094 Lines
  • ipython
    Python 3.9.2 (default, Feb 28 2021, 17:03:44)
    Type 'copyright', 'credits' or 'license' for more information
    IPython 7.27.0 -- An enhanced Interactive Python. Type '?' for help.
    ...
    • Feb 16 2022, 4:14 PM
    • 235 Lines
  • [2022-02-15 13:42:59,440: INFO/MainProcess] Task swh.loader.cvs.tasks.LoadCvsRepository[9ea64ca6-744b-46b0-a5dd-754ad1a7ae99] received
    swh-loader_1 | [2022-02-15 13:43:02,110: INFO/ForkPoolWorker-1] Load origin 'pserver://anonymous:anonymous@a.cvs.sourceforge.net/cvsroot/mx4j/mx4j' with type 'cvs'
    swh-loader_1 | Request: BEGIN AUTH REQUEST
    swh-loader_1 | /cvsroot/mx4j
    swh-loader_1 | anonymous
    ...
    • Feb 15 2022, 2:46 PM
    • 22 Lines
  • $ psql service=staging-swh-deposit -c '\x' -c 'select now(), * from deposit order by id desc limit 5;'
    ```
    +-------------------------------+-----+-------------------------------+-------------------------------+------------------------------------------+----------------------------------------------------+--------+-----------+---------------+-----------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+--------------+---------------------------------------------------------------------------+
    | now | id | reception_date | complete_date | external_id | swhid | status | client_id | collection_id | parent_id | status_detail | swhid_context | check_task_id | load_task_id | origin_url |
    ...
    • Feb 15 2022, 10:33 AM
    • 16 Lines
  • 10:04:07 swh-scheduler@db1:5432=> select first_seen, url from listed_origins where visit_type='bzr' and url like 'http://%' order by first_seen desc;
    select first_seen, url from listed_origins where visit_type='bzr' and url like 'http://%' order by first_seen desc;
    +-------------------------------+--------------------------------------------------------------------+
    | first_seen | url |
    +-------------------------------+--------------------------------------------------------------------+
    ...
    • Feb 15 2022, 10:05 AM
    • 83 Lines
  • diff ok.org ko.org
    9c9
    < charset-normalizer==2.0.11
    ---
    > charset-normalizer==2.0.12
    ...
    • Feb 14 2022, 3:52 PM
    • 25 Lines
  • dpkg: error processing archive /var/cache/apt/archives/python3-swh.loader.bzr_1.1.0-1~swh2~bpo10+1_all.deb (--unpack):
    trying to overwrite '/usr/lib/python3/dist-packages/swh/loader/__init__.py', which is also in package python3-swh.loader.core 2.5.0-1~swh1~bpo10+1
    Errors were encountered while processing:
    /var/cache/apt/archives/python3-swh.loader.bzr_1.1.0-1~swh2~bpo10+1_all.deb
    • Feb 11 2022, 5:43 PM
    • 4 Lines
  • sbuild --nolog --batch --no-clean-source --no-run-lintian --arch-all --source --build-dep-resolver=aptitude --extra-repository='deb http://incoming.debian.org/debian-buildd/ buildd-unstable main' --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'
    dpkg-source: info: using source format '3.0 (quilt)'
    dpkg-source: info: building breezy using existing ./breezy_3.2.1+bzr7585.orig.tar.gz
    dpkg-source: warning: upstream signing key but no upstream tarball signature
    dpkg-source: info: using patch list from debian/patches/series
    ...
    • Feb 11 2022, 5:14 PM
    • 49,805 Lines
  • Feb 11 13:46:52 kelvingrove keycloak[1436401]: 13:46:52,362 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-109) Uncaught server error: org.keycloak.authentication.AuthenticationFlowException: authenticator: direct-grant-validate-password
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.DefaultAuthenticationFlow.processSingleFlowExecutionModel(DefaultAuthenticationFlow.java:423)
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:247)
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:976)
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.protocol.oidc.endpoints.TokenEndpoint.resourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:592)
    ...
    • Feb 11 2022, 2:51 PM
    • 77 Lines
  • $ swh auth --client-id swh-web generate-token vlorentz
    Password:
    Traceback (most recent call last):
    File "/home/dev/.local/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core', 'console_scripts', 'swh')())
    ...
    • Feb 11 2022, 2:43 PM
    • 30 Lines
  • def origin_visit_find_latest_status_by_date(
    origin_url: str, visit_date: datetime.datetime
    ) -> Optional[OriginVisitInfo]:
    """Retrieve origin visit status whose date is most recent than the provided visit_date.
    ...
    • Feb 10 2022, 9:22 AM
    • 30 Lines
    • Python
  • 0: swh:1:dir:d24c739c57c68a5cfa7a0042bfbd796f875d1e0c
    successors:
    swh:1:dir:bd1898f4ea14c895bb34219c5495f1c9840eab2a asm [perms: 16384]
    swh:1:dir:294d460f1960619445897b62d8aa0bddb6d81f40 uapi [perms: 16384]
    ...
    • Feb 9 2022, 4:05 PM
    • 51 Lines
  • package org.softwareheritage.graph.utils;
    import it.unimi.dsi.big.webgraph.NodeIterator;
    import org.softwareheritage.graph.SwhUnidirectionalGraph;
    import org.softwareheritage.graph.labels.DirEntry;
    ...
    • Feb 9 2022, 3:58 PM
    • 77 Lines
    • Java
  • swh-lister_1 | [2022-02-08 10:29:25,384: INFO/MainProcess] Task swh.lister.maven.tasks.FullMavenLister[95fa1dcb-6fea-41b7-82e3-e07162d45f2d] received
    swh-lister_1 | [2022-02-08 10:29:45,424: INFO/ForkPoolWorker-1] Downloading text index from http://swh-lister-maven-nginx/export.fld.
    swh-lister_1 | [2022-02-08 10:30:05,455: ERROR/ForkPoolWorker-1] Task swh.lister.maven.tasks.FullMavenLister[95fa1dcb-6fea-41b7-82e3-e07162d45f2d] raised unexpected: ValueError("can not serialize 'datetime.datetime' object where tzinfo=None")
    swh-lister_1 | Traceback (most recent call last):
    swh-lister_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 450, in trace_task
    ...
    • Feb 8 2022, 11:31 AM
    • 37 Lines
  • public class DumpProperties {
    public static void main(String[] args) throws IOException {
    String graphPath = args[0];
    SwhBidirectionalGraph graph = SwhBidirectionalGraph.load(graphPath);
    ...
    • Feb 7 2022, 5:56 PM
    • 50 Lines
  • 17:49:20 swh-scheduler@db1:5432=> select now(), count(*) from listed_origins where visit_type='bzr';
    select now(), count(*) from listed_origins where visit_type='bzr';
    +-------------------------------+-------+
    | now | count |
    +-------------------------------+-------+
    ...
    • Feb 7 2022, 5:50 PM
    • 309 Lines
  • ```
    @remote_api_endpoint("origin/visit_status/find_by_date")
    def origin_visit_status_find_by_date(
    self, origin: str, visit_date: datetime.datetime
    ) -> Optional[OriginVisit]:
    ...
    • Feb 4 2022, 3:55 PM
    • 18 Lines
  • diff --git a/swh/web/common/origin_visits.py b/swh/web/common/origin_visits.py
    index 6dc2201a..023c7aa8 100644
    --- a/swh/web/common/origin_visits.py
    +++ b/swh/web/common/origin_visits.py
    @@ -42,43 +42,45 @@ def get_origin_visits(origin_info: OriginInfo) -> List[OriginVisitInfo]:
    ...
    • Feb 3 2022, 6:07 PM
    • 96 Lines
  • diff --git a/swh/web/common/archive.py b/swh/web/common/archive.py
    index e9e877f6..dca07e30 100644
    --- a/swh/web/common/archive.py
    +++ b/swh/web/common/archive.py
    @@ -703,7 +703,7 @@ def lookup_revision_log(rev_sha1_git, limit):
    ...
    • Feb 3 2022, 3:56 PM
    • 13 Lines
  • time curl https://webapp1.internal.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/
    [{"origin_url":"https://foss.heptapod.net/mercurial/hgview","visit_type":"hg","save_request_date":"2022-02-02T10:15:01.448199+00:00","save_request_status":"accepted","save_task_status":"failed","visit_status":"failed","visit_date":"2022-02-02T10:15:17.559967+00:00","loading_task_id":405935267,"note":null},{"origin_url":"https://foss.heptapod.net/mercurial/hgview","visit_type":"hg","save_request_date":"2022-02-02T09:48:23.326128+00:00","save_request_status":"accepted","save_task_status":"succeeded","visit_status":"full","visit_date":"2022-02-02T09:48:36.296855+00:00","loading_task_id":405934604,"note":null},{"origin_url":"https://foss.heptapod.net/mercurial/hgview","visit_type":"hg","save_request_date":"2022-02-02T09:44:36.829049+00:00","save_request_status":"accepted","save_task_status":"succeeded","visit_status":"full","visit_date":"2022-02-02T09:44:46.020548+00:00","loading_task_id"...
    • Feb 2 2022, 11:36 AM
    • 2 Lines
  • $ varnishncsa -F '%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" %D' | grep -E " [0-9]{7,}$"
    ...
    128.93.166.14 - - [02/Feb/2022:09:35:11 +0000] "GET http://archive.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/ HTTP/1.1" 502 341 "-" "python-requests/2.21.0" 60058264
    128.93.166.14 - - [02/Feb/2022:09:41:16 +0000] "POST http://archive.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/ HTTP/1.1" 502 341 "-" "python-requests/2.21.0" 60057275
    128.93.166.14 - - [02/Feb/2022:09:44:36 +0000] "POST http://archive.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/ HTTP/1.1" 502 341 "-" "python-requests/2.21.0" 60060106
    ...
    • Feb 2 2022, 11:04 AM
    • 6 Lines
  • root@pergamon:~# sudo -u nobody /bin/bash
    nobody@pergamon:/root$ chronic /usr/local/bin/weekly-planning-bot
    + DEST=ardumont@softwareheritage.org
    + AUTHOR=swh-team@inria.fr
    + TEMPLATE_ID=6YKT5osoST-amJQ0MusH2Q
    ...
    • Jan 31 2022, 11:40 AM
    • 103 Lines
  • root@dali:~# cat backup.sh
    #!/bin/bash
    i=0
    dataset=data/postgresql
    ...
    • Jan 27 2022, 10:25 AM
    • 20 Lines
    • Bash Scripting
  • $ sudo virsh list
    Id Name State
    ----------------------------------------------
    2 puppet-environment_pergamon running
    4 puppet-environment_riverside running
    ...
    • Jan 27 2022, 10:15 AM
    • 15 Lines
  • use anyhow::Result;
    use tempfile::TempDir;
    use std::{
    env, io,
    ...
    • Jan 26 2022, 9:46 PM
    • 377 Lines
    • rust
  • write(4, "0032have 16617535684faf9de30620de83667214297a36b8\n0032have ee8f3955c0ef4c7a90a5f707c37c4cb3711ff626\n"..., 1604) = 1604
    read(3, "0008", 4) = 4
    read(3, "NAK\n", 4) = 4
    brk(0x5558db682000) = 0x5558db682000
    write(4, "0032have 1e1490a38504419e349caa1b7d55d5c141a9bccb\n0032have 90e94b160c7f647ddffda707f5e3c0c66c170df8\n"..., 1604) = 1604
    ...
    • Jan 26 2022, 2:42 PM
    • 81 Lines
  • softwareheritage=> copy (select distinct date_offset, encode(date_offset_bytes, 'escape') from revision) to stdout;
    69 +0109
    765 +1245
    -137 -0217
    360 +0600
    ...
    • Jan 21 2022, 9:21 AM
    • 446 Lines
  • class Statsd:
    # ...
    @contextmanager
    def status_gauge(self, metric_name: str, statuses: Collection[str], common_tags: Optional[Dict[str, str]] = None):
    current_status = Optional[str] = None
    ...
    • Jan 20 2022, 1:46 PM
    • 19 Lines
    • Python
  • Jan 19 16:11:15 worker0 python3[3375647]: [2022-01-19 16:11:15,120: INFO/MainProcess] Received task: swh.loader.svn.tasks.DumpMountAndLoadSvnRepository[8028cd34-c06f-4dda-b109-a1b315dafabb]
    Jan 19 16:11:18 worker0 python3[3375658]: [2022-01-19 16:11:18,753: INFO/ForkPoolWorker-1] Load origin 'https://plugins.svn.wordpress.org/wp-activity-log-for-woocommerce' with type 'svn'
    Jan 19 21:12:10 worker0 python3[3375658]: [2022-01-19 21:12:10,701: INFO/ForkPoolWorker-1] Processing revisions [1-168825] for {'swh-origin': 'https://plugins.svn.wordpress.org/wp-activity-log-for-woocommerce', 'remote_url': 'file:///tmp/swh.loader.svn.dxrrwh6j-3375658/swh.loader.svn.fw2r9j6a-3375658/tmpiw_ugkd8', 'local_url': b'/tmp/swh.loader.svn.ft2fxai2-3375658/tmpiw_ugkd8', 'uuid': b'b8457f37-d9ea-0310-8a92-e5e31aec5664'}
    Jan 19 21:18:07 worker0 python3[3375658]: [2022-01-19 21:18:07,339: INFO/ForkPoolWorker-1] Task swh.loader.svn.tasks.DumpMountAndLoadSvnRepository[8028cd34-c06f-4dda-b109-a1b315dafabb] succeeded in 18412.19621137576s...
    • Jan 20 2022, 9:59 AM
    • 4 Lines
  • git,https://scm.gforge.inria.fr/anonscm/git/simty/simty.git
    git,https://github.com/unitystation/unitystation
    svn,http://svn.code.sf.net/p/sauerbraten/code
    git,https://gitlab.com/inkscape/inkscape
    git,https://github.com/keybase/client
    ...
    • Jan 17 2022, 1:54 PM
    • 25 Lines
  • 12:37 $ git diff
    diff --git a/swh/web/common/origin_save.py b/swh/web/common/origin_save.py
    index 3749d134..6696427b 100644
    --- a/swh/web/common/origin_save.py
    +++ b/swh/web/common/origin_save.py
    ...
    • Jan 13 2022, 12:38 PM
    • 20 Lines
  • insert into new_task(id, type, arguments, next_run, current_interval, status, policy,
    retries_left, priority)
    select id, type, arguments, next_run, current_interval, status, policy,
    retries_left, priority
    from task
    ...
    • Jan 12 2022, 1:47 PM
    • 33 Lines
  • -- SWH DB schema upgrade
    -- from_version: 32
    -- to_version: 33
    -- description: Archive old new_task/new_task-run tasks
    ...
    • Jan 10 2022, 3:52 PM
    • 223 Lines
  • >>> s = '<entry xmlns:testns="https://example.org"> <testns:foo> <testns:bar url="http://example.org/1" /> <testns:bar url="http://example.org/2" /></testns:foo> </entry>'
    >>> pprint(xmltodict.parse(s, process_namespaces=True, dict_constructor=dict))
    {'entry': {'https://example.org:foo': {'https://example.org:bar': [{'@url': 'http://example.org/1',
    '@xmlns': OrderedDict([('testns',
    'https://example.org')])},
    ...
    • Jan 10 2022, 2:25 PM
    • 6 Lines
    • Python
  • ```
    16:28:34 softwareheritage-scheduler@belvedere:5432=> select type, policy, count(*) from task where type like 'load-%' and status !='disabled' and priority is null group by type, policy;
    +--------------------+-----------+---------+
    | type | policy | count |
    +--------------------+-----------+---------+
    ...
    • Jan 7 2022, 5:13 PM
    • 21 Lines
  • 16:45:09 swh-web@belvedere:5432=> select * from save_origin_request where origin_url='https://github.com/globalbioticinteractions/usnm-ixodes' order by id desc limit 3;
    +--------+-------------------------------+------------+---------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+--------+
    | id | request_date | visit_type | origin_url | status | loading_task_id | visit_date | loading_task_status | visit_status | user_ids | note |
    +--------+-------------------------------+------------+---------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+--------+
    | 105121 | 2022-01-07 15:14:52.39695+00 | git | https://github.com/globalbioticinteractions/usnm-ixodes | accepted | 404639723 | (null) | succeeded | (null) | (null) | (null) |
    ...
    • Jan 7 2022, 4:46 PM
    • 8 Lines
  • 11:03:56 Build Architecture: amd64
    11:03:56 Build Type: full
    11:03:56 Build-Space: 109680
    11:03:56 Build-Time: 288
    11:03:56 Distribution: unstable-swh
    ...
    • Jan 4 2022, 12:14 PM
    • 26 Lines
  • (.venv) jenkins@30f908325208:~/swh-environment/swh-search$ python3 -c "from frozendict import frozendict; frozendict({})"
    Segmentation fault (core dumped)
    (.venv) jenkins@30f908325208:~/swh-environment/swh-search$ ls
    AUTHORS CONTRIBUTORS MANIFEST.in README.md docs mypy.ini package.json pytest.ini requirements-test.txt setup.cfg swh tox.ini
    CODE_OF_CONDUCT.md LICENSE Makefile core es_config node_modules pyproject.toml requirements-swh.txt requirements.txt setup.py swh.search.egg-info yarn.lock
    ...
    • Dec 21 2021, 4:52 PM
    • 62 Lines
  • pytest -s
    ============================================================================================================================= test session starts =============================================================================================================================
    platform linux -- Python 3.7.3, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
    rootdir: /home/test/swh-environment/swh-search, configfile: pytest.ini
    plugins: django-test-migrations-1.2.0, postgresql-2.6.1, swh.core-1.0.0, xdist-2.5.0, forked-1.4.0, redis-2.3.0, flask-1.2.0, asyncio-0.16.0, dash-2.0.0, django-4.5.2, requests-mock-1.9.3, mock-3.6.1, hypothesis-6.31.6, swh.journal-0.9.2.dev2+g4e5e009
    ...
    • Dec 21 2021, 1:30 PM
    • 95 Lines
  • ./bin/update
    Already up to date.
    perl: warning: Setting locale failed.
    perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    ...
    • Dec 21 2021, 12:43 PM
    • 161 Lines
  • vagrant plugin install vagrant-snapshot
    Installing the 'vagrant-snapshot' plugin. This can take a few minutes...
    Building native extensions. This could take a while...
    Vagrant failed to properly resolve required dependencies. These
    errors can commonly be caused by misconfigured plugin installations
    ...
    • Dec 14 2021, 10:01 AM
    • 20 Lines
  • #!/bin/bash
    id=$1
    echo Getting the logs...
    ...
    • Dec 14 2021, 9:46 AM
    • 21 Lines
    • Bash Scripting
  • 15:59:24 softwareheritage-scheduler@belvedere:5432=> select * from listed_origins where visit_type='cvs' limit 10;
    +--------------------------------------+----------------------------------------------+------------+------------------------+---------+-------------------------------+-------------------------------+------------------------+
    | lister_id | url | visit_type | extra_loader_arguments | enabled | first_seen | last_seen | last_update |
    +--------------------------------------+----------------------------------------------+------------+------------------------+---------+-------------------------------+-------------------------------+------------------------+
    | b678cfc3-2780-4186-9186-d78a14bd4958 | https://cvs.code.sf.net/p/a-a-p/code | cvs | {} | t | 2021-06-01 20:24:22.899139+00 | 2021-10-15 04:34:05.365731+00 | 2018-08-12 00:00:00+00 |
    ...
    • Dec 9 2021, 3:59 PM
    • 16 Lines
  • *** swh-deploy: starting test run on moma.internal.softwareheritage.org...
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Dec 8 2021, 3:03 PM
    • 27 Lines
  • diff --git a/swh/loader/svn/ra.py b/swh/loader/svn/ra.py
    index 8a51c49..d88496f 100644
    --- a/swh/loader/svn/ra.py
    +++ b/swh/loader/svn/ra.py
    @@ -11,6 +11,7 @@ from __future__ import annotations
    ...
    • Dec 8 2021, 1:42 PM
    • 245 Lines
  • ```
    Dec 08 08:31:43 thyssen package-helper[6666]: /etc/unbound/unbound.conf.d/listen-docker.conf:1: error: syntax error
    Dec 08 08:31:43 thyssen package-helper[6666]: read /etc/unbound/unbound.conf failed: 1 errors in configuration file
    Dec 08 08:31:43 thyssen package-helper[6669]: /etc/unbound/unbound.conf.d/listen-docker.conf:1: error: syntax error
    Dec 08 08:31:43 thyssen package-helper[6669]: read /etc/unbound/unbound.conf failed: 1 errors in configuration file
    ...
    • Dec 8 2021, 9:36 AM
    • 10 Lines
  • Somehow for now, the pergamon manifest is not complete/incorrect (and it's not clear where exactly we fall short).
    At boot time, we have the puppet master running while it should not. Puppet master must be served through apache (ssl configured and what not).
    So we need to make the provisioning step pass twice (for some unknown reason as of yet as well).
    Then connect to the node, stop and deactivate the puppet master service.
    Then restart the apache service so it serves properly puppet (as master).
    ...
    • Dec 7 2021, 10:27 AM
    • 25 Lines
  • vagrant vm:
    ```
    root@thyssen:~# puppet agent --test
    Info: Using configured environment 'production'
    ...
    • Dec 7 2021, 10:19 AM
    • 76 Lines
  • production:
    ```
    List of databases
    +------------------------------------+--------------+----------+---------+---------+-----------------------+
    | Name | Owner | Encoding | Collate | Ctype | Access privileges |
    ...
    • Dec 6 2021, 5:57 PM
    • 65 Lines
  • 15:04:47 swh-scheduler@db1:5432=> select * from task where type = 'list-debian-distribution';
    +-[ RECORD 1 ]-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | id | 1182486 |
    | type | list-debian-distribution |
    | arguments | {"args": [], "kwargs": {"suites": ["stretch", "buster", "bullseye"], "components": ["main", "contrib", "non-free"], "mirror_url": "http://deb.debian.org/debian/", "distribution": "Debian"}} |
    ...
    • Dec 6 2021, 3:05 PM
    • 24 Lines
  • root@pergamon:~# clush -b -w @swh-workers systemctl status swh-worker@loader_nixguix | grep succeeded
    Dec 05 18:28:52 worker15 python3[2936713]: [2021-12-05 18:28:52,509: INFO/ForkPoolWorker-1] Task swh.loader.package.nixguix.tasks.LoadNixguix[d38e7d14-4894-4851-9ec3-7fa533d74049] succeeded in 28061.734233491123s: {'status': 'eventful', 'snapshot_id': '3568e78b54169b3995efa0acfa7c88a45d0056aa'}
    Dec 05 10:40:20 worker16 python3[2974229]: [2021-12-05 10:40:20,535: INFO/ForkPoolWorker-1] Task swh.loader.package.nixguix.tasks.LoadNixguix[51f86173-0b8e-496f-8d05-fb396247d197] succeeded in 9895.689101913944s: {'status': 'eventful', 'snapshot_id': '0415952acb92d5a8ad03c1f9d60a81459cb78aa5'}
    • Dec 6 2021, 9:33 AM
    • 3 Lines
  • (swh) ✔ ~/swh/swh-environment/docker [master|✚ 1⚑ 41]
    16:01 $ doco exec swh-scheduler /bin/bash
    swh@ab68bb37fea4:/$ psql service=swh-scheduler
    psql (12.9 (Debian 12.9-1.pgdg110+1), server 12.8 (Debian 12.8-1.pgdg100+1))
    Type "help" for help.
    ...
    • Dec 3 2021, 4:02 PM
    • 22 Lines
  • 14:09:12 swh-scheduler@db1:5432=> begin;
    BEGIN
    Time: 5.818 ms
    14:09:14 *swh-scheduler@db1:5432=> update listed_origins set extra_loader_arguments = extra_loader_arguments - 'date' where visit_type='deb';
    UPDATE 35105
    ...
    • Dec 3 2021, 2:09 PM
    • 15 Lines
  • 13:38:27 softwareheritage-scheduler@belvedere:5432=> select * from task where type = 'list-gitlab-incremental';
    +-----------+-------------------------+--------------------------------------------------------------------------------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------+
    | id | type | arguments | next_run | current_interval | status | policy | retries_left | priority |
    +-----------+-------------------------+--------------------------------------------------------------------------------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------+
    | 399914374 | list-gitlab-incremental | {"args": [], "kwargs": {"url": "https://foss.heptapod.net/api/v4/", "name": "heptapod"}} | 2021-12-04 08:56:02.284533+00 | 1 day | next_run_not_scheduled | recurring | 3 | (null) |
    ...
    • Dec 3 2021, 1:38 PM
    • 16 Lines
  • root@riverside:~# for volume in data redis kafka zookeeper symbolicator clickhouse; do echo docker volume create --name=sentry-$volume; done
    docker volume create --name=sentry-data
    docker volume create --name=sentry-redis
    docker volume create --name=sentry-kafka
    docker volume create --name=sentry-zookeeper
    ...
    • Dec 3 2021, 10:11 AM
    • 26 Lines
  • from functools import wraps
    def handle_raise_on_commit(f):
    @wraps(f)
    ...
    • Dec 2 2021, 5:02 PM
    • 16 Lines
    • Python
  • swh-loader_1 | [2021-12-02 13:43:44,472: ERROR/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[fb2923f8-a344-4a4d-9340-0897f6bab6fc] raised unexpected: TypeError("__init__() missing 1 required positional argument: 'date'")
    swh-loader_1 | Traceback (most recent call last):
    swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 450, in trace_task
    swh-loader_1 | R = retval = fun(*args, **kwargs)
    swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/task.py", line 55, in __call__
    ...
    • Dec 2 2021, 2:46 PM
    • 33 Lines
  • class DirectoryData:
    date: datetime
    """
    Date of the directory in the isochrone frontier
    """
    ...
    • Dec 1 2021, 5:13 PM
    • 11 Lines
    • Python
  • @dataclass(eq=True, frozen=True)
    class DirectoryData:
    """Object representing the data associated to a directory in the provenance model,
    where `date` is the date of the directory in the isochrone frontier, and `flat` is a
    flag acknowledging that a flat model for the elements outside the frontier has
    ...
    • Dec 1 2021, 5:12 PM
    • 10 Lines
    • Python
  • P1225 tryout
    $ ipython
    In [1]: 1626111425534 / 1e3
    Out[1]: 1626111425.534
    ...
    • Nov 26 2021, 10:40 AM
    • 22 Lines
  • worker11: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 553
    worker11: Error: /Stage[main]/Profile::Systemd_journal::Journalbeat/User[journalbeat]/ensure: change from 'present' to 'absent' failed: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 553
    worker10: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 522
    worker10: Error: /Stage[main]/Profile::Systemd_journal::Journalbeat/User[journalbeat]/ensure: change from 'present' to 'absent' failed: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 522
    worker12: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 516
    ...
    • Nov 26 2021, 10:20 AM
    • 31 Lines
  • root@bardo:~# puppet agent --test
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Nov 24 2021, 4:22 PM
    • 149 Lines