Page MenuHomeSoftware Heritage
Create Paste
Paste Active Pastes

Active Pastes
Use Results
Edit Query

  • P1009 not created save code now submitted and failing with a more clear error
    ```
    head -1 /tmp/reschedule-save-code-now.not-created.csv | swh web save submit-request | jq . | tee /tmp/reschedule-save-code-now.txt WARNING:swh.core.cli:Could not load subcommand scanner: swh-scanner
    --- Logging error ---
    Traceback (most recent call last):
    File "/home/tony/work/inria/repo/swh/swh-environment/swh-web-client/swh/web/client/cli.py", line 165, in submit_request
    ...
    • Apr 20 2021, 1:02 PM
    • 53 Lines
    Author: ardumont
  • P1008 Pending running save code now requests
    now | id | request_date | visit_type | origin_url | status | loading_task_id | visit_date | loading_task$
    -------------------------------+-------+-------------------------------+------------+---------------------------------------------------------------------------------+----------+-----------------+----------------------------+-------------$
    2021-04-20 07:12:06.315517+00 | 74388 | 2021-04-20 07:04:53.803404+00 | git | https://github.com/Anuken/Mindustry | accepted | 380600429 | | running
    2021-04-20 07:12:06.315517+00 | 74387 | 2021-04-20 07:04:30.629639+00 | git | https://github.com/tobspr/shapez.io | accepted | 380600405 | | running
    2021-04-20 07:12:06.315517+00 | 71357 | 2021-04-01 16:57:50.205+00 | git | https://scm.gforge.inria.fr/anonscm/git/simty/simty.git | accepted | 379069991 | | running
    ...
    • Apr 20 2021, 9:11 AM
    • 49 Lines
    Author: ardumont
  • P1007 can't read randomized origin visit status because... <- insert reason here ;)
    ```
    (Pdb++) self._swh_storage.origin_visit_status_get_random('git')
    *** swh.core.api.RemoteException: <RemoteException 500 TypeError: ["can not serialize '_GenericAlias' object"]>
    Traceback (most recent call last):
    File "/home/tony/work/inria/repo/swh/swh-environment/swh-core/swh/core/api/__init__.py", line 181, in meth_
    ...
    • Apr 15 2021, 5:35 PM
    • 25 Lines
    Author: ardumont
  • P1006 tryout fixture but this hung so meh
    Staged
    modified swh/scheduler/pytest_plugin.py
    @@ -1,13 +1,15 @@
    -# Copyright (C) 2020 The Software Heritage developers
    +# Copyright (C) 2020-2021 The Software Heritage developers
    ...
    • Apr 15 2021, 1:29 PM
    • 85 Lines
    Author: ardumont
  • P1005 Convert webapp historical data to historical json
    from swh.web.common.utils import parse_iso8601_date_to_utc
    import json
    _stat_counters_backfill = {
    "2015-09-01": {"revision": 0, "origin": 0, "content": 0},
    ...
    • Apr 13 2021, 11:17 AM
    • 38 Lines
    • Python
    Author: vsellier
  • P1004 elasticsearch index freeze/close
    import datetime
    import iso8601
    import elasticsearch
    ...
    • Apr 12 2021, 10:31 AM
    • 33 Lines
    • Python
    Author: olasd
  • P1003 (An Untitled Masterwork)
    /home/ddouard/.virtualenvs/swh/lib/python3.9/site-packages/django/db/backends/postgresql/base.py
    161a162,164
    > if settings_dict.get('NAME', '').startswith('postgresql://'):
    > return {'dsn': settings_dict['NAME']}
    >
    • Apr 9 2021, 4:04 PM
    • 5 Lines
    Author: douardda
  • P1002 save request duration metric draft
    # HELP swh_web_save_requests_delay_seconds Save Requests Duration
    # TYPE swh_web_save_requests_delay_seconds gauge
    swh_web_save_requests_delay_seconds{load_task_status="failed",visit_type="git"} 0.0
    swh_web_save_requests_delay_seconds{load_task_status="failed",visit_type="hg"} 0.0
    swh_web_save_requests_delay_seconds{load_task_status="failed",visit_type="svn"} 0.0
    ...
    • Apr 8 2021, 11:26 AM
    • 8 Lines
    Author: ardumont
  • P1001 save code now average ingestion time
    ```
    swh-web=> select loading_task_status, avg(visit_date - request_date) from save_origin_request where loading_task_status in ('succeeded', 'failed') group by loading_task_status;
    loading_task_status | avg
    ---------------------+-----------------
    succeeded | 06:13:01.904513
    ...
    • Apr 7 2021, 3:54 PM
    • 34 Lines
    Author: ardumont
  • P1000 deposit-ipol.not-done.csv
    587,2020-05-06 08:38:15.900328+00,2020-05-06 08:38:15.890146+00,ipol.2018.236,,rejected,7,6,,"{""archive"": [{""summary"": ""Deposit without archive is rejected""}], ""metadata"": [{""fields"": [""author""], ""summary"": ""Mandatory fields are missing""}, {""fields"": [""name or title""], ""summary"": ""Mandatory alternate fields are missing""}]}",,333174385,,https://doi.org/10.5201/ipol.2018.236
    589,2020-05-06 10:44:11.614131+00,2020-05-06 10:44:11.606561+00,ipol.2018.236,,rejected,7,6,,"{""archive"": [{""summary"": ""Deposit without archive is rejected""}], ""metadata"": [{""fields"": [""author""], ""summary"": ""Mandatory fields are missing""}, {""fields"": [""name or title""], ""summary"": ""Mandatory alternate fields are missing""}]}",,333176410,,https://doi.org/10.5201/ipol.2018.236
    632,2020-05-26 13:47:09.857387+00,2020-05-26 13:47:09.82403+00,ipol.2011.blm-cdf,,rejected,7,6,,"{""archive"": [{""fields"": [1180], ""summary"": ""Mandatory archive type is not supported""}]}",,334942017,,https://doi.org/10.5201/ipol.2011.blm-cdf
    636,2020-05-27 22:53:07.665269+00,2020-05-27 22:53:07.627188+00,ipol.2011.blm-cdf,,rejected,7,6,,"{""archive"": [{""fields"": [1188], ""summary"": ""Mandatory archive type is not supported""}]}",,335056086,,https://doi.org/10.5201/ipol.2011.blm-cdf
    1245,2020-12-19 14:46:03.412595+00,2020-12-19 14:46:03.411527+00,ipol.2014.71,,rejected,7,6,,"{""metadata"": [{""fields"": [""codemeta:dateCreated""], ""summary"": ""Invalid date format""}]}",,349310053,349310062,https://doi.org/10.5201/ipol.2014.71
    ...
    • Apr 2 2021, 2:57 PM
    • 68 Lines
    Author: ardumont
  • P999 deposit-ipol.done.csv
    551,2020-04-20 23:11:20.74476+00,2020-04-20 23:11:20.731795+00,87afca27-ebbb-408a-845d-5277b6a92f49,swh:1:dir:a10423592dd061a00f7d34e4a3c102ba00c3d2ab,done,7,6,,,swh:1:dir:a10423592dd061a00f7d34e4a3c102ba00c3d2ab;origin=https://www.ipol.im/87afca27-ebbb-408a-845d-5277b6a92f49;visit=swh:1:snp:98b5853bb907a1d931d46521c26f753ddf4de4d3;anchor=swh:1:rev:86e6c7e3ad0699601f73f343344a65f65b6f76ce;path=/,325543487,325543502,https://doi.org/10.5201/87afca27-ebbb-408a-845d-5277b6a92f49
    595,2020-05-06 22:48:30.487078+00,2020-05-06 22:48:30.447662+00,ipol.2018.236,swh:1:dir:a10423592dd061a00f7d34e4a3c102ba00c3d2ab,done,7,6,,,swh:1:dir:a10423592dd061a00f7d34e4a3c102ba00c3d2ab;origin=https://doi.org/10.5201/ipol.2018.236;visit=swh:1:snp:206991563b219d18cdfc998d0e35716ba142c12e;anchor=swh:1:rev:baabf12542cd6df80af9594de8922b5281f07272;path=/,333184241,333184251,https://doi.org/10.5201/ipol.2018.236
    596,2020-05-06 22:50:06.271071+00,2020-05-06 22:50:06.267703+00,ipol.2018.236,swh:1:dir:a10423592dd061a00f7d34e4a3c102ba00c3d2ab,done,7,6,,,swh:1:dir:a10423592dd061a00f7d34e4a3c102ba00c3d2ab;origin=https://doi.org/10.5201/ipol.2018.236;visit=swh:1:snp:3ceae6836c8fa31e09d66133934dbc2c68ac7860;anchor=swh:1:rev:17ba798378f39d47e0525ac436856976262ac733;path=/,333184242,333184252,https://doi.org/10.5201/ipol.2018.236
    597,2020-05-06 22:50:14.110159+00,2020-05-06 22:50:14.105915+00,ipol.2018.236,swh:1:dir:a10423592dd061a00f7d34e4a3c102ba00c3d2ab,done,7,6,,,swh:1:dir:a10423592dd061a00f7d34e4a3c102ba00c3d2ab;origin=https://doi.org/10.5201/ipol.2018.236;visit=swh:1:snp:f7decde6a26a4fa5f0886d71c010ceae827bae92;anchor=swh:1:rev:1bb3dbaf98e4e775ed84a9e2d428246a47ab12d6;path=/,333184243,333184254,https://doi.org/10.5201/ipol.2018.236
    602,2020-05-07 17:32:54.537693+00,2020-05-07 17:32:54.517556+00,ipol.2018.236,swh:1:dir:a10423592dd061a00f7d34e4a3c102ba00c3d2ab,done,7,6,597,,swh:1:dir:a10423592dd061a00f7d34e4a3c102ba00c3d2ab;origin=https://doi.org/10.5201/ipol.2018.236;visit=swh:1:snp:994f6ca7c49b1012768c4a5a6470f17f28d0e294;anchor=swh:1:rev:cd8faa35b66794fecfc5c1e49c826d1a287ab410;path=/,333382926,333382931,https://doi.org/10.5201/ipol.2018.236
    ...
    • Apr 2 2021, 2:56 PM
    • 115 Lines
    Author: ardumont
  • P998 (An Untitled Masterwork)
    2021-04-02T07:33:09.081346338Z swh_storage.1.yjtfpr7ambpm@libra | ERROR:root:unsupported Unicode escape sequence
    2021-04-02T07:33:09.081377763Z swh_storage.1.yjtfpr7ambpm@libra | DETAIL: \u0000 cannot be converted to text.
    2021-04-02T07:33:09.081383405Z swh_storage.1.yjtfpr7ambpm@libra | CONTEXT: JSON data, line 1: ...ng prices from the Steam market.", "description":...
    2021-04-02T07:33:09.081387993Z swh_storage.1.yjtfpr7ambpm@libra | COPY tmp_revision, line 10, column metadata: "{"intrinsic": {"tool": "PKG-INFO", "raw": {"metadata_version": "2.1", "name": "steammarket", "versio..."
    2021-04-02T07:33:09.081392498Z swh_storage.1.yjtfpr7ambpm@libra | Traceback (most recent call last):
    ...
    • Apr 2 2021, 9:34 AM
    • 32 Lines
    Author: douardda
  • P997 irc log
    14:01 * hakb[m] < https://matrix.org/_matrix/media/r0/download/matrix.org/sWuFMMJQwliKvENXMZBPCYuc/message.txt >
    14:02 * hakb[m] < https://matrix.org/_matrix/media/r0/download/matrix.org/ZVjcnGQYAvfmxpMyGzdahqbh/message.txt >
    14:02 * hakb[m] < https://matrix.org/_matrix/media/r0/download/matrix.org/NUHFuDYZBNRXTTXEkXhlmLGS/message.txt >
    14:03 * hakb[m] < https://matrix.org/_matrix/media/r0/download/matrix.org/hpRqGUJfkoikaajfTfXGPkmI/message.txt >
    • Apr 1 2021, 4:12 PM
    • 4 Lines
    Author: vlorentz
  • P996 (An Untitled Masterwork)
    fullset=# select * from revision order by id desc limit 10;
    id | sha1 | date | org
    --------+--------------------------------------------+------------------------+-----
    338447 | \x3a5c899326e18b639a8f09dea58c035ade73a8ea | 1970-01-01 01:00:00+01 |
    338446 | \x3bf3d78487e1700fb4b13396c8b1d41fd6b64446 | 1970-01-01 01:00:00+01 |
    ...
    • Apr 1 2021, 12:48 PM
    • 14 Lines
    Author: douardda
  • P995 (An Untitled Masterwork)
    C D R C-D C-R D-R Loc pos
    03-29T12:30 1,949,158 264 338,263 5022 498,545,729 101,905 32,647,347 9,302,016
    03-29T14:25 1,949,158 264 338,263 5022 498,545,729 101,905 32,897,808
    03-30T10:12 1,949,230 264 338,287 5022 661,970,360 117,289 34,102,420 9,302,016
    ...
    • Apr 1 2021, 11:11 AM
    • 6 Lines
    Author: douardda
  • P994 dsc metadata
    {'files': {'mapbox-geometry_1.0.0-1.debian.tar.xz': {'md5sum': '8c38df38641cad35883e2b207016709c',
    'name': 'mapbox-geometry_1.0.0-1.debian.tar.xz',
    'sha256': 'fab7bc559d03929685a47ad036ccee4151d5d0a5fbec522cde341908bbaeaf2a',
    'size': 3756,
    'uri': 'http://deb.debian.org/debian//pool/main/m/mapbox-geometry/mapbox-geometry_1.0.0-1.debian.tar.xz'},
    ...
    • Mar 30 2021, 1:10 PM
    • 19 Lines
    • Python
    Author: vlorentz
  • P993 swh/loader/mercurial/tests/test_partial_storage.py
    from datetime import datetime
    from hashlib import sha1
    import os
    import pytest
    ...
    • Mar 29 2021, 1:56 PM
    • 96 Lines
    • Python
    Author: vlorentz
  • P992 (An Untitled Masterwork)
    swh/graph/tests/test_cli.py::test_pipeline
    ------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------
    DEBUG swh.core.config:config.py:88 Loading config file /tmp/tmp2xijfg30.swh-graph-test/config.yml
    INFO root:config.py:33 using swh-graph JAR: /home/anlambert/swh/swh-environment/swh-graph/java/target/swh-graph-0.3.0.jar
    INFO root:webgraph.py:205 starting compression at 2021-03-26 12:41:38.621097
    ...
    • Mar 26 2021, 12:42 PM
    • 115 Lines
    Author: anlambert
  • P991 (An Untitled Masterwork)
    ddouard@petitpalais:~$ cat revision-cvs-extract.py
    #!/usr/bin/env python
    import sys
    import iso8601
    ...
    • Mar 26 2021, 9:52 AM
    • 25 Lines
    Author: douardda
  • P990 pytest wat
    def test_foo():
    from unittest.mock import MagicMock, call
    f = MagicMock()
    ...
    • Mar 25 2021, 5:45 PM
    • 33 Lines
    • Python
    Author: vlorentz
  • P989 class overload wat
    def test_load_get_known_extids() -> None:
    """Checks PackageLoader.load() fetches known extids efficiently"""
    storage = Mock(wraps=get_storage("memory"))
    class StubPackageInfo(BasePackageInfo):
    ...
    • Mar 25 2021, 5:11 PM
    • 26 Lines
    • Python
    Author: vlorentz
  • P988 swh-docs: make -C docs clean; make -C docs
    swh  tony  yavin4  ~  work  …  swh  swh-environment  swh-auth   master  1⬆  1✎  1⚑  %  make -C docs clean
    make: Entering directory '/home/tony/work/inria/repo/swh/swh-environment/swh-auth/docs'
    sphinx-build -M clean . _build
    Removing everything under '_build'...
    rm -rf apidoc
    ...
    • Mar 25 2021, 4:53 PM
    • 36 Lines
    Author: ardumont
  • P987 docker-keycloak 10.0.2 failures on start
    keycloak_1 | 16:25:55,702 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-1) Uncaught server error: java.lang.RuntimeException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "additionalParameters" (class org.keycloak.models.credential.dto.PasswordCredentialData), not marked as ignorable (2 known properties: "hashIterations", "algorithm"])
    keycloak_1 | at [Source: (String)"{"hashIterations":27500,"algorithm":"pbkdf2-sha256","additionalParameters":{}}"; line: 1, column: 77] (through reference chain: org.keycloak.models.credential.dto.PasswordCredentialData["additionalParameters"])
    keycloak_1 | at org.keycloak.keycloak-server-spi@10.0.2//org.keycloak.models.credential.PasswordCredentialModel.createFromCredentialModel(PasswordCredentialModel.java:55)
    keycloak_1 | at org.keycloak.keycloak-services@10.0.2//org.keycloak.credential.PasswordCredentialProvider.getPassword(PasswordCredentialProvider.java:71)
    keycloak_1 | at org.keycloak.keycloak-services@10.0.2//org.keycloak.credential.PasswordCredentialProvider.isValid(PasswordCredentialProvider.java:243)
    ...
    • Mar 24 2021, 5:32 PM
    • 122 Lines
    Author: ardumont
  • P986 (An Untitled Masterwork)
    ddouard@petitpalais:/data/aeviso$ ipython3
    Python 3.7.3 (default, Jul 25 2020, 13:03:44)
    Type "copyright", "credits" or "license" for more information.
    IPython 5.8.0 -- An enhanced Interactive Python.
    ...
    • Mar 24 2021, 2:46 PM
    • 15 Lines
    Author: douardda
  • P985 keycloak staging realm error from time to time...
    Mar 24 11:37:30 kelvingrove standalone.sh[946]: 11:37:30,352 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-216) Uncaught server error: org.keycloak.authentication.AuthenticationFlowException: authenticator: direct-grant-validate-password
    Mar 24 11:37:30 kelvingrove standalone.sh[946]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.DefaultAuthenticationFlow.processSingleFlowExecutionModel(DefaultAuthenticationFlow.java:423)
    Mar 24 11:37:30 kelvingrove standalone.sh[946]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:247)
    Mar 24 11:37:30 kelvingrove standalone.sh[946]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:976)
    Mar 24 11:37:30 kelvingrove standalone.sh[946]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.protocol.oidc.endpoints.TokenEndpoint.resourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:592)
    ...
    • Mar 24 2021, 12:41 PM
    • 77 Lines
    Author: ardumont
  • P984 (An Untitled Masterwork)
    return datetime.datetime(
    1970, 1, 1, tzinfo=datetime.timezone.utc
    ) + datetime.timedelta(
    seconds=obj["timestamp"]["seconds"],
    > microseconds=obj["timestamp"]["microseconds"],
    ...
    • Mar 24 2021, 11:04 AM
    • 7 Lines
    Author: seirl
  • P983 staging-users.yaml with creds redacted
    - username: hal
    password: ""
    email: ""
    - username: hal-preprod
    password: ""
    ...
    • Mar 23 2021, 12:17 PM
    • 21 Lines
    Author: ardumont
  • P982 (An Untitled Masterwork)
    11:17 $ curl -i -H "Authorization: Bearer ${TOKEN}" https://archive.softwareheritage.org/api/1/stat/counters/
    HTTP/1.1 403 Forbidden
    Date: Tue, 23 Mar 2021 10:17:27 GMT
    Server: gunicorn/19.9.0
    Content-Type: application/json
    ...
    • Mar 23 2021, 11:18 AM
    • 17 Lines
    Author: anlambert
  • P981 KeycloakError format is a json-dict bytes encoded
    swh-deposit_1 | [2021-03-23 09:29:57 +0000] [114] [DEBUG] GET /deposit/1/servicedocument/
    swh-deposit_1 | EEEEE KeycloakError: e: 401: b'{"error":"invalid_grant","error_description":"Invalid user credentials"}'
    swh-deposit_1 | EEEEE KeycloakError: e.error_message: b'{"error":"invalid_grant","error_description":"Invalid user credentials"}'
    swh-deposit_1 | EEEEE KeycloakError: e.code: 401
    swh-deposit_1 | EEEEE KeycloakError: e.body: b'{"error":"invalid_grant","error_description":"Invalid user credentials"}'
    • Mar 23 2021, 10:36 AM
    • 5 Lines
    Author: ardumont
  • P980 deposit client: parsing error returned by deposit server ends up in a cascading key error
    swh deposit upload --url http://localhost:5080/deposit \
    --username test \
    --password test \
    --metadata ../deposit-swh.update-metadata.xml --format json
    WARNING:swh.deposit.cli.client:The metadata file provided should contain "<swh:create_origin>" or "<swh:add_to_origin>" tag
    ...
    • Mar 22 2021, 4:07 PM
    • 33 Lines
    Author: ardumont
  • P979 Command-Line Input
    (base) [zacchiro@rioc ~]$ ls -al /scratch/apietri/graph-2020-12-15/
    total 1108962828
    drwxr-xr-x 2 apietri irill 5 Jan 3 04:49 .
    drwxr-xr-x 11 apietri irill 14 Jan 6 18:45 ..
    -rw-r--r-- 1 apietri irill 917013631214 Jan 1 23:43 graph-bv.graph
    ...
    • Mar 19 2021, 2:26 PM
    • 12 Lines
    Author: zack
  • P978 (An Untitled Masterwork)
    def test_provenance_revision_add(provenance, swh_storage_with_objects, archive_pg):
    """Test the ProvenanceDB.revision_add() method"""
    for revision in TEST_OBJECTS["revision"]:
    ...
    • Mar 18 2021, 6:43 PM
    • 19 Lines
    Author: douardda
  • P977 raw_extrinsic_metadata discrepancies
    softwareheritage=> select target, discovery_date from raw_extrinsic_metadata rem where not exists (select 1 from raw_extrinsic_metadata_new remn where date_trunc('second', remn.discovery_date) = date_trunc('second', rem.discovery_date) and remn.target = case when rem.target LIKE 'swh:1:%' then rem.target else 'swh:1:ori:' || encode(digest(rem.target, 'sha1'), 'hex') end) order by target;
    target | discovery_date
    ----------------------------------------------------+-------------------------------
    swh:1:dir:d83b7dda887dc790f7207608474650d4344b8df9 | 2021-03-16 00:03:03.765938+00
    swh:1:dir:d83b7dda887dc790f7207608474650d4344b8df9 | 2021-03-16 16:41:37.093553+00
    ...
    • Mar 17 2021, 7:46 PM
    • 10 Lines
    Author: olasd
  • P976 python3-mirakuru update tryouts
    $ gbp import-orig --uscan
    gbp:info: Launching uscan...
    gbp:info: Using uscan downloaded tarball ../mirakuru_2.3.0.orig.tar.gz
    What is the upstream version? [2.3.0]
    gbp:info: Importing '../mirakuru_2.3.0.orig.tar.gz' to branch 'debian/upstream'...
    ...
    • Mar 17 2021, 10:01 AM
    • 770 Lines
    Author: ardumont
  • P975 docker: auth_time not present in decoded_token
    ???
    swh-deposit_1 | Internal Server Error: /deposit/1/servicedocument/
    swh-deposit_1 | Traceback (most recent call last):
    swh-deposit_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
    ...
    • Mar 13 2021, 10:25 AM
    • 38 Lines
    Author: ardumont
  • P974 (An Untitled Masterwork)
    tox -e py3-django2-dev -- -s -k test_backend_
    GLOB sdist-make: /home/tony/work/inria/repo/swh/swh-environment/swh-deposit/setup.py
    py3-django2-dev inst-nodeps: /home/tony/work/inria/repo/swh/swh-environment/swh-deposit/.tox/.tmp/package/1/swh.deposit-0.11.2.dev3+gf42e3dfe.d20210312.zip
    py3-django2-dev installed: aiohttp==3.7.4.post0,aiohttp-utils==3.1.1,amqp==5.0.5,async-timeout==3.0.1,attrs==20.3.0,attrs-strict==0.2.1,beautifulsoup4==4.9.3,billiard==3.6.3.0,blinker==1.4,cassandra-driver==3.24.0,celery==5.0.5,certifi==2020.12.5,cffi==1.14.5,chardet==4.0.0,click==7.1.2,click-didyoumean==0.0.3,click-plugins==1.1.1,click-repl==0.1.6,confluent-kafka==1.6.0,coreapi==2.3.3,coreschema==0.0.4,coverage==5.5,cryptography==3.4.6,decorator==4.4.2,Deprecated==1.2.11,distro==1.5.0,Django==2.2.19,django-stubs==1.7.0,django-test-migrations==1.1.0,djangorestframework==3.12.2,djangorestframework-stubs==1.4.0,ecdsa==0.14.1,elasticsearch==7.11.0,fancycompleter==0.9.1,Flask==1.1.2,geomet==0.2.1.post1,gunicorn==20.0.4,httplib2==0.19.0,humanize==3.2.0,hypothesis==6.8.0,idna==2.10,importlib-metadata==3.7.2,iniconfig==1.1.1,iso8601==0.1.14,itsdangerous==1.1.0,itypes==1.2.0,jeepney==0.6.0,Jinja2==2.11.3,keyring==23.0.0,kombu==5.0.2,launchpadlib==1.10.13,lazr.restfulclient==0.14.3,lazr.uri==1.0.5,MarkupSafe==1.1.1,mirakuru==2.3.0,msgpack==1.0.2,multidict==5.1.0,mypy==0.812,mypy-extensions==0.4.3,oauthlib==3.1.0,packaging==20.9,pbr==5.5.1,pdbpp==0.10.2,pika==1.2.0,pkginfo==1.7.0,plotille==3.7.2,pluggy==0.13.1,port-for==0.4,prompt-toolkit==3.0.17,psutil==5.8.0,psycopg2==2.8.6,py==1.10.0,pyasn1==0.4.8,pycparser==2.20,Pygments==2.8.1,pyparsing==2.4.7,pyrepl==0.9.0,pytest==6.2.2,pytest-cov==2.11.1,pytest-django==4.1.0,pytest-mock==3.5.1,pytest-postgresql==2.6.1,python-dateutil==2.8.1,python-debian==0.1.39,python-jose==3.2.0,python-keycloak==0.24.0,python-mimeparse==1.6.0,pytz==2021.1,PyYAML==5.4.1,requests==2.25.1,requests-mock==1.8.0,rsa==4.7.2,SecretStorage==3.3.1,sentry-sdk==1.0.0,simpy==3.0.13,six==1.15.0,sortedcontainers==2.3.0,soupsieve==2.2,sqlparse==0.4.1,swh.auth @ file:///home/tony/work/inria/repo/swh/swh-environment/swh-auth,swh.core==0.12.0,swh.deposit @ file:///home/tony/work/inria/repo/swh/swh-environment/swh-deposit/.tox/.tmp/package/1/swh.deposit-0.11.2.dev3%2Bgf42e3dfe.d20210312.zip,swh.journal==0.7.1,swh.lister==0.10.0,swh.loader.core==0.20.0,swh.model==2.1.0,swh.objstorage==0.2.2,swh.scheduler==0.10.0,swh.storage==0.24.1,tenacity==7.0.0,testresources==2.0.1,toml==0.10.2,typed-ast==1.4.2,typing-extensions==3.7.4.3,uritemplate==3.0.1,urllib3==1.26.3,vine==5.0.0,wadllib==1.3.5,wcwidth==0.2.5,Werkzeug==1.0.1,wmctrl==0.3,wrapt==1.12.1,xmltodict==0.12.0,yarl==1.6.3,zipp==3.4.1
    py3-django2-dev run-test-pre: PYTHONHASHSEED='3082583648'
    ...
    • Mar 12 2021, 4:10 PM
    • 177 Lines
    Author: ardumont
  • P973 (An Untitled Masterwork)
    http https://archive.softwareheritage.org/api/1/snapshot/fc7706e4c177714475a4886831486ad0979983ea/ "Authorization: Bearer ${SWH_TOKEN}"
    HTTP/1.1 403 Forbidden
    Age: 0
    Allow: OPTIONS, GET, HEAD, OPTIONS
    Connection: keep-alive
    ...
    • Mar 12 2021, 2:54 PM
    • 21 Lines
    Author: douardda
  • P972 mypy wat
    from typing import List
    def f() -> None:
    l1: List[str] = ["foo"]
    l2: List[int] = [42]
    ...
    • Mar 11 2021, 4:28 PM
    • 7 Lines
    • Python
    Author: vlorentz
  • P971 (An Untitled Masterwork)
    ______________ test_swh_model_todict_fromdict[metadata_authority-objects3] _______________
    object_type = 'metadata_authority'
    objects = [MetadataAuthority(type=<MetadataAuthorityType.FORGE: 'forge'>, url='http://example.org/', metadata=ImmutableDict({}))]
    ...
    • Mar 8 2021, 2:13 PM
    • 67 Lines
    Author: douardda
  • P970 deposit loader error in docker environment
    swh-loader-deposit_1 | [2021-03-05 13:29:50,864: INFO/MainProcess] Received task: swh.loader.package.deposit.tasks.LoadDeposit[29c39c67-9d10-4ee7-be6b-9a03c5dbbb8c]
    swh-loader-deposit_1 | [2021-03-05 13:29:50,867: DEBUG/ForkPoolWorker-1] Loading config file /loader-deposit.yml
    swh-loader-deposit_1 | [2021-03-05 13:29:51,384: DEBUG/MainProcess] heartbeat_tick : for connection 4c6e0d55bab340648017acbce1649821
    swh-loader-deposit_1 | [2021-03-05 13:29:51,384: DEBUG/MainProcess] heartbeat_tick : Prev sent/recv: 39/2290, now - 41/2446, monotonic - 845447.299477598, last_heartbeat_sent - 845447.299474569, heartbeat int. - 60 for connection 4c6e0d55bab340648017acbce1649821
    swh-loader-deposit_1 | [2021-03-05 13:29:52,977: DEBUG/ForkPoolWorker-1] last snapshot: None
    ...
    • Mar 5 2021, 2:33 PM
    • 43 Lines
    Author: anlambert
  • P969 swh.fuse failure in jenkins
    jenkins@bfa1ac925ae0:/var/lib/jenkins/workspace/DFUSE/tests-on-diff@tmp/swh-fuse$ swh --log-level=DEBUG fs mount -f /tmp/fuse
    WARNING:root:Using default configuration (cannot load custom one)
    Traceback (most recent call last):
    File "/home/jenkins/.local/lib/python3.7/site-packages/swh/fuse/cli.py", line 72, in fuse
    raise ValueError(f"Cannot parse configuration file: {config_file}")
    ...
    • Mar 4 2021, 6:00 PM
    • 47 Lines
    Author: olasd
  • P968 swhgraph.sh
    #!/bin/bash
    mkdir -p /srv/softwareheritage/ssd/seirl/tmp
    java -Xmx400G -server \
    -Djava.io.tmpdir=/srv/softwareheritage/ssd/seirl/tmp \
    ...
    • Mar 4 2021, 12:36 PM
    • 10 Lines
    • Bash Scripting
    Author: seirl
  • P967 (An Untitled Masterwork)
    print("content_raw middle")
    for chunk in r.iter_content(chunk_size=None, decode_unicode=False):
    print("content_raw chunk", repr(chunk))
    yield chunk
    ...
    • Mar 4 2021, 11:40 AM
    • 7 Lines
    Author: vlorentz
  • P966 Snapshot branches queries performances
    Executing the snapshot_get_branches query on db.internal.softwareheritage.org gives unexpected
    performance results:
    - limiting the query to 10 results is fast and responsive
    ...
    • Mar 3 2021, 2:07 PM
    • 150 Lines
    Author: anlambert
  • P965 Belvedere zfs errors
    ```
    The number of checksum errors associated with a ZFS device
    exceeded acceptable levels. ZFS has marked the device as
    degraded.
    ...
    • Mar 3 2021, 11:00 AM
    • 68 Lines
    Author: vsellier
  • P964 Deposit-keycloak tryouts from docker
    Start docker container with docker-compose override on the deposit repository:
    In the following:
    #+begin_src sh
    alias swh-doco-keycloak="docker-compose -f docker-compose.yml -f docker-compose.keycloak.yml -f docker-compose.override.yml"
    ...
    • Feb 25 2021, 11:37 AM
    • 71 Lines
    Author: ardumont
  • P963 FindEarliestRevision
    package org.softwareheritage.graph.utils;
    import it.unimi.dsi.big.webgraph.LazyLongIterator;
    import it.unimi.dsi.fastutil.BigArrays;
    import it.unimi.dsi.fastutil.io.BinIO;
    ...
    • Feb 24 2021, 3:03 PM
    • 61 Lines
    • Java
    Author: seirl
  • P962 loader run through docker
    $ doco exec swh-loader /bin/bash
    swh@2d4da9949cde:/$ swh loader run mercurial https://mercurial-scm.org/repo/hello
    INFO:swh.loader.mercurial.Bundle20Loader:Load origin 'https://mercurial-scm.org/repo/hello' with type 'hg'
    {'status': 'eventful'}
    swh@2d4da9949cde:/$
    • Feb 23 2021, 5:39 PM
    • 5 Lines
    Author: ardumont
  • P961 docker: deposit-keycloak: connection ok
    ```
    from typing import Any, Dict
    import requests
    ...
    • Feb 23 2021, 2:27 PM
    • 123 Lines
    Author: ardumont
  • P960 nomossa on binary stuff
    ardumont@worker3:~% file repo.tar.gz
    repo.tar.gz: POSIX tar archive (GNU)
    ardumont@worker3:~% nomossa repo.tar.gz
    File repo.tar.gz contains license(s) GPL-2.0,LGPL
    ardumont@worker3:~% time nomossa info.txt.gpg
    ...
    • Feb 22 2021, 5:32 PM
    • 15 Lines
    Author: ardumont
  • P959 (An Untitled Masterwork)
    diff --git a/swh/clearlydefined/orchestrator.py b/swh/clearlydefined/orchestrator.py
    index 41beb9b..c35259e 100644
    --- a/swh/clearlydefined/orchestrator.py
    +++ b/swh/clearlydefined/orchestrator.py
    @@ -121,16 +121,16 @@ def map_previously_unmapped_data(storage: StorageInterface, cursor, connection)
    ...
    • Feb 22 2021, 3:45 PM
    • 25 Lines
    • Diff
    Author: vlorentz
  • P958 fossology-nomossa license computations status
    softwareheritage-indexer=> select count(*) from content_fossology_license;
    now | count
    ------------------------------- | ------------
    2021-02-22 10:55:14.408757+00 | 4459106964
    (1 row)
    ...
    • Feb 22 2021, 11:55 AM
    • 829 Lines
    Author: ardumont
  • P957 chardet contextual version conflict for some reasons...
    FAILED .tox/py3/lib/python3.7/site-packages/swh/loader/tests/test_cli.py::test_get_loader - pkg_resources.ContextualVersionConflict: (chardet 4.0.0 (/home/tony/work/inria/repo/swh/swh-environment/swh-loader-core/.tox/py3/lib/python3.7/s...
    FAILED .tox/py3/lib/python3.7/site-packages/swh/loader/tests/test_cli.py::test_run_pypi - AssertionError: assert 1 == 0
    FAILED .tox/py3/lib/python3.7/site-packages/swh/loader/tests/test_cli.py::test_list_help_npm - AssertionError: assert 1 == 0
    ERROR .tox/py3/lib/python3.7/site-packages/swh/loader/package/archive/tests/test_tasks.py::test_tasks_archive_loader - pkg_resources.ContextualVersionConflict: (chardet 4.0.0 (/home/tony/work/inria/repo/swh/swh-environment/swh-loader-co...
    ERROR .tox/py3/lib/python3.7/site-packages/swh/loader/package/cran/tests/test_tasks.py::test_tasks_cran_loader - pkg_resources.ContextualVersionConflict: (chardet 4.0.0 (/home/tony/work/inria/repo/swh/swh-environment/swh-loader-core/.to...
    ...
    • Feb 19 2021, 6:00 PM
    • 11 Lines
    Author: ardumont
  • P956 (An Untitled Masterwork)
    swh-web_1 | django.urls.exceptions.Resolver404: {'tried': [[<URLResolver <module 'swh.web.admin.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/admin/urls.py'> (None:None) '^admin/'>], [<URLPattern '^favicon\.ico$'>], [<URLResolver <module 'swh.web.api.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/api/urls.py'> (None:None) '^api/'>], [<URLResolver <module 'swh.web.browse.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/browse/urls.py'> (None:None) '^browse/'>], [<URLPattern '^$' [name='swh-web-homepage']>], [<URLPattern '^jsreverse/$' [name='js_reverse']>], [<URLPattern '^(?P<swhid>swh:[0-9]+:[a-z]+:[0-9a-f]+.*)/$' [name='browse-swhid']>], [<URLResolver <module 'swh.web.misc.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/urls.py'> (None:None) '^'>, <URLResolver <module 'swh.web.misc.coverage' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/coverage.py'> (None:None) '^'>, <URLPattern '^coverage/$' [name='swh-coverage']>], [<URLResolver <module 'swh.web.misc.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/urls.py'> (None:None) '^'>, <URLPattern '^jslicenses/$' [name='jslicenses']>], [<URLResolver <module 'swh.web.misc.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/urls.py'> (None:None) '^'>, <URLResolver <module 'swh.web.misc.origin_save' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/origin_save.py'> (None:None) '^'>, <URLPattern '^save/$' [name='origin-save']>], [<URLResolver <module 'swh.web.misc.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/urls.py'> (None:None) '^'>, <URLResolver <module 'swh.web.misc.origin_save' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/origin_save.py'> (None:None) '^'>, <URLPattern '^save/(?P<visit_type>.+)/url/(?P<origin_url>.+)/$' [name='origin-save-request']>], [<URLResolver <module 'swh.web.misc.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/urls.py'> (None:None) '^'>, <URLResolver <module 'swh.web.misc.origin_save' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/origin_save.py'> (None:None) '^'>, <URLPattern '^save/types/list/$' [name='origin-save-types-list']>], [<URLResolver <module 'swh.web.misc.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/urls.py'> (None:None) '^'>, <URLResolver <module 'swh.web.misc.origin_save' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/origin_save.py'> (None:None) '^'>, <URLPattern '^save/requests/list/(?P<status>.+)/$' [name='origin-save-requests-list']>], [<URLResolver <module 'swh.web.misc.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/urls.py'> (None:None) '^'>, <URLResolver <module 'swh.web.misc.origin_save' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/origin_save.py'> (None:None) '^'>, <URLPattern '^save/task/info/(?P<save_request_id>.+)/' [name='origin-save-task-info']>], [<URLResolver <module 'swh.web.misc.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/urls.py'> (None:None) '^'>, <URLPattern '^stat_counters/' [name='stat-counters']>], [<URLResolver <module 'swh.web.misc.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/urls.py'> (None:None) '^'>, <URLResolver <module 'swh.web.misc.badges' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/badges.py'> (None:None) '^'>, <URLPattern '^badge/(?P<object_type>[a-z]+)/(?P<object_id>.+)/$' [name='swh-badge']>], [<URLResolver <module 'swh.web.misc.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/urls.py'> (None:None) '^'>, <URLResolver <module 'swh.web.misc.badges' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/badges.py'> (None:None) '^'>, <URLPattern '^badge/(?P<object_swhid>swh:[0-9]+:[a-z]+:[0-9a-f]+.*)/$' [name='swh-badge-swhid']>], [<URLResolver <module 'swh.web.misc.urls' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/misc/urls.py'> (None:None) '^'>, <URLPattern '^metrics/prometheus/$' [name='metrics-prometheus']>], [<URLResolver <module 'swh.web.auth.views' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/auth/views.py'> (None:None) '^'>, <URLPattern '^oidc/login/$' [name='oidc-login']>], [<URLResolver <module 'swh.web.auth.views' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/auth/views.py'> (None:None) '^'>, <URLPattern '^oidc/login-complete/$' [name='oidc-login-complete']>], [<URLResolver <module 'swh.web.auth.views' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/auth/views.py'> (None:None) '^'>, <URLPattern '^oidc/logout/$' [name='oidc-logout']>], [<URLResolver <module 'swh.web.auth.views' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/auth/views.py'> (None:None) '^'>, <URLPattern '^oidc/generate-bearer-token/$' [name='oidc-generate-bearer-token']>], [<URLResolver <module 'swh.web.auth.views' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/auth/views.py'> (None:None) '^'>, <URLPattern '^oidc/generate-bearer-token-complete/$' [name='oidc-generate-bearer-token-complete']>], [<URLResolver <module 'swh.web.auth.views' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/auth/views.py'> (None:None) '^'>, <URLPattern '^oidc/list-bearer-token/$' [name='oidc-list-bearer-tokens']>], [<URLResolver <module 'swh.web.auth.views' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/auth/views.py'> (None:None) '^'>, <URLPattern '^oidc/get-bearer-token/$' [name='oidc-get-bearer-token']>], [<URLResolver <module 'swh.web.auth.views' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/auth/views.py'> (None:None) '^'>, <URLPattern '^oidc/revoke-bearer-tokens/$' [name='oidc-revoke-bearer-tokens']>], [<URLResolver <module 'swh.web.auth.views' from '/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/auth/views.py'> (None:None) '^'>, <URLPattern '^oidc/profile/$' [name='oidc-profile']>], [<URLPattern '^logout/$' [name='logout']>], [<URLPattern '^static/(?P<path>.*)$'>]], 'path': 'favicon.ico/'}
    swh-web_1 |
    swh-web_1 | During handling of the above exception, another exception occurred:
    swh-web_1 |
    swh-web_1 | Traceback (most recent call last):
    ...
    • Feb 19 2021, 3:33 PM
    • 29 Lines
    Author: ardumont
  • P955 swh-search journal-client offsets before T3060
    "swh.journal.objects.origin_visit_status",26,386129
    "swh.journal.objects.origin_visit_status",12,386575
    "swh.journal.objects.origin_visit_status",35,385683
    "swh.journal.objects.origin",54,8166
    "swh.journal.objects.origin_visit",55,195150
    ...
    • Feb 18 2021, 4:25 PM
    • 192 Lines
    Author: vsellier
  • P954 TestElasticsearchSearch.test_origin_intrinsic_metadata_date
    self = <swh.search.tests.test_elasticsearch.TestElasticsearchSearch testMethod=test_origin_intrinsic_metadata_date>
    def test_origin_intrinsic_metadata_date(self):
    """Checks inserting a date-like in a field does not update the mapping to
    require every document uses a date in that field; or that search queries
    ...
    • Feb 18 2021, 11:32 AM
    • 69 Lines
    • Python
    Author: vlorentz
  • P953 origin index mapping with metadata
    {
    "origin" : {
    "mappings" : {
    "properties" : {
    "has_visits" : {
    ...
    • Feb 18 2021, 11:14 AM
    • 644 Lines
    • JSON
    Author: vsellier
  • P952 (An Untitled Masterwork)
    swh-loader_1 | [2021-02-17 12:59:37,002: INFO/MainProcess] Received task: swh.loader.mercurial.tasks.LoadMercurial[d8a7ffea-56e0-4c93-9c88-6d1efe05a5b1]
    swh-loader_1 | [2021-02-17 12:59:37,005: DEBUG/ForkPoolWorker-1] Loading config file /loader.yml
    swh-loader_1 | [2021-02-17 12:59:39,814: INFO/ForkPoolWorker-1] Load origin 'https://hg.sr.ht/~zimoun/hello-example' with type 'hg'
    swh-loader_1 | [2021-02-17 12:59:39,815: DEBUG/ForkPoolWorker-1] Cloning https://hg.sr.ht/~zimoun/hello-example to /tmp/swh.loader.mercurial.0bko1hw5-46 with timeout 7200 seconds
    swh-loader_1 | [2021-02-17 12:59:40,664: ERROR/ForkPoolWorker-1] Loading failure, updating to `not_found` status
    ...
    • Feb 17 2021, 2:01 PM
    • 16 Lines
    Author: anlambert
  • P951 Cassandra dependencies's hell
    ```
    root@73dd2e784003:/# aptitude why openjdk-17-jre
    i cassandra Depends openjdk-8-jre-headless | java8-runtime
    i A openjdk-17-jre Provides java8-runtime
    ```
    ...
    • Feb 16 2021, 12:41 PM
    • 12 Lines
    Author: vsellier
  • P950 (An Untitled Masterwork)
    13:47 $ swh --log-level DEBUG search -C ~/.config/swh/search.yml journal-client objects -o origin_visit
    DEBUG:swh.core.config:Loading config file /home/anlambert/.config/swh/search.yml
    DEBUG:swh.journal.client:Consumer settings: {'debug': 'consumer', 'bootstrap.servers': 'journal0.internal.staging.swh.network', 'auto.offset.reset': 'earliest', 'group.id': 'anlambert.search', 'on_commit': <function _on_commit at 0x7f7168a54268>, 'error_cb': <function _error_cb at 0x7f7168a541e0>, 'enable.auto.commit': False, 'logger': <Logger swh.journal.client.rdkafka (DEBUG)>}
    Traceback (most recent call last):
    File "/home/anlambert/.virtualenvs/swh/bin/swh", line 8, in <module>
    ...
    • Feb 15 2021, 2:14 PM
    • 31 Lines
    Author: anlambert
  • P949 (An Untitled Masterwork)
    [15/Feb/2021 13:02:16] [DEBUG] urllib3.connectionpool._new_conn:231 - Starting new HTTP connection (1): search-esnode0.internal.staging.swh.network:9200
    [15/Feb/2021 13:02:17] [WARNING] elasticsearch.log_request_fail:281 - POST http://search-esnode0.internal.staging.swh.network:9200/origin/_search?size=100 [status:N/A request:0.013s]
    Traceback (most recent call last):
    File "/home/anlambert/.virtualenvs/swh/lib/python3.7/site-packages/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
    ...
    • Feb 15 2021, 2:11 PM
    • 10 Lines
    Author: anlambert
  • P948 (An Untitled Masterwork)
    [15/Feb/2021 12:58:32] [DEBUG] urllib3.connectionpool._new_conn:231 - Starting new HTTP connection (1): search0.internal.staging.swh.network:5010
    [15/Feb/2021 12:58:32] [DEBUG] urllib3.util.retry.increment:575 - Incremented Retry for (url='/origin/search'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
    [15/Feb/2021 12:58:32] [WARNING] urllib3.connectionpool.urlopen:781 - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdd08e4a240>: Failed to establish a new connection: [Errno 111] Connection refused')': /origin/search
    [15/Feb/2021 12:58:32] [DEBUG] urllib3.connectionpool._new_conn:231 - Starting new HTTP connection (2): search0.internal.staging.swh.network:5010
    [15/Feb/2021 12:58:32] [DEBUG] urllib3.util.retry.increment:575 - Incremented Retry for (url='/origin/search'): Retry(total=1, connect=None, read=None, redirect=None, status=None)
    ...
    • Feb 15 2021, 2:09 PM
    • 11 Lines
    Author: anlambert
  • P947 mercurial errors when no repository is found
    In [19]: exc
    Out[19]:
    hglib.error.CommandError('hg',
    'clone',
    b'--',
    ...
    • Feb 11 2021, 6:42 PM
    • 63 Lines
    Author: ardumont
  • P946 production deployment of swh-search
    @startuml
    title swh-search environment (production)
    node moma {
    component webapp
    ...
    • Feb 11 2021, 11:29 AM
    • 54 Lines
    Author: vsellier
  • P945 origin_visit_stats scenario
    # S1
    ## Old code
    | date | snapshot | | last_ev | last_unev | Snap |
    | ---- | -------- | --- | -------- | --------- | ---- |
    | 2022 | S2 | | 2022 | | S2 |
    ...
    • Feb 10 2021, 10:00 AM
    • 55 Lines
    Author: vsellier
  • P944 search_journal_client @origin offsets matching the index backup (csv)
    "swh.journal.objects.origin_visit_status",26,335718
    "swh.journal.objects.origin_visit_status",12,336502
    "swh.journal.objects.origin_visit_status",35,335346
    "swh.journal.objects.origin",54,8082
    "swh.journal.objects.origin_visit",55,169851
    ...
    • Feb 9 2021, 6:33 PM
    • 192 Lines
    Author: vsellier
  • P943 swh-fuse 1.0 pip freeze
    aiosqlite==0.16.1
    async-generator==1.10
    attrs==20.3.0
    attrs-strict==0.2.0
    certifi==2020.12.5
    ...
    • Feb 9 2021, 2:34 PM
    • 33 Lines
    Author: haltode
  • P942 jitsiroom - get a fresh URL for a Jitsi room
    #!/bin/bash
    # SPDX-FileCopyrightText: © 2021 Stefano Zacchiroli <zack@upsilon.cc>
    # SPDX-License-Identifier: GPL-3.0-or-later
    ...
    • Feb 8 2021, 10:31 AM
    • 29 Lines
    • Bash Scripting
    Author: zack
  • P941 storage build stuck
    last which worked:
    ```
    tox -- -x -s
    GLOB sdist-make: /home/tony/work/inria/repo/swh/swh-environment/swh-storage/setup.py
    black installed: appdirs==1.4.4,attrs==20.3.0,black==19.10b0,click==7.1.2,pathspec==0.8.1,regex==2020.11.13,toml==0.10.2,typed-ast==1.4.2
    ...
    • Feb 5 2021, 3:48 PM
    • 325 Lines
    Author: ardumont
  • P940 Elasticsearch field conversion from string to integer
    #!/usr/bin/env bash
    # set -x
    set -e
    export ES_NODE="192.168.100.61:9200"
    ...
    • Feb 4 2021, 7:46 PM
    • 66 Lines
    • Bash Scripting
    Author: vsellier
  • P939 Command-Line Input
    zack@noziere:/tmp/t$ tree
    .
    ├── a
    ├── b
    ├── c
    ...
    • Feb 4 2021, 12:00 PM
    • 61 Lines
    Author: zack
  • P938 timestamp normalization
    $Timestamp is a decimal representation of the number of seconds since
    1970-01-01 00:00:00 UTC, normalized "the git way"
    (ie. no leading 0 unless the integer part is zero, always a dot, no trailing zero);
    and at most six digits in the decimal part.
    • Feb 3 2021, 3:40 PM
    • 4 Lines
    Author: vlorentz
  • P937 pytest-postgresql recursive dependency
    $ cat /home/dev/swh-environment/swh-storage/swh/storage/tests/test_postgresql_migration.py
    # Copyright (C) 2021 The Software Heritage developers
    # See the AUTHORS file at the top-level directory of this distribution
    # License: GNU General Public License version 3, or any later version
    # See top-level LICENSE file for more information
    ...
    • Feb 3 2021, 2:31 PM
    • 71 Lines
    • Python
    Author: vlorentz
  • P936 Command-Line Input
    #!/bin/bash
    DEST=swh-team@inria.fr
    TEMPLATE_ID="dGY2yP6bSa65aTwxyZSwXg"
    URL="https://hedgedoc.softwareheritage.org"
    ...
    • Feb 2 2021, 3:59 PM
    • 49 Lines
    Author: douardda
  • P935 staging: migrate origin-visit-status script run
    time ./T2968-migrate-origin-visit-status.sh
    Timing is on.
    UPDATE 100000
    Time: 2571.613 ms (00:02.572)
    Timing is on.
    ...
    • Feb 2 2021, 12:10 PM
    • 455 Lines
    Author: ardumont
  • P934 backfill.sh for origin-visit-status
    #!/bin/bash
    objtype=origin_visit_status
    first=0
    ...
    • Feb 1 2021, 11:51 AM
    • 22 Lines
    • Bash Scripting
    Author: ardumont
  • P933 Black error wrong line
    def foo():
    f({**d})
    def bar():
    pass
    ...
    • Jan 29 2021, 3:03 PM
    • 8 Lines
    • Python
    Author: vlorentz
  • P932 launchpad lister incremental run failing
    swhworker@worker0:~$ time SWH_CONFIG_FILENAME=/etc/softwareheritage/lister.yml swh lister run --lister launchpad incremental=True
    Traceback (most recent call last):
    File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.11.0', 'console_scripts', 'swh')()
    ...
    • Jan 28 2021, 5:42 PM
    • 50 Lines
    Author: ardumont
  • P931 staging: cgit instance eclipse: remote server closed connection
    swhworker@worker0:~$ SWH_CONFIG_FILENAME=lister.yml swh lister run --lister cgit url=https://git.eclipse.org/c/ instance=eclipse
    WARNING:swh.lister.cgit.lister:Unexpected HTTP status code 500 on https://git.eclipse.org/c/osbp/org.eclipse.osbp.runtime.functionlibrary.validation.git/
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    ...
    • Jan 28 2021, 5:21 PM
    • 89 Lines
    Author: ardumont
  • P930 (An Untitled Masterwork)
    [
    {"origin":"https://github.com/torvalds/linux","visit:":"67","date":"2020-09-21 21:55:01.586191+00","status":"full","snapshot":"c7beb2432b7e93c4cf6ab09cd194c7c1998df2f9","type":"git"},
    {"origin":"https://github.com/torvalds/linux","visit:":"67","date":"2020-09-21 19:15:24.238712+00","status":"created","snapshot":"","type":"git"},
    {"origin":"https://github.com/torvalds/linux","visit:":"66","date":"2020-09-21 17:12:11.930011+00","status":"partial","snapshot":"","type":"git"},
    {"origin":"https://github.com/torvalds/linux","visit:":"66","date":"2020-09-21 17:07:41.94459+00","status":"created","snapshot":"","type":"git"},
    ...
    • Jan 28 2021, 10:30 AM
    • 72 Lines
    Author: vsellier
  • P929 Staging infrastructure
    ' Related to T2920
    @startuml
    title Staging environment
    actor user
    ...
    • Jan 27 2021, 6:07 PM
    • 231 Lines
    Author: vsellier
  • P928 cgit instances
    softwareheritage-scheduler=> select arguments from task where type='list-cgit';
    arguments
    ------------------------------------------------------------------------------------------------------------
    {"args": [], "kwargs": {"url": "https://git.joeyh.name/"}}
    {"args": [], "kwargs": {"url": "https://git.savannah.gnu.org/cgit/", "instance": "gnu-savannah"}}
    ...
    • Jan 25 2021, 11:17 AM
    • 19 Lines
    Author: ardumont
  • P927 backfill script with the right backfill.yml and logging.yml configuration files
    #!/bin/bash
    objtype=origin_visit_status
    first=0
    ...
    • Jan 20 2021, 6:45 PM
    • 22 Lines
    Author: vsellier
  • P926 (An Untitled Masterwork)
    $ curl 'https://archive.internal.softwareheritage.org/api/1/graph/neighbors/swh:1:cnt:c839dea9e8e6f0528b468214348fee8669b305b2/?direction=backward'
    swh:1:dir:ded70c63b81f585ad410703aab9a2a75c090a655
    swh:1:dir:b94a90cd37b572636244856a02214da21504e754
    swh:1:dir:fa8c090895a208acd20c68f1f7760bac95178725
    swh:1:dir:6ca2e4447601e1ba60c298635407f00b926cb23e
    ...
    • Jan 19 2021, 6:37 PM
    • 9 Lines
    Author: zack
  • P925 scheduler - task archive error
    swhscheduler@scheduler0:~$ /usr/bin/swh scheduler --config-file /etc/softwareheritage/backend/elastic.yml task archive
    Traceback (most recent call last):
    File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.11.0', 'console_scripts', 'swh')()
    File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
    ...
    • Jan 19 2021, 2:40 PM
    • 25 Lines
    Author: vsellier
  • P924 journal as optional deps for swh.scheduler ends up in error
    tox -r -- --pdb --no-cov -x -s -k test_cli_journal_client
    GLOB sdist-make: /home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/setup.py
    black recreate: /home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/black
    black installdeps: black==19.10b0
    black installed: appdirs==1.4.4,attrs==20.3.0,black==19.10b0,click==7.1.2,pathspec==0.8.1,regex==2020.11.13,toml==0.10.2,typed-ast==1.4.2
    ...
    • Jan 19 2021, 11:34 AM
    • 59 Lines
    Author: ardumont
  • P923 Command-Line Input
    This is the mail system at host relay8-d.mail.gandi.net.
    I'm sorry to have to inform you that your message could not
    be delivered to one or more recipients. It's attached below.
    ...
    • Jan 19 2021, 9:40 AM
    • 36 Lines
    Author: zack
  • P922 (An Untitled Masterwork)
    In [8]: def t1(l):
    ...: for i in range(10000):
    ...: l = attr.evolve(l, instance_name=f"titi {i}")
    ...: return l
    ...
    • Jan 15 2021, 11:44 AM
    • 18 Lines
    Author: douardda
  • P921 (An Untitled Masterwork)
    dev@desktop5  ~/swh-environment/swh-scheduler   scheduling-policy ●  python3 -m swh.scheduler.simulator run $((10 * 24 * 3600))
    ^Ctotal time: 7 days, 3:17:46.016887
    Total visits: 11124
    Useless visits: 1259
    Eventful visits: 6347
    ...
    • Jan 15 2021, 10:06 AM
    • 48 Lines
    Author: vlorentz
  • P920 pytest lister celery timeout
    (swh) ~/dev/swh-env/swh-lister$ pytest swh/lister/bitbucket/tests/test_tasks.py
    ==================================================================================== test session starts =====================================================================================
    platform linux -- Python 3.7.3, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
    rootdir: /home/tenma/dev/swh-env/swh-lister, configfile: pytest.ini
    plugins: flask-1.1.0, requests-mock-1.8.0, hypothesis-5.42.3, cov-2.10.1, xdist-2.1.0, django-4.1.0, postgresql-2.5.2, forked-1.3.0, asyncio-0.14.0, mock-3.3.1, dash-1.18.0, swh.journal-0.5.2.dev2+g8dd9712, swh.core-0.11.1.dev7+gd33457e
    ...
    • Jan 12 2021, 8:11 PM
    • 212 Lines
    Author: tenma
  • P919 tox packages celery timeout pkg_resources.ContextualVersionConflict
    py3 installed: aiohttp==3.7.3,aiohttp-utils==3.1.1,amqp==5.0.2,arrow==0.17.0,async-timeout==3.0.1,attrs==20.3.0,attrs-strict==0.2.0,beautifulsoup4==4.9.3,billiard==3.6.3.0,blinker==1.4,cassandra-driver==3.24.0,celery==5.0.5,certifi==2020.12.5,cffi==1.14.4,chardet==4.0.0,click==7.1.2,click-didyoumean==0.0.3,click-plugins==1.1.1,click-repl==0.1.6,coverage==5.3.1,cryptography==3.3.1,decorator==4.4.2,Deprecated==1.2.10,distro==1.5.0,elasticsearch==7.10.1,Flask==1.1.2,geomet==0.2.1.post1,gunicorn==20.0.4,httplib2==0.18.1,humanize==3.2.0,hypothesis==6.0.0,idna==2.10,importlib-metadata==3.4.0,iniconfig==1.1.1,iso8601==0.1.13,itsdangerous==1.1.0,jeepney==0.6.0,Jinja2==2.11.2,keyring==21.8.0,kombu==5.0.2,launchpadlib==1.10.13,lazr.restfulclient==0.14.3,lazr.uri==1.0.5,MarkupSafe==1.1.1,mirakuru==2.3.0,msgpack==1.0.2,multidict==5.1.0,mypy==0.790,mypy-extensions==0.4.3,oauthlib==3.1.0,packaging==20.8,pbr==5.5.1,pg8000==1.16.6,pika==1.1.0,pluggy==0.13.1,port-for==0.4,prompt-toolkit==3.0.10,psutil==5.8.0,psycopg2==2.8.6...
    • Jan 12 2021, 8:03 PM
    • 1 Line
    Author: tenma
  • P918 2021 New lister template
    # Copyright (C) 2021 The Software Heritage developers
    # See the AUTHORS file at the top-level directory of this distribution
    # License: GNU General Public License version 3, or any later version
    # See top-level LICENSE file for more information
    ...
    • Jan 11 2021, 5:39 PM
    • 93 Lines
    • Python
    Author: tenma
  • P917 Black crash
    (b"a" f"{b}")
    (
    "foo",
    "bar"
    ...
    • Jan 11 2021, 4:47 PM
    • 6 Lines
    Author: vlorentz
  • P916 (An Untitled Masterwork)
    ============================================== ERRORS ==============================================
    _____________________ ERROR at setup of TestCassandraStorage.test_content_add ______________________
    cassandra_cluster = (['127.0.0.1'], 58071)
    ...
    • Jan 7 2021, 3:25 PM
    • 94 Lines
    Author: douardda
  • P915 (An Untitled Masterwork)
    SWH_CASSANDRA_LOG=toto.log pytest swh/storage/tests/test_cassandra.py -k test_extid -x -s
    ======================================= test session starts ========================================
    platform linux -- Python 3.9.1, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
    rootdir: /home/ddouard/swh/swh-environment/swh-storage, configfile: pytest.ini
    plugins: dash-1.18.1, postgresql-2.5.2, asyncio-0.14.0, django-4.1.0, requests-mock-1.8.0, forked-1.3.0, mock-3.3.1, hypothesis-5.43.3, xdist-2.1.0, flask-1.1.0, swh.core-0.11.1.dev3+g777ea18, swh.journal-0.6.1
    ...
    • Jan 7 2021, 3:05 PM
    • 15 Lines
    Author: douardda
  • P914 (An Untitled Masterwork)
    Hello,
    Quoting and replying to your questions below.
    >> 1) is it possible to recover the SWHID information using the 'slug'?
    ...
    • Jan 5 2021, 10:41 AM
    • 51 Lines
    Author: ardumont
  • P913 random es_search with word typing simulation
    #!/bin/bash -eu
    searched=1
    total=5
    ...
    • Jan 4 2021, 9:40 AM
    • 73 Lines
    • Bash Scripting
    Author: vsellier
  • P912 Command-Line Input
    <?xml version="1.0" encoding="utf-8"?>
    <entry xmlns="http://www.w3.org/2005/Atom"
    xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0"
    xmlns:swh="https://www.softwareheritage.org/schema/2018/deposit">
    <title>GNU Hello</title>
    ...
    • Dec 30 2020, 1:40 PM
    • 30 Lines
    Author: olasd
  • P911 Command-Line Input
    $ du -shc /srv/softwareheritage/hdd/graph/2020-05-20/{content,directory,origin_visit,release,revision,snapshot}/
    175G /srv/softwareheritage/hdd/graph/2020-05-20/content
    4,6T /srv/softwareheritage/hdd/graph/2020-05-20/directory
    54G /srv/softwareheritage/hdd/graph/2020-05-20/origin_visit
    919M /srv/softwareheritage/hdd/graph/2020-05-20/release
    ...
    • Dec 23 2020, 11:17 PM
    • 8 Lines
    Author: zack
  • P910 Command-Line Input
    $ nmcli connection export Software\ Heritage
    client
    remote 'louvre.softwareheritage.org'
    ca '/home/zack/swh/vpn/softwareheritage-ca.crt'
    cert '/home/zack/swh/vpn/softwareheritage.crt'
    ...
    • Dec 23 2020, 6:49 PM
    • 17 Lines
    Author: zack
First
Prev
Next
About · Developer information · Legal