Page MenuHomeSoftware Heritage
Create Paste
Paste Active Pastes

Active Pastes
Use Results
Edit Query

  • P938 timestamp normalization
    $Timestamp is a decimal representation of the number of seconds since
    1970-01-01 00:00:00 UTC, normalized "the git way"
    (ie. no leading 0 unless the integer part is zero, always a dot, no trailing zero);
    and at most six digits in the decimal part.
    • Feb 3 2021, 3:40 PM
    • 4 Lines
    Author: vlorentz
  • P937 pytest-postgresql recursive dependency
    $ cat /home/dev/swh-environment/swh-storage/swh/storage/tests/test_postgresql_migration.py
    # Copyright (C) 2021 The Software Heritage developers
    # See the AUTHORS file at the top-level directory of this distribution
    # License: GNU General Public License version 3, or any later version
    # See top-level LICENSE file for more information
    ...
    • Feb 3 2021, 2:31 PM
    • 71 Lines
    • Python
    Author: vlorentz
  • P936 Command-Line Input
    #!/bin/bash
    DEST=swh-team@inria.fr
    TEMPLATE_ID="dGY2yP6bSa65aTwxyZSwXg"
    URL="https://hedgedoc.softwareheritage.org"
    ...
    • Feb 2 2021, 3:59 PM
    • 49 Lines
    Author: douardda
  • P935 staging: migrate origin-visit-status script run
    time ./T2968-migrate-origin-visit-status.sh
    Timing is on.
    UPDATE 100000
    Time: 2571.613 ms (00:02.572)
    Timing is on.
    ...
    • Feb 2 2021, 12:10 PM
    • 455 Lines
    Author: ardumont
  • P934 backfill.sh for origin-visit-status
    #!/bin/bash
    objtype=origin_visit_status
    first=0
    ...
    • Feb 1 2021, 11:51 AM
    • 22 Lines
    • Bash Scripting
    Author: ardumont
  • P933 Black error wrong line
    def foo():
    f({**d})
    def bar():
    pass
    ...
    • Jan 29 2021, 3:03 PM
    • 8 Lines
    • Python
    Author: vlorentz
  • P932 launchpad lister incremental run failing
    swhworker@worker0:~$ time SWH_CONFIG_FILENAME=/etc/softwareheritage/lister.yml swh lister run --lister launchpad incremental=True
    Traceback (most recent call last):
    File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.11.0', 'console_scripts', 'swh')()
    ...
    • Jan 28 2021, 5:42 PM
    • 50 Lines
    Author: ardumont
  • P931 staging: cgit instance eclipse: remote server closed connection
    swhworker@worker0:~$ SWH_CONFIG_FILENAME=lister.yml swh lister run --lister cgit url=https://git.eclipse.org/c/ instance=eclipse
    WARNING:swh.lister.cgit.lister:Unexpected HTTP status code 500 on https://git.eclipse.org/c/osbp/org.eclipse.osbp.runtime.functionlibrary.validation.git/
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    ...
    • Jan 28 2021, 5:21 PM
    • 89 Lines
    Author: ardumont
  • P930 (An Untitled Masterwork)
    [
    {"origin":"https://github.com/torvalds/linux","visit:":"67","date":"2020-09-21 21:55:01.586191+00","status":"full","snapshot":"c7beb2432b7e93c4cf6ab09cd194c7c1998df2f9","type":"git"},
    {"origin":"https://github.com/torvalds/linux","visit:":"67","date":"2020-09-21 19:15:24.238712+00","status":"created","snapshot":"","type":"git"},
    {"origin":"https://github.com/torvalds/linux","visit:":"66","date":"2020-09-21 17:12:11.930011+00","status":"partial","snapshot":"","type":"git"},
    {"origin":"https://github.com/torvalds/linux","visit:":"66","date":"2020-09-21 17:07:41.94459+00","status":"created","snapshot":"","type":"git"},
    ...
    • Jan 28 2021, 10:30 AM
    • 72 Lines
    Author: vsellier
  • P929 Staging infrastructure
    ' Related to T2920
    @startuml
    title Staging environment
    actor user
    ...
    • Jan 27 2021, 6:07 PM
    • 231 Lines
    Author: vsellier
  • P928 cgit instances
    softwareheritage-scheduler=> select arguments from task where type='list-cgit';
    arguments
    ------------------------------------------------------------------------------------------------------------
    {"args": [], "kwargs": {"url": "https://git.joeyh.name/"}}
    {"args": [], "kwargs": {"url": "https://git.savannah.gnu.org/cgit/", "instance": "gnu-savannah"}}
    ...
    • Jan 25 2021, 11:17 AM
    • 19 Lines
    Author: ardumont
  • P927 backfill script with the right backfill.yml and logging.yml configuration files
    #!/bin/bash
    objtype=origin_visit_status
    first=0
    ...
    • Jan 20 2021, 6:45 PM
    • 22 Lines
    Author: vsellier
  • P926 (An Untitled Masterwork)
    $ curl 'https://archive.internal.softwareheritage.org/api/1/graph/neighbors/swh:1:cnt:c839dea9e8e6f0528b468214348fee8669b305b2/?direction=backward'
    swh:1:dir:ded70c63b81f585ad410703aab9a2a75c090a655
    swh:1:dir:b94a90cd37b572636244856a02214da21504e754
    swh:1:dir:fa8c090895a208acd20c68f1f7760bac95178725
    swh:1:dir:6ca2e4447601e1ba60c298635407f00b926cb23e
    ...
    • Jan 19 2021, 6:37 PM
    • 9 Lines
    Author: zack
  • P925 scheduler - task archive error
    swhscheduler@scheduler0:~$ /usr/bin/swh scheduler --config-file /etc/softwareheritage/backend/elastic.yml task archive
    Traceback (most recent call last):
    File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.11.0', 'console_scripts', 'swh')()
    File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
    ...
    • Jan 19 2021, 2:40 PM
    • 25 Lines
    Author: vsellier
  • P924 journal as optional deps for swh.scheduler ends up in error
    tox -r -- --pdb --no-cov -x -s -k test_cli_journal_client
    GLOB sdist-make: /home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/setup.py
    black recreate: /home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/.tox/black
    black installdeps: black==19.10b0
    black installed: appdirs==1.4.4,attrs==20.3.0,black==19.10b0,click==7.1.2,pathspec==0.8.1,regex==2020.11.13,toml==0.10.2,typed-ast==1.4.2
    ...
    • Jan 19 2021, 11:34 AM
    • 59 Lines
    Author: ardumont
  • P923 Command-Line Input
    This is the mail system at host relay8-d.mail.gandi.net.
    I'm sorry to have to inform you that your message could not
    be delivered to one or more recipients. It's attached below.
    ...
    • Jan 19 2021, 9:40 AM
    • 36 Lines
    Author: zack
  • P922 (An Untitled Masterwork)
    In [8]: def t1(l):
    ...: for i in range(10000):
    ...: l = attr.evolve(l, instance_name=f"titi {i}")
    ...: return l
    ...
    • Jan 15 2021, 11:44 AM
    • 18 Lines
    Author: douardda
  • P921 (An Untitled Masterwork)
    dev@desktop5  ~/swh-environment/swh-scheduler   scheduling-policy ●  python3 -m swh.scheduler.simulator run $((10 * 24 * 3600))
    ^Ctotal time: 7 days, 3:17:46.016887
    Total visits: 11124
    Useless visits: 1259
    Eventful visits: 6347
    ...
    • Jan 15 2021, 10:06 AM
    • 48 Lines
    Author: vlorentz
  • P920 pytest lister celery timeout
    (swh) ~/dev/swh-env/swh-lister$ pytest swh/lister/bitbucket/tests/test_tasks.py
    ==================================================================================== test session starts =====================================================================================
    platform linux -- Python 3.7.3, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
    rootdir: /home/tenma/dev/swh-env/swh-lister, configfile: pytest.ini
    plugins: flask-1.1.0, requests-mock-1.8.0, hypothesis-5.42.3, cov-2.10.1, xdist-2.1.0, django-4.1.0, postgresql-2.5.2, forked-1.3.0, asyncio-0.14.0, mock-3.3.1, dash-1.18.0, swh.journal-0.5.2.dev2+g8dd9712, swh.core-0.11.1.dev7+gd33457e
    ...
    • Jan 12 2021, 8:11 PM
    • 212 Lines
    Author: tenma
  • P919 tox packages celery timeout pkg_resources.ContextualVersionConflict
    py3 installed: aiohttp==3.7.3,aiohttp-utils==3.1.1,amqp==5.0.2,arrow==0.17.0,async-timeout==3.0.1,attrs==20.3.0,attrs-strict==0.2.0,beautifulsoup4==4.9.3,billiard==3.6.3.0,blinker==1.4,cassandra-driver==3.24.0,celery==5.0.5,certifi==2020.12.5,cffi==1.14.4,chardet==4.0.0,click==7.1.2,click-didyoumean==0.0.3,click-plugins==1.1.1,click-repl==0.1.6,coverage==5.3.1,cryptography==3.3.1,decorator==4.4.2,Deprecated==1.2.10,distro==1.5.0,elasticsearch==7.10.1,Flask==1.1.2,geomet==0.2.1.post1,gunicorn==20.0.4,httplib2==0.18.1,humanize==3.2.0,hypothesis==6.0.0,idna==2.10,importlib-metadata==3.4.0,iniconfig==1.1.1,iso8601==0.1.13,itsdangerous==1.1.0,jeepney==0.6.0,Jinja2==2.11.2,keyring==21.8.0,kombu==5.0.2,launchpadlib==1.10.13,lazr.restfulclient==0.14.3,lazr.uri==1.0.5,MarkupSafe==1.1.1,mirakuru==2.3.0,msgpack==1.0.2,multidict==5.1.0,mypy==0.790,mypy-extensions==0.4.3,oauthlib==3.1.0,packaging==20.8,pbr==5.5.1,pg8000==1.16.6,pika==1.1.0,pluggy==0.13.1,port-for==0.4,prompt-toolkit==3.0.10,psutil==5.8.0,psycopg2==2.8.6...
    • Jan 12 2021, 8:03 PM
    • 1 Line
    Author: tenma
  • P918 2021 New lister template
    # Copyright (C) 2021 The Software Heritage developers
    # See the AUTHORS file at the top-level directory of this distribution
    # License: GNU General Public License version 3, or any later version
    # See top-level LICENSE file for more information
    ...
    • Jan 11 2021, 5:39 PM
    • 93 Lines
    • Python
    Author: tenma
  • P917 Black crash
    (b"a" f"{b}")
    (
    "foo",
    "bar"
    ...
    • Jan 11 2021, 4:47 PM
    • 6 Lines
    Author: vlorentz
  • P916 (An Untitled Masterwork)
    ============================================== ERRORS ==============================================
    _____________________ ERROR at setup of TestCassandraStorage.test_content_add ______________________
    cassandra_cluster = (['127.0.0.1'], 58071)
    ...
    • Jan 7 2021, 3:25 PM
    • 94 Lines
    Author: douardda
  • P915 (An Untitled Masterwork)
    SWH_CASSANDRA_LOG=toto.log pytest swh/storage/tests/test_cassandra.py -k test_extid -x -s
    ======================================= test session starts ========================================
    platform linux -- Python 3.9.1, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
    rootdir: /home/ddouard/swh/swh-environment/swh-storage, configfile: pytest.ini
    plugins: dash-1.18.1, postgresql-2.5.2, asyncio-0.14.0, django-4.1.0, requests-mock-1.8.0, forked-1.3.0, mock-3.3.1, hypothesis-5.43.3, xdist-2.1.0, flask-1.1.0, swh.core-0.11.1.dev3+g777ea18, swh.journal-0.6.1
    ...
    • Jan 7 2021, 3:05 PM
    • 15 Lines
    Author: douardda
  • P914 (An Untitled Masterwork)
    Hello,
    Quoting and replying to your questions below.
    >> 1) is it possible to recover the SWHID information using the 'slug'?
    ...
    • Jan 5 2021, 10:41 AM
    • 51 Lines
    Author: ardumont
  • P913 random es_search with word typing simulation
    #!/bin/bash -eu
    searched=1
    total=5
    ...
    • Jan 4 2021, 9:40 AM
    • 73 Lines
    • Bash Scripting
    Author: vsellier
  • P912 Command-Line Input
    <?xml version="1.0" encoding="utf-8"?>
    <entry xmlns="http://www.w3.org/2005/Atom"
    xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0"
    xmlns:swh="https://www.softwareheritage.org/schema/2018/deposit">
    <title>GNU Hello</title>
    ...
    • Dec 30 2020, 1:40 PM
    • 30 Lines
    Author: olasd
  • P911 Command-Line Input
    $ du -shc /srv/softwareheritage/hdd/graph/2020-05-20/{content,directory,origin_visit,release,revision,snapshot}/
    175G /srv/softwareheritage/hdd/graph/2020-05-20/content
    4,6T /srv/softwareheritage/hdd/graph/2020-05-20/directory
    54G /srv/softwareheritage/hdd/graph/2020-05-20/origin_visit
    919M /srv/softwareheritage/hdd/graph/2020-05-20/release
    ...
    • Dec 23 2020, 11:17 PM
    • 8 Lines
    Author: zack
  • P910 Command-Line Input
    $ nmcli connection export Software\ Heritage
    client
    remote 'louvre.softwareheritage.org'
    ca '/home/zack/swh/vpn/softwareheritage-ca.crt'
    cert '/home/zack/swh/vpn/softwareheritage.crt'
    ...
    • Dec 23 2020, 6:49 PM
    • 17 Lines
    Author: zack
  • P909 Command-Line Input
    11547 lun 21 root (0,6K) SMART error (OfflineUncorrectableSector) detected on host: db1
    11548 lun 21 root (0,6K) SMART error (CurrentPendingSector) detected on host: esnode1
    • Dec 22 2020, 3:46 PM
    • 2 Lines
    Author: zack
  • P908 (An Untitled Masterwork)
    stat("/tmp/pip-pip-egg-info-b5go659r/subvertpy.egg-info/PKG-INFO", {st_mode=S_IFREG|0644, st_size=1040, ...}) = 0
    openat(AT_FDCWD, "/tmp/pip-pip-egg-info-b5go659r/subvertpy.egg-info/PKG-INFO", O_RDONLY|O_CLOEXEC) = 4
    fstat(4, {st_mode=S_IFREG|0644, st_size=1040, ...}) = 0
    ioctl(4, TCGETS, 0x7ffc2aa5cc80) = -1 ENOTTY (Inappropriate ioctl for device)
    lseek(4, 0, SEEK_CUR) = 0
    ...
    • Dec 22 2020, 3:00 PM
    • 10 Lines
    Author: vlorentz
  • P907 failing deposit...
    Dec 15 12:32:27 deposit python3[4016621]: 2020-12-15 12:32:27 [4016621] swh.deposit.api.common:WARNING ########### swhid: swh:1:dir:3b276e18c9b1e70eb84068e3109c0945fc2404eb
    Dec 15 12:32:27 deposit python3[4016621]: 2020-12-15 12:32:27 [4016621] swh.deposit.api.common:WARNING ########## metadata-only deposit
    Dec 15 12:32:27 deposit python3[4016621]: 2020-12-15 12:32:27 [4016621] django.request:ERROR Internal Server Error: /1/swh/
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 34, in inner
    ...
    • Dec 15 2020, 1:50 PM
    • 33 Lines
    Author: ardumont
  • P906 metadata-only deposit with deposit v0.7.3 on staging
    ```
    $ cat deposit-swh-metadata-only-xml
    <?xml version="1.0" encoding="utf-8"?>
    <entry xmlns="http://www.w3.org/2005/Atom"
    xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0"
    ...
    • Dec 15 2020, 12:22 PM
    • 55 Lines
    Author: ardumont
  • P905 swh.search 0.3.3 latest failure
    Dec 11 16:37:34 search0 swh[20181]: Traceback (most recent call last):
    Dec 11 16:37:34 search0 swh[20181]: File "/usr/bin/swh", line 11, in <module>
    Dec 11 16:37:34 search0 swh[20181]: load_entry_point('swh.core==0.11.0', 'console_scripts', 'swh')()
    Dec 11 16:37:34 search0 swh[20181]: File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
    Dec 11 16:37:34 search0 swh[20181]: return swh(auto_envvar_prefix="SWH")
    ...
    • Dec 11 2020, 5:48 PM
    • 60 Lines
    Author: ardumont
  • P904 scheduled nixpkgs and guix sources
    softwareheritage-scheduler=> select * from task where type='load-nixguix' and status != 'disabled';
    -[ RECORD 1 ]----+-----------------------------------------------------------------------------------------------------
    id | 334411727
    type | load-nixguix
    arguments | {"args": [], "kwargs": {"url": "https://nix-community.github.io/nixpkgs-swh/sources-unstable.json"}}
    ...
    • Dec 11 2020, 3:52 PM
    • 21 Lines
    Author: ardumont
  • P903 Command-Line Input
    * **2020-03-01:** significantly improved the efficiency of our scheduler of
    crawler tasks, resulting in much faster archival of GitHub_ repositories,
    both newly created and updated ones
    • Dec 11 2020, 3:29 PM
    • 4 Lines
    Author: zack
  • P902 (An Untitled Masterwork)
    diff --git a/htdocs/.htaccess b/htdocs/.htaccess
    index e5b631c8..5f3882b8 100644
    --- a/htdocs/.htaccess
    +++ b/htdocs/.htaccess
    @@ -6,11 +6,10 @@
    ...
    • Dec 11 2020, 2:58 PM
    • 18 Lines
    Author: anlambert
  • P901 list-gnu periodically
    softwareheritage-scheduler=> select * from task where type='list-gnu-full';
    id | type | arguments | next_run | current_interval | status | policy | retries_left | priority
    -----------+---------------+----------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------
    167876594 | list-gnu-full | {"args": [], "kwargs": {}} | 2020-12-14 18:24:54.097392+00 | 7 days | next_run_not_scheduled | recurring | 3 |
    • Dec 10 2020, 11:20 AM
    • 4 Lines
    Author: ardumont
  • P900 swh-docker-dev error truncated
    03:58:01 tests/test_deposit.py ...FF [ 71%]
    04:04:01 tests/test_git_loader.py . [ 85%]
    04:06:32 tests/test_vault.py . [100%]
    04:07:00
    04:07:00 =================================== FAILURES ===================================
    ...
    • Dec 10 2020, 11:11 AM
    • 78 Lines
    Author: ardumont
  • P899 stacktrace key error `origin_url` ~> should be `id` instead
    swh-search-journal-client-indexed_1 | DEBUG:swh.journal.client.rdkafka:CLOSE [rdkafka#consumer-1] [thrd:app]: Closing consumer
    swh-search-journal-client-indexed_1 | DEBUG:swh.journal.client.rdkafka:CLOSE [rdkafka#consumer-1] [thrd:app]: Waiting for close events
    swh-search-journal-client-indexed_1 | DEBUG:swh.journal.client.rdkafka:REBALANCE [rdkafka#consumer-1] [thrd:main]: Group "swh.search.journal_client" is rebalancing in state up (join-state started) with assignment: unsubscribe
    swh-search-journal-client-indexed_1 | DEBUG:swh.journal.client.rdkafka:UNASSIGN [rdkafka#consumer-1] [thrd:main]: Group "swh.search.journal_client": unassigning 1 partition(s) (v5)
    swh-search-journal-client-indexed_1 | DEBUG:swh.journal.client.rdkafka:LEAVE [rdkafka#consumer-1] [thrd:main]: kafka:9092/1001: Leaving group
    ...
    • Dec 9 2020, 10:36 PM
    • 44 Lines
    Author: ardumont
  • P898 swh-search data out of the swh.journal.indexed.origin_intrinsic_metadata topic
    [{'from_revision': b'"\xcfL+\x83j\x00&VW\x92\xb5\xd1\xb3\xc5\xffl\x1f\xc8+',
    'id': 'https://github.com/codemeta/codemeta',
    'mappings': ('codemeta',),
    'metadata': {'@context': 'https://doi.org/10.5063/schema/codemeta-2.0',
    'author': ({'email': 'cboettig@gmail.com',
    ...
    • Dec 9 2020, 10:25 PM
    • 527 Lines
    Author: ardumont
  • P897 "expected" (?) directory not found first then it goes on cooking that stuff...
    Dec 08 16:28:10 vault python3[2301262]: 2020-12-08 16:28:10 [2301262] root:ERROR directory 822f1edbaaa2ad833ea939c31715ed18f43d6e99 was not found.
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/core/api/asynchronous.py", line 71, in middleware_handler
    return await handler(request)
    File "/usr/lib/python3/dist-packages/swh/core/api/asynchronous.py", line 178, in decorated_meth
    ...
    • Dec 8 2020, 5:42 PM
    • 22 Lines
    Author: ardumont
  • P896 (An Untitled Masterwork)
    diff --git swh/dataset/graph.py swh/dataset/graph.py
    index 52165fe..51e9bb9 100644
    --- swh/dataset/graph.py
    +++ swh/dataset/graph.py
    @@ -46,7 +46,7 @@ def process_messages(messages, config, node_writer, edge_writer, node_set):
    ...
    • Dec 8 2020, 4:26 PM
    • 13 Lines
    • Diff
    Author: seirl
  • P895 Command-Line Input
    _id: pypi/pypi/-/cymem/2.0.5
    _meta:
    schemaVersion: 1.6.1
    updated: '2020-12-07T07:11:19.820Z'
    coordinates:
    ...
    • Dec 8 2020, 2:48 PM
    • 152 Lines
    Author: zack
  • P894 Command-Line Input
    diff --git a/swh/core/api/serializers.py b/swh/core/api/serializers.py
    index 30abc9c..1a6d58f 100644
    --- a/swh/core/api/serializers.py
    +++ b/swh/core/api/serializers.py
    @@ -12,7 +12,6 @@ import types
    ...
    • Dec 8 2020, 12:18 PM
    • 30 Lines
    • Diff
    Author: douardda
  • P893 (An Untitled Masterwork)
    curl -XPUT -H "Content-Type: application/json" http://10.168.130.80:9200/_index_template/origin\?pretty -d '{"index_patterns": "origin", "template": {"settings": { "index": { "number_of_replicas":0, "number_of_shards": 10 } } } } '
    • Dec 4 2020, 5:23 PM
    • 1 Line
    Author: vsellier
  • P892 IO retry unreliable remote
    import logging
    from tenacity import (
    before_sleep_log,
    retry,
    stop_after_attempt,
    ...
    • Dec 4 2020, 3:02 PM
    • 41 Lines
    • Python
    Author: tenma
  • P889 (An Untitled Masterwork)
    version: '2'
    services:
    swh-storage-db:
    volumes:
    ...
    • Dec 2 2020, 12:33 PM
    • 9 Lines
    Author: douardda
  • P888 Counters pipeline
    @startuml
    title Object counters pipeline
    actor browser
    ...
    • Dec 2 2020, 9:15 AM
    • 73 Lines
    Author: vsellier
  • P887 Masterwork From Distant Lands
    diff --git a/swh/vault/backend.py b/swh/vault/backend.py
    index 3009cdd..8712302 100644
    --- a/swh/vault/backend.py
    +++ b/swh/vault/backend.py
    @@ -5,6 +5,7 @@
    ...
    • Dec 1 2020, 6:26 PM
    • 40 Lines
    Author: tenma
  • P886 Complete statistics with temporary tables
    ```
    relid | schemaname | relname | heap_blks_read | heap_blks_hit | idx_blks_read | idx_blks_hit | toast_blks_read | toast_blks_hit | tidx_blks_read | tidx_blks_hit
    ------------+-------------+-----------------------------+----------------+---------------+---------------+--------------+-----------------+----------------+----------------+---------------
    17019 | public | content | 14703454269 | 188847965966 | 24118242299 | 78730108709 | 0 | 0 | 0 | 0
    17086 | public | dbversion | 46 | 32 | 8 | 16 | 0 | 0 | 0 | 0
    ...
    • Dec 1 2020, 10:05 AM
    • 959 Lines
    Author: vsellier
  • P885 indexer error
    ```
    Nov 27 16:17:32 storage1 python3[28595]: 2020-11-27 16:17:32 [28595] root:ERROR there is no unique or exclusion constraint matching the ON CONFLICT specification
    CONTEXT: SQL statement "insert into indexer_configuration(tool_name, tool_version, tool_configuration)
    select tool_name, tool_version, tool_configuration from tmp_indexer_configuration tmp
    on conflict(tool_name, tool_version, tool_configuration) do nothing"
    ...
    • Nov 27 2020, 5:21 PM
    • 35 Lines
    Author: vsellier
  • P884 (An Untitled Masterwork)
    # worker :
    ```
    c9042-d63b-43a3-8eb1-2ed479703417]
    Nov 27 11:41:37 worker0 python3[20266]: [2020-11-27 11:41:37,976: ERROR/ForkPoolWorker-1] Problem when reading contents metadata.
    Traceback (most recent call last):
    ...
    • Nov 27 2020, 12:53 PM
    • 57 Lines
    Author: vsellier
  • P883 Plantuml diagram for origin visits to swh search pipeline
    @startuml
    title Origin visit -> Indexer -> swh search pipeline
    queue swh.journal.objects.origin as o #LightBlue
    queue swh.journal.objects.origin_visit as ov #LightBlue
    ...
    • Nov 27 2020, 10:13 AM
    • 34 Lines
    Author: vsellier
  • P882 swh-indexer-journal-client error on origin_visit
    root@journal0:~# swh indexer --config-file=/etc/softwareheritage/indexer/journal_client.yml journal-client
    Traceback (most recent call last):
    File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.9.1', 'console_scripts', 'swh')()
    File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 135, in main
    ...
    • Nov 26 2020, 3:17 PM
    • 33 Lines
    Author: vsellier
  • P881 (An Untitled Masterwork)
    cache:
    blob:
    path: !!python/object/apply:pathlib.PosixPath
    - /
    - home
    ...
    • Nov 26 2020, 10:03 AM
    • 30 Lines
    Author: haltode
  • P880 (An Untitled Masterwork)
    http https://archive.softwareheritage.org/api/1/vault/directory/f557a108b5878c7d7a46e94f538d6bb65379e62e/
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Age: 0
    Allow: POST, GET, OPTIONS
    ...
    • Nov 25 2020, 10:43 AM
    • 31 Lines
    Author: douardda
  • P879 swh-fuse user testing session 2020-11-24
    Testing session swh-graph:
    - ENOENT shouldn't be logged
    - Use yaml.dump() to display the config (only in debug instead of info)
    - Document how to change the logging level
    ...
    • Nov 24 2020, 3:39 PM
    • 15 Lines
    Author: zack
  • P878 Config library WIP (does not run for now)
    # Copyright (C) 2015-2020 The Software Heritage developers
    # See the AUTHORS file at the top-level directory of this distribution
    # License: GNU General Public License version 3, or any later version
    # See top-level LICENSE file for more information
    ...
    • Nov 24 2020, 2:50 PM
    • 456 Lines
    • Python
    Author: tenma
  • P877 (An Untitled Masterwork)
    (jupyterhub) ddouard@desktop6:~/src/repo2docker$ repo2docker --config cfg.json --image-name test:pystuff --user-name swh --user-id 1000 swh:1:rev:94dca98c006b80309704c717b5d83dff3c1fa3a0
    Picked Swhid content provider.
    Fetching revision 94dca98c006b80309704c717b5d83dff3c1fa3a0 from https://archive.softwareheritage.org/api/1/revision/94dca98c006b80309704c717b5d83dff3c1fa3a0/
    Fetching directory f557a108b5878c7d7a46e94f538d6bb65379e62e from https://archive.softwareheritage.org/api/1/directory/f557a108b5878c7d7a46e94f538d6bb65379e62e/
    Fetching /tmp/repo2docker4onz44zc/.travis.yml from https://archive.softwareheritage.org/api/1/content/sha1_git:eb1b383f0a3ee7a983df05604a5b454448c28d2c/raw/
    ...
    • Nov 24 2020, 2:18 PM
    • 152 Lines
    Author: douardda
  • P876 (An Untitled Masterwork)
    (python:.venv) ↪ tox -e py3 -- -v
    GLOB sdist-make: /home/haltode/work/swh/swh-environment/swh-fuse/setup.py
    py3 inst-nodeps: /home/haltode/work/swh/swh-environment/swh-fuse/.tox/.tmp/package/1/swh.fuse-0.9.2.dev3+gbb4e535.zip
    py3 installed: aiosqlite==0.16.0,async-generator==1.10,attrs==20.3.0,attrs-strict==0.2.0,certifi==2020.11.8,chardet==3.0.4,click==7.1.2,coverage==5.3,Deprecated==1.2.10,docutils==0.16,hypothesis==5.41.2,idna==2.10,iniconfig==1.1.1,iso8601==0.1.13,lockfile==0.12.2,outcome==1.1.0,packaging==20.4,pluggy==0.13.1,psutil==5.7.3,py==1.9.0,pyfuse3==3.1.1,pyparsing==2.4.7,pytest==6.1.2,pytest-cov==2.10.1,python-daemon==2.2.4,python-dateutil==2.8.1,PyYAML==5.3.1,requests==2.25.0,requests-mock==1.8.0,sentry-sdk==0.19.3,six==1.15.0,sniffio==1.2.0,sortedcontainers==2.3.0,swh.core==0.8.0,swh.fuse @ file:///home/haltode/work/swh/swh-environment/swh-fuse/.tox/.tmp/package/1/swh.fuse-0.9.2.dev3%2Bgbb4e535.zip,swh.model==0.9.0,swh.web.client==0.2.2,toml==0.10.2,trio==0.17.0,typing-extensions==3.7.4.3,urllib3==1.26.2,wrapt==1.12.1
    py3 run-test-pre: PYTHONHASHSEED='3708312725'
    ...
    • Nov 24 2020, 1:39 PM
    • 69 Lines
    Author: haltode
  • P875 (An Untitled Masterwork)
    diff --git a/swh/core/db/tests/test_db.py b/swh/core/db/tests/test_db.py
    index bb7f80b..6b7529c 100644
    --- a/swh/core/db/tests/test_db.py
    +++ b/swh/core/db/tests/test_db.py
    @@ -7,11 +7,8 @@ from dataclasses import dataclass
    ...
    • Nov 23 2020, 6:07 PM
    • 147 Lines
    Author: anlambert
  • P873 error during npm loader
    Nov 20 11:30:14 saam python3[2819769]: 2020-11-20 11:30:14 [2819769] root:ERROR __init__() got an unexpected keyword argument 'id'
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/core/api/serializers.py", line 272, in msgpack_loads
    strict_map_key=False,
    File "msgpack/_unpacker.pyx", line 161, in msgpack._unpacker.unpackb
    ...
    • Nov 20 2020, 12:33 PM
    • 52 Lines
    Author: vsellier
  • P872 ES 7.5 tokenization
    $ curl -X PUT "localhost:9200/origin/"
    {"acknowledged":true,"shards_acknowledged":true,"index":"origin"}
    $ curl -X PUT "localhost:9200/origin/_mapping" -H 'Content-Type: application/json' -d'
    ...
    • Nov 20 2020, 12:25 PM
    • 511 Lines
    Author: vlorentz
  • P871 ES 7.10 tokenization
    $ curl -X PUT "localhost:9200/origin/"
    {"acknowledged":true,"shards_acknowledged":true,"index":"origin"}
    $ curl -X PUT "localhost:9200/origin/_mapping" -H 'Content-Type: application/json' -d'
    ...
    • Nov 20 2020, 12:24 PM
    • 432 Lines
    Author: vlorentz
  • P870 loader npm
    root@worker01:~# SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_npm.yml swh loader run npm https://www.npmjs.com/vue
    Traceback (most recent call last):
    File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.8.0', 'console_scripts', 'swh')()
    File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 135, in main
    ...
    • Nov 20 2020, 12:09 PM
    • 27 Lines
    Author: vsellier
  • P869 Command-Line Input
    (swh) zack@noziere:~/swh-environment/swhfs/archive/swh:1:rev:9d76c0b163675505d1a901e5fe5249a2c55609bc$ ls -f history/by-date/2009/12/
    31 22 30 21 19 18 17 15 16 14 11 10 09 08 07 06 05 01 04 03 02
    (swh) zack@noziere:~/swh-environment/swhfs/archive/swh:1:rev:9d76c0b163675505d1a901e5fe5249a2c55609bc$ ls -f history/by-date/2009/12/31
    (swh) zack@noziere:~/swh-environment/swhfs/archive/swh:1:rev:9d76c0b163675505d1a901e5fe5249a2c55609bc$ ls -f history/by-date/2009/12/31
    (swh) zack@noziere:~/swh-environment/swhfs/archive/swh:1:rev:9d76c0b163675505d1a901e5fe5249a2c55609bc$ ls -f history/by-date/2009/12/01
    ...
    • Nov 19 2020, 11:30 AM
    • 6 Lines
    Author: zack
  • P867 Opening staging to the world email
    Hello,
    We recently opened the staging archive [1] and deposit [2] to the world.
    Note that it will be mostly used to preview new features and allow third
    ...
    • Nov 17 2020, 10:16 AM
    • 32 Lines
    Author: ardumont
  • P866 staging: from hal staging to our staging deposit: deposits ok!
    id | reception_date | complete_date | external_id | swhid | status | client_id | collection_id | parent_id | status_detail | swhid_context | check_task_id | load_task_id
    ----+-------------------------------+-------------------------------+--------------------------------------+----------------------------------------------------+---------+-----------+---------------+-----------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+--------------
    21 |...
    • Nov 16 2020, 3:41 PM
    • 5 Lines
    Author: ardumont
  • P864 Command-Line Input
    n.n.n / 2020-11-13
    ==================
    * cache: replace asizeof() call with simpler heuristic
    ...
    • Nov 13 2020, 5:19 PM
    • 11 Lines
    Author: zack
  • P863 (An Untitled Masterwork)
    $ cat .config/swh/global.yml
    swh:
    fuse:
    web-api:
    url: "https://archive.internal.softwareheritage.org/api/1/"
    • Nov 12 2020, 1:16 PM
    • 5 Lines
    Author: zack
  • P862 dns issue
    root@pergamon:~# puppet agent --test
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Nov 9 2020, 12:14 PM
    • 17 Lines
    Author: ardumont
  • P861 staging: rp0: some firewall rules are off
    (cleaned up)
    22:58 <swhbot> icinga PROBLEM: service swh deposit on rp0.internal.staging.swh.network is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 395 bytes in 0.149 second response time
    23:55 <swhbot> icinga PROBLEM: service swh webapp revision log end to end on webapp.internal.staging.swh.network is CRITICAL: HTTP CRITICAL: HTTP/1.1 404 Not Found - pattern not found - 10146 bytes in 0.010 second response time
    09:43 <swhbot> icinga PROBLEM: service journalbeat on rp0.internal.staging.swh.network is WARNING: JOURNAL LAG WARNING - monotonic_lag is 1363s (outside range 0:1200)
    ...
    • Nov 7 2020, 4:56 PM
    • 14 Lines
    Author: ardumont
  • P860 (An Untitled Masterwork)
    ______________________________________________________________________________________ test_api_endpoints_have_cors_header _______________________________________________________________________________________
    client = <django.test.client.Client object at 0x7f0a94071a20>
    @given(directory(), revision())
    ...
    • Nov 6 2020, 4:13 PM
    • 58 Lines
    Author: anlambert
  • P859 (An Untitled Masterwork)
     ddouard  (e) swh   master ✚ 1 ⚑ 1  …/s/s/docs  make
    sphinx-apidoc --ext-viewcode --separate -o apidoc .. . ../setup.py ../swh/*/tests ../swh/*/tests/* ../swh/*/*/tests/* ../swh/*/*/*/tests/* ../swh/*/migrations ../swh/*/migrations/* ../swh/*/*/migrations/* ../swh/*/*/*/migrations/* ../bin/* ../utils/* ../node_modules/* ../swh/*/settings/*
    Creating file apidoc/conftest.rst.
    Creating file apidoc/swh.rst.
    Creating file apidoc/swh.deposit.rst.
    ...
    • Nov 6 2020, 10:02 AM
    • 95 Lines
    Author: douardda
  • P858 (An Untitled Masterwork)
     ddouard  (e) swh   master ✚ 1 ⚑ 1  …/s/s/docs  make
    sphinx-build -b html . _build/html
    Running Sphinx v3.3.0
    loading translations [en]... done
    WARNING: logo file '_static/software-heritage-logo-title-motto-vertical-white.png' does not exist
    ...
    • Nov 6 2020, 10:00 AM
    • 159 Lines
    Author: douardda
  • P857 diff buster-preseed.cfg buster-preseed.vagrant.cfg
    diff buster-preseed.cfg buster-preseed.vagrant.cfg
    296c296
    < grub-pc grub-pc/install_devices multiselect /dev/vda
    ---
    > grub-pc grub-pc/install_devices multiselect /dev/disk/by-id/ata-VBOX_HARDDISK_VBbc5508de-d37f68c1
    ...
    • Nov 5 2020, 4:50 PM
    • 23 Lines
    Author: ardumont
  • P856 new and clearer swh webapps puppet roles
    @startuml
    title Webapp and Deposit roles with and without reverse proxy
    note top of role_swh_rp_webapp : production webapp0 (azure)
    note top of role_swh_rp_webapps : production moma (rocq)
    ...
    • Nov 5 2020, 1:54 PM
    • 25 Lines
    Author: ardumont
  • P855 current swh web apis puppet roles
    @startuml
    title Webapp and Deposit roles with and without reverse proxy
    role_swh_base_api *-- profile_swh_webapp
    note top of role_swh_base_api: - production webapp0 (azure)\n- staging webapp (rocq)
    ...
    • Nov 5 2020, 1:53 PM
    • 19 Lines
    Author: ardumont
  • P854 Command-Line Input
    (swh) ✘  ~/swh-environment/swhfs/archive  time (lf | wc -l)
    519
    ( ls --color=tty -f | wc -l; ) 0,01s user 0,00s system 4% cpu 0,362 total
    (swh) ~/swh-environment/swhfs/archive  time (lf | wc -l)
    519
    ...
    • Nov 5 2020, 10:28 AM
    • 12 Lines
    Author: zack
  • P853 Target staging reverse proxy
    @startuml
    title Staging Reverse Proxy
    participant User
    box "Firewall" #Orange
    ...
    • Nov 4 2020, 3:39 PM
    • 51 Lines
    Author: ardumont
  • P852 current hitch varnish apache gunicorn status on web frontend
    @startuml
    participant User
    box "webserver: moma (webapp, deposit), webapp.staging, deposit.staging..." #LightBlue
    participant FrontendHitch
    participant Varnish
    ...
    • Nov 4 2020, 2:02 PM
    • 41 Lines
    Author: ardumont
  • P851 (An Untitled Masterwork)
    Selecting previously unselected package linux-image-5.9.0-fusebuf-amd64-unsigned.
    (Reading database ... 478590 files and directories currently installed.)
    Preparing to unpack linux-image-5.9.0-fusebuf-amd64-unsigned_5.9.1-2_amd64.deb ...
    Unpacking linux-image-5.9.0-fusebuf-amd64-unsigned (5.9.1-2) ...
    Setting up linux-image-5.9.0-fusebuf-amd64-unsigned (5.9.1-2) ...
    ...
    • Nov 4 2020, 10:25 AM
    • 56 Lines
    Author: haltode
  • P850 rebuild a kernel Debian package with larger FUSE getdirplus() buffer
    apt-get source linux
    cd linux-*/
    $EDITOR debian/config/defines
    # change "debug-info:" line to "false"
    ...
    • Nov 3 2020, 8:08 PM
    • 14 Lines
    Author: zack
  • P849 revision metadata tasks with deprecated policy_update...
    softwareheritage-scheduler=> \x
    softwareheritage-scheduler=> select * from task where type='index-revision-metadata' limit 10;
    -[ RECORD 1 ]----+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------$
    id | 114244463
    type | index-revision-metadata
    ...
    • Nov 3 2020, 2:44 PM
    • 66 Lines
    Author: ardumont
  • P848 revision metadata tasks with deprecated policy_update...
    -[ RECORD 1 ]----+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------$
    id | 114244463
    type | index-revision-metadata
    arguments | {"args": [], "kwargs": {"ids": ["2ade866be83193ef461c8f661bbbaa8cea3a6ac5"], "next_step": {"type": "indexer_origin_metadata", "policy": "oneshot", "arguments": {"args": [], "kwargs": {"origin_head": {"14342257": "2ade86$
    next_run | 2019-01-29 11:11:29.196488+00
    ...
    • Nov 3 2020, 2:42 PM
    • 50 Lines
    Author: ardumont
  • P847 (An Untitled Masterwork)
    import asyncio
    from typing import AsyncGenerator, AsyncIterator
    async def gen() -> AsyncGenerator[int, None]:
    ...
    • Nov 3 2020, 9:48 AM
    • 22 Lines
    • Python
    Author: haltode
  • P846 clean up raw extrinsic metadata
    delete from raw_extrinsic_metadata
    where id >= 'swh:1:snp:0000000000000000000000000000000000000000' and id < 'swh:1:snp:1111111111111111111111111111111111111111'
    and (format = 'replicate-npm-package-json'
    or format = 'pypi-project-json');
    ...
    • Nov 2 2020, 5:20 PM
    • 74 Lines
    Author: ardumont
  • P845 (An Untitled Masterwork)
    14:16 $ ./bin/deploy-on --apt --test moma.internal.softwareheritage.org
    *** swh-deploy: deploying recipes on pergamon.internal.softwareheritage.org...
    INFO -> Using Puppetfile '/etc/puppet/code/environments/production/Puppetfile'
    INFO -> Using Puppetfile '/etc/puppet/code/environments/staging/Puppetfile'
    INFO -> Deploying environment /etc/puppet/code/environments/production
    ...
    • Nov 2 2020, 2:18 PM
    • 133 Lines
    Author: anlambert
  • P844 Command-Line Input
    @startuml arch-context
    ' !includeurl https://raw.githubusercontent.com/RicardoNiepel/C4-PlantUML/master/C4_Container.puml
    !include /home/zack/dati/src/plantuml/C4-PlantUML/C4_Container.puml
    ' title System Context diagram for Software Heritage Filesystem (SwhFS)
    ...
    • Oct 31 2020, 5:50 PM
    • 19 Lines
    Author: zack
  • P843 (An Untitled Masterwork)
    swh-deposit_1 | [2020-10-28 13:52:51 +0000] [34] [DEBUG] GET /deposit/1/servicedocument/
    swh-deposit_1 | [2020-10-28 13:52:51 +0000] [34] [DEBUG] Closing connection.
    swh-deposit_1 | [2020-10-28 13:52:51 +0000] [35] [DEBUG] POST /deposit/1/test/
    swh-deposit_1 | [2020-10-28 13:52:52 +0000] [35] [DEBUG] Closing connection.
    swh-deposit_1 | [2020-10-28 13:52:55 +0000] [35] [DEBUG] GET /1/private/test/1/check/
    ...
    • Oct 28 2020, 4:40 PM
    • 12 Lines
    Author: douardda
  • P842 Command-Line Input
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
    ...
    • Oct 28 2020, 10:52 AM
    • 44 Lines
    Author: zack
  • P841 Command-Line Input
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
    ...
    • Oct 28 2020, 10:49 AM
    • 44 Lines
    Author: zack
  • P840 Command-Line Input
    8: veth61ae6d9@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-d0d7f9beaec2 state UP group default
    link/ether 42:03:28:76:3b:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::4003:28ff:fe76:3b03/64 scope link
    valid_lft forever preferred_lft forever
    • Oct 28 2020, 8:57 AM
    • 4 Lines
    Author: zack
  • P839 (An Untitled Masterwork)
    [Question] getdents() calls use very small buffer when FUSE mount
    Hello all!
    I am currently working on a FUSE filesystem where a directory can
    ...
    • Oct 27 2020, 10:07 AM
    • 32 Lines
    Author: haltode
  • P838 time ls -f history/
    on master:
    Impossible to list history/ for more than a few hundreds commits because ls
    getdents() syscalls are done with a quite small buffer (19 entries on my
    machine), hence the readdir is regularly called with a different offset and the
    ...
    • Oct 26 2020, 5:15 PM
    • 15 Lines
    Author: haltode
  • P837 ingesting_cpython_connreset
    $ doco exec swh-scheduler swh scheduler task add load-git -p oneshot url=https://github.com/python/cpython
    $ doco logs -f --tail 0 | grep -vE "heartbeat|statsd"
    swh-scheduler-runner_1 | INFO:swh.scheduler.celery_backend.runner:Grabbed 1 tasks load-git
    amqp_1 |
    amqp_1 | =INFO REPORT==== 26-Oct-2020::14:58:45 ===
    ...
    • Oct 26 2020, 4:28 PM
    • 748 Lines
    Author: tenma
  • P836 (An Untitled Masterwork)
    [swh]
    dbname=softwareheritage
    host=db.internal.softwareheritage.org
    user=guest
    • Oct 26 2020, 2:29 PM
    • 4 Lines
    Author: douardda
  • P835 (An Untitled Masterwork)
    class Bar:
    def __init__(self):
    self.x = randint(1, 1000)
    self.y = [randint(1, 1000) for _ in range(100)]
    ...
    • Oct 26 2020, 1:59 PM
    • 23 Lines
    • Python
    Author: haltode
  • P834 (An Untitled Masterwork)
    pympler asizeof 25880
    vlorentz total_size 28
    original total_size 604
    pympler asizeof 32648
    ...
    • Oct 26 2020, 1:08 PM
    • 23 Lines
    Author: haltode
First
Prev
Next
About · Developer information · Legal