Page MenuHomeSoftware Heritage
Paste Active Pastes
  • sbuild (Debian sbuild) 0.79.0 (05 February 2020) on mirzakhani.olasd.eu
    +==============================================================================+
    | swh-web 0.0.226-1~swh2 (amd64) Tue, 07 Apr 2020 12:37:33 +0000 |
    +==============================================================================+
    ...
    • Tue, Apr 7, 2:39 PM
    • 2,871 Lines
  • Apr 07 03:33:13 worker0 python3[16392]: [2020-04-07 03:33:13,434: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).')
    Apr 07 03:33:13 worker0 python3[16392]: Traceback (most recent call last):
    Apr 07 03:33:13 worker0 python3[16392]: File "/usr/lib/python3/dist-packages/billiard/pool.py", line 1267, in mark_as_worker_lost
    Apr 07 03:33:13 worker0 python3[16392]: human_status(exitcode)),
    Apr 07 03:33:13 worker0 python3[16392]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).
    ...
    • Tue, Apr 7, 9:58 AM
    • 10 Lines
  • dir_to_dir
    old: 48341950415
    new: 49543338345
    +2.48%
    ...
    • Sun, Apr 5, 2:12 PM
    • 42 Lines
  • origin_visit
    old: 194970670
    new: 1009322644
    snapshot
    ...
    • Sat, Apr 4, 12:20 PM
    • 19 Lines
  • $ cat foo.py
    from tenacity import retry
    @retry
    def foo(n):
    ...
    • Thu, Apr 2, 6:59 PM
    • 18 Lines
  • @overload
    def _get_key(self, object_type: str, object_: Union[Revision, Release, Directory, Snapshot]) -> bytes:
    ...
    ...
    • Thu, Apr 2, 4:22 PM
    • 37 Lines
    • Python
  • def _get_key(
    self,
    object_type: str,
    object_: BaseModel) -> Union[bytes, Dict]:
    if object_type in ('revision', 'release', 'directory', 'snapshot'):
    ...
    • Thu, Apr 2, 4:03 PM
    • 28 Lines
    • Python
  • Info: Loading facts
    Info: Caching catalog for 9fb17b6df4b3.test
    Info: Applying configuration version '1585756962'
    Notice: /Stage[main]/Apt/File[preferences]/ensure: created
    Info: /Stage[main]/Apt/File[preferences]: Scheduling refresh of Class[Apt::Update]
    ...
    • Wed, Apr 1, 6:08 PM
    • 279 Lines
  • 14:59 $ doco exec puppet-agent puppet agent -t
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Wed, Apr 1, 5:06 PM
    • 307 Lines
  • WARNING:swh.core.cli:Could not load subcommand journal: cannot import name 'HashCollision' from 'swh.storage' (/home/danseraf/swh-environment/swh-storage/swh/storage/__init__.py)
    WARNING:swh.core.cli:Could not load subcommand indexer: cannot import name 'HashCollision' from 'swh.storage' (/home/danseraf/swh-environment/swh-storage/swh/storage/__init__.py)
    WARNING:swh.core.cli:Could not load subcommand search: cannot import name 'HashCollision' from 'swh.storage' (/home/danseraf/swh-environment/swh-storage/swh/storage/__init__.py)
    • Thu, Mar 26, 2:32 PM
    • 3 Lines
  • (swh) ~/swh-environment/swh-scanner   master  make check
    python3 -m flake8 swh
    swh/scanner/model.py:70:47: E226 missing whitespace around arithmetic operator
    swh/scanner/scanner.py:18:9: E126 continuation line over-indented for hanging indent
    swh/scanner/scanner.py:25:9: E123 closing bracket does not match indentation of opening bracket's line
    ...
    • Wed, Mar 25, 11:00 AM
    • 14 Lines
  • ```
    id | type | arguments | next_run | current_interval | status | policy | retries_left | priorit$
    ---------+-----------------+------------------------------------------------------------------------------------------------------+-------------------------------+------------------+--------------------+-----------+--------------+--------$
    1186662 | load-functional | {"args": [], "kwargs": {"url": "https://nix-community.github.io/nixpkgs-swh/sources.json"}} | 2020-03-25 02:40:59.571373+00 | 1 day | disabled | recurring | 0 |
    1186665 | load-functional | {"args": [], "kwargs": {"url": "https://nix-community.github.io/nixpkgs-swh/sources-unstable.json"}} | 2020-03-25 08:30:08.666509+00 | 1 day | next_run_scheduled | recurring | 0 |
    ...
    • Wed, Mar 25, 9:32 AM
    • 8 Lines
  • { config, lib, pkgs, unstable-pkgs, mypkgs, ... }:
    with lib;
    let xsession-enable = config.my.xsession.enable;
    optional-dependencies = config.my.emacs.optional-dependencies;
    ...
    • Tue, Mar 24, 8:02 PM
    • 242 Lines
  • from typing import Callable
    class Foo:
    def __init__(self, f: Callable[[int], None]):
    self.f: Callable[[int], None] = f
    ...
    • Tue, Mar 24, 5:15 PM
    • 8 Lines
  • from typing import Callable
    class Foo:
    f: Callable[[int], None]
    ...
    • Tue, Mar 24, 5:06 PM
    • 15 Lines
  • from typing import Callable
    class Foo:
    f: Callable
    ...
    • Tue, Mar 24, 5:02 PM
    • 7 Lines
  • *** rdkafka_cgrp.c:3037:rd_kafka_cgrp_op_serve: assert: rktp->rktp_assigned ***
    rd_kafka_t 0x2121b30: rdkafka#consumer-2
    producer.msg_cnt 0 (0 bytes)
    rk_rep reply queue: 2259 ops
    brokers:
    ...
    • Mon, Mar 23, 9:45 PM
    • 12 Lines
  • tony  yavin4  ~  %  my-hm build
    ~/repo/private/home ~
    querying info about '/nix/store/34ckk0vslnar8xj70sa0cgpiyg1r96r5-emacs-with-packages-26.3' on 'https://cache.nixos.org'...
    downloading 'https://cache.nixos.org/34ckk0vslnar8xj70sa0cgpiyg1r96r5.narinfo'...
    querying info about '/nix/store/dmjkw4hlf56x4wg2g5kgw0cj96qjqdp3-emacs-rust-mode-20191208.1654' on 'https://cache.nixos.org'...
    ...
    • Mon, Mar 23, 6:28 PM
    • 80 Lines
  • {
    "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
    "@type": "SoftwareSourceCode",
    "name": "Foo Software",
    "author": [
    ...
    • Sat, Mar 21, 12:06 PM
    • 25 Lines
    • JSON
  • {
    "total-collisions-raised-in-sentry": 9677,
    "total-collisions": 3,
    "total-falsy-collisions": 11,
    "detailed-collisions": {
    ...
    • Fri, Mar 20, 3:49 PM
    • 489 Lines
  • Do you prefer:
    @attr.s(frozen=True)
    class Person(BaseModel):
    ...
    • Fri, Mar 20, 12:18 PM
    • 24 Lines
  • DEBUG:swh.loader.package.loader:Number of skipped contents: 0 [58/4699]
    DEBUG:urllib3.connectionpool:http://storage0.internal.staging.swh.network:5002 "POST /content/skipped/missing HTTP/1.1" 200 1
    DEBUG:swh.loader.package.loader:Number of contents: 261
    DEBUG:urllib3.connectionpool:http://storage0.internal.staging.swh.network:5002 "POST /content/missing HTTP/1.1" 200 1
    DEBUG:swh.loader.package.loader:Number of directories: 22
    ...
    • Fri, Mar 20, 11:16 AM
    • 57 Lines
  • Traceback (most recent call first):
    <built-in method acquire of _thread.lock object at remote 0x7f3c084e89b8>
    File "/usr/lib/python3.7/threading.py", line 300, in wait
    gotit = waiter.acquire(True, timeout)
    File "/usr/lib/python3.7/threading.py", line 552, in wait
    ...
    • Wed, Mar 18, 4:33 PM
    • 92 Lines
  • #!/usr/bin/env bash
    # to run as swhworker
    export SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_functional.yml
    ...
    • Wed, Mar 18, 2:31 PM
    • 10 Lines
  • DEBUG:swh.journal.client:Consumer settings: {'security.protocol': 'SASL_SSL', 'sasl.mechanisms': 'SCRAM-SHA-512', 'sasl.username': 'seirl', 'sasl.password': 'CENSORED', 'debug': 'all', 'bootstrap.servers': 'kafka01.euwest.azure.internal.softwareheritage.org:9094,kafka02.euwest.azure.internal.softwareheritage.org:9094,kafka03.euwest.azure.internal.softwareheritage.org:9094,kafka04.euwest.azure.internal.softwareheritage.org:9094,kafka05.euwest.azure.internal.softwareheritage.org:9094,kafka06.euwest.azure.internal.softwareheritage.org:9094', 'auto.offset.reset': 'earliest', 'group.id': 'swh-dataset-export-seirl-test-51', 'on_commit': <function _on_commit at 0x7f4bc5db85e0>, 'error_cb': <function _error_cb at 0x7f4bc5dfbe50>, 'enable.auto.commit': False, 'logger': <Logger swh.journal.client.rdkafka (DEBUG)>}
    DEBUG:swh.journal.client:Subscribing to: ['swh.journal.objects.origin_visit', 'swh.journal.objects.snapshot', 'swh.journal.objects.release', 'swh.journal.objects.revision', 'swh.journal.objects.directory']
    DEBUG:swh.journal.client.rdkafka:SASL [rdkafka#consumer-1] [thrd:app]: Selected provider SCRAM (builtin) for SASL mechanism SCRAM-SHA-512
    DEBUG:swh.journal.client.rdkafka:OPENSSL [rdkafka#consumer-1] [thrd:app]: librdkafka built with OpenSSL version 0x1000212f
    DEBUG:swh.journal.client.rdkafka:MEMBERID [rdkafka#consumer-1] [thrd:app]: Group "swh-dataset-export-seirl-test-51": updating member id "(not-set)" -> ""
    ...
    • Tue, Mar 17, 6:01 PM
    • 564 Lines
  • ```
    visits: Iterable[OriginVisit] = [
    _fix_origin_visit(v) for v in objects
    if _fix_origin_visit(v) is not None
    ]
    ...
    • Tue, Mar 17, 10:43 AM
    • 11 Lines
  • > {}['foo']
    [ 'foo' ]
    > obj = {}
    {}
    > obj['foo']
    ...
    • Thu, Mar 12, 5:59 PM
    • 6 Lines
    • Javascript
  • python analyze.py ~/sources.json
    There are 17599 sources in nixpkgs
    Sources are coming from 1529 different hosts
    ...
    • Thu, Mar 12, 3:59 PM
    • 78 Lines
  • def attrib_typecheck(default: Any = attr.NOTHING,
    type: Optional[Type] = None,
    validator: Collection[Callable] = ()):
    "A 'partial' of attr.ib that prefill the validator with type_validator"
    return attr.attrib(
    ...
    • Thu, Mar 12, 3:33 PM
    • 8 Lines
  • def test_timestamp_seconds():
    attr.validate(Timestamp(seconds=0, microseconds=0))
    with pytest.raises(AttributeTypeError):
    attr.validate(Timestamp(seconds='0', microseconds=0))
    ...
    • Thu, Mar 12, 10:52 AM
    • 12 Lines
  • diff --git a/swh/model/tests/test_model.py b/swh/model/tests/test_model.py
    index 8bffa80..3f4de69 100644
    --- a/swh/model/tests/test_model.py
    +++ b/swh/model/tests/test_model.py
    @@ -298,6 +298,7 @@ def test_release_model_id_computation():
    ...
    • Wed, Mar 11, 2:11 PM
    • 10 Lines
  • seirl@granet ~/swh-environment (git)-[master] % kafkacat -b kafka01.euwest.azure.softwareheritage.org:9093 -d broker -L :(
    %7|1583865197.850|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
    %7|1583865197.850|BROKER|rdkafka#producer-1| [thrd:app]: kafka01.euwest.azure.softwareheritage.org:9093/bootstrap: Added new broker with NodeId -1
    %7|1583865197.850|CONNECT|rdkafka#producer-1| [thrd:app]: kafka01.euwest.azure.softwareheritage.org:9093/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
    %7|1583865197.850|BRKMAIN|rdkafka#producer-1| [thrd:kafka01.euwest.azure.softwareheritage.org:9093/bootstrap]: kafka01.euwest.azure.softwareheritage.org:9093/bootstrap: Enter main broker thread
    ...
    • Tue, Mar 10, 7:33 PM
    • 24 Lines
  • python3 -m pytest .
    Traceback (most recent call last):
    File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
    File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    ...
    • Tue, Mar 10, 5:11 PM
    • 59 Lines
  • make test
    python3 -m pytest .
    usage: pytest.py [options] [file_or_dir] [file_or_dir] [...]
    pytest.py: error: unrecognized arguments: --no-start-live-server --live-server-port .
    inifile: /home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/pytest.ini
    ...
    • Tue, Mar 10, 5:07 PM
    • 7 Lines
  • $ sudo smartctl -a /dev/nvme0
    smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-4-amd64] (local build)
    Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
    === START OF INFORMATION SECTION ===
    ...
    • Mon, Mar 9, 9:25 AM
    • 67 Lines
  • #!/bin/bash
    # Copyright (C) 2019 Stefano Zacchiroli <zack@upsilon.cc>
    # License: GNU General Public License (GPL), version 3 or above
    #
    ...
    • Sun, Mar 8, 4:00 PM
    • 62 Lines
    • Bash Scripting
  • | Traceback (most recent call last):
    | File "/usr/bin/swh", line 11, in <module>
    | load_entry_point('swh.core==0.0.94', 'console_scripts', 'swh')()
    | File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 111, in main
    | return swh(auto_envvar_prefix='SWH')
    ...
    • Mar 6 2020, 5:36 PM
    • 59 Lines
  • diff --git a/swh/storage/tests/test_storage.py b/swh/storage/tests/test_storage.py
    index c68f28c..20959fc 100644
    --- a/swh/storage/tests/test_storage.py
    +++ b/swh/storage/tests/test_storage.py
    @@ -1087,6 +1087,12 @@ class TestStorage:
    ...
    • Mar 4 2020, 3:53 PM
    • 17 Lines
    • Diff
  • 2020-03-04T14:13:06.142010302Z swh_graph-replayer-release.1.8shri9bb8q0h@mirror-replay01 | Starting the SWH mirror graph replayer
    2020-03-04T14:13:21.173425515Z swh_graph-replayer-release.1.8shri9bb8q0h@mirror-replay01 | Traceback (most recent call last):
    2020-03-04T14:13:21.173475615Z swh_graph-replayer-release.1.8shri9bb8q0h@mirror-replay01 | File "/usr/bin/swh", line 11, in <module>
    2020-03-04T14:13:21.173483615Z swh_graph-replayer-release.1.8shri9bb8q0h@mirror-replay01 | load_entry_point('swh.core==0.0.94', 'console_scripts', 'swh')()
    2020-03-04T14:13:21.173487415Z swh_graph-replayer-release.1.8shri9bb8q0h@mirror-replay01 | File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 111, in main
    ...
    • Mar 4 2020, 3:20 PM
    • 48 Lines
  • 2020-03-04T12:55:32.779118726Z swh_graph-replayer-revision.1.ihrszqxqznya@mirror-replay03 | DEBUG:swh.journal.client.rdkafka:CGRPOP [rdkafka#consumer-1] [thrd:main]: Group "test-graph-replayer-b70e8cf435c0" received op PARTITION_LEAVE in state up (join state wait-unassign, v840) for swh.journal.objects.revision [0]
    2020-03-04T12:55:32.779122726Z swh_graph-replayer-revision.1.ihrszqxqznya@mirror-replay03 | DEBUG:swh.journal.client.rdkafka:PARTDEL [rdkafka#consumer-1] [thrd:main]: Group "test-graph-replayer-b70e8cf435c0": delete swh.journal.objects.revision [0]
    2020-03-04T12:55:32.779134326Z swh_graph-replayer-revision.1.ihrszqxqznya@mirror-replay03 | DEBUG:swh.journal.client.rdkafka:CGRPOP [rdkafka#consumer-1] [thrd:main]: Group "test-graph-replayer-b70e8cf435c0" received op REPLY:FETCH_STOP in state up (join state wait-unassign, v840) for swh.journal.objects.revision [0]
    2020-03-04T12:55:32.779138526Z swh_graph-replayer-revision.1.ihrszqxqznya@mirror-replay03 | DEBUG:swh.journal.client.rdkafka:UNASSIGN [rdkafka#consumer-1] [thrd:main]: Unassign not done yet (255 wait_unassign, 255 assigned, 0 wait commit, join state wait-unassign): FETCH_STOP done
    ...
    • Mar 4 2020, 2:04 PM
    • 16 Lines
  • storage:
    cls: remote
    args:
    url: http://localhost:5002/
    ...
    • Mar 3 2020, 5:10 PM
    • 17 Lines
  • -- SWH Indexer DB schema upgrade
    -- from_version: 130
    -- to_version: 131
    -- description:
    ...
    • Mar 2 2020, 11:16 AM
    • 116 Lines
  • Traceback (most recent call last):
    File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/aiohttp/connector.py", line 955, in _create_direct_connection
    traces=traces), loop=self._loop)
    File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/aiohttp/connector.py", line 825, in _resolve_host
    self._resolver.resolve(host, port, family=self._family)
    ...
    • Feb 28 2020, 4:08 PM
    • 61 Lines
  • with lost_task_runs as (
    select task_run.id
    from task
    inner join task_run on task.id = task_run.task
    where task.policy = 'recurring' and
    ...
    • Feb 27 2020, 10:09 AM
    • 14 Lines
    • SQL
  • import datetime
    import gzip
    import hashlib
    import multiprocessing
    import os
    ...
    • Feb 11 2020, 5:55 PM
    • 87 Lines
    • Python
  • import datetime
    import gc
    import gzip
    import hashlib
    import multiprocessing
    ...
    • Feb 11 2020, 5:54 PM
    • 108 Lines
    • Python
  • diff --git swh/journal/client.py swh/journal/client.py
    index 0d481b0..42e2b96 100644
    --- swh/journal/client.py
    +++ swh/journal/client.py
    @@ -76,7 +76,7 @@ class JournalClient:
    ...
    • Feb 11 2020, 4:44 PM
    • 39 Lines
    • Diff
  • from swh.journal.client import JournalClient
    import logging
    logging.basicConfig(level=logging.INFO)
    logging.info('Running test')
    ...
    • Feb 10 2020, 6:03 PM
    • 33 Lines
    • Python
  • [2020-02-10 17:37:13] INFO:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:GroupCoordinator]: GroupCoordinator/6: Timed out HeartbeatRequest in flight (after 10580ms, timeout #0)
    [2020-02-10 17:37:13] WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:GroupCoordinator]: GroupCoordinator/6: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
    [2020-02-10 17:37:13] INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_TIMED_OUT,val=-185,str="GroupCoordinator: 1 request(s) timed out: disconnect (after 29019ms in state UP)"}
    [2020-02-10 17:37:14] INFO:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:GroupCoordinator]: GroupCoordinator/6: Timed out HeartbeatRequest in flight (after 10536ms, timeout #0): possibly held back by preceeding OffsetCommitRequest with timeout in 47805ms
    [2020-02-10 17:37:14] WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:GroupCoordinator]: GroupCoordinator/6: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
    ...
    • Feb 10 2020, 5:39 PM
    • 68 Lines
  • INFO:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:kafka06.euwest.azure.internal.softwareheritage.org:9092/bootstr]: kafka06.euwest.azure.internal.softwareheritage.org:9092/6: Timed out FetchRequest in flight (after 60337ms, timeout #0)
    WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:kafka06.euwest.azure.internal.softwareheritage.org:9092/bootstr]: kafka06.euwest.azure.internal.softwareheritage.org:9092/6: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
    INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_TIMED_OUT,val=-185,str="kafka06.euwest.azure.internal.softwareheritage.org:9092/6: 1 request(s) timed out: disconnect (after 71045ms in state UP)"}
    INFO:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:kafka06.euwest.azure.internal.softwareheritage.org:9092/bootstr]: kafka06.euwest.azure.internal.softwareheritage.org:9092/6: Timed out FetchRequest in flight (after 60682ms, timeout #0)
    WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:kafka06.euwest.azure.internal.softwareheritage.org:9092/bootstr]: kafka06.euwest.azure.internal.softwareheritage.org:9092/6: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
    ...
    • Feb 10 2020, 4:55 PM
    • 46 Lines
  • swh/storage/tests/test_retry.py F
    ================================================================================================================================== FAILURES ===================================================================================================================================
    ___________________________________________________________________________________________________________________ test_retrying_proxy_storage_content_add ___________________________________________________________________________________________________________________
    ...
    • Feb 5 2020, 6:40 PM
    • 39 Lines
  • WARNING cassandra.cluster:cql.py:45 Downgrading core protocol version from 66 to 65 for 127.0.0.1:59387. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
    WARNING cassandra.cluster:cql.py:45 Downgrading core protocol version from 65 to 4 for 127.0.0.1:59387. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
    ERROR cassandra.cluster:thread.py:57 Exception refreshing schema in response to schema change:
    Traceback (most recent call last):
    File "cassandra/cluster.py", line 4044, in cassandra.cluster.refresh_schema_and_set_result
    ...
    • Feb 4 2020, 1:16 PM
    • 12 Lines
  • >>> pprint.pprint(s.revision_get([b'Y\xd7\xa5=\xe5\x980\x8eN\x1f\xffy\x19Z\xe8Z{#\xea\x0e']))
    [{'author': {'email': b'me@nanx.me',
    'fullname': b'Nan Xiao <me@nanx.me>',
    'name': b'Nan Xiao'},
    'committer': {'email': b'me@nanx.me',
    ...
    • Jan 30 2020, 3:11 PM
    • 76 Lines
  • elasticsearch_1 | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
    elasticsearch_1 | {"type": "server", "timestamp": "2020-01-27T12:45:13,203Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "959d65b52b05", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/nvme0n1p3)]], net usable_space [199.1gb], net total_space [438.6gb], types [ext4]" }
    elasticsearch_1 | {"type": "server", "timestamp": "2020-01-27T12:45:13,206Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "959d65b52b05", "message": "heap size [989.8mb], compressed ordinary object pointers [true]" }
    elasticsearch_1 | {"type": "server", "timestamp": "2020-01-27T12:45:13,210Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "959d65b52b05", "message": "node name [959d65b52b05], node ID [jwb72D4vRxa9uqgxR7PSbg], cluster name [docker-cluster]" }
    elasticsearch_1 | {"type": "server", "timestamp": "2020-01-27T12:45:13,210Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "959d65b52b05", "message": "version[7.5.0], pid[1], build[default/docker/e9ccaed468e2fac2275a3761849cbee64b39519f/2019-11-26T01:06:52.518245Z], OS[Linux/4.19.0-6-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]" }
    ...
    • Jan 27 2020, 1:46 PM
    • 60 Lines
  • > assert results == {cont['sha1']: cont}
    E assert {b'4\x972t\xcc\xefj\xb4\xdf\xaa\xf8e\x99y/\xa9\xc3\xfeF\x89': [{'blake2s256': b'\xd5\xfe\x199'\n b"We'\xe4"\n b',\xfdv\xa9'\n b'EZ$2'\n b'\xfe\x7fVf'\n b'\x95dW}'\n b'\xd9<B\x80'\n b'\xe7mf\x1d',\n 'length': 3,\n 'sha1': b'4\x972t'\n...
    • Jan 24 2020, 4:16 PM
    • 2 Lines
  • root@551c92280895:/# aptitude why g++
    i python3-swh.web Depends python3-pypandoc
    i A python3-pypandoc Depends python3-pip
    i A python3-pip Recommends build-essential
    i A build-essential Depends g++ (>= 4:8.3)
    • Jan 23 2020, 4:36 PM
    • 5 Lines
  • delete from origin_visit where type='cran';
    delete from origin where url like 'https://cran.r-project.org/%';
    • Jan 16 2020, 2:59 PM
    • 2 Lines
  • #!/usr/bin/env bash
    set -xe
    USER=$1
    ...
    • Jan 16 2020, 1:32 PM
    • 29 Lines
  • [testenv:xdist]
    deps =
    pytest-cov
    pytest-xdist
    commands =
    ...
    • Jan 15 2020, 3:26 PM
    • 6 Lines
  • SystemCheckError: System check identified some issues:
    ERRORS:
    ?: (urls.E007) The custom handler400 view 'swh.web.common.exc.swh_handle400' does not take the correct number of arguments (request, exception).
    ?: (urls.E007) The custom handler403 view 'swh.web.common.exc.swh_handle403' does not take the correct number of arguments (request, exception).
    ...
    • Jan 14 2020, 11:35 AM
    • 6 Lines
  • content : 22% 1544836073 / 7031830448
    directory : 9% 308855808 / 3528500560
    origin : 100% 91383460 / 91383581
    origin_visit : 98% 1088929623 / 1110533233
    release : 100% 12108512 / 12108527
    ...
    • Jan 6 2020, 1:17 PM
    • 7 Lines
  • def test_create_deposit_multipart(host):
    deposit = host.check_output(
    'swh deposit upload --format json --username test --password test '
    ...
    • Dec 20 2019, 2:12 PM
    • 32 Lines
    • Python
  • url2 = 'http://deb.debian.org/debian//pool/main/l/lxqt-config/lxqt-config_0.14.1.orig.tar.xz.asc'
    # patched download to not check anything
    In [10]: download(url2, dest='/tmp')
    Out[10]:
    ...
    • Dec 20 2019, 12:57 PM
    • 37 Lines
  • Dec 19 14:47:07 worker2 python3[30717]: [2019-12-19 14:47:07,898: ERROR/ForkPoolWorker-1] Fail to load https://softwareheritage.org/swh-ddev
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/core/tarball.py", line 72, in uncompress
    shutil.unpack_archive(tarpath, extract_dir=dest)
    File "/usr/lib/python3.7/shutil.py", line 999, in unpack_archive
    ...
    • Dec 19 2019, 3:51 PM
    • 20 Lines
  • PACKAGE_FILES3 = {
    'bullseye/main/0.10-1': {
    'files': {
    'libbarcode-datamatrix-perl_0.10-1.debian.tar.xz': {
    'md5sum': '30bd8e44db00610333af39ccd0805110',
    ...
    • Dec 19 2019, 1:20 PM
    • 40 Lines
  • <?xml version="1.0" encoding="utf-8"?>
    <entry xmlns="http://www.w3.org/2005/Atom"
    xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0">
    <title>Je suis GPL</title>
    <client>swh</client>
    ...
    • Dec 17 2019, 2:26 PM
    • 25 Lines
  • Notice: /Stage[main]/Profile::Swh::Deploy::Webapp/Gunicorn::Instance[swh-webapp]/File[/etc/gunicorn/instances/swh-webapp.cfg]/content:
    --- /etc/gunicorn/instances/swh-webapp.cfg 2018-03-06 18:52:38.179007424 +0000
    +++ /tmp/puppet-file20191213-2435-qv2yh7 2019-12-13 14:53:05.846920036 +0000
    @@ -1,6 +1,13 @@
    # Gunicorn instance configuration.
    ...
    • Dec 13 2019, 3:56 PM
    • 97 Lines
  • swh-web_1 | Traceback (most recent call last):
    swh-web_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/django/core/handlers/exception.py", line 41, in inner
    swh-web_1 | response = get_response(request)
    swh-web_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/django/core/handlers/base.py", line 172, in _get_response
    swh-web_1 | resolver_match = resolver.resolve(request.path_info)
    ...
    • Dec 12 2019, 2:47 PM
    • 395 Lines
  • create index on task(type, status, policy);
    update task
    set arguments=jsonb_set(arguments, '{kwargs}', json_build_object('url', arguments#>>'{kwargs,package_url}')::jsonb)
    where type = 'load-npm' and
    ...
    • Dec 10 2019, 6:40 PM
    • 87 Lines
  • with swh_count_origins as (
    select value
    from object_counts
    where object_type='origin'
    ),
    ...
    • Dec 6 2019, 12:14 PM
    • 9 Lines
    • SQL
  • * scheduler
    Following module migration to their own namespace (D2395):
    #+BEGIN_SRC sh
    ...
    • Dec 5 2019, 8:38 AM
    • 16 Lines
  • self.search.origin_update([
    {'url': 'https://bitbucket.org/bitbucket0145/bitbucket_repo.git'},
    {'url': 'https://gitorious.org/railstutorial/railstutorial.git'},
    {'url': 'https://bitbucket.org/bittelc/railstutorial.git'},
    ])
    ...
    • Dec 4 2019, 4:40 PM
    • 11 Lines
    • Python
  • [b'3\xbe\x07S\xcf(~j\xc0\xc0C\xd6\xea\xe6-\x1f\xd3&7;',
    b'\xa2\x82b\xfcGdS\x82O\xe0\x00\xf9\xda{\x85!\x1b\x82\x9d\x07',
    b'\x1c\xff\x7f\xb26\xdb[\xda3\xde\x11\xe5H\xa04\x02\x12\x9b\x8c\xf4',
    b'?J\x9c]\xc1\x1c\x13|\xb9\xd3.\x0eO\xf0\x9e2\xaf\x15M\x15',
    b'-q\x82E\x03\xd1\xf6\xa0G\x14S\xf8\xa0\t\xfdu?\x9e\xb02',
    ...
    • Nov 28 2019, 4:04 PM
    • 20 Lines
  • import json
    from pprint import pprint
    import re
    import elasticsearch
    ...
    • Nov 25 2019, 2:07 PM
    • 46 Lines
    • Python
  • default: &default_settings
    memory: 200G
    java_tool_options: -XXlol
    compress:
    <<: *default_settings
    ...
    • Nov 8 2019, 2:49 PM
    • 6 Lines
  • default:
    memory: 200G
    java_tool_options: -XXlol
    compress:
    memory: 1000G
    • Nov 8 2019, 2:47 PM
    • 5 Lines
    • YAML
  • 11:18 <+ardumont> douardda: D2237 draft to check for missing task types
    11:18 -- Notice(swhbot): D2237 (author: ardumont, Needs Review) on swh-lister: lister: Add checks on expected scheduler's output tasks <https://forge.softwareheritage.org/D2237>
    11:18 <+ardumont> i'm not sure when/where to plug that check though
    11:21 <+olasd> if we're doing that, we might just as well create the task type with the proper settings
    11:25 <+ardumont> mmm, unsure
    ...
    • Nov 8 2019, 12:00 PM
    • 33 Lines
  • swh/graph/tests/test_cli.py::TestCompress::test_pipeline
    -------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------
    webgraph.py 233 INFO starting compression
    webgraph.py 242 INFO starting compression step MPH (1/11)
    webgraph.py 153 INFO running: java it.unimi.dsi.sux4j.mph.GOVMinimalPerfectHashFunction --zipped /tmp/tmp465gdf9e.swh-graph-test/example.mph --temp-dir /tmp/tmp465gdf9e.swh-graph-test/tmp /home/antoine/swh/swh-environment/swh-graph/swh/graph/tests/dataset/example.nodes.csv.gz
    ...
    • Nov 4 2019, 12:01 PM
    • 49 Lines
  • url
    ------------------------------------------------------------------------------------------------
    https://github.com/rootpy/root_numpy
    https://github.com/stevengj/mpb
    https://github.com/barbagroup/pygbe
    ...
    • Oct 15 2019, 2:49 PM
    • 263 Lines
  • package org.softwareheritage.graph;
    import org.softwareheritage.graph.algo.Traversal;
    import java.io.OutputStream;
    ...
    • Oct 14 2019, 1:52 PM
    • 67 Lines
    • Java
  • ERR-CONDUIT-CORE: Graph cycle detected (type=5, cycle=PHID-DREV-vvlfxmyjkkrcdnbfzb5l, PHID-DREV-sz2tjc63iowtoyay6mey, PHID-DREV-ni5tcyma6542fypzaalx, PHID-DREV-q6y4zi7eciqddatexzxl, PHID-DREV-tsal5vzfklovorhkqvzn, PHID-DREV-rtmmo2kgo7wacl4warhv, PHID-DREV-5muna6xupz7kvfkz2yuo, PHID-DREV-ee7627jtcof4i73qqrfm, PHID-DREV-kf4oob54dw4sttlkv2az, PHID-DREV-vvlfxmyjkkrcdnbfzb5l).
    • Oct 14 2019, 12:28 PM
    • 1 Line
  • git pull --rebase
    remote: Enumerating objects: 21, done.
    remote: Counting objects: 100% (21/21), done.
    remote: Compressing objects: 100% (11/11), done.
    remote: Total 12 (delta 6), reused 0 (delta 0)
    ...
    • Oct 9 2019, 2:58 PM
    • 15 Lines
  • P545 Data
    {'_id': ObjectId('5ad99f9fbd95630dfc4b9a4e'),
    'graphql': {'user': {'biography': 'International educator, traveler, music '
    'lover, photographer, Californian.',
    'blocked_by_viewer': False,
    'connected_fb_page': None,
    ...
    • Oct 8 2019, 6:30 PM
    • 640 Lines
  • 17:09 <+douardda> and the pb should not happen if we put the plugin out of the swh package (i.e. ar the root of swh-core directory)
    17:10 <+douardda> (I've reproduced the issue in a minimal src repo)
    17:12 <+douardda> the problem seems to be that when the plugin lives under our package's hat, it's loaded very soon, thus swh.core is loaded but from the installed location (in the tox venv here)
    17:13 <+douardda> so when the subsequent import statement for a conftest or a testfile occurs, it's looked under this package's root directory first
    17:14 <+douardda> so the simple solution is to put this plugin in a dedicated python module not under the swh's (and especially the swh.core one I think) package
    ...
    • Oct 8 2019, 11:49 AM
    • 6 Lines
  • Bad:
    Returns
    List of tarball urls and their associated metadata (time, length).
    For example:
    ...
    • Oct 8 2019, 11:33 AM
    • 21 Lines
  • ```
    $ sudo apt install r-base
    ```
    ```
    ...
    • Oct 6 2019, 11:49 AM
    • 34 Lines
  • Hello,
    It's the time of the year where I ask you (again!) for your help to better archive GNU source code in the Software Heritage archive.
    Would it be possible to change the format of the GNU file listing [1] to also include SHA256 checksums?
    ...
    • Oct 1 2019, 12:10 PM
    • 16 Lines
  • indexes:
    - swh_workers-2018.03.*
    size: 100
    from: 0
    ...
    • Sep 30 2019, 7:47 PM
    • 27 Lines
  • diff --git a/Makefile b/Makefile
    index 524175c..be63f09 100644
    --- a/Makefile
    +++ b/Makefile
    @@ -3,3 +3,12 @@
    ...
    • Sep 27 2019, 7:10 PM
    • 29 Lines
    • Diff
  • ✘ ⚙ dev@desktop5  ~/swh-environment/swh-journal   bencode-key ●  pytest
    ========================================================================================================= test session starts =========================================================================================================
    platform linux -- Python 3.5.3, pytest-5.0.1, py-1.7.0, pluggy-0.12.0
    hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/dev/swh-environment/swh-journal/.hypothesis/examples')
    rootdir: /home/dev/swh-environment/swh-journal, inifile: pytest.ini
    ...
    • Sep 23 2019, 3:00 PM
    • 51 Lines
  • -- gitlab (renamed key "api_baseurl" to "url")
    update task set arguments='{"args": [], "kwargs": {"instance": "inria", "url": "https://gitlab.inria.fr/api/v4"}}' where arguments#>>'{kwargs,instance}' = 'inria' and type in ('list-gitlab-full', 'list-gitlab-incremental');
    update task set arguments='{"args": [], "kwargs": {"instance": "framagit", "url": "https://framagit.org/api/v4"}}' where arguments#>>'{kwargs,instance}' = 'framagit' and type in ('list-gitlab-full', 'list-gitlab-incremental');
    update task set arguments='{"args": [], "kwargs": {"instance": "riseup", "url": "https://0xacab.org/api/v4"}}' where arguments#>>'{kwargs,instance}' = 'riseup' and type in ('list-gitlab-full', 'list-gitlab-incremental');
    update task set arguments='{"args": [], "kwargs": {"instance": "gitlab", "url": "https://gitlab.com/api/v4"}}' where arguments#>>'{kwargs,instance}' = 'gitlab' and type in ('list-gitlab-full', 'list-gitlab-incremental');
    ...
    • Sep 11 2019, 3:54 PM
    • 35 Lines
    • SQL
  • messages: 168300
    messages: 168400
    messages: 168500
    messages: 168600
    messages: 168700
    ...
    • Sep 11 2019, 3:18 PM
    • 23 Lines
  • swh_graph-replayer.1.khy6rzsfohdz@mirror-node-3 | INFO:swh.journal.cli:Processed 35000 messages.
    swh_graph-replayer.1.khy6rzsfohdz@mirror-node-3 | Traceback (most recent call last):
    swh_graph-replayer.1.khy6rzsfohdz@mirror-node-3 | File "/usr/bin/swh", line 11, in <module>
    swh_graph-replayer.1.khy6rzsfohdz@mirror-node-3 | load_entry_point('swh.core==0.0.67', 'console_scripts', 'swh')()
    swh_graph-replayer.1.khy6rzsfohdz@mirror-node-3 | File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 56, in main
    ...
    • Sep 10 2019, 5:14 PM
    • 36 Lines
  • swh_graph-replayer.1.p18dcgg4lwq5@mirror-node-3 | INFO:swh.journal.cli:Processed 124000 messages.
    swh_graph-replayer.1.p18dcgg4lwq5@mirror-node-3 | INFO:kafka.client:Closing idle connection 1, last active 540038 ms ago
    swh_graph-replayer.1.p18dcgg4lwq5@mirror-node-3 | INFO:kafka.conn:<BrokerConnection node_id=1 host=kafka01.euwest.azure.internal.softwareheritage.org:9092 <connected> [IPv4 ('192.168.200.24', 9092)]>: Closing connection.
    swh_graph-replayer.1.p18dcgg4lwq5@mirror-node-3 | INFO:kafka.client:Closing idle connection 3, last active 540139 ms ago
    swh_graph-replayer.1.p18dcgg4lwq5@mirror-node-3 | INFO:kafka.conn:<BrokerConnection node_id=3 host=kafka03.euwest.azure.internal.softwareheritage.org:9092 <connected> [IPv4 ('192.168.200.31', 9092)]>: Closing connection.
    ...
    • Sep 10 2019, 1:50 PM
    • 9 Lines
  • Sep 10 11:59:01 desktop5 replayer-12301[2746]: INFO:kafka.conn:<BrokerConnection node_id=11 host=esnode1.internal.softwareheritage.org:9092 <connecting> [IPv4 ('192.168.100.61', 9092)]>: connecting to esnode1.internal.softwareheritage.org:9092 [('192.168.100.61', 9092) IPv4]
    Sep 10 11:59:06 desktop5 replayer-12301[2746]: INFO:kafka.conn:<BrokerConnection node_id=12 host=esnode2.internal.softwareheritage.org:9092 <connecting> [IPv4 ('192.168.100.62', 9092)]>: connecting to esnode2.internal.softwareheritage.org:9092 [('192.168.100.62', 9092) IPv4]
    Sep 10 11:59:06 desktop5 replayer-12301[2746]: INFO:kafka.conn:<BrokerConnection node_id=11 host=esnode1.internal.softwareheritage.org:9092 <connecting> [IPv4 ('192.168.100.61', 9092)]>: Connection complete.
    Sep 10 11:59:06 desktop5 replayer-12301[2746]: INFO:kafka.conn:<BrokerConnection node_id=12 host=esnode2.internal.softwareheritage.org:9092 <connecting> [IPv4 ('192.168.100.62', 9092)]>: Connection complete.
    Sep 10 11:59:19 desktop5 replayer-12301[2746]: WARNING:kafka.coordinator:Heartbeat session expired, marking coordinator dead
    ...
    • Sep 10 2019, 12:00 PM
    • 43 Lines
  • ERROR:root:An error occurred while calling o0.visit.
    : java.lang.ArrayIndexOutOfBoundsException: Index 272313807 out of bounds for length 121824
    at it.unimi.dsi.bits.LongArrayBitVector.getBoolean(LongArrayBitVector.java:374)
    at org.softwareheritage.graph.algo.Traversal.visitNodesVisitor(Traversal.java:160)
    at org.softwareheritage.graph.Entry.visit(Entry.java:28)
    ...
    • Sep 9 2019, 7:33 PM
    • 108 Lines
  • /browse/origin/26984/latest_snapshot/
    /browse/origin/34423/latest_snapshot/
    /browse/origin/21387/latest_snapshot/
    /browse/origin/48567/latest_snapshot/
    /browse/origin/29526/latest_snapshot/
    ...
    • Sep 9 2019, 2:59 PM
    • 851 Lines
  • import json
    import requests
    from pprint import pprint
    ...
    • Sep 9 2019, 2:53 PM
    • 74 Lines
    • Python
  • swh/indexer/tests/test_metadata.py .................................F
    ================================================================================================================== FAILURES ===================================================================================================================
    ___________________________________________________________________________________________________ Metadata.test_revision_metadata_indexer ___________________________________________________________________________________________________
    ...
    • Sep 5 2019, 2:55 PM
    • 92 Lines
  • # Graph compression output
    Compression script and environment used: https://forge.softwareheritage.org/source/swh-graph/browse/master/dockerfiles/
    - Direct compressed graph: `all.{graph,obl,offsets,properties}`
    ...
    • Aug 27 2019, 9:20 PM
    • 8 Lines