Page MenuHomeSoftware Heritage
Paste Active Pastes
  • Command : $ django-admin shell --settings=swh.web.settings.tests
    /home/kalpitk/.virtualenvs/swh/lib/python3.6/site-packages/swh/scheduler/__init__.py:69: DeprecationWarning: Call to deprecated class SWHRemoteAPI. (Use the RPCClient instead) -- Deprecated since version 0.0.64.
    return SchedulerBackend(**args)
    Traceback (most recent call last):
    ...
    • Thu, Aug 22, 11:45 AM
    • 38 Lines
    • Bash Scripting
  • #!/usr/bin/env python3
    import sys
    import dateutil.parser
    ...
    • Tue, Aug 20, 2:40 PM
    • 36 Lines
    • Python
  • On rioc:
    (base) [zacchiro@rioc graph]$ cat all+ori.*.count
    164513699014
    11683687950
    ...
    • Mon, Aug 19, 10:44 AM
    • 10 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-lister/swh/lister$ python jjj.py
    DEBUG:swh.lister.core.lister_base:Loading config from lister_gnu
    INFO:swh.core.config:Loading config file /home/archit/.config/swh/lister_gnu.yml
    DEBUG:swh.lister.core.lister_base:<swh.lister.gnu.lister.GNULister object at 0x7ffacb9f7a20> CONFIG={'content_size_limit': 104857600, 'log_db': 'dbname=softwareheritage-log', 'storage': {'cls': 'remote', 'args': {'url': 'http://localhost:5002/'}}, 'scheduler': {'cls': 'remote', 'args': {'url': 'http://localhost:5008/'}}, 'lister': {'cls': 'local', 'args': {'db': 'postgresql:///lister-gnu'}}, 'credentials': [], 'cache_responses': True, 'cache_dir': '/home/archit/.cache/swh/lister/gnu/'}
    DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
    ...
    • Sun, Aug 18, 2:35 PM
    • 91 Lines
  • Done on rioc:
    Used 1000 random nodes (results are in seconds):
    'git bundle' use-case
    ...
    • Thu, Aug 15, 9:46 PM
    • 15 Lines
  • swh-storage=# select * from origin_visit where origin=424 ;
    origin | visit | date | type | status | metadata | snapshot
    --------+-------+-------------------------------+------+---------+----------+--------------------------------------------
    424 | 1 | 2019-08-14 10:02:03.190532+00 | gnu | partial | |
    424 | 2 | 2019-08-14 10:02:46.964868+00 | gnu | full | | \x1f3305edbd687c27ca005f7cafea3d6f809c38d1
    ...
    • Wed, Aug 14, 1:32 PM
    • 16 Lines
  • Done on rioc:
    Used 100000 random nodes (results are in seconds):
    'ls' use-case
    ...
    • Wed, Aug 14, 11:22 AM
    • 27 Lines
  • Print of `revisions` at this line
    https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/storage.py$692
    [{'author': {'email': 'robot@softwareheritage.org',
    'fullname': 'Software Heritage',
    ...
    • Wed, Aug 14, 10:32 AM
    • 182 Lines
  • swh-storage_1 | [2019-08-13 20:10:39 +0000] [41] [DEBUG] POST /content/missing
    swh-storage_1 | [2019-08-13 20:10:39 +0000] [41] [DEBUG] POST /directory/missing
    swh-storage_1 | [2019-08-13 20:10:39 +0000] [41] [DEBUG] POST /revision/missing
    swh-storage_1 | [2019-08-13 20:10:39 +0000] [41] [DEBUG] POST /revision/add
    swh-storage_1 | ERROR:root:Object of type bytes is not JSON serializable
    ...
    • Tue, Aug 13, 10:12 PM
    • 39 Lines
  • utkarsh@G3:~$ workon swh
    Usage:: command not found
    Command: command not found
    Options:: command not found
    INFO: command not found
    ...
    • Mon, Aug 12, 3:26 PM
    • 12 Lines
  • (swh) utkarsh@G3:~/swh-environment$ swh scheduler task-type add list-packagist-full2 "swh.lister.packagist.tasks.PackagistListerTask" "Full PACKAGIST lister" --default-interval '1 day' --backoff-factor 1
    /home/utkarsh/.virtualenvs/swh/lib/python3.7/site-packages/swh/scheduler/__init__.py:69: DeprecationWarning: Call to deprecated class SWHRemoteAPI. (Use the RPCClient instead) -- Deprecated since version 0.0.64.
    return SchedulerBackend(**args)
    OK
    • Mon, Aug 12, 2:48 PM
    • 4 Lines
  • There were two revisions for this package
    {'author': {'email': b'robot@softwareheritage.org',
    'fullname': b'Software Heritage',
    'name': b'Software Heritage'},
    ...
    • Sun, Aug 11, 9:10 PM
    • 104 Lines
  • swh-loader_1 | [2019-08-11 19:02:59,149: INFO/MainProcess] Received task: swh.loader.package.tasks.LoadGNU[11fa45c7-ffc8-47ad-9e18-dcff159f1bd2]
    swh-loader_1 | [2019-08-11 19:02:59,152: INFO/ForkPoolWorker-1] Loading config file /loader.yml
    swh-loader_1 | [2019-08-11 19:02:59,169: DEBUG/ForkPoolWorker-1] Creating gnu origin for https://ftp.gnu.org/gnu/hello/
    swh-loader_1 | [2019-08-11 19:02:59,304: DEBUG/ForkPoolWorker-1] Done creating gnu origin for https://ftp.gnu.org/gnu/hello/
    swh-loader_1 | [2019-08-11 19:02:59,304: DEBUG/ForkPoolWorker-1] Creating origin_visit for origin https://ftp.gnu.org/gnu/hello/ at time 2019-08-11 19:02:59.304333+00:00
    ...
    • Sun, Aug 11, 9:06 PM
    • 82 Lines
  • iled'}
    swh-loader_1 | [2019-08-11 18:50:23,779: INFO/MainProcess] Received task: swh.loader.package.tasks.LoadGNU[7a7e43e1-dd5b-4ebe-be5b-2ea10fd0bdbc]
    swh-loader_1 | [2019-08-11 18:50:23,871: INFO/ForkPoolWorker-1] Loading config file /loader.yml
    swh-loader_1 | [2019-08-11 18:50:23,915: DEBUG/ForkPoolWorker-1] Creating gnu origin for https://ftp.gnu.org/gnu/dap/
    swh-loader_1 | [2019-08-11 18:50:23,924: DEBUG/ForkPoolWorker-1] Done creating gnu origin for https://ftp.gnu.org/gnu/dap/
    ...
    • Sun, Aug 11, 8:58 PM
    • 91 Lines
  • P497 logs
    wh-loader_1 | Using pip from /srv/softwareheritage/venv/bin/pip
    swh-loader_1 | Processing /src/swh-loader
    swh-loader_1 | Requirement already satisfied: vcversioner in /srv/softwareheritage/venv/lib/python3.7/site-packages (from swh.loader.core==0.0.44.post3) (2.16.0.0)
    swh-loader_1 | Requirement already satisfied: retrying in /srv/softwareheritage/venv/lib/python3.7/site-packages (from swh.loader.core==0.0.44.post3) (1.3.3)
    swh-loader_1 | Requirement already satisfied: psutil in /srv/softwareheritage/venv/lib/python3.7/site-packages (from swh.loader.core==0.0.44.post3) (5.6.3)
    ...
    • Fri, Aug 9, 4:35 PM
    • 295 Lines
  • NPM Revision
    {'synthetic': True,
    'metadata': {'package_source': {'name': 'ja','version': '0.0.1', 'filename': 'ja-0.0.1.tgz', 'sha1': '31399c51d3024f6eb91c626a31a175dc30f343e5', 'date': '2014-04-07T16:02:07.453Z', 'url': 'https://registry.npmjs.org/ja/-/ja-0.0.1.tgz', 'sha256': '8101e284d5846e77f9698628da660e92f8627535c266a3aef28549b25e63a597', 'blake2s256': 'fcf5438822e12c8a6973b9ecb40df4978947edd675c4269d728d5279dc82b203'},
    'package': {'author': 'Goldeneye Solutions', 'name': 'ja', 'description': 'Compose a cross-platform application from various things.',
    ...
    • Fri, Aug 9, 3:56 PM
    • 95 Lines
  • id | type | arguments | next_run | current_interval | status | policy | retries_left | priority
    --------+---------------+----------------------------+-------------------------------+------------------+--------------------+---------+--------------+----------
    225784 | list-gnu-full | {"args": [], "kwargs": {}} | 2019-08-08 19:22:27.338807+00 | 90 days | next_run_scheduled | oneshot | 0 |
    225786 | list-gnu-full | {"args": [], "kwargs": {}} | 2019-08-08 19:25:28.450559+00 | 90 days | next_run_scheduled | oneshot | 0 |
    225787 | list-gnu-full | {"args": [], "kwargs": {}} | 2019-08-08 19:29:55.666438+00 | 90 days | disabled | oneshot | 0 |
    ...
    • Fri, Aug 9, 7:47 AM
    • 8 Lines
  • swh-lister_1 | [2019-08-09 05:32:01,482: ERROR/ForkPoolWorker-1] Task swh.lister.gnu.tasks.GNUListerTask[e997c3cf-0dfc-423e-9460-3e8d685321b7] raised unexpected: NotNullViolation('null value in column "retries_left" violates not-null constraint\nDETAIL: Failing row contains (225791, load-tar, {"args": ["gcal", "https://ftp.gnu.org/gnu/gcal/"], "kwargs": {"..., 2019-08-09 05:32:01.13707+00, null, next_run_not_scheduled, recurring, null, null).\nCONTEXT: SQL statement "insert into task (type, arguments, next_run, status, current_interval, policy,\n retries_left, priority)\n select type, arguments, next_run, status, current_interval, policy,\n retries_left, priority\n from tmp_task t\n where not exists(select 1\n from task\n where type = t.type and\n arguments->\'args\' = t.arguments->\'args\' and\n arguments->\'kwargs\' = t.arguments->\'kwargs\' and\n policy = t.policy and\n priority is not distinct from t.priority and\n status = t.status)"\nPL/pgSQL function swh_scheduler_create_tasks_from_temp() line 12 at SQL statement\n')
    swh-lister_1 | Traceback (most recent call last):
    swh-lister_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
    swh-lister_1 | R = retval = fun(*args, **kwargs)
    swh-lister_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/task.py", line 45, in __call__
    ...
    • Fri, Aug 9, 7:35 AM
    • 41 Lines
  • Notice: /Stage[main]/Profile::Ssh::Server/Sshkey[ssh-worker1.internal.staging.swh.network-ecdsa-sha2-nistp256]/ensure: current_value absent, should be present (noop)
    Notice: /Stage[main]/Profile::Ssh::Server/Sshkey[ssh-worker0.internal.staging.swh.network-rsa]/ensure: current_value absent, should be present (noop)
    Notice: /Stage[main]/Profile::Ssh::Server/Sshkey[ssh-worker0.internal.staging.swh.network-ecdsa-sha2-nistp256]/ensure: current_value absent, should be present (noop)
    Notice: /Stage[main]/Profile::Ssh::Server/Sshkey[ssh-worker0.internal.staging.swh.network-ed25519]/ensure: current_value absent, should be present (noop)
    Notice: /Stage[main]/Profile::Ssh::Server/Sshkey[ssh-worker0.internal.staging.swh.network-dsa]/ensure: current_value absent, should be present (noop)
    ...
    • Thu, Aug 8, 7:28 PM
    • 172 Lines
  • version: '2'
    services:
    swh-objstorage:
    volumes:
    ...
    • Thu, Aug 8, 3:41 PM
    • 15 Lines
  • Benchmark results for content_find:
    hash_algo = sha1 (sample size=263):
    cassandra: avg = 9 ms, stdev = 2.5 ms
    postgres: avg = 14 ms, stdev = 14.9 ms
    ...
    • Thu, Aug 8, 1:47 PM
    • 72 Lines
  • from collections import defaultdict
    import csv
    import itertools
    import os
    from pprint import pprint
    ...
    • Thu, Aug 8, 12:21 PM
    • 207 Lines
    • Python
  • Benchmark results for content_find:
    hash_algo = sha1 (sample size=248):
    avg cassandra = 5 ms
    avg postgres = 14 ms
    hash_algo = sha1_git (sample size=242):
    ...
    • Thu, Aug 8, 12:16 PM
    • 25 Lines
  • DEBUG:swh.loader.package.GNULoader:Sending 2 revisions
    DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
    DEBUG:urllib3.connectionpool:http://localhost:5002 "POST /revision/add HTTP/1.1" 400 85
    ERROR:swh.loader.package.GNULoader:Loading failure, updating to `partial` status
    Traceback (most recent call last):
    ...
    • Wed, Aug 7, 1:12 PM
    • 38 Lines
  • pip install $( ./bin/pip-swh-packages --with-testing )
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-core
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-model
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-core
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-objstorage
    ...
    • Fri, Aug 2, 12:03 PM
    • 52 Lines
  • variable "region" {
    type = "string"
    default = "northeurope"
    }
    ...
    • Thu, Aug 1, 5:40 PM
    • 68 Lines
  • Bonjour,
    ci-dessous mes reponses.
    Toutefois, seriez-vous disponible demain pour en discuter par telephone?
    ...
    • Thu, Aug 1, 11:58 AM
    • 91 Lines
  • # Get an available port number
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind(('127.0.0.1', 0))
    self.port = sock.getsockname()[1]
    sock.close()
    • Thu, Aug 1, 10:05 AM
    • 5 Lines
    • Python
  • Info: Using configured environment 'new_staging'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Loading facts
    Info: Applying configuration version '1564591379'
    ...
    • Wed, Jul 31, 6:45 PM
    • 59 Lines
  • Package: DZEXPM
    Type: Package
    Title: Estimation and Prediction of Skewed Spatial Processes
    Version: 1.0
    Date: 2017-06-24
    ...
    • Thu, Jul 25, 6:56 PM
    • 14 Lines
  • root@pergamon:~# puppet agent --test --noop
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Loading facts
    ...
    • Wed, Jul 24, 4:27 PM
    • 68 Lines
  • $ git commit --date '1999-01-01' -m "foo" --allow-empty | grep Date
    Date: Fri Jan 1 12:59:34 1999 +0100
    $ git commit --date '1973-03-03' -m "foo" --allow-empty | grep Date
    Date: Sat Mar 3 13:00:21 1973 +0100
    $ git commit --date '1973-03-02' -m "foo" --allow-empty | grep Date
    ...
    • Jul 23 2019, 2:02 PM
    • 10 Lines
  • ardumont@pergamon:~% cat /etc/fstab
    # /etc/fstab: static file system information.
    #
    # Use 'blkid' to print the universally unique identifier for a
    # device; this may be used with UUID= as a more robust way to name devices
    ...
    • Jul 22 2019, 11:41 AM
    • 32 Lines
  • /
    -----master----. b
    \ /
    ...
    • Jul 19 2019, 2:10 PM
    • 18 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-loader-core/swh/loader/gnu$ tox .
    GLOB sdist-make: /home/archit/swh-environment/swh-loader-core/setup.py
    flake8 installed: entrypoints==0.3,flake8==3.7.8,mccabe==0.6.1,pycodestyle==2.5.0,pyflakes==2.1.1,swh.loader.core==0.0.44.post2
    flake8 run-test-pre: PYTHONHASHSEED='1804426209'
    flake8 run-test: commands[0] | /home/archit/swh-environment/swh-loader-core/.tox/flake8/bin/python -m flake8
    ...
    • Jul 19 2019, 9:48 AM
    • 65 Lines
  • swh_content-replayer.1.bgt2xzycn7wo@desktop6 | Starting the SWH mirror content replayer
    swh_content-replayer.1.bgt2xzycn7wo@desktop6 | /usr/lib/python3/dist-packages/swh/storage/api/server.py:20: DeprecationWarning: Call to deprecated class SWHServerAPIApp. (Use the RPCServerApp instead) -- Deprecated since version 0.0.64.
    swh_content-replayer.1.bgt2xzycn7wo@desktop6 | app = SWHServerAPIApp(__name__)
    swh_content-replayer.1.bgt2xzycn7wo@desktop6 | Traceback (most recent call last):
    swh_content-replayer.1.bgt2xzycn7wo@desktop6 | File "/usr/bin/swh", line 11, in <module>
    ...
    • Jul 18 2019, 3:42 PM
    • 15 Lines
  • kafka:
    image: wurstmeister/kafka
    environment:
    KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
    ports:
    ...
    • Jul 14 2019, 1:30 PM
    • 9 Lines
  • (swh) archit@work-pc:~/swh-environment$ doco logs kafka
    kafka_1 | log.cleaner.min.cleanable.ratio = 0.5
    kafka_1 | log.cleaner.min.compaction.lag.ms = 0
    ...
    • Jul 14 2019, 1:23 PM
    • 177 Lines
  • (swh) archit@work-pc:~/swh-environment$ doco logs swh-storage
    ...
    • Jul 14 2019, 1:21 PM
    • 47 Lines
  • INFO:swh.core.config:Loading config file /home/archit/.config/swh/loader/cran.yml
    /home/archit/swh-environment/swh-storage/swh/storage/api/client.py:13: DeprecationWarning: Call to deprecated class MetaRPCClient. (Use the MetaRPCClient instead) -- Deprecated since version 0.0.64.
    class RemoteStorage(SWHRemoteAPI):
    /home/archit/swh-environment/swh-storage/swh/storage/__init__.py:43: DeprecationWarning: Call to deprecated class RPCClient. (Use the RPCClient instead) -- Deprecated since version 0.0.64.
    return Storage(**args)
    ...
    • Jul 14 2019, 12:55 PM
    • 48 Lines
  • Traceback (most recent call last):
    File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/browse/views/utils/snapshot_context.py", line 239, in browse_snapshot_directory
    browse_context='directory') # noqa
    File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/browse/views/utils/snapshot_context.py", line 135, in _process_snapshot_request
    origin_url, timestamp, visit_id)
    ...
    • Jul 13 2019, 9:00 PM
    • 10 Lines
  • ttaching to swh-docker-dev_swh-storage_1
    swh-storage_1 | Using pip from /srv/softwareheritage/venv/bin/pip
    swh-storage_1 | Installed Python packages:
    swh-storage_1 | Package Version
    swh-storage_1 | ----------------------------- ---------
    ...
    • Jul 11 2019, 5:09 PM
    • 107 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-loader-core/swh/loader$ pip list
    Package Version Location
    ----------------------------- -------------- -------------------------------------------------
    aiohttp 3.5.4
    alabaster 0.7.12
    ...
    • Jul 11 2019, 5:05 PM
    • 164 Lines
  • DEBUG:urllib3.connectionpool:http://localhost:5002 "POST /origin/visit/get_latest HTTP/1.1" 400 74
    ERROR:swh.loader.pypi.PyPILoader:Loading failure, updating to `partial` status
    Traceback (most recent call last):
    File "/home/archit/swh-environment/swh-loader-core/swh/loader/core/loader.py", line 868, in load
    self.prepare(*args, **kwargs)
    ...
    • Jul 11 2019, 3:21 PM
    • 71 Lines
  • FROM debian:stretch
    RUN apt-get update
    RUN apt-get install -y python3-pip tox git make
    RUN git clone https://github.com/t-b/sphinxcontrib-images.git
    ...
    • Jul 11 2019, 2:57 PM
    • 18 Lines
  • Attaching to swh-docker-dev_swh-web_1
    swh-web_1 | Using pip from /srv/softwareheritage/venv/bin/pip
    swh-web_1 | Installed Python packages:
    swh-web_1 | Package Version
    swh-web_1 | ----------------------------- ---------
    ...
    • Jul 11 2019, 11:09 AM
    • 491 Lines
  • (swh) archit@work-pc:~/swh-environment$ pip uninstall sphinx
    (swh) archit@work-pc:~/swh-environment$ ./bin/update
    (swh) archit@work-pc:~/swh-environment$ pip install $(./bin/pip-swh-packages --with-testing)
    Stored in directory: /home/archit/.cache/pip/wheels/7b/6c/23/eb26369b77904c8963fae9e64338b0f0b948b4d59710760834
    ...
    • Jul 10 2019, 10:58 PM
    • 9 Lines
  • (swh) ~/swh-environment$ pip install $(./bin/pip-swh-packages --with-testing)
    Successfully built PyYAML systemd-python psycopg2 vcversioner python-cephlibs retrying pyld fastimport dulwich SQLAlchemy python-hglib sqlitedict subvertpy htmlmin pypandoc sphinxcontrib-images wrapt yarl billiard snowballstemmer future
    ERROR: sphinxcontrib-images 0.8.0 has requirement sphinx<2,>=1.1.3, but you'll have sphinx 2.1.2 which is incompatible.
    ERROR: swh-docs 0.0.0.post156 has requirement sphinx<2,>=1.3, but you'll have sphinx 2.1.2 which is incompatible.
    ...
    • Jul 10 2019, 9:08 PM
    • 6 Lines
  • Traceback (most recent call last):
    File "/home/twitu/Code/swh-environment/swh-indexer/.tox/py3/bin/pytest", line 10, in <module>
    sys.exit(main())
    File "/home/twitu/Code/swh-environment/swh-indexer/.tox/py3/lib/python3.5/site-packages/_pytest/config/__init__.py", line 58, in main
    config = _prepareconfig(args, plugins)
    ...
    • Jul 10 2019, 6:11 PM
    • 77 Lines
  • This document lists use cases and benchmark scenarii for the Software Heritage
    graph service.
    Conventions
    ...
    • Jul 7 2019, 1:20 PM
    • 170 Lines
    • reStructuredText
  • ```
    swh-lister=> select * from bitbucket_repo order by indexable desc limit 10;
    name | full_name | html_url | origin_url | origin_type | last_seen | task_id | uid | indexable
    ------------------------------+----------------------------------------------+--------------------------------------------------------------------+------------------------------------------------------------------------+-------------+----------------------------+-----------+----------------------------------------+-------------------------------
    issues_crud_hg | bb-synthetic_2-bot/issues_crud_hg | https://bitbucket.org/bb-synthetic_2-bot/issues_crud_hg | https://bitbucket.org/bb-synthetic_2-bot/issues_crud_hg | hg | 2019-07-06 07:23:56.735599 | 169799503 | {bf9c4d33-712b-4da3-b420-a959f53db27f} | 2019-07-06 07:23:52.331212+00
    ...
    • Jul 6 2019, 9:31 AM
    • 17 Lines
  • on worker09:
    ```
    @app.task(name=__name__ + '.IncrementalBitBucketLister')
    def incremental_bitbucket_lister(**lister_args):
    lister = new_lister(**lister_args)
    ...
    • Jul 4 2019, 4:53 PM
    • 13 Lines
  • swhstorage@getty:~$ swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
    INFO:swh.core.config:Loading config file /etc/softwareheritage/indexer/journal_client.yml
    Usage: swh indexer journal-client [OPTIONS]
    Error: You must specify at least one kafka broker.
    • Jul 3 2019, 4:20 PM
    • 5 Lines
  • ========== ============== ================================== ====================================== ==========
    type name url format Example
    ========== ============== ================================= ====================================== ===========
    deposit hal https://hal.archives-ouvertes.fr/ codemeta-ish [1]
    deposit intel https://software.intel.com/ codemeta-ish [2]
    ...
    • Jul 3 2019, 2:24 PM
    • 17 Lines
  • Done in 5.97s.
    python3 swh/web/manage.py migrate -v0 2>/dev/null
    python3 swh/web/manage.py createcachetable -v0 2>/dev/null
    bash -c "trap 'trap - SIGINT SIGTERM ERR EXIT; jobs -p | head -1 | xargs pkill -P' SIGINT SIGTERM ERR EXIT; python3 swh/web/manage.py runserver --nostatic --settings=swh.web.settings.tests & sleep 10 && yarn run cypress open"
    Loading failure, updating to `partial` status
    ...
    • Jul 3 2019, 12:23 PM
    • 88 Lines
  • {"exception":"BadInputExc","reason":"Error when parsing identifier. Wrong format: There should be 4 mandatory values","traceback":"Traceback (most recent call last):\n File \"/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/common/utils.py\", line 235, in resolve_swh_persistent_id\n swh_id_parsed = parse_persistent_identifier(swh_id)\n File \"/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/model/identifiers.py\", line 747, in parse_persistent_identifier\n 'Wrong format: There should be 4 mandatory values')\nswh.model.exceptions.ValidationError: ['Wrong format: There should be 4 mandatory values']\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/api/apidoc.py\", line 307, in documented_view\n response = f(request, **kwargs)\n File \"/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/api/views/identifiers.py\", line 57, in...
    • Jun 26 2019, 3:33 PM
    • 1 Line
  • {
    "data": {
    "origins": [
    {
    "id": 53607151,
    ...
    • Jun 25 2019, 4:42 PM
    • 142 Lines
    • JSON
  • query {
    origins(urls: ["https://github.com/SoftwareHeritage/swh-model"]) {
    id
    type
    url
    ...
    • Jun 25 2019, 4:42 PM
    • 24 Lines
    • Plain Text
  • swh-environment/swh-web/.tox/py3/lib/python3.5/site-packages/swh/web/tests/api/views/test_origin.py(330)test_api_origin_intrinsic_metadata_get()
    -> def test_api_origin_intrinsic_metadata_get(self, origin):
    swh-environment/swh-web/.tox/py3/lib/python3.5/site-packages/hypothesis/core.py(1023)wrapped_test()
    -> raise the_error_hypothesis_found
    • Jun 23 2019, 2:55 PM
    • 4 Lines
    • Bash Scripting
  • /swh-environment/swh-web/.tox/py3/bin/pytest --hypothesis-profile=swh-web-fast --cov .tox/py3/lib/python3.5/site-packages/swh/web --cov-branch .tox/py3/lib/python3.5/site-packages/swh/web/tests/api/views/test_origin.py
    • Jun 22 2019, 12:26 PM
    • 1 Line
    • Bash Scripting
  • with (
    patch('swh.web.common.service.idx_storage') as mock_idx_storage,
    patch('swh.web.common.service.storage' as mock_storage
    ):
    • Jun 22 2019, 9:50 AM
    • 4 Lines
    • Python
  • LinkNode([('', LinkNode()),
    ('1',
    LinkNode([('content',
    LinkNode([('ctags', LinkNode()),
    ('filetype', LinkNode()),
    ...
    • Jun 20 2019, 1:57 PM
    • 49 Lines
  • info:
    description: ''
    title: Bookings API
    version: ''
    openapi: 3.0.0
    ...
    • Jun 20 2019, 1:38 PM
    • 639 Lines
    • YAML
  • diff --git a/swh/web/api/urls.py b/swh/web/api/urls.py
    index afbcaf44..4267c73d 100644
    --- a/swh/web/api/urls.py
    +++ b/swh/web/api/urls.py
    @@ -15,6 +15,25 @@ import swh.web.api.views.snapshot # noqa
    ...
    • Jun 20 2019, 1:37 PM
    • 30 Lines
    • Diff
  • P442 code
    from bs4 import BeautifulSoup
    from collections import defaultdict
    import requests
    base_url = "https://git.savannah.gnu.org/cgit/"
    ...
    • Jun 19 2019, 11:27 PM
    • 81 Lines
  • (swh) ddouard@desktop6:~/src/swh-environment/swh-journal$ pyarc diff -c
    Accepted D1608
    Repo: swh-journal (DJNL)
    Author: douardda
    Created: 3 hours ago
    ...
    • Jun 19 2019, 2:38 PM
    • 26 Lines
  • alter table gnu_repo drop column origin_id;
    alter table gnu_repo drop column description;
    alter table phabricator_repo drop column origin_id;
    alter table phabricator_repo drop column description;
    ...
    • Jun 19 2019, 10:34 AM
    • 24 Lines
    • SQL
  • swh-lister git:(master) git push
    Counting objects: 5, done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (5/5), done.
    Writing objects: 100% (5/5), 629 bytes | 0 bytes/s, done.
    ...
    • Jun 19 2019, 7:59 AM
    • 17 Lines
  • Jun 18 11:51:45 worker01 python3[29020]: [2019-06-18 11:51:45,602: INFO/MainProcess] Received task: swh.lister.phabricator.tasks.IncrementalPhabricatorLister[6511673e-9881-494d-bc32-e2d5631809df]
    Jun 18 11:51:45 worker01 python3[29026]: [2019-06-18 11:51:45,820: ERROR/ForkPoolWorker-3] Task swh.lister.phabricator.tasks.IncrementalPhabricatorLister[6511673e-9881-494d-bc32-e2d5631809df] raised unexpected: ValueError('Problem during information fetch: ERR-CONDUIT-CORE',)
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
    ...
    • Jun 18 2019, 1:57 PM
    • 23 Lines
  • Jun 18 11:24:42 worker01 python3[12919]: [2019-06-18 11:24:42,238: INFO/MainProcess] Received task: swh.lister.phabricator.tasks.FullPhabricatorLister[551ff6d5-473b-4e6f-8e70-9d8041ab5fce]
    Jun 18 11:27:32 worker01 python3[12934]: [2019-06-18 11:27:32,544: ERROR/ForkPoolWorker-5] Task swh.lister.phabricator.tasks.FullPhabricatorLister[551ff6d5-473b-4e6f-8e70-9d8041ab5fce] raised unexpected: ChunkedEncodingError(ProtocolError('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)),)
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/urllib3/response.py", line 601, in _update_chunk_length
    self.chunk_left = int(line, 16)
    ...
    • Jun 18 2019, 1:34 PM
    • 69 Lines
  • Jun 18 10:25:08 worker06 python3[14188]: [2019-06-18 10:25:08,661: WARNING/ForkPoolWorker-5] Failed to fetch https://developers.blender.org/api/diffusion.repository.search?order=oldest&attachments[uris]=1&after=&order=oldest&limit=1: HTTPSConnectionPool(host='developers.blender.org', port=443): Max retries exceeded with url: /api/diffusion.repository.search?order=oldest&attachment
    s[uris]=1&after=&order=oldest&limit=1&api.token=api-e45yly42hzfvmxi62nssekngtl5w (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8109bbbbe0>: Failed to establish a new connection: [Errno -2] Name or service not known',))
    Jun 18 10:25:08 worker06 python3[14188]: [2019-06-18 10:25:08,661: WARNING/ForkPoolWorker-5] connection error on &order=oldest&limit=1: sleep for 10 seconds
    Jun 18 10:25:18 worker06 python3[14188]: [2019-06-18 10:25:18,671: WARNING/ForkPoolWorker-5] Failed to fetch https://developers.blender.org/api/diffusion.repository.search?order=oldest&attachments[uris]=1&after=&order=oldest&limit=1: HTTPSConnectionPool(host='developers.blender.org', port=443): Max retries exceeded with url: /api/diffusion.repository.search?order=oldest&attachment
    s[uris]=1&after=&order=oldest&limit=1&api.token=api-e45yly42hzfvmxi62nssekngtl5w (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8109bbba20>: Failed to establish a new connection: [Errno -2] Name or service not known',))
    ...
    • Jun 18 2019, 1:11 PM
    • 31 Lines
  • (swh) archit@work-pc:~/swh-environment$ doco up swh-lister
    ...
    swh-lister_1 | swh.model 0.0.36
    swh-lister_1 | swh.objstorage 0.0.33
    swh-lister_1 | swh.scheduler 0.0.56
    ...
    • Jun 18 2019, 12:39 PM
    • 97 Lines
  • -- Align table names to singular (phabricator, bitbucket, etc...) ~> same convention as in the main archive for instance
    alter table phabricator_repos rename to phabricator_repo;
    alter table bitbucket_repos rename to bitbucket_repo;
    alter table github_repos rename to github_repo;
    -- Add the missing instance column
    ...
    • Jun 18 2019, 7:05 AM
    • 10 Lines
  • P433 error
    (swh) archit@work-pc:~/swh-environment/swh-lister$ python3 setup.py develop
    running develop
    running egg_info
    writing swh.lister.egg-info/PKG-INFO
    writing dependency_links to swh.lister.egg-info/dependency_links.txt
    ...
    • Jun 17 2019, 3:50 PM
    • 284 Lines
  • rubygems_development=# SELECT count(*) FROM linksets where code != '' ;
    count
    -------
    17306
    ...
    • Jun 15 2019, 10:21 PM
    • 19 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-lister/swh/lister/packagist$ tox --recreate .
    GLOB sdist-make: /home/archit/swh-environment/swh-lister/setup.py
    flake8 recreate: /home/archit/swh-environment/swh-lister/.tox/flake8
    flake8 installdeps: flake8
    flake8 installed: entrypoints==0.3,flake8==3.7.7,mccabe==0.6.1,pycodestyle==2.5.0,pyflakes==2.1.1,swh.lister==0.0.25.post1
    ...
    • Jun 14 2019, 1:09 PM
    • 130 Lines
  • ______________________________________________ TestGitLoader.test_load_unchanged ______________________________________________
    self = <swh.loader.git.tests.test_loader.TestGitLoader testMethod=test_load_unchanged>
    def test_load_unchanged(self):
    ...
    • Jun 12 2019, 6:27 PM
    • 48 Lines
  • Here is a short report of my work on listers:
    Listers I proposed to implement in my proposal -
    GNU Lister
    Completed and merged
    ...
    • Jun 11 2019, 12:41 PM
    • 24 Lines
  • [haltode@sexus ~]$ jmap -histo 27604 | head -n 20
    num #instances #bytes class name (module)
    -------------------------------------------------------
    1: 556765035 583484484400 [B (java.base@12.0.1)
    2: 2 5047568000 [[B (java.base@12.0.1)
    ...
    • Jun 9 2019, 7:05 AM
    • 24 Lines
  • P426 cli
    # Copyright (C) 2018 The Software Heritage developers
    # See the AUTHORS file at the top-level directory of this distribution
    # License: GNU General Public License version 3, or any later version
    # See top-level LICENSE file for more information
    ...
    • Jun 8 2019, 7:11 PM
    • 155 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-lister$ swh scheduler task list
    Traceback (most recent call last):
    File "/home/archit/.virtualenvs/swh/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2419, in resolve
    return functools.reduce(getattr, self.attrs, module)
    AttributeError: module 'swh.lister.cli' has no attribute 'lister'
    ...
    • Jun 8 2019, 5:48 PM
    • 18 Lines
  • Task 1772
    Next run: in 3 months (2019-09-06 13:00:14+00:00)
    Interval: 90 days, 0:00:00
    Type: load-cran
    Policy: recurring
    ...
    • Jun 8 2019, 3:05 PM
    • 13 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-lister$ python ttesting.py
    DEBUG:swh.lister.core.lister_base:Loading config from lister_gnu
    INFO:swh.core.config:Loading config file /home/archit/.config/swh/lister_gnu.yml
    DEBUG:swh.lister.core.lister_base:<swh.lister.gnu.lister.GNULister object at 0x7fb7397e49e8> CONFIG={'content_size_limit': 104857600, 'log_db': 'dbname=softwareheritage-log', 'storage': {'cls': 'remote', 'args': {'url': 'http://localhost:5002/'}}, 'scheduler': {'cls': 'remote', 'args': {'url': 'http://localhost:5008/'}}, 'lister': {'cls': 'local', 'args': {'db': 'postgresql:///lister-gnu'}}, 'credentials': [], 'cache_responses': True, 'cache_dir': '/home/archit/.cache/swh/lister/gnu/'}
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ftp.gnu.org:443
    ...
    • Jun 7 2019, 5:53 PM
    • 40 Lines
  • wh-scheduler-api_1 | Using pip from /srv/softwareheritage/venv/bin/pip
    swh-scheduler-api_1 | Installed Python packages:
    swh-scheduler-api_1 | Package Version
    swh-scheduler-api_1 | ----------------------------- --------
    swh-scheduler-api_1 | aiohttp 3.5.4
    ...
    • Jun 6 2019, 10:42 PM
    • 505 Lines
  • Name Command State Ports
    --------------------------------------------------------------------------------
    swh-docker- docker-entrypoint.sh Up 15671/tcp,
    dev_amqp_1 rabbi ... 15672/tcp,
    25672/tcp,
    ...
    • Jun 6 2019, 10:39 PM
    • 65 Lines
  • python ttesting.py
    DEBUG:swh.lister.core.lister_base:Loading config from lister_gnu
    INFO:swh.core.config:Loading config file /home/archit/.config/swh/lister_gnu.yml
    DEBUG:swh.lister.core.lister_base:<swh.lister.gnu.lister.GNULister object at 0x7f526d50dac8> CONFIG={'content_size_limit': 104857600, 'log_db': 'dbname=softwareheritage-log', 'storage': {'cls': 'remote', 'args': {'url': 'http://localhost:5002/'}}, 'scheduler': {'cls': 'remote', 'args': {'url': 'http://localhost:5008/'}}, 'lister': {'cls': 'local', 'args': {'db': 'postgresql:///lister-gnu'}}, 'credentials': [], 'cache_responses': True, 'cache_dir': '/home/archit/.cache/swh/lister/gnu/'}
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ftp.gnu.org:443
    ...
    • Jun 6 2019, 9:45 PM
    • 35 Lines
  • import java.math.BigInteger;
    import static java.math.BigInteger.ZERO;
    import static java.nio.charset.StandardCharsets.US_ASCII;
    import static java.util.Arrays.copyOf;
    import static java.lang.Math.max;
    ...
    • Jun 6 2019, 2:29 PM
    • 48 Lines
    • Java
  • import java.math.BigInteger;
    import static java.math.BigInteger.ZERO;
    import static java.nio.charset.StandardCharsets.US_ASCII;
    import static java.util.Arrays.copyOf;
    import static java.lang.Math.max;
    ...
    • Jun 6 2019, 12:03 PM
    • 48 Lines
    • Java
  • (swh) archit@work-pc:~/swh-environment/swh-lister$ python3 -m swh.lister.cli --db-url postgres:///lister-gnu gnu
    Traceback (most recent call last):
    File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
    File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    ...
    • Jun 5 2019, 10:42 AM
    • 36 Lines
  • version: '2'
    services:
    swh-lister:
    volumes:
    ...
    • Jun 5 2019, 10:19 AM
    • 6 Lines
  • $ gem list -r --all
    ...
    zulip (0.0.2, 0.0.1)
    zulip-client (0.2.4, 0.2.3, 0.2.2, 0.2.1, 0.2.0, 0.1.0)
    zulip_machine (0.0.1)
    ...
    • Jun 2 2019, 8:23 PM
    • 82 Lines
  • [haltode@sexus software-heritage]$ zgrep "f39d7d78b70e0f39facb1e4fab77ad3df5c52a35" rev_to_rev.edges.csv.gz
    f39d7d78b70e0f39facb1e4fab77ad3df5c52a35 3150ce629cf66d313453fd9a5e20b3f4884911eb
    f39d7d78b70e0f39facb1e4fab77ad3df5c52a35 30a7acd573899fd8b8ac39236eff6468b195ac7d
    52c90f2d32bfa7d6eccd66a56c44ace1f78fbadd f39d7d78b70e0f39facb1e4fab77ad3df5c52a35
    a31e58e129f73ab5b04016330b13ed51fde7a961 f39d7d78b70e0f39facb1e4fab77ad3df5c52a35
    • May 31 2019, 9:35 AM
    • 5 Lines
  • P411 Error
    arc diff --update D1482 --less-context origin/master
    You have untracked files in this working copy.
    Working copy: /home/archit/swh-environment/swh-lister/
    ...
    • May 28 2019, 4:47 PM
    • 35 Lines
  • arc diff --update D1482 origin/master
    You have untracked files in this working copy.
    Working copy: /home/archit/swh-environment/swh-lister/
    ...
    • May 28 2019, 4:43 PM
    • 36 Lines
  • 3dldf
    https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.3.tar.gz
    https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.4.tar.gz
    https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.5.1.tar.gz
    https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.5.tar.gz
    ...
    • May 27 2019, 11:42 PM
    • 8,990 Lines
  • {'name': '3dldf', 'url': 'https://ftp.gnu.org/gnu/3dldf/', 'list_of_tarballs': [{'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.3.tar.gz', 'date': '1071002600'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.4.tar.gz', 'date': '1071078759'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.5.1.tar.gz', 'date': '1074278633'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.5.tar.gz', 'date': '1074281010'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-2.0.1.tar.gz', 'date': '1384100514'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-2.0.2.tar.gz', 'date': '1384108703'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-2.0.3-examples.tar.gz', 'date': '1386961043'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-2.0.3.tar.gz', 'date': '1386954662'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-2.0.tar.gz', 'date': '1383935767'}]}
    {'name': '8sync', 'url': 'https://ftp.gnu.org/gnu/8sync/', 'list_of_tarballs': [{'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.1.0.tar.gz', 'date': '1461357336'}, {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.2.0.tar.gz', 'date': '1480991830'}, {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.3.0.tar.gz', 'date': '1482181784'}, {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.4.0.tar.gz', 'date': '1483894300'}, {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.4.1.tar.gz', 'date': '1483911251'}, {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.4.2.tar.gz', 'date': '1489817041'}]}
    {'name': 'a2ps', 'url': 'https://ftp.gnu.org/gnu/a2ps/', 'list_of_tarballs': [{'archive': 'https://ftp.gnu.org/gnu/a2ps/a2ps-4.10.4.tar.gz', 'date': '907035300'}, {'archive': 'https://ftp.gnu.org/gnu/a2ps/a2ps-4.12.tar.gz', 'date': '919729234'}, {'archive': 'https://ftp.gnu.org/gnu/a2ps/a2ps-4.13b.tar.gz', 'date': '1015264004'}, {'archive': 'https://ftp.gnu.org/gnu/a2ps/a2ps-4.14.tar.gz', 'date': '1198900401'}]}
    {'name': 'acct', 'url': 'https://ftp.gnu.org/gnu/acct/', 'list_of_tarballs': [{'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.3.2.tar.gz', 'date': '889516800'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.3.tar.gz', 'date': '872060400'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.5.1.tar.gz', 'date': '1252260451'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.5.3.tar.gz', 'date': '1264282921'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.5.4.tar.gz', 'date': '1265992809'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.5.5.tar.gz', 'date': '1288986081'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.5.tar.gz', 'date': '1251999322'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.6.1.tar.gz', 'date': '1362788330'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.6.2.tar.gz', 'date': '1415424027'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.6.3.tar.gz', 'date': '1492450287'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.6.4.tar.gz', 'date': '1499010720'}]}
    {'name': 'acm', 'url': 'https://ftp.gnu.org/gnu/acm/', 'list_of_tarballs': [{'archive': 'https://ftp.gnu.org/gnu/acm/acm-5.1.tar.gz', 'date': '1291834535'}]}
    ...
    • May 27 2019, 11:12 PM
    • 378 Lines
  • [
    {
    "Package": "A3",
    "Version": "1.0.0",
    "Depends": "R (>= 2.15.0), xtable, pbapply",
    ...
    • May 26 2019, 7:44 PM
    • 99 Lines
  • #+title: dbs
    Dbs inventory so far.
    Order by node name/cluster:port
    ...
    • May 23 2019, 3:46 PM
    • 94 Lines
    • Plain Text
  • <?xml version='1.0' encoding='UTF-8'?>
    <index>
    <info>
    <indexPath>/home/archit/Downloads</indexPath>
    <fields count='31'>
    ...
    • May 23 2019, 1:02 PM
    • 1,598 Lines
  • Found host vangogh.euwest.azure.internal.softwareheritage.org
    *** Running octocatalog-diff on host vangogh.euwest.azure.internal.softwareheritage.org
    diff origin/production/vangogh.euwest.azure.internal.softwareheritage.org current/vangogh.euwest.azure.internal.softwareheritage.org
    *******************************************
    + Concat::Fragment[nginx-swh-objstorage-500-6666cd76f96956469e7be39d750cc7d9] =>
    ...
    • May 22 2019, 10:05 AM
    • 1,994 Lines
    • Bash Scripting