Page MenuHomeSoftware Heritage
Paste Active Pastes
  • Old :
    CREATE TABLE public.directory (
    id public.sha1_git NOT NULL,
    dir_entries bigint[],
    ...
    • Tue, Feb 19, 11:48 AM
    • 33 Lines
  • Feb 14 17:46:13 worker01 python3[2344]: [2019-02-14 17:46:13,609: ERROR/ForkPoolWorker-1] Task swh.deposit.loader.tasks.ChecksDepositTsk[e201d7a8-6f19-4248-a08c-d10874c2e6a3] raised unexpected: AttributeError("'DepositChecker' object has no attribute 'log'",)
    Feb 14 17:46:13 worker01 python3[2344]: Traceback (most recent call last):
    Feb 14 17:46:13 worker01 python3[2344]: File "/usr/lib/python3/dist-packages/swh/deposit/loader/checker.py", line 21, in check
    Feb 14 17:46:13 worker01 python3[2344]: self.client.check(deposit_check_url)
    Feb 14 17:46:13 worker01 python3[2344]: File "/usr/lib/python3/dist-packages/swh/deposit/client/__init__.py", line 208, in check
    ...
    • Thu, Feb 14, 7:01 PM
    • 42 Lines
  • Feb 14 14:24:05 worker01 python3[4534]: [2019-02-14 14:24:05,077: ERROR/ForkPoolWorker-1] Task swh.lister.gitlab.tasks.IncrementalGitLabLister[f7f2d92d-bfa9-4994-a637-e3859a73f432] raised unexpected: TypeError('incremental_gitlab_lister() takes 0 positional arguments but 1 was given',)
    Feb 14 14:24:05 worker01 python3[4534]: Traceback (most recent call last):
    Feb 14 14:24:05 worker01 python3[4534]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 382, in trace_task
    Feb 14 14:24:05 worker01 python3[4534]: R = retval = fun(*args, **kwargs)
    Feb 14 14:24:05 worker01 python3[4534]: File "/usr/lib/python3/dist-packages/swh/scheduler/task.py", line 45, in __call__
    ...
    • Thu, Feb 14, 3:39 PM
    • 18 Lines
  • rng | frq | bar
    -------------+----------+--------------------------------
    [4,73) | 22473062 | ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    [72,142) | 4225673 | ■■■■■■
    [141,211) | 3351946 | ■■■■
    ...
    • Tue, Feb 12, 4:41 PM
    • 22 Lines
  • goal: Make the stretch-swh's build for the scheduler ok
    Relevant commit:
    ```
    cd swh-scheduler
    ...
    • Mon, Feb 11, 4:24 PM
    • 38 Lines
  • ============================= test session starts ==============================
    platform linux -- Python 3.5.3, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
    rootdir: /<<PKGBUILDDIR>>, inifile:
    plugins: postgresql-1.3.4, hypothesis-3.6.1, celery-4.2.1
    collected 42 items
    ...
    • Mon, Feb 11, 11:15 AM
    • 68 Lines
  • Started by timer
    Running in Durability level: MAX_SURVIVABILITY
    Loading library swh@master
    Attempting to resolve master from remote references...
    > git --version # timeout=10
    ...
    • Thu, Feb 7, 3:38 PM
    • 138 Lines
  • gemspec | codemeta | pkginfo | npm | maven | total | total_nonempty
    ---------+----------+---------+---------+--------+----------+----------------
    143688 | 139 | 3525 | 1313459 | 447697 | 17730303 | 2033490
    • Mon, Feb 4, 10:55 AM
    • 3 Lines
  • #!/bin/bash
    # wrapper to run GitHub Licensee license detection tool form a git clone of its
    # repo, setting up the appropriate Ruby load path
    ...
    • Sat, Feb 2, 1:35 PM
    • 8 Lines
  • Jan 31 13:04:36 storage0 python3 [2434334]: 2019-01-31 13:04:36 [2434334] [ERROR] canceling statement due to statement timeout
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request ()
    File "/usr/lib/python3/dist-packages/flask/app.py", line 1598, in dispatch_request
    ...
    • Thu, Jan 31, 2:11 PM
    • 31 Lines
  • *** swh-deploy: deploying on moma.internal.softwareheritage.org...
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Loading facts
    ...
    • Fri, Jan 25, 2:49 PM
    • 51 Lines
  • swh-scheduler-api_1 | [INFO] werkzeug -- 172.20.0.15 - - [24/Jan/2019 10:48:57] "POST /create_tasks HTTP/1.1" 200 -
    swh-scheduler-api_1 | [INFO] werkzeug -- 172.20.0.15 - - [24/Jan/2019 10:48:57] "POST /create_tasks HTTP/1.1" 200 -
    swh-scheduler-api_1 | [INFO] werkzeug -- 172.20.0.15 - - [24/Jan/2019 10:48:57] "POST /create_tasks HTTP/1.1" 200 -
    swh-scheduler-api_1 | [ERROR] root -- relation "tmp_task" already exists
    ...
    • Thu, Jan 24, 11:56 AM
    • 47 Lines
  • $ pip3 show pyld | grep Version
    Version: 1.0.3
    $ python3
    Python 3.5.3 (default, Sep 27 2018, 17:25:39)
    [GCC 6.3.0 20170516] on linux
    ...
    • Jan 16 2019, 5:57 PM
    • 29 Lines
  • >>> import pprint
    >>> import swh.indexer.storage.api.client
    >>> s = swh.indexer.storage.api.client.RemoteStorage(url='http://uffizi.internal.softwareheritage.org:5007/')
    >>> pprint.pprint(s.origin_intrinsic_metadata_search_fulltext(['James']))
    [{'from_revision': b'\xd4bM\xa6\x9eH\x06\x15\x0c\x1ap\xbc\x84~\x11\x17'
    ...
    • Jan 14 2019, 1:40 PM
    • 85 Lines
    • Python
  • import sys
    import time
    #import kafka
    ...
    • Jan 10 2019, 1:38 PM
    • 32 Lines
  • # Copyright (C) 2018 The Software Heritage developers
    # See the AUTHORS file at the top-level directory of this distribution
    # License: GNU General Public License version 3, or any later version
    # See top-level LICENSE file for more information
    ...
    • Jan 10 2019, 11:53 AM
    • 50 Lines
    • Python
  • -- DONE
    CREATE TABLE ctas_dataset_dir_to_rev
    WITH (format = 'TEXTFILE', external_location =
    's3://softwareheritage/edges_dataset/dir_to_rev/', field_delimiter = ' ')
    AS SELECT to_hex(directory.id) as source, to_hex(target) as dest
    ...
    • Jan 7 2019, 7:15 PM
    • 72 Lines
    • SQL
  • swh-environment $ pip install $( ./bin/pip-swh-packages --with-testing )
    swh-storage[schemata,listener][testing] should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+
    • Dec 20 2018, 3:31 PM
    • 2 Lines
  • ```
    $ pifpaf run postgresql -- pytest
    WARNING [pifpaf.drivers] `psutil.Popen(pid=23360, status='terminated')` is already gone, sending SIGKILL to its process group
    ERROR [pifpaf] sequence item 0: expected str instance, bytes found
    ```
    ...
    • Dec 20 2018, 3:24 PM
    • 45 Lines
  • swh-scheduler-api_1 | ERROR:root:fe_sendauth: no password supplied
    swh-scheduler-api_1 | Traceback (most recent call last):
    swh-scheduler-api_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    swh-scheduler-api_1 | rv = self.dispatch_request()
    swh-scheduler-api_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
    ...
    • Dec 17 2018, 3:22 PM
    • 19 Lines
  • ✘ dev@desktop5  ~/swh-environment/swh-docs   master  git pull
    Already up-to-date.
    dev@desktop5  ~/swh-environment/swh-docs   master  tox -r -e sphinx-dev
    GLOB sdist-make: /home/dev/swh-environment/swh-docs/setup.py
    sphinx-dev recreate: /home/dev/swh-environment/swh-docs/.tox/sphinx-dev
    ...
    • Nov 29 2018, 6:29 PM
    • 46 Lines
  • When i have something like:
    ```
    if a:
    r = do_something(a)
    ...
    • Nov 22 2018, 11:46 AM
    • 21 Lines
  • Hypothesis issue with old version: https://github.com/HypothesisWorks/hypothesis/issues/290
    build output:
    ```
    ============================= test session starts ==============================
    ...
    • Nov 19 2018, 11:38 AM
    • 98 Lines
  • pytest swh/indexer/tests/storage/test_storage.py -x
    ========================================================================================================= test session starts =========================================================================================================
    platform linux -- Python 3.5.3, pytest-3.9.3, py-1.7.0, pluggy-0.8.0
    hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/dev/swh-environment/swh-indexer/.hypothesis/examples')
    rootdir: /home/dev/swh-environment/swh-indexer, inifile: pytest.ini
    ...
    • Nov 15 2018, 10:52 AM
    • 99 Lines
  • diff <(grep 'def ' swh/storage/in_memory.py | grep -v 'def _' | sed -e 's/(.*//' | sort) <(grep 'def ' swh/storage/storage.py | grep -v 'def _' | sed -e 's/(.*//' | sort)
    0a1
    > def add_to_objstorage
    3a5
    > def content_get
    ...
    • Nov 15 2018, 10:07 AM
    • 21 Lines
    • Diff
  • | context | file_name | counted | percentage | percentage | percentage on 3,424,000,000 files |
    |-------------------------------|-------------------|----------------------|------------|------------|-----------------------------------|
    | CodeMeta | CODE | 320 | 0.00% | | 8.85E-06 |
    | haskell | .cabal | 676053 | 1.27% | 0.01% | 0.01870298068 |
    | java- Maven | pom.xml | 15509125 | 29.03% | 0.43% | 0.4290593566 |
    ...
    • Nov 12 2018, 5:11 PM
    • 33 Lines
  • def _naive_sig(param_names):
    return inspect.Signature([
    inspect.Parameter(name, inspect.Parameter.POSITIONAL_OR_KEYWORD)
    for name in param_names])
    ...
    • Nov 9 2018, 1:07 PM
    • 82 Lines
    • Python
  • def content_mimetype_missing(self, mimetypes, db=None, cur=None):
    """Generates mimetypes missing from storage.
    Args:
    mimetypes (iterable): iterable of dict with keys:
    ...
    • Nov 8 2018, 12:03 PM
    • 13 Lines
    • Python
  • commit edebe6a4a42bae99a1819898b14bb0951cfe6b8b
    Author: Valentin Lorentz <vlorentz@softwareheritage.org>
    Date: Mon Nov 5 14:54:06 2018 +0100
    Remove testrepo.zip.
    ...
    • Nov 5 2018, 2:52 PM
    • 64 Lines
    • Diff
  • GLOB sdist-make: /home/morane/Documents/code/swh-environment/swh-indexer/setup.py
    flake8 recreate: /home/morane/Documents/code/swh-environment/swh-indexer/.tox/flake8
    flake8 installdeps: flake8
    flake8 installed: flake8==3.6.0,mccabe==0.6.1,pkg-resources==0.0.0,pycodestyle==2.4.0,pyflakes==2.0.0
    flake8 runtests: PYTHONHASHSEED='3997505493'
    ...
    • Oct 30 2018, 4:31 PM
    • 3,664 Lines
  • dev@desktop5  ~/swh-environment/swh-indexer   master  git pull
    remote: Counting objects: 14, done.
    remote: Compressing objects: 100% (14/14), done.
    remote: Total 14 (delta 10), reused 0 (delta 0)
    Unpacking objects: 100% (14/14), done.
    ...
    • Oct 29 2018, 10:22 AM
    • 14 Lines
  • If it's urgent to redeploy indexers, here is what i foresee in the
    current state of affairs,
    - After review and acceptance, merge the diffs:
    ...
    • Oct 27 2018, 11:16 AM
    • 58 Lines
  • celery.worker.strategy: INFO: Received task: swh.indexer.tests.test_origin_metadata.test_revision_metadata_task[785b1155-89f2-4aec-ac50-c2d3eb34b4d3]
    celery.app.trace: ERROR: Task swh.indexer.tests.test_origin_metadata.test_revision_metadata_task[785b1155-89f2-4aec-ac50-c2d3eb34b4d3] raised unexpected: EncodeError(TypeError("b'8dbb6aeb036e7fd80664eb8bfd1507881af1ba9f' is not JSON serializable",),)
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
    ...
    • Oct 25 2018, 3:08 PM
    • 81 Lines
  • ======================================================================
    FAIL: test_pipeline (swh.indexer.tests.test_origin_metadata.TestOriginMetadata)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    ...
    • Oct 25 2018, 2:47 PM
    • 26 Lines
  • def test_compute_metadata_codemeta(self):
    """
    test that a codemeta file is not altered with translation
    """
    ...
    • Oct 25 2018, 12:06 PM
    • 50 Lines
  • pre-requisite:
    ```
    arc patch D582
    ```
    ...
    • Oct 24 2018, 11:12 AM
    • 1,748 Lines
  • KeyboardInterrupt
    Exception ignored in: <bound method AsyncResult.__del__ of <AsyncResult: 8731ccc1-5068-41ff-9b83-e0e2ef930843>>
    Traceback (most recent call last):
    File "/home/morane/.local/lib/python3.5/site-packages/celery/result.py", line 385, in __del__
    self.backend.remove_pending_result(self)
    ...
    • Oct 23 2018, 5:01 PM
    • 60 Lines
  • ---
    errors:
    hg:
    - "Failed to uncompress archive"
    - "OSError: [Errno 12] Cannot allocate memory"
    ...
    • Oct 18 2018, 11:13 AM
    • 116 Lines
  • origin-update-pypi;oneshot;["manhattan_seo", "https://pypi.org/project/manhattan_seo/"];{"project_metadata_url": "https://pypi.org/pypi/manhattan_seo/json"}
    origin-update-pypi;oneshot;["hypermax", "https://pypi.org/project/hypermax/"];{"project_metadata_url": "https://pypi.org/pypi/hypermax/json"}
    origin-update-pypi;oneshot;["Flask-Security-Bundle", "https://pypi.org/project/Flask-Security-Bundle/"];{"project_metadata_url": "https://pypi.org/pypi/Flask-Security-Bundle/json"}
    origin-update-pypi;oneshot;["collective.dms.mailcontent", "https://pypi.org/project/collective.dms.mailcontent/"];{"project_metadata_url": "https://pypi.org/pypi/collective.dms.mailcontent/json"}
    origin-update-pypi;oneshot;["certbot-dns-cloudxns", "https://pypi.org/project/certbot-dns-cloudxns/"];{"project_metadata_url": "https://pypi.org/pypi/certbot-dns-cloudxns/json"}
    ...
    • Oct 17 2018, 4:21 PM
    • 1,411 Lines
  • return self.RevisionMetadataTask().apply_async(
    kwargs={
    'ids': [res['revision_id'] for res in results],
    'policy_update': 'update-dups',
    },
    ...
    • Oct 16 2018, 2:25 PM
    • 9 Lines
    • Python
  • jq . pypi.group-output.txt | grep -v 'Reason: 404'
    {
    "googlecode": {
    "total": 3933,
    "errors": {
    ...
    • Oct 16 2018, 1:50 PM
    • 178 Lines
  • ```
    export PYTHONPATH=$SWH_ENVIRONMENT_HOME/snippets/ardumont:$PYTHONPATH
    python3 -m kibana_fetch_logs > output.txt
    cat output.txt | python3 -m group_by_exception --loader-type hg > output-group-by.txt
    ```
    ...
    • Oct 15 2018, 4:22 PM
    • 7 Lines
  • curl -XPOST 'http://esnode3.internal.softwareheritage.org:9200/swh_workers-2018.10.11,swh_workers-2018.10.12,swh_workers-2018.10.13,swh_workers-2018.10.14/_search' -d '{
    "from": 10,
    "_source": [
    "message",
    ...
    • Oct 15 2018, 2:33 PM
    • 36 Lines
  • {
    "swh_workers-2018.09.30": {
    "mappings": {
    "doc": {
    "properties": {
    ...
    • Oct 15 2018, 1:36 PM
    • 617 Lines
  • {
    "order": 0,
    "index_patterns": [
    "swh_workers-*"
    ],
    ...
    • Oct 15 2018, 1:33 PM
    • 16 Lines
  •  tony  (e) .venv   arcpatch-D505_1  …  swh  swh-environment  swh-indexer  2  make test
    python3 -m nose -sv --with-doctest .
    Failure: TypeError (metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases) ... ERROR
    Failure: TypeError (metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases) ... ERROR
    Failure: TypeError (metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases) ... ERROR
    ...
    • Oct 11 2018, 10:46 AM
    • 465 Lines
  • http://kibana0.internal.softwareheritage.org:5601/app/kibana#/dashboard/32632370-c0bd-11e8-8222-07f3ec376cd5?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:%272018-08-31T22:00:00.000Z%27,mode:absolute,to:%272018-10-05T21:59:59.999Z%27))&_a=(description:%27This%20is%20a%20general%20dashboard%20listing%20the%20full%20logs%20of%20the%20swh-workers%27,filters:!((%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:%2720de3150-c0bb-11e8-8222-07f3ec376cd5%27,key:systemd_unit,negate:!f,params:(query:%27swh-worker@swh_loader_pypi.service%27,type:phrase),type:phrase,value:%27swh-worker@swh_loader_pypi.service%27),query:(match:(systemd_unit:(query:%27swh-worker@swh_loader_pypi.service%27,type:phrase)))),(%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:%2720de3150-c0bb-11e8-8222-07f3ec376cd5%27,key:priority,negate:!f,params:(query:%273%27,type:phrase),type:phrase,value:%273%27),query:(match:(priority:(query:%273%27,type:phrase)))),(%27$state%27:(store:appState),meta:(alias:!n...
    • Oct 5 2018, 6:29 PM
    • 1 Line
  • dev@desktop5  ~/swh-environment/swh-indexer   origin-head-indexer  grep content_metadata_add **/*.{py,sql}
    swh/indexer/metadata.py: self.idx_storage.content_metadata_add(
    swh/indexer/storage/__init__.py: def content_metadata_add(self, metadata, conflict_update=False, db=None,
    swh/indexer/storage/__init__.py: db.content_metadata_add_from_temp(conflict_update, cur)
    swh/indexer/storage/api/client.py: def content_metadata_add(self, metadata, conflict_update=False):
    ...
    • Oct 5 2018, 4:22 PM
    • 20 Lines
  • Traceback (most recent call last):
    File "origin_head.py", line 121, in <module>
    main()
    File "/usr/lib/python3/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
    ...
    • Oct 5 2018, 2:51 PM
    • 26 Lines
  • ```
    test_revision_metadata_indexer (swh.indexer.tests.test_metadata.Metadata) ... ERROR
    ...
    ======================================================================
    ...
    • Oct 4 2018, 4:19 PM
    • 28 Lines
  • # swhpass and completion mechanism
    SWH_PASSWORD_STORE_DIR=${HOME}/work/inria/repo/swh/credentials/
    function swhpass() {
    PASSWORD_STORE_DIR=$SWH_PASSWORD_STORE_DIR pass $@
    }
    ...
    • Oct 4 2018, 3:00 PM
    • 12 Lines
  • diff --git a/setup.py b/setup.py
    index 42e154f..93d3192 100644
    --- a/setup.py
    +++ b/setup.py
    @@ -34,6 +34,7 @@ setup(
    ...
    • Oct 1 2018, 2:55 PM
    • 12 Lines
    • Diff
  • http://acsccg.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/acsccg/acsccg-repo.svndump.gz
    http://anarchintosh-projects.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/anarchintosh-projects/anarchintosh-projects-repo.svndump.gz
    http://bastian.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/b/bastian/bastian-repo.svndump.gz
    http://calculapdf.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/c/calculapdf/calculapdf-repo.svndump.gz
    http://dagamers.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/d/dagamers/dagamers-repo.svndump.gz
    ...
    • Sep 28 2018, 4:04 PM
    • 23 Lines
  • http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 240, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 742, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 308, in store_data\n self.last_known_swh_revision)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 496, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 217, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 161, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 266, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 333, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 318, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 742, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 308, in store_data\n self.last_known_swh_revision)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 496, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 217, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 161, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 266, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 333, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 318, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 732, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 308, in store_data\n self.last_known_swh_revision)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 496, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 217, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 161, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 266, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 336, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 321, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 628, in load\n """Detailed visit status.\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 300, in store_data\n Note:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 489, in process_repository\n self.log.info(msg)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 229, in process_swh_revisions\n _id = known_swh_rev.get('id')\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 213, in process_swh_revisions\n revs = []\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 155, in process_svn_revisions\n gen_revs = svnrepo.swh_hash_data_per_revision(\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 453, in compute_hashes\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 411, in replay\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    ...
    • Sep 27 2018, 3:47 PM
    • 81 Lines
  • $ python3 -m venv .venv
    $ source .venv/bin/activate
    $ pip install $( bin/pip-swh-packages )
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-core
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-model
    ...
    • Sep 25 2018, 5:35 PM
    • 68 Lines
  • root@scratch01-euwest ~ # du -sh /srv/hdd/swh-parquet/origin_visit/*
    128M /srv/hdd/swh-parquet/origin_visit/019ed7acd0d64f309dcfb3a977f11480.parquet
    113M /srv/hdd/swh-parquet/origin_visit/05160c9af2bf41379f605259a3e1cb24.parquet
    122M /srv/hdd/swh-parquet/origin_visit/07e9ce57dd544eebb539c376f7c81aaa.parquet
    136M /srv/hdd/swh-parquet/origin_visit/2ccd24604c194508909a3ae7f431d731.parquet
    ...
    • Sep 25 2018, 1:33 PM
    • 36 Lines
  • Table origin_visit...
    0%|▏ | 500000/230786000 [00:10<1:22:12, 46691.30it/s]
    Table origin...
    1%|▍ | 500000/85109300 [00:04<13:35, 103756.07it/s]
    Table occurrence_history...
    ...
    • Sep 20 2018, 2:32 PM
    • 34 Lines
  • Sep 12 14:20:04 worker12 python3[25846]: [2018-09-12 14:20:04,287: ERROR/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[a129a228-79b1-47e1-8c37-70e913c1f544] raised unexpected: ConnectionResetError(104, 'Connection reset by peer')
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 438, in __protected_call__
    ...
    • Sep 19 2018, 1:59 PM
    • 38 Lines
  • begin;
    create or replace function swh_count_total_indexes_size()
    returns bigint
    language plpgsql
    ...
    • Sep 6 2018, 2:43 PM
    • 27 Lines
  • logstash0: Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install logstash=6.3.2' returned 100: Reading package lists...
    logstash0: Building dependency tree...
    logstash0: Reading state information...
    logstash0: E: Version '6.3.2' for 'logstash' was not found
    logstash0: Error: /Stage[main]/Profile::Logstash/Package[logstash]/ensure: change from 1:6.3.2-1 to 6.3.2 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--$
    ...
    • Aug 31 2018, 10:10 AM
    • 31 Lines
  • #+BEGIN_SRC emacs-lisp
    (defun swh-percent (current total)
    (* 100 (/ current (* total 1.0))))
    ...
    • Aug 24 2018, 11:49 AM
    • 25 Lines
  • alpha
    Asuka
    Automiko
    appReviewToSlack
    aio2gis
    ...
    • Aug 22 2018, 4:03 PM
    • 402 Lines
  • allspark
    archiveorg
    audit
    astar
    apicheckr
    ...
    • Aug 22 2018, 2:53 PM
    • 378 Lines
  • {'info': {'author': 'Nathan Harrington',
    'author_email': 'nharrington@wasatchphotonics.com',
    'bugtrack_url': None,
    'classifiers': [],
    'description': 'UNKNOWN',
    ...
    • Aug 2 2018, 3:07 PM
    • 96 Lines
  • {
    "info": {
    "author": "bernardfrk",
    "author_email": "bernard.frk@gmail.com",
    "bugtrack_url": null,
    ...
    • Aug 2 2018, 3:03 PM
    • 95 Lines
  • python3
    Python 3.6.6 (default, Jun 27 2018, 14:44:17)
    [GCC 8.1.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import logging; logging.basicConfig(level=logging.DEBUG); from swh.loader.pypi.tasks import LoadPyPiTsk; LoadPyPiTsk().run('arrow', 'https://pypi.org/pypi/arrow/', project_metadata_url='https://pypi.org/pypi/arrow/json')
    ...
    • Aug 1 2018, 2:29 PM
    • 309 Lines
  • begin;
    -- identify release and snapshots to mark them for cleanup
    create or replace function mark_wrong_release_and_snapshot_for_cleanup()
    returns void
    ...
    • Jul 25 2018, 6:23 PM
    • 180 Lines
    • SQL
  • @istest
    def empty_merge(self):
    """
    """
    full_dict = {
    ...
    • Jul 24 2018, 2:03 PM
    • 146 Lines
  • Makefile.local:30: warning: overriding recipe for target 'test'
    ../Makefile.python:22: warning: ignoring old recipe for target 'test'
    ./swh/manage.py test
    Without authentication, endpoint refuses access with 401 response ... ok
    Without authentication, endpoint refuses access with 401 response ... ok
    ...
    • Jul 24 2018, 1:43 PM
    • 283 Lines
  • ======================================================================
    ERROR: Existing status for update with info should return a 204 response
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/home/morane/Documents/code/swh-environment/swh-deposit/swh/deposit/tests/api/test_deposit_update_status.py", line 80, in update_deposit_status_with_info
    ...
    • Jul 24 2018, 1:41 PM
    • 92 Lines
  • <?xml version="1.0" encoding="utf-8"?>
    <entry xmlns="http://www.w3.org/2005/Atom"
    xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0">
    <title>Je suis GPL</title>
    <client>swh</client>
    ...
    • Jul 23 2018, 2:03 PM
    • 25 Lines
  • This diff is against commit f29e3f49ad973df93bfbf9e543b808fe43003bf9, but
    the commit is nowhere in the working copy. Try to apply it against the
    current working copy state? (9b4eb1e1b43349e4356307e1b5f924c870fc0b25)
    [Y/n] Y
    ...
    • Jul 16 2018, 11:41 AM
    • 25 Lines
  • Setup apt archive
    -----------------
    Merged Build-Depends: dose-distcheck
    Filtered Build-Depends: dose-distcheck
    ...
    • Jul 2 2018, 2:35 PM
    • 78 Lines
  • Jul 02 11:44:49 moma gunicorn3[27165]: [02/Jul/2018 11:44:49] ERROR [django.request:132] Internal Server Error: /1/hal/131/media/
    Jul 02 11:44:49 moma gunicorn3[27165]: Traceback (most recent call last):
    Jul 02 11:44:49 moma gunicorn3[27165]: File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 42, in inner
    Jul 02 11:44:49 moma gunicorn3[27165]: response = get_response(request)
    Jul 02 11:44:49 moma gunicorn3[27165]: File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 187, in _get_response
    ...
    • Jul 2 2018, 1:59 PM
    • 58 Lines
  • elasticsearch | [2018-06-22T14:17:31,321][WARN ][o.e.l.LicenseService ] [-7edhpM]
    elasticsearch | #
    elasticsearch | # LICENSE [EXPIRED] ON [THURSDAY, JUNE 21, 2018]. IF YOU HAVE A NEW LICENSE, PLEASE UPDATE IT.
    elasticsearch | # OTHERWISE, PLEASE REACH OUT TO YOUR SUPPORT CONTACT.
    elasticsearch | #
    ...
    • Jun 22 2018, 4:19 PM
    • 26 Lines
  • # master (somerset)
    ## bootstrap db data replication from master to slave:
    ```
    ...
    • Jun 20 2018, 11:31 AM
    • 142 Lines
  • select distinct content.sha1 from (
    select distinct walk.target as target
    from (
    select distinct snapshot_branch.target as target
    from (
    ...
    • Jun 19 2018, 3:21 PM
    • 25 Lines
    • SQL
  • select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from (
    select snapshot.object_id as object_id
    from origin tablesample system (0.00001)
    inner join origin_visit on origin_visit.origin = origin.id
    ...
    • Jun 18 2018, 7:28 PM
    • 18 Lines
    • SQL
  • select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from origin tablesample system (0.00001)
    inner join origin_visit on origin_visit.origin = origin.id
    inner join snapshot on origin_visit.snapshot_id = snapshot.object_id
    inner join snapshot_branches on snapshot.object_id = snapshot_branches.snapshot_id
    ...
    • Jun 18 2018, 7:21 PM
    • 14 Lines
    • SQL
  • select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from origin
    inner join origin_visit on origin_visit.origin = origin.id
    inner join snapshot on origin_visit.snapshot_id = snapshot.object_id
    inner join snapshot_branches on snapshot.object_id = snapshot_branches.snapshot_id
    ...
    • Jun 18 2018, 7:20 PM
    • 15 Lines
    • SQL
  • select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from snapshot
    inner join snapshot_branches on snapshot.object_id = snapshot_branches.snapshot_id
    inner join snapshot_branch on snapshot_branch.object_id = snapshot_branches.branch_id
    inner join revision on snapshot_branch.target = revision.id
    ...
    • Jun 18 2018, 7:14 PM
    • 13 Lines
    • PostgreSQL
  • select *
    from snapshot tablesample system (0.0002) repeatable (564)
    inner join snapshot_branches on snapshot_branches.snapshot_id = snapshot.object_id
    inner join snapshot_branch on snapshot_branch.object_id = snapshot_branches.branch_id
    • Jun 18 2018, 7:03 PM
    • 4 Lines
    • SQL
  • select entries.target
    from
    (
    select swh_directory_walk(revision.directory)
    from snapshot tablesample system (0.001)
    ...
    • Jun 18 2018, 6:33 PM
    • 11 Lines
    • SQL
  • Info: /Stage[main]/Profile::Prometheus::Node/File[/etc/default/prometheus-node-exporter]: Scheduling refresh of Service[prometheus-node-exporter]
    Notice: /Stage[main]/Profile::Prometheus::Node/Service[prometheus-node-exporter]: Triggered 'refresh' from 2 events
    Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install prometheus-sql-exporter' returned 100: Reading package lists...
    Building dependency tree...
    Reading state information...
    ...
    • Jun 13 2018, 6:10 PM
    • 74 Lines
  • Jun 13 13:56:38 worker11 python3[663]: [2018-06-13 13:56:38,481: INFO/MainProcess] Received task: swh.vault.cooking_tasks.SWHBatchCookingTask[85c43f4b-b97c-4d2f-b6f9-ebbae74d8f7e]
    Jun 13 13:56:40 worker11 python3[682]: [2018-06-13 13:56:40,263: ERROR/Worker-11] Bundle cooking failed.
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/vault/cookers/base.py", line 114, in cook
    self.prepare_bundle()
    ...
    • Jun 13 2018, 3:57 PM
    • 17 Lines
  • Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,767: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[6a8b530a-2ae6-4d01-bc13-b04314896256] succeeded in 11380.885421179235s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,769: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[fd5ef220-830b-4d78-88c7-b5494ca4cd82] succeeded in 11400.2180494708s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,771: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[d87679ec-3ea7-4c5f-be2e-2ab71b9f92d6] succeeded in 11406.673143764958s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,772: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[64d1ce65-bb21-40cf-bd16-89b44faa352c] succeeded in 11318.264325065073s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,773: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
    ...
    • Jun 13 2018, 3:55 PM
    • 59 Lines
  • Jun 13 10:07:28 moma python3[26617]: 2018-06-13 10:07:28 [26617] [ERROR] Internal Server Error: /browse/directory/977fc4b98c0e85816348cebd3b12026407c368b6/
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 42, in inner
    response = get_response(request)
    File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 187, in _get_response
    ...
    • Jun 13 2018, 12:08 PM
    • 15 Lines
  • swh-dedup=# select count(*) from chunk tablesample system (0.0003) left join chunked_content_chunk on chunk_id = chunk.id;
    count
    -------
    4181
    (1 row)
    ...
    • Jun 5 2018, 3:41 PM
    • 15 Lines
  • swh-dedup=# explain
    select
    count(distinct chunk.id) as count_chunk,
    sum(chunk.length) as total_size,
    sum(chunk.compressed_length) as total_compressed_size
    ...
    • May 24 2018, 9:13 PM
    • 25 Lines
    • PostgreSQL
  • On this directory: https://archive.softwareheritage.org/browse/directory/31b5c8cc985d190b5a7ef4878128ebfdc2358f49/
    The readme didn't show up and when i clicked to see the content i got the error below.
    After I backed to the directory view- i have the error text in the readme placeholder (I have a local screenshot of this)
    ----------------------------------------------------------------------------------------------------------
    ...
    • May 23 2018, 3:51 PM
    • 14 Lines
  • swh-dedup=# explain select count(chunk.length)
    from chunk tablesample system(0.001)
    left join chunked_content_chunk on chunk_id = chunk.id;
    QUERY PLAN
    --------------------------------------------------------------------------------------------------
    ...
    • May 7 2018, 6:47 PM
    • 29 Lines
  • Traceback (most recent call last):
    File "/home/morane/.venv/lib/python3.5/site-packages/sphinx/config.py", line 161, in __init__
    execfile_(filename, config)
    File "/home/morane/.venv/lib/python3.5/site-packages/sphinx/util/pycompat.py", line 150, in execfile_
    exec_(code, _globals)
    ...
    • May 4 2018, 4:28 PM
    • 29 Lines
  • report:
    -
    package: sbuild-build-depends-swh-web-dummy
    version: 0.invalid.0
    architecture: amd64
    ...
    • May 4 2018, 3:10 PM
    • 42 Lines
  • python3 -m nose -sv --with-doctest ./swh/vault/tests
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-storage » existe déjà
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-vault » existe déjà
    test_available (test_backend.TestBackend) ... ok
    test_cache_expire_oldest (test_backend.TestBackend) ... ok
    ...
    • Apr 26 2018, 2:44 PM
    • 206 Lines
  • python3 -m nose -sv --with-doctest ./swh/vault/tests
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-vault » existe déjà
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-storage » existe déjà
    pg_restore: [programme d'archivage (db)] Erreur pendant le traitement de la TOC (« PROCESSING TOC ») :
    pg_restore: [programme d'archivage (db)] Erreur à partir de l'entrée TOC 240 ; 1259 2506400 SEQUENCE metadata_provider_id_seq ndandrim
    ...
    • Apr 26 2018, 2:36 PM
    • 415 Lines
  • python3 -m nose -sv --with-doctest ./swh/vault/tests
    createdb: database creation failed: ERROR: database "softwareheritage-test-vault" already exists
    Time: 2.739 ms
    Time: 0.344 ms
    Time: 0.454 ms
    ...
    • Apr 26 2018, 10:32 AM
    • 506 Lines
  • In [51]: c.batch_progress(4)
    Out[51]:
    {'bundles': [{'id': 3,
    'obj_id': '7d4aecffc20478ea6807b9649b25b71e22ebbcb6',
    'obj_type': 'revision_gitfast',
    ...
    • Apr 24 2018, 6:23 PM
    • 22 Lines
    • JSON
  • softwareheritage=> \conninfo
    You are connected to database "softwareheritage" as user "guest" on host "somerset.internal.softwareheritage.org" at port "5433".
    SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
    softwareheritage=> select * from origin where type='deposit' and url like '%hal-01767263%';
    ┌────┬──────┬─────┬────────┬─────────┐
    ...
    • Apr 17 2018, 9:35 AM
    • 23 Lines
  • P248 .psqlrc
    \pset linestyle unicode
    \pset border 2
    \pset null ¤
    \set PROMPT1 '%[%033[1;;40m%]%`date +%H:%M:%S`%[%033[0m%] %[%033[33;1m%]%x%[%033[0m%]%[%033[1m%]%/%[%033[0m%]@%[%033[33;1m%]%m%[%033[0m%]:%[%033[33;1m%]%>%[%033[0m%]%R%# '
    \timing
    ...
    • Apr 13 2018, 2:41 PM
    • 7 Lines
    • PostgreSQL
  • worker09.euwest.azure: Apr 12 07:45:28 worker09 python3[128430]: [2018-04-12 07:45:27,737: ERROR/Worker-65266] Problem when reading contents metadata.
    worker09.euwest.azure: Traceback (most recent call last):
    worker09.euwest.azure: File "/usr/lib/python3/dist-packages/swh/indexer/indexer.py", line 359, in run
    worker09.euwest.azure: raw_content = self.objstorage.get(sha1)
    worker09.euwest.azure: File "/usr/lib/python3/dist-packages/swh/objstorage/multiplexer/multiplexer_objstorage.py", line 137, in get
    ...
    • Apr 12 2018, 9:47 AM
    • 18 Lines
    • Plain Text