Page MenuHomeSoftware Heritage
Paste Active Pastes
  • ✘ dev@desktop5  ~/swh-environment/swh-docs   master  git pull
    Already up-to-date.
    dev@desktop5  ~/swh-environment/swh-docs   master  tox -r -e sphinx-dev
    GLOB sdist-make: /home/dev/swh-environment/swh-docs/setup.py
    sphinx-dev recreate: /home/dev/swh-environment/swh-docs/.tox/sphinx-dev
    ...
    • Thu, Nov 29, 6:29 PM
    • 46 Lines
  • When i have something like:
    ```
    if a:
    r = do_something(a)
    ...
    • Thu, Nov 22, 11:46 AM
    • 21 Lines
  • Hypothesis issue with old version: https://github.com/HypothesisWorks/hypothesis/issues/290
    build output:
    ```
    ============================= test session starts ==============================
    ...
    • Mon, Nov 19, 11:38 AM
    • 98 Lines
  • pytest swh/indexer/tests/storage/test_storage.py -x
    ========================================================================================================= test session starts =========================================================================================================
    platform linux -- Python 3.5.3, pytest-3.9.3, py-1.7.0, pluggy-0.8.0
    hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/dev/swh-environment/swh-indexer/.hypothesis/examples')
    rootdir: /home/dev/swh-environment/swh-indexer, inifile: pytest.ini
    ...
    • Thu, Nov 15, 10:52 AM
    • 99 Lines
  • diff <(grep 'def ' swh/storage/in_memory.py | grep -v 'def _' | sed -e 's/(.*//' | sort) <(grep 'def ' swh/storage/storage.py | grep -v 'def _' | sed -e 's/(.*//' | sort)
    0a1
    > def add_to_objstorage
    3a5
    > def content_get
    ...
    • Thu, Nov 15, 10:07 AM
    • 21 Lines
    • Diff
  • | context | file_name | counted | percentage | percentage | percentage on 3,424,000,000 files |
    |-------------------------------|-------------------|----------------------|------------|------------|-----------------------------------|
    | CodeMeta | CODE | 320 | 0.00% | | 8.85E-06 |
    | haskell | .cabal | 676053 | 1.27% | 0.01% | 0.01870298068 |
    | java- Maven | pom.xml | 15509125 | 29.03% | 0.43% | 0.4290593566 |
    ...
    • Mon, Nov 12, 5:11 PM
    • 33 Lines
  • def _naive_sig(param_names):
    return inspect.Signature([
    inspect.Parameter(name, inspect.Parameter.POSITIONAL_OR_KEYWORD)
    for name in param_names])
    ...
    • Nov 9 2018, 1:07 PM
    • 82 Lines
    • Python
  • def content_mimetype_missing(self, mimetypes, db=None, cur=None):
    """Generates mimetypes missing from storage.
    Args:
    mimetypes (iterable): iterable of dict with keys:
    ...
    • Nov 8 2018, 12:03 PM
    • 13 Lines
    • Python
  • commit edebe6a4a42bae99a1819898b14bb0951cfe6b8b
    Author: Valentin Lorentz <vlorentz@softwareheritage.org>
    Date: Mon Nov 5 14:54:06 2018 +0100
    Remove testrepo.zip.
    ...
    • Nov 5 2018, 2:52 PM
    • 64 Lines
    • Diff
  • GLOB sdist-make: /home/morane/Documents/code/swh-environment/swh-indexer/setup.py
    flake8 recreate: /home/morane/Documents/code/swh-environment/swh-indexer/.tox/flake8
    flake8 installdeps: flake8
    flake8 installed: flake8==3.6.0,mccabe==0.6.1,pkg-resources==0.0.0,pycodestyle==2.4.0,pyflakes==2.0.0
    flake8 runtests: PYTHONHASHSEED='3997505493'
    ...
    • Oct 30 2018, 4:31 PM
    • 3,664 Lines
  • dev@desktop5  ~/swh-environment/swh-indexer   master  git pull
    remote: Counting objects: 14, done.
    remote: Compressing objects: 100% (14/14), done.
    remote: Total 14 (delta 10), reused 0 (delta 0)
    Unpacking objects: 100% (14/14), done.
    ...
    • Oct 29 2018, 10:22 AM
    • 14 Lines
  • If it's urgent to redeploy indexers, here is what i foresee in the
    current state of affairs,
    - After review and acceptance, merge the diffs:
    ...
    • Oct 27 2018, 11:16 AM
    • 58 Lines
  • celery.worker.strategy: INFO: Received task: swh.indexer.tests.test_origin_metadata.test_revision_metadata_task[785b1155-89f2-4aec-ac50-c2d3eb34b4d3]
    celery.app.trace: ERROR: Task swh.indexer.tests.test_origin_metadata.test_revision_metadata_task[785b1155-89f2-4aec-ac50-c2d3eb34b4d3] raised unexpected: EncodeError(TypeError("b'8dbb6aeb036e7fd80664eb8bfd1507881af1ba9f' is not JSON serializable",),)
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
    ...
    • Oct 25 2018, 3:08 PM
    • 81 Lines
  • ======================================================================
    FAIL: test_pipeline (swh.indexer.tests.test_origin_metadata.TestOriginMetadata)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    ...
    • Oct 25 2018, 2:47 PM
    • 26 Lines
  • def test_compute_metadata_codemeta(self):
    """
    test that a codemeta file is not altered with translation
    """
    ...
    • Oct 25 2018, 12:06 PM
    • 50 Lines
  • pre-requisite:
    ```
    arc patch D582
    ```
    ...
    • Oct 24 2018, 11:12 AM
    • 1,748 Lines
  • KeyboardInterrupt
    Exception ignored in: <bound method AsyncResult.__del__ of <AsyncResult: 8731ccc1-5068-41ff-9b83-e0e2ef930843>>
    Traceback (most recent call last):
    File "/home/morane/.local/lib/python3.5/site-packages/celery/result.py", line 385, in __del__
    self.backend.remove_pending_result(self)
    ...
    • Oct 23 2018, 5:01 PM
    • 60 Lines
  • ---
    errors:
    hg:
    - "Failed to uncompress archive"
    - "OSError: [Errno 12] Cannot allocate memory"
    ...
    • Oct 18 2018, 11:13 AM
    • 116 Lines
  • origin-update-pypi;oneshot;["manhattan_seo", "https://pypi.org/project/manhattan_seo/"];{"project_metadata_url": "https://pypi.org/pypi/manhattan_seo/json"}
    origin-update-pypi;oneshot;["hypermax", "https://pypi.org/project/hypermax/"];{"project_metadata_url": "https://pypi.org/pypi/hypermax/json"}
    origin-update-pypi;oneshot;["Flask-Security-Bundle", "https://pypi.org/project/Flask-Security-Bundle/"];{"project_metadata_url": "https://pypi.org/pypi/Flask-Security-Bundle/json"}
    origin-update-pypi;oneshot;["collective.dms.mailcontent", "https://pypi.org/project/collective.dms.mailcontent/"];{"project_metadata_url": "https://pypi.org/pypi/collective.dms.mailcontent/json"}
    origin-update-pypi;oneshot;["certbot-dns-cloudxns", "https://pypi.org/project/certbot-dns-cloudxns/"];{"project_metadata_url": "https://pypi.org/pypi/certbot-dns-cloudxns/json"}
    ...
    • Oct 17 2018, 4:21 PM
    • 1,411 Lines
  • return self.RevisionMetadataTask().apply_async(
    kwargs={
    'ids': [res['revision_id'] for res in results],
    'policy_update': 'update-dups',
    },
    ...
    • Oct 16 2018, 2:25 PM
    • 9 Lines
    • Python
  • jq . pypi.group-output.txt | grep -v 'Reason: 404'
    {
    "googlecode": {
    "total": 3933,
    "errors": {
    ...
    • Oct 16 2018, 1:50 PM
    • 178 Lines
  • ```
    export PYTHONPATH=$SWH_ENVIRONMENT_HOME/snippets/ardumont:$PYTHONPATH
    python3 -m kibana_fetch_logs > output.txt
    cat output.txt | python3 -m group_by_exception --loader-type hg > output-group-by.txt
    ```
    ...
    • Oct 15 2018, 4:22 PM
    • 7 Lines
  • curl -XPOST 'http://esnode3.internal.softwareheritage.org:9200/swh_workers-2018.10.11,swh_workers-2018.10.12,swh_workers-2018.10.13,swh_workers-2018.10.14/_search' -d '{
    "from": 10,
    "_source": [
    "message",
    ...
    • Oct 15 2018, 2:33 PM
    • 36 Lines
  • {
    "swh_workers-2018.09.30": {
    "mappings": {
    "doc": {
    "properties": {
    ...
    • Oct 15 2018, 1:36 PM
    • 617 Lines
  • {
    "order": 0,
    "index_patterns": [
    "swh_workers-*"
    ],
    ...
    • Oct 15 2018, 1:33 PM
    • 16 Lines
  •  tony  (e) .venv   arcpatch-D505_1  …  swh  swh-environment  swh-indexer  2  make test
    python3 -m nose -sv --with-doctest .
    Failure: TypeError (metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases) ... ERROR
    Failure: TypeError (metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases) ... ERROR
    Failure: TypeError (metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases) ... ERROR
    ...
    • Oct 11 2018, 10:46 AM
    • 465 Lines
  • http://kibana0.internal.softwareheritage.org:5601/app/kibana#/dashboard/32632370-c0bd-11e8-8222-07f3ec376cd5?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:%272018-08-31T22:00:00.000Z%27,mode:absolute,to:%272018-10-05T21:59:59.999Z%27))&_a=(description:%27This%20is%20a%20general%20dashboard%20listing%20the%20full%20logs%20of%20the%20swh-workers%27,filters:!((%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:%2720de3150-c0bb-11e8-8222-07f3ec376cd5%27,key:systemd_unit,negate:!f,params:(query:%27swh-worker@swh_loader_pypi.service%27,type:phrase),type:phrase,value:%27swh-worker@swh_loader_pypi.service%27),query:(match:(systemd_unit:(query:%27swh-worker@swh_loader_pypi.service%27,type:phrase)))),(%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:%2720de3150-c0bb-11e8-8222-07f3ec376cd5%27,key:priority,negate:!f,params:(query:%273%27,type:phrase),type:phrase,value:%273%27),query:(match:(priority:(query:%273%27,type:phrase)))),(%27$state%27:(store:appState),meta:(alias:!n...
    • Oct 5 2018, 6:29 PM
    • 1 Line
  • dev@desktop5  ~/swh-environment/swh-indexer   origin-head-indexer  grep content_metadata_add **/*.{py,sql}
    swh/indexer/metadata.py: self.idx_storage.content_metadata_add(
    swh/indexer/storage/__init__.py: def content_metadata_add(self, metadata, conflict_update=False, db=None,
    swh/indexer/storage/__init__.py: db.content_metadata_add_from_temp(conflict_update, cur)
    swh/indexer/storage/api/client.py: def content_metadata_add(self, metadata, conflict_update=False):
    ...
    • Oct 5 2018, 4:22 PM
    • 20 Lines
  • Traceback (most recent call last):
    File "origin_head.py", line 121, in <module>
    main()
    File "/usr/lib/python3/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
    ...
    • Oct 5 2018, 2:51 PM
    • 26 Lines
  • ```
    test_revision_metadata_indexer (swh.indexer.tests.test_metadata.Metadata) ... ERROR
    ...
    ======================================================================
    ...
    • Oct 4 2018, 4:19 PM
    • 28 Lines
  • # swhpass and completion mechanism
    SWH_PASSWORD_STORE_DIR=${HOME}/work/inria/repo/swh/credentials/
    function swhpass() {
    PASSWORD_STORE_DIR=$SWH_PASSWORD_STORE_DIR pass $@
    }
    ...
    • Oct 4 2018, 3:00 PM
    • 12 Lines
  • diff --git a/setup.py b/setup.py
    index 42e154f..93d3192 100644
    --- a/setup.py
    +++ b/setup.py
    @@ -34,6 +34,7 @@ setup(
    ...
    • Oct 1 2018, 2:55 PM
    • 12 Lines
    • Diff
  • http://acsccg.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/acsccg/acsccg-repo.svndump.gz
    http://anarchintosh-projects.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/anarchintosh-projects/anarchintosh-projects-repo.svndump.gz
    http://bastian.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/b/bastian/bastian-repo.svndump.gz
    http://calculapdf.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/c/calculapdf/calculapdf-repo.svndump.gz
    http://dagamers.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/d/dagamers/dagamers-repo.svndump.gz
    ...
    • Sep 28 2018, 4:04 PM
    • 23 Lines
  • http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 240, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 742, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 308, in store_data\n self.last_known_swh_revision)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 496, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 217, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 161, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 266, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 333, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 318, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 742, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 308, in store_data\n self.last_known_swh_revision)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 496, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 217, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 161, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 266, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 333, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 318, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 732, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 308, in store_data\n self.last_known_swh_revision)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 496, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 217, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 161, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 266, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 336, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 321, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 628, in load\n """Detailed visit status.\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 300, in store_data\n Note:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 489, in process_repository\n self.log.info(msg)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 229, in process_swh_revisions\n _id = known_swh_rev.get('id')\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 213, in process_swh_revisions\n revs = []\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 155, in process_svn_revisions\n gen_revs = svnrepo.swh_hash_data_per_revision(\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 453, in compute_hashes\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 411, in replay\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    ...
    • Sep 27 2018, 3:47 PM
    • 81 Lines
  • $ python3 -m venv .venv
    $ source .venv/bin/activate
    $ pip install $( bin/pip-swh-packages )
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-core
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-model
    ...
    • Sep 25 2018, 5:35 PM
    • 68 Lines
  • root@scratch01-euwest ~ # du -sh /srv/hdd/swh-parquet/origin_visit/*
    128M /srv/hdd/swh-parquet/origin_visit/019ed7acd0d64f309dcfb3a977f11480.parquet
    113M /srv/hdd/swh-parquet/origin_visit/05160c9af2bf41379f605259a3e1cb24.parquet
    122M /srv/hdd/swh-parquet/origin_visit/07e9ce57dd544eebb539c376f7c81aaa.parquet
    136M /srv/hdd/swh-parquet/origin_visit/2ccd24604c194508909a3ae7f431d731.parquet
    ...
    • Sep 25 2018, 1:33 PM
    • 36 Lines
  • Table origin_visit...
    0%|▏ | 500000/230786000 [00:10<1:22:12, 46691.30it/s]
    Table origin...
    1%|▍ | 500000/85109300 [00:04<13:35, 103756.07it/s]
    Table occurrence_history...
    ...
    • Sep 20 2018, 2:32 PM
    • 34 Lines
  • Sep 12 14:20:04 worker12 python3[25846]: [2018-09-12 14:20:04,287: ERROR/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[a129a228-79b1-47e1-8c37-70e913c1f544] raised unexpected: ConnectionResetError(104, 'Connection reset by peer')
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 438, in __protected_call__
    ...
    • Sep 19 2018, 1:59 PM
    • 38 Lines
  • begin;
    create or replace function swh_count_total_indexes_size()
    returns bigint
    language plpgsql
    ...
    • Sep 6 2018, 2:43 PM
    • 27 Lines
  • logstash0: Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install logstash=6.3.2' returned 100: Reading package lists...
    logstash0: Building dependency tree...
    logstash0: Reading state information...
    logstash0: E: Version '6.3.2' for 'logstash' was not found
    logstash0: Error: /Stage[main]/Profile::Logstash/Package[logstash]/ensure: change from 1:6.3.2-1 to 6.3.2 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--$
    ...
    • Aug 31 2018, 10:10 AM
    • 31 Lines
  • #+BEGIN_SRC emacs-lisp
    (defun swh-percent (current total)
    (* 100 (/ current (* total 1.0))))
    ...
    • Aug 24 2018, 11:49 AM
    • 25 Lines
  • alpha
    Asuka
    Automiko
    appReviewToSlack
    aio2gis
    ...
    • Aug 22 2018, 4:03 PM
    • 402 Lines
  • allspark
    archiveorg
    audit
    astar
    apicheckr
    ...
    • Aug 22 2018, 2:53 PM
    • 378 Lines
  • {'info': {'author': 'Nathan Harrington',
    'author_email': 'nharrington@wasatchphotonics.com',
    'bugtrack_url': None,
    'classifiers': [],
    'description': 'UNKNOWN',
    ...
    • Aug 2 2018, 3:07 PM
    • 96 Lines
  • {
    "info": {
    "author": "bernardfrk",
    "author_email": "bernard.frk@gmail.com",
    "bugtrack_url": null,
    ...
    • Aug 2 2018, 3:03 PM
    • 95 Lines
  • python3
    Python 3.6.6 (default, Jun 27 2018, 14:44:17)
    [GCC 8.1.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import logging; logging.basicConfig(level=logging.DEBUG); from swh.loader.pypi.tasks import LoadPyPiTsk; LoadPyPiTsk().run('arrow', 'https://pypi.org/pypi/arrow/', project_metadata_url='https://pypi.org/pypi/arrow/json')
    ...
    • Aug 1 2018, 2:29 PM
    • 309 Lines
  • begin;
    -- identify release and snapshots to mark them for cleanup
    create or replace function mark_wrong_release_and_snapshot_for_cleanup()
    returns void
    ...
    • Jul 25 2018, 6:23 PM
    • 180 Lines
    • SQL
  • @istest
    def empty_merge(self):
    """
    """
    full_dict = {
    ...
    • Jul 24 2018, 2:03 PM
    • 146 Lines
  • Makefile.local:30: warning: overriding recipe for target 'test'
    ../Makefile.python:22: warning: ignoring old recipe for target 'test'
    ./swh/manage.py test
    Without authentication, endpoint refuses access with 401 response ... ok
    Without authentication, endpoint refuses access with 401 response ... ok
    ...
    • Jul 24 2018, 1:43 PM
    • 283 Lines
  • ======================================================================
    ERROR: Existing status for update with info should return a 204 response
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/home/morane/Documents/code/swh-environment/swh-deposit/swh/deposit/tests/api/test_deposit_update_status.py", line 80, in update_deposit_status_with_info
    ...
    • Jul 24 2018, 1:41 PM
    • 92 Lines
  • <?xml version="1.0" encoding="utf-8"?>
    <entry xmlns="http://www.w3.org/2005/Atom"
    xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0">
    <title>Je suis GPL</title>
    <client>swh</client>
    ...
    • Jul 23 2018, 2:03 PM
    • 25 Lines
  • This diff is against commit f29e3f49ad973df93bfbf9e543b808fe43003bf9, but
    the commit is nowhere in the working copy. Try to apply it against the
    current working copy state? (9b4eb1e1b43349e4356307e1b5f924c870fc0b25)
    [Y/n] Y
    ...
    • Jul 16 2018, 11:41 AM
    • 25 Lines
  • Setup apt archive
    -----------------
    Merged Build-Depends: dose-distcheck
    Filtered Build-Depends: dose-distcheck
    ...
    • Jul 2 2018, 2:35 PM
    • 78 Lines
  • Jul 02 11:44:49 moma gunicorn3[27165]: [02/Jul/2018 11:44:49] ERROR [django.request:132] Internal Server Error: /1/hal/131/media/
    Jul 02 11:44:49 moma gunicorn3[27165]: Traceback (most recent call last):
    Jul 02 11:44:49 moma gunicorn3[27165]: File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 42, in inner
    Jul 02 11:44:49 moma gunicorn3[27165]: response = get_response(request)
    Jul 02 11:44:49 moma gunicorn3[27165]: File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 187, in _get_response
    ...
    • Jul 2 2018, 1:59 PM
    • 58 Lines
  • elasticsearch | [2018-06-22T14:17:31,321][WARN ][o.e.l.LicenseService ] [-7edhpM]
    elasticsearch | #
    elasticsearch | # LICENSE [EXPIRED] ON [THURSDAY, JUNE 21, 2018]. IF YOU HAVE A NEW LICENSE, PLEASE UPDATE IT.
    elasticsearch | # OTHERWISE, PLEASE REACH OUT TO YOUR SUPPORT CONTACT.
    elasticsearch | #
    ...
    • Jun 22 2018, 4:19 PM
    • 26 Lines
  • # master (somerset)
    ## bootstrap db data replication from master to slave:
    ```
    ...
    • Jun 20 2018, 11:31 AM
    • 142 Lines
  • select distinct content.sha1 from (
    select distinct walk.target as target
    from (
    select distinct snapshot_branch.target as target
    from (
    ...
    • Jun 19 2018, 3:21 PM
    • 25 Lines
    • SQL
  • select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from (
    select snapshot.object_id as object_id
    from origin tablesample system (0.00001)
    inner join origin_visit on origin_visit.origin = origin.id
    ...
    • Jun 18 2018, 7:28 PM
    • 18 Lines
    • SQL
  • select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from origin tablesample system (0.00001)
    inner join origin_visit on origin_visit.origin = origin.id
    inner join snapshot on origin_visit.snapshot_id = snapshot.object_id
    inner join snapshot_branches on snapshot.object_id = snapshot_branches.snapshot_id
    ...
    • Jun 18 2018, 7:21 PM
    • 14 Lines
    • SQL
  • select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from origin
    inner join origin_visit on origin_visit.origin = origin.id
    inner join snapshot on origin_visit.snapshot_id = snapshot.object_id
    inner join snapshot_branches on snapshot.object_id = snapshot_branches.snapshot_id
    ...
    • Jun 18 2018, 7:20 PM
    • 15 Lines
    • SQL
  • select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from snapshot
    inner join snapshot_branches on snapshot.object_id = snapshot_branches.snapshot_id
    inner join snapshot_branch on snapshot_branch.object_id = snapshot_branches.branch_id
    inner join revision on snapshot_branch.target = revision.id
    ...
    • Jun 18 2018, 7:14 PM
    • 13 Lines
    • PostgreSQL
  • select *
    from snapshot tablesample system (0.0002) repeatable (564)
    inner join snapshot_branches on snapshot_branches.snapshot_id = snapshot.object_id
    inner join snapshot_branch on snapshot_branch.object_id = snapshot_branches.branch_id
    • Jun 18 2018, 7:03 PM
    • 4 Lines
    • SQL
  • select entries.target
    from
    (
    select swh_directory_walk(revision.directory)
    from snapshot tablesample system (0.001)
    ...
    • Jun 18 2018, 6:33 PM
    • 11 Lines
    • SQL
  • Info: /Stage[main]/Profile::Prometheus::Node/File[/etc/default/prometheus-node-exporter]: Scheduling refresh of Service[prometheus-node-exporter]
    Notice: /Stage[main]/Profile::Prometheus::Node/Service[prometheus-node-exporter]: Triggered 'refresh' from 2 events
    Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install prometheus-sql-exporter' returned 100: Reading package lists...
    Building dependency tree...
    Reading state information...
    ...
    • Jun 13 2018, 6:10 PM
    • 74 Lines
  • Jun 13 13:56:38 worker11 python3[663]: [2018-06-13 13:56:38,481: INFO/MainProcess] Received task: swh.vault.cooking_tasks.SWHBatchCookingTask[85c43f4b-b97c-4d2f-b6f9-ebbae74d8f7e]
    Jun 13 13:56:40 worker11 python3[682]: [2018-06-13 13:56:40,263: ERROR/Worker-11] Bundle cooking failed.
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/vault/cookers/base.py", line 114, in cook
    self.prepare_bundle()
    ...
    • Jun 13 2018, 3:57 PM
    • 17 Lines
  • Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,767: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[6a8b530a-2ae6-4d01-bc13-b04314896256] succeeded in 11380.885421179235s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,769: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[fd5ef220-830b-4d78-88c7-b5494ca4cd82] succeeded in 11400.2180494708s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,771: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[d87679ec-3ea7-4c5f-be2e-2ab71b9f92d6] succeeded in 11406.673143764958s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,772: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[64d1ce65-bb21-40cf-bd16-89b44faa352c] succeeded in 11318.264325065073s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,773: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
    ...
    • Jun 13 2018, 3:55 PM
    • 59 Lines
  • Jun 13 10:07:28 moma python3[26617]: 2018-06-13 10:07:28 [26617] [ERROR] Internal Server Error: /browse/directory/977fc4b98c0e85816348cebd3b12026407c368b6/
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 42, in inner
    response = get_response(request)
    File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 187, in _get_response
    ...
    • Jun 13 2018, 12:08 PM
    • 15 Lines
  • swh-dedup=# select count(*) from chunk tablesample system (0.0003) left join chunked_content_chunk on chunk_id = chunk.id;
    count
    -------
    4181
    (1 row)
    ...
    • Jun 5 2018, 3:41 PM
    • 15 Lines
  • swh-dedup=# explain
    select
    count(distinct chunk.id) as count_chunk,
    sum(chunk.length) as total_size,
    sum(chunk.compressed_length) as total_compressed_size
    ...
    • May 24 2018, 9:13 PM
    • 25 Lines
    • PostgreSQL
  • On this directory: https://archive.softwareheritage.org/browse/directory/31b5c8cc985d190b5a7ef4878128ebfdc2358f49/
    The readme didn't show up and when i clicked to see the content i got the error below.
    After I backed to the directory view- i have the error text in the readme placeholder (I have a local screenshot of this)
    ----------------------------------------------------------------------------------------------------------
    ...
    • May 23 2018, 3:51 PM
    • 14 Lines
  • swh-dedup=# explain select count(chunk.length)
    from chunk tablesample system(0.001)
    left join chunked_content_chunk on chunk_id = chunk.id;
    QUERY PLAN
    --------------------------------------------------------------------------------------------------
    ...
    • May 7 2018, 6:47 PM
    • 29 Lines
  • Traceback (most recent call last):
    File "/home/morane/.venv/lib/python3.5/site-packages/sphinx/config.py", line 161, in __init__
    execfile_(filename, config)
    File "/home/morane/.venv/lib/python3.5/site-packages/sphinx/util/pycompat.py", line 150, in execfile_
    exec_(code, _globals)
    ...
    • May 4 2018, 4:28 PM
    • 29 Lines
  • report:
    -
    package: sbuild-build-depends-swh-web-dummy
    version: 0.invalid.0
    architecture: amd64
    ...
    • May 4 2018, 3:10 PM
    • 42 Lines
  • python3 -m nose -sv --with-doctest ./swh/vault/tests
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-storage » existe déjà
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-vault » existe déjà
    test_available (test_backend.TestBackend) ... ok
    test_cache_expire_oldest (test_backend.TestBackend) ... ok
    ...
    • Apr 26 2018, 2:44 PM
    • 206 Lines
  • python3 -m nose -sv --with-doctest ./swh/vault/tests
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-vault » existe déjà
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-storage » existe déjà
    pg_restore: [programme d'archivage (db)] Erreur pendant le traitement de la TOC (« PROCESSING TOC ») :
    pg_restore: [programme d'archivage (db)] Erreur à partir de l'entrée TOC 240 ; 1259 2506400 SEQUENCE metadata_provider_id_seq ndandrim
    ...
    • Apr 26 2018, 2:36 PM
    • 415 Lines
  • python3 -m nose -sv --with-doctest ./swh/vault/tests
    createdb: database creation failed: ERROR: database "softwareheritage-test-vault" already exists
    Time: 2.739 ms
    Time: 0.344 ms
    Time: 0.454 ms
    ...
    • Apr 26 2018, 10:32 AM
    • 506 Lines
  • In [51]: c.batch_progress(4)
    Out[51]:
    {'bundles': [{'id': 3,
    'obj_id': '7d4aecffc20478ea6807b9649b25b71e22ebbcb6',
    'obj_type': 'revision_gitfast',
    ...
    • Apr 24 2018, 6:23 PM
    • 22 Lines
    • JSON
  • softwareheritage=> \conninfo
    You are connected to database "softwareheritage" as user "guest" on host "somerset.internal.softwareheritage.org" at port "5433".
    SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
    softwareheritage=> select * from origin where type='deposit' and url like '%hal-01767263%';
    ┌────┬──────┬─────┬────────┬─────────┐
    ...
    • Apr 17 2018, 9:35 AM
    • 23 Lines
  • P248 .psqlrc
    \pset linestyle unicode
    \pset border 2
    \pset null ¤
    \set PROMPT1 '%[%033[1;;40m%]%`date +%H:%M:%S`%[%033[0m%] %[%033[33;1m%]%x%[%033[0m%]%[%033[1m%]%/%[%033[0m%]@%[%033[33;1m%]%m%[%033[0m%]:%[%033[33;1m%]%>%[%033[0m%]%R%# '
    \timing
    ...
    • Apr 13 2018, 2:41 PM
    • 7 Lines
    • PostgreSQL
  • worker09.euwest.azure: Apr 12 07:45:28 worker09 python3[128430]: [2018-04-12 07:45:27,737: ERROR/Worker-65266] Problem when reading contents metadata.
    worker09.euwest.azure: Traceback (most recent call last):
    worker09.euwest.azure: File "/usr/lib/python3/dist-packages/swh/indexer/indexer.py", line 359, in run
    worker09.euwest.azure: raw_content = self.objstorage.get(sha1)
    worker09.euwest.azure: File "/usr/lib/python3/dist-packages/swh/objstorage/multiplexer/multiplexer_objstorage.py", line 137, in get
    ...
    • Apr 12 2018, 9:47 AM
    • 18 Lines
    • Plain Text
  • swh-dedup=# select algo, avg(duration_us) from chunked_content left join chunking_method on chunking_method.id = chunked_content.method_id group by algo;
    algo | avg
    ---------+-----------------------
    buzhash | 976.7281333333333333
    rabin | 1222.3993333333333333
    ...
    • Apr 10 2018, 6:35 PM
    • 6 Lines
  • https://github.com/Vishcn/vishWeb,ded9ec3e8ccc8f84d3a0541ed0c5a4a8606038ea
    https://github.com/YxhWife/firstPoj,19f0650345e0ff2cade69f6a105908ea47d0afe5
    https://github.com/ZergyPoo/VolunteerManager,e7cdde6ddc8e4500648de72cfce3aa6bd28d0a12
    https://github.com/adriancosma88/test-java-ocp,f8381b7ea001fe1db9b55c4fbdd6f7bacae74ef6
    https://github.com/ansell/bio2rdf-helpers,9ea852f3a0c73cab0403509f67d4b4164f1b30b2
    ...
    • Apr 10 2018, 4:15 PM
    • 10 Lines
  • 17:52:45 :
    550 new
    17:54:03 :
    66 done
    ...
    • Mar 29 2018, 5:58 PM
    • 18 Lines
  • +---------------------------------------------------------------------------------------------+
    | UNIBO NETWORK |
    | |
    | |
    | +-----------------------------------------+ +-------------------------------+ |
    ...
    • Mar 28 2018, 1:36 PM
    • 45 Lines
  • begin;
    create type task_record as (
    task_id bigint,
    task_policy task_policy,
    task_status task_status,
    ...
    • Mar 22 2018, 12:07 PM
    • 64 Lines
    • SQL
  • {
    "took": 1,
    "timed_out": false,
    "_shards": {
    "total": 1,
    ...
    • Mar 22 2018, 10:38 AM
    • 43 Lines
    • JSON
  • {
    "order": 0,
    "index_patterns": ["swh-tasks-*"],
    "settings": {
    "index": {
    ...
    • Mar 21 2018, 5:09 PM
    • 48 Lines
    • JSON
  • {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 15:30:21,909: ERROR/Worker-15] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 1001, in store_data\n self.send_batch_revisions(self.get_revisions())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 654, in send_batch_revisions\n send_in_packets(revisions, self.send_revisions, packet_size)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 46, in send_in_packets\n sender(formatted_objects)\n File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f\n return Retrying(*dargs, **dkw).call(f, *args, **kw)\n File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 447, in send_revisions\n self.storage.revision_add(revision_list)\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 80, in revision_add\n return self.post(\'revision/add\', {\'revisions\': revisions})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 62, in post\n return self._decode_response(response)\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 91, in _decode_response\n raise pickle.loads(decode_response(response))\npsycopg2.extensions.TransactionRollbackError: deadlock detected\nDETAIL: Process 6294 waits for ShareLock on transaction 1703510102; blocked by process 6221.\nProcess 6221 waits for ShareLock on transaction 1703510103; blocked by process 6294.\nHINT: See server log for query details.\nCONTEXT: while inserting index tuple (28015479,4) in relation "revision_pkey"\nSQL statement "insert into revision (id, date, date_offset, date_neg_utc_offset, committer_date, committer_date_offset, committer_date_neg_utc_offset, type, directory, message, author, committer, metadata, synthetic)\n select t.id, t.date, t.date_offset, t.date_neg_utc_offset, t.committer_date, t.committer_date_offset, t.committer_date_neg_utc_offset, t.type, t.directory, t.message, a.id, c.id, t.metadata, t.synthetic\n from tmp_revision t\n left join person a on a.fullname = t.author_fullname\n left join person c on c.fullname = t.committer_fullname"\nPL/pgSQL function swh_revision_add() line 5 at SQL statement\n'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 19:26:48,882: ERROR/Worker-54] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 997, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 644, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 41, in send_in_packets\n sender(formatted_objects)\n File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f\n return Retrying(*dargs, **dkw).call(f, *args, **kw)\n File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 397, in send_contents\n self.storage.content_add(content_list)\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 21, in content_add\n return self.post(\'content/add\', {\'content\': content})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 62, in post\n return self._decode_response(response)\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 96, in _decode_response\n response.content,\nswh.core.api.RemoteException: Unexpected status code for API request: 504 (b\'<html>\\r\\n<head><title>504 Gateway Time-out</title></head>\\r\\n<body bgcolor="white">\\r\\n<center><h1>504 Gateway Time-out</h1></center>\\r\\n<hr><center>nginx/1.10.3</center>\\r\\n</body>\\r\\n</html>\\r\\n\')'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 19:29:31,436: ERROR/Worker-84] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 999, in store_data\n self.send_batch_directories(self.get_directories())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 649, in send_batch_directories\n send_in_packets(directories, self.send_directories, packet_size)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 329, in get_directories\n for _, _, new_dirs in self.load_directories():\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 314, in load_directories\n perms_code\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/objects.py", line 157, in add_blob\n node[fbase] = SimpleBlob(file_hash, is_symlink, file_perms)\nTypeError: \'SimpleBlob\' object does not support item assignment'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 20:25:25,808: ERROR/Worker-76] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 999, in store_data\n self.send_batch_directories(self.get_directories())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 649, in send_batch_directories\n send_in_packets(directories, self.send_directories, packet_size)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 329, in get_directories\n for _, _, new_dirs in self.load_directories():\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 307, in load_directories\n tree = tree.remove_tree_node_for_path(path)\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/objects.py", line 124, in remove_tree_node_for_path\n node[first] = node[first].remove_tree_node_for_path(rest)\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/objects.py", line 124, in remove_tree_node_for_path\n node[first] = node[first].remove_tree_node_for_path(rest)\nKeyError: b\'\''}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 22:38:50,016: ERROR/Worker-128] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen\n chunked=chunked)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File "/usr/lib/python3.5/http/client.py", line 1107, in request\n self._send_request(method, url, body, headers)\n File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request\n self.endheaders(body)\n File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders\n self._send_output(message_body)\n File "/usr/lib/python3.5/http/client.py", line 936, in _send_output\n self.send(message_body)\n File "/usr/lib/python3.5/http/client.py", line 908, in send\n self.sock.sendall(data)\nBrokenPipeError: [Errno 32] Broken pipe\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/requests/adapters.py", line 423, in send\n timeout=timeout\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen\n _stacktrace=sys.exc_info()[2])\n File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 334, in increment\n raise six.reraise(type(error), error, _stacktrace)\n File "/usr/lib/python3/dist-packages/six.py", line 685, in reraise\n raise value.with_traceback(tb)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen\n chunked=chunked)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File "/usr/lib/python3.5/http/client.py", line 1107, in request\n self._send_request(method, url, body, headers)\n File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request\n self.endheaders(body)\n File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders\n self._send_output(message_body)\n File "/usr/lib/python3.5/http/client.py", line 936, in _send_output\n self.send(message_body)\n File "/usr/lib/python3.5/http/client.py", line 908, in send\n self.sock.sendall(data)\nrequests.packages.urllib3.exceptions.ProtocolError: (\'Connection aborted.\', BrokenPipeError(32, \'Broken pipe\'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 42, in raw_post\n **opts\n File "/usr/lib/python3/dist-packages/requests/sessions.py", line 535, in post\n return self.request(\'POST\', url, data=data, json=json, **kwargs)\n File "/usr/lib/python3/dist-packages/requests/sessions.py", line 488, in request\n resp = self.send(prep, **send_kwargs)\n File "/usr/lib/python3/dist-packages/requests/sessions.py", line 609, in send\n r = adapter.send(request, **kwargs)\n File "/usr/lib/python3/dist-packages/requests/adapters.py", line 473, in send\n raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: (\'Connection aborted.\', BrokenPipeError(32, \'Broken pipe\'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 997, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 644, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 41, in send_in_packets\n sender(formatted_objects)\n File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f\n return Retrying(*dargs, **dkw).call(f, *args, **kw)\n File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 397, in send_contents\n self.storage.content_add(content_list)\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 21, in content_add\n return self.post(\'content/add\', {\'content\': content})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 61, in post\n headers={\'content-type\': \'application/x-msgpack\'})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 45, in raw_post\n raise self.api_exception(e)\nswh.storage.exc.StorageAPIError: An unexpected error occurred in the api backend: (\'Connection aborted.\', BrokenPipeError(32, \'Broken pipe\'))'}
    ...
    • Mar 21 2018, 2:45 PM
    • 56 Lines
  • I usually do something like this:
    ```
    make -C $SWH_ENVIRONMENT_HOME rebuild-testdata
    ...
    • Mar 14 2018, 12:19 PM
    • 65 Lines
  • # storage to read sha1's metadata (path)
    # storage:
    # cls: local
    # args:
    # db: "service=swh-dev"
    ...
    • Mar 14 2018, 10:39 AM
    • 36 Lines
    • YAML
  • # storage to read sha1's metadata (path)
    # storage:
    # cls: local
    # args:
    # db: "service=swh-dev"
    ...
    • Mar 14 2018, 10:37 AM
    • 32 Lines
    • YAML
  • indexers:
    # language:
    # batch_size: 10
    # check_presence: false
    fossology_license:
    ...
    • Mar 14 2018, 10:36 AM
    • 10 Lines
    • YAML
  • indexers:
    mimetype:
    check_presence: false
    batch_size: 100
    • Mar 14 2018, 10:36 AM
    • 4 Lines
    • YAML
  • storage:
    cls: remote
    args:
    url: http://localhost:5002/
    ...
    • Mar 14 2018, 10:12 AM
    • 25 Lines
    • YAML
  • #!/usr/bin/env bash
    psql service=swh-dev -c "copy (select sha1 from content) to stdin" | sed -e 's/^\\\\x//g'
    • Mar 14 2018, 10:09 AM
    • 3 Lines
    • Bash Scripting
  • Mar 08 00:39:41 saatchi python3[16096]: Traceback (most recent call last):
    Mar 08 00:39:41 saatchi python3[16096]: File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    Mar 08 00:39:41 saatchi python3[16096]: "__main__", mod_spec)
    Mar 08 00:39:41 saatchi python3[16096]: File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    Mar 08 00:39:41 saatchi python3[16096]: exec(code, run_globals)
    ...
    • Mar 8 2018, 9:52 AM
    • 76 Lines
  • CREATE DOMAIN sha1 AS bytea CHECK (length(value) = 20);
    CREATE TABLE content (
    id sha1 PRIMARY KEY, -- SHA1 checksum
    length integer
    ...
    • Feb 28 2018, 7:18 PM
    • 32 Lines
    • SQL
  • CREATE DOMAIN sha1 AS bytea CHECK (length(value) = 20);
    CREATE TABLE content (
    id sha1 PRIMARY KEY, -- SHA1 checksum
    length integer
    ...
    • Feb 28 2018, 7:14 PM
    • 32 Lines
    • SQL
  • $ python3
    Python 3.6.4 (default, Jan 5 2018, 02:13:53)
    [GCC 7.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> archive_name = 'latest/jackalopetears-wifi-propeller-load-source-archive.zip' # worker lost -> big archive
    ...
    • Feb 19 2018, 6:02 PM
    • 66 Lines
  • diff --git a/bin/deploy-on b/bin/deploy-on
    index 46d18e0..8e86207 100755
    --- a/bin/deploy-on
    +++ b/bin/deploy-on
    @@ -29,12 +29,16 @@ die_usage () {
    ...
    • Feb 16 2018, 2:38 PM
    • 33 Lines
    • Diff