Page MenuHomeSoftware Heritage
Create Paste
Paste Active Pastes

Active Pastes
Use Results
Edit Query

  • P317 pypi errors group by end of exception message (~50 characters from the end).
    jq . pypi.group-output.txt | grep -v 'Reason: 404'
    {
    "googlecode": {
    "total": 3933,
    "errors": {
    ...
    • Oct 16 2018, 1:50 PM
    • 178 Lines
    Author: ardumont
  • P316 fetch logs and compute group by end of exception message to have a sensible output of error occurrences
    ```
    export PYTHONPATH=$SWH_ENVIRONMENT_HOME/snippets/ardumont:$PYTHONPATH
    python3 -m kibana_fetch_logs > output.txt
    cat output.txt | python3 -m group_by_exception --loader-type hg > output-group-by.txt
    ```
    ...
    • Oct 15 2018, 4:22 PM
    • 7 Lines
    Author: ardumont
  • P315 querying mercurial loader errors
    curl -XPOST 'http://esnode3.internal.softwareheritage.org:9200/swh_workers-2018.10.11,swh_workers-2018.10.12,swh_workers-2018.10.13,swh_workers-2018.10.14/_search' -d '{
    "from": 10,
    "_source": [
    "message",
    ...
    • Oct 15 2018, 2:33 PM
    • 36 Lines
    Author: ardumont
  • P314 elasticsearch: indices: swh_workers-2018.09.30 mapping
    {
    "swh_workers-2018.09.30": {
    "mappings": {
    "doc": {
    "properties": {
    ...
    • Oct 15 2018, 1:36 PM
    • 617 Lines
    Author: ardumont
  • P313 elasticsearch: indices: template template_swh_workers
    {
    "order": 0,
    "index_patterns": [
    "swh_workers-*"
    ],
    ...
    • Oct 15 2018, 1:33 PM
    • 16 Lines
    Author: ardumont
  • P312 D505/507 test errors
     tony  (e) .venv   arcpatch-D505_1  …  swh  swh-environment  swh-indexer  2  make test
    python3 -m nose -sv --with-doctest .
    Failure: TypeError (metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases) ... ERROR
    Failure: TypeError (metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases) ... ERROR
    Failure: TypeError (metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases) ... ERROR
    ...
    • Oct 11 2018, 10:46 AM
    • 465 Lines
    Author: ardumont
  • P311 loader-pypi errors on first round
    http://kibana0.internal.softwareheritage.org:5601/app/kibana#/dashboard/32632370-c0bd-11e8-8222-07f3ec376cd5?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:%272018-08-31T22:00:00.000Z%27,mode:absolute,to:%272018-10-05T21:59:59.999Z%27))&_a=(description:%27This%20is%20a%20general%20dashboard%20listing%20the%20full%20logs%20of%20the%20swh-workers%27,filters:!((%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:%2720de3150-c0bb-11e8-8222-07f3ec376cd5%27,key:systemd_unit,negate:!f,params:(query:%27swh-worker@swh_loader_pypi.service%27,type:phrase),type:phrase,value:%27swh-worker@swh_loader_pypi.service%27),query:(match:(systemd_unit:(query:%27swh-worker@swh_loader_pypi.service%27,type:phrase)))),(%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:%2720de3150-c0bb-11e8-8222-07f3ec376cd5%27,key:priority,negate:!f,params:(query:%273%27,type:phrase),type:phrase,value:%273%27),query:(match:(priority:(query:%273%27,type:phrase)))),(%27$state%27:(store:appState),meta:(alias:!n...
    • Oct 5 2018, 6:29 PM
    • 1 Line
    Author: ardumont
  • P310 (An Untitled Masterwork)
    dev@desktop5  ~/swh-environment/swh-indexer   origin-head-indexer  grep content_metadata_add **/*.{py,sql}
    swh/indexer/metadata.py: self.idx_storage.content_metadata_add(
    swh/indexer/storage/__init__.py: def content_metadata_add(self, metadata, conflict_update=False, db=None,
    swh/indexer/storage/__init__.py: db.content_metadata_add_from_temp(conflict_update, cur)
    swh/indexer/storage/api/client.py: def content_metadata_add(self, metadata, conflict_update=False):
    ...
    • Oct 5 2018, 4:22 PM
    • 20 Lines
    Author: vlorentz
  • P309 OriginHead click error
    Traceback (most recent call last):
    File "origin_head.py", line 121, in <module>
    main()
    File "/usr/lib/python3/dist-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
    ...
    • Oct 5 2018, 2:51 PM
    • 26 Lines
    Author: moranegg
  • P308 indexer: make test ko
    ```
    test_revision_metadata_indexer (swh.indexer.tests.test_metadata.Metadata) ... ERROR
    ...
    ======================================================================
    ...
    • Oct 4 2018, 4:19 PM
    • 28 Lines
    Author: ardumont
  • P307 swhpass alias
    # swhpass and completion mechanism
    SWH_PASSWORD_STORE_DIR=${HOME}/work/inria/repo/swh/credentials/
    function swhpass() {
    PASSWORD_STORE_DIR=$SWH_PASSWORD_STORE_DIR pass $@
    }
    ...
    • Oct 4 2018, 3:00 PM
    • 12 Lines
    Author: ardumont
  • P305 (An Untitled Masterwork)
    diff --git a/setup.py b/setup.py
    index 42e154f..93d3192 100644
    --- a/setup.py
    +++ b/setup.py
    @@ -34,6 +34,7 @@ setup(
    ...
    • Oct 1 2018, 2:55 PM
    • 12 Lines
    • Diff
    Author: seirl
  • P304 mixed crlf origin in error
    http://acsccg.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/acsccg/acsccg-repo.svndump.gz
    http://anarchintosh-projects.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/a/anarchintosh-projects/anarchintosh-projects-repo.svndump.gz
    http://bastian.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/b/bastian/bastian-repo.svndump.gz
    http://calculapdf.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/c/calculapdf/calculapdf-repo.svndump.gz
    http://dagamers.googlecode.com/svn/ /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/d/dagamers/dagamers-repo.svndump.gz
    ...
    • Sep 28 2018, 4:04 PM
    • 23 Lines
    Author: ardumont
  • P303 svn loader: all time origins in error with UnicodeDecodeError - 81 (with duplicates)
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 240, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 742, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 308, in store_data\n self.last_known_swh_revision)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 496, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 217, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 161, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 266, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 333, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 318, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 742, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 308, in store_data\n self.last_known_swh_revision)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 496, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 217, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 161, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 266, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 333, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 318, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 732, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 308, in store_data\n self.last_known_swh_revision)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 496, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 217, in process_swh_revisions\n self.config['revision_packet_size']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 161, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 266, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 336, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 321, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    http://9i00.googlecode.com/svn/ f Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 628, in load\n """Detailed visit status.\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 300, in store_data\n Note:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 489, in process_repository\n self.log.info(msg)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 229, in process_swh_revisions\n _id = known_swh_rev.get('id')\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 213, in process_swh_revisions\n revs = []\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 155, in process_svn_revisions\n gen_revs = svnrepo.swh_hash_data_per_revision(\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 453, in compute_hashes\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 411, in replay\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte\n
    ...
    • Sep 27 2018, 3:47 PM
    • 81 Lines
    Author: ardumont
  • P302 Problem during venv install without libsystemd-dev
    $ python3 -m venv .venv
    $ source .venv/bin/activate
    $ pip install $( bin/pip-swh-packages )
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-core
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-model
    ...
    • Sep 25 2018, 5:35 PM
    • 68 Lines
    Author: ardumont
  • P301 (An Untitled Masterwork)
    root@scratch01-euwest ~ # du -sh /srv/hdd/swh-parquet/origin_visit/*
    128M /srv/hdd/swh-parquet/origin_visit/019ed7acd0d64f309dcfb3a977f11480.parquet
    113M /srv/hdd/swh-parquet/origin_visit/05160c9af2bf41379f605259a3e1cb24.parquet
    122M /srv/hdd/swh-parquet/origin_visit/07e9ce57dd544eebb539c376f7c81aaa.parquet
    136M /srv/hdd/swh-parquet/origin_visit/2ccd24604c194508909a3ae7f431d731.parquet
    ...
    • Sep 25 2018, 1:33 PM
    • 36 Lines
    Author: seirl
  • P300 (An Untitled Masterwork)
    Table origin_visit...
    0%|▏ | 500000/230786000 [00:10<1:22:12, 46691.30it/s]
    Table origin...
    1%|▍ | 500000/85109300 [00:04<13:35, 103756.07it/s]
    Table occurrence_history...
    ...
    • Sep 20 2018, 2:32 PM
    • 34 Lines
    Author: seirl
  • P299 (An Untitled Masterwork)
    Sep 12 14:20:04 worker12 python3[25846]: [2018-09-12 14:20:04,287: ERROR/MainProcess] Task swh.vault.cooking_tasks.SWHCookingTask[a129a228-79b1-47e1-8c37-70e913c1f544] raised unexpected: ConnectionResetError(104, 'Connection reset by peer')
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 438, in __protected_call__
    ...
    • Sep 19 2018, 1:59 PM
    • 38 Lines
    Author: seirl
  • P296 count total size of indexes of the db
    begin;
    create or replace function swh_count_total_indexes_size()
    returns bigint
    language plpgsql
    ...
    • Sep 6 2018, 2:43 PM
    • 27 Lines
    Author: ardumont
  • P293 logstash0: puppet agent --test strangeness
    logstash0: Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install logstash=6.3.2' returned 100: Reading package lists...
    logstash0: Building dependency tree...
    logstash0: Reading state information...
    logstash0: E: Version '6.3.2' for 'logstash' was not found
    logstash0: Error: /Stage[main]/Profile::Logstash/Package[logstash]/ensure: change from 1:6.3.2-1 to 6.3.2 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--$
    ...
    • Aug 31 2018, 10:10 AM
    • 31 Lines
    Author: ardumont
  • P292 estimation of gitlab ingestion remaining days
    #+BEGIN_SRC emacs-lisp
    (defun swh-percent (current total)
    (* 100 (/ current (* total 1.0))))
    ...
    • Aug 24 2018, 11:49 AM
    • 25 Lines
    Author: ardumont
  • P291 empty pypi project release (402 out of 6077, 6.62%)
    alpha
    Asuka
    Automiko
    appReviewToSlack
    aio2gis
    ...
    • Aug 22 2018, 4:03 PM
    • 402 Lines
    Author: ardumont
  • P290 pypi project with no sdist (378 out of 6077, 6.22 %)
    allspark
    archiveorg
    audit
    astar
    apicheckr
    ...
    • Aug 22 2018, 2:53 PM
    • 378 Lines
    Author: ardumont
  • P289 pypi sample 2
    {'info': {'author': 'Nathan Harrington',
    'author_email': 'nharrington@wasatchphotonics.com',
    'bugtrack_url': None,
    'classifiers': [],
    'description': 'UNKNOWN',
    ...
    • Aug 2 2018, 3:07 PM
    • 96 Lines
    Author: ardumont
  • P288 pypi metadata sample
    {
    "info": {
    "author": "bernardfrk",
    "author_email": "bernard.frk@gmail.com",
    "bugtrack_url": null,
    ...
    • Aug 2 2018, 3:03 PM
    • 95 Lines
    Author: ardumont
  • P287 loader pypi taking form - wip
    python3
    Python 3.6.6 (default, Jun 27 2018, 14:44:17)
    [GCC 8.1.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import logging; logging.basicConfig(level=logging.DEBUG); from swh.loader.pypi.tasks import LoadPyPiTsk; LoadPyPiTsk().run('arrow', 'https://pypi.org/pypi/arrow/', project_metadata_url='https://pypi.org/pypi/arrow/json')
    ...
    • Aug 1 2018, 2:29 PM
    • 309 Lines
    Author: ardumont
  • P286 mercurial loading release issue - Cleanup release/snapshot/origin_visits procedure
    begin;
    -- identify release and snapshots to mark them for cleanup
    create or replace function mark_wrong_release_and_snapshot_for_cleanup()
    returns void
    ...
    • Jul 25 2018, 6:23 PM
    • 180 Lines
    • SQL
    Author: ardumont
  • P285 D400 added tests
    @istest
    def empty_merge(self):
    """
    """
    full_dict = {
    ...
    • Jul 24 2018, 2:03 PM
    • 146 Lines
    Author: moranegg
  • P284 D400 output-text
    Makefile.local:30: warning: overriding recipe for target 'test'
    ../Makefile.python:22: warning: ignoring old recipe for target 'test'
    ./swh/manage.py test
    Without authentication, endpoint refuses access with 401 response ... ok
    Without authentication, endpoint refuses access with 401 response ... ok
    ...
    • Jul 24 2018, 1:43 PM
    • 283 Lines
    Author: moranegg
  • P283 D400 errors
    ======================================================================
    ERROR: Existing status for update with info should return a 204 response
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/home/morane/Documents/code/swh-environment/swh-deposit/swh/deposit/tests/api/test_deposit_update_status.py", line 80, in update_deposit_status_with_info
    ...
    • Jul 24 2018, 1:41 PM
    • 92 Lines
    Author: moranegg
  • P282 Je-suis-gpl metadata
    <?xml version="1.0" encoding="utf-8"?>
    <entry xmlns="http://www.w3.org/2005/Atom"
    xmlns:codemeta="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0">
    <title>Je suis GPL</title>
    <client>swh</client>
    ...
    • Jul 23 2018, 2:03 PM
    • 25 Lines
    Author: moranegg
  • P281 Arc patch D380 resulting in an error
    This diff is against commit f29e3f49ad973df93bfbf9e543b808fe43003bf9, but
    the commit is nowhere in the working copy. Try to apply it against the
    current working copy state? (9b4eb1e1b43349e4356307e1b5f924c870fc0b25)
    [Y/n] Y
    ...
    • Jul 16 2018, 11:41 AM
    • 25 Lines
    Author: moranegg
  • P280 python3-swh.core build issue due to python3-psycopg2 deps
    Setup apt archive
    -----------------
    Merged Build-Depends: dose-distcheck
    Filtered Build-Depends: dose-distcheck
    ...
    • Jul 2 2018, 2:35 PM
    • 78 Lines
    Author: ardumont
  • P279 deposit error 500
    Jul 02 11:44:49 moma gunicorn3[27165]: [02/Jul/2018 11:44:49] ERROR [django.request:132] Internal Server Error: /1/hal/131/media/
    Jul 02 11:44:49 moma gunicorn3[27165]: Traceback (most recent call last):
    Jul 02 11:44:49 moma gunicorn3[27165]: File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 42, in inner
    Jul 02 11:44:49 moma gunicorn3[27165]: response = get_response(request)
    Jul 02 11:44:49 moma gunicorn3[27165]: File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 187, in _get_response
    ...
    • Jul 2 2018, 1:59 PM
    • 58 Lines
    Author: olasd
  • P277 kibana logs warn
    elasticsearch | [2018-06-22T14:17:31,321][WARN ][o.e.l.LicenseService ] [-7edhpM]
    elasticsearch | #
    elasticsearch | # LICENSE [EXPIRED] ON [THURSDAY, JUNE 21, 2018]. IF YOU HAVE A NEW LICENSE, PLEASE UPDATE IT.
    elasticsearch | # OTHERWISE, PLEASE REACH OUT TO YOUR SUPPORT CONTACT.
    elasticsearch | #
    ...
    • Jun 22 2018, 4:19 PM
    • 26 Lines
    Author: ardumont
  • P276 replication note (somerset - swh-indexer -> dbreplica1 - swh-indexer)
    # master (somerset)
    ## bootstrap db data replication from master to slave:
    ```
    ...
    • Jun 20 2018, 11:31 AM
    • 142 Lines
    Author: ardumont
  • P275 (An Untitled Masterwork)
    select distinct content.sha1 from (
    select distinct walk.target as target
    from (
    select distinct snapshot_branch.target as target
    from (
    ...
    • Jun 19 2018, 3:21 PM
    • 25 Lines
    • SQL
    Author: seirl
  • P274 (An Untitled Masterwork)
    select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from (
    select snapshot.object_id as object_id
    from origin tablesample system (0.00001)
    inner join origin_visit on origin_visit.origin = origin.id
    ...
    • Jun 18 2018, 7:28 PM
    • 18 Lines
    • SQL
    Author: seirl
  • P273 (An Untitled Masterwork)
    select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from origin tablesample system (0.00001)
    inner join origin_visit on origin_visit.origin = origin.id
    inner join snapshot on origin_visit.snapshot_id = snapshot.object_id
    inner join snapshot_branches on snapshot.object_id = snapshot_branches.snapshot_id
    ...
    • Jun 18 2018, 7:21 PM
    • 14 Lines
    • SQL
    Author: seirl
  • P272 (An Untitled Masterwork)
    select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from origin
    inner join origin_visit on origin_visit.origin = origin.id
    inner join snapshot on origin_visit.snapshot_id = snapshot.object_id
    inner join snapshot_branches on snapshot.object_id = snapshot_branches.snapshot_id
    ...
    • Jun 18 2018, 7:20 PM
    • 15 Lines
    • SQL
    Author: seirl
  • P271 (An Untitled Masterwork)
    select snapshot.id as snapshot_id, snapshot_branch.name as branch_name, content.sha1, walk.*
    from snapshot
    inner join snapshot_branches on snapshot.object_id = snapshot_branches.snapshot_id
    inner join snapshot_branch on snapshot_branch.object_id = snapshot_branches.branch_id
    inner join revision on snapshot_branch.target = revision.id
    ...
    • Jun 18 2018, 7:14 PM
    • 13 Lines
    • PostgreSQL
    Author: olasd
  • P270 (An Untitled Masterwork)
    select *
    from snapshot tablesample system (0.0002) repeatable (564)
    inner join snapshot_branches on snapshot_branches.snapshot_id = snapshot.object_id
    inner join snapshot_branch on snapshot_branch.object_id = snapshot_branches.branch_id
    • Jun 18 2018, 7:03 PM
    • 4 Lines
    • SQL
    Author: seirl
  • P269 (An Untitled Masterwork)
    select entries.target
    from
    (
    select swh_directory_walk(revision.directory)
    from snapshot tablesample system (0.001)
    ...
    • Jun 18 2018, 6:33 PM
    • 11 Lines
    • SQL
    Author: seirl
  • P267 first puppet run not enough
    Info: /Stage[main]/Profile::Prometheus::Node/File[/etc/default/prometheus-node-exporter]: Scheduling refresh of Service[prometheus-node-exporter]
    Notice: /Stage[main]/Profile::Prometheus::Node/Service[prometheus-node-exporter]: Triggered 'refresh' from 2 events
    Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install prometheus-sql-exporter' returned 100: Reading package lists...
    Building dependency tree...
    Reading state information...
    ...
    • Jun 13 2018, 6:10 PM
    • 74 Lines
    Author: ardumont
  • P266 (An Untitled Masterwork)
    Jun 13 13:56:38 worker11 python3[663]: [2018-06-13 13:56:38,481: INFO/MainProcess] Received task: swh.vault.cooking_tasks.SWHBatchCookingTask[85c43f4b-b97c-4d2f-b6f9-ebbae74d8f7e]
    Jun 13 13:56:40 worker11 python3[682]: [2018-06-13 13:56:40,263: ERROR/Worker-11] Bundle cooking failed.
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/vault/cookers/base.py", line 114, in cook
    self.prepare_bundle()
    ...
    • Jun 13 2018, 3:57 PM
    • 17 Lines
    Author: seirl
  • P265 (An Untitled Masterwork)
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,767: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[6a8b530a-2ae6-4d01-bc13-b04314896256] succeeded in 11380.885421179235s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,769: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[fd5ef220-830b-4d78-88c7-b5494ca4cd82] succeeded in 11400.2180494708s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,771: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[d87679ec-3ea7-4c5f-be2e-2ab71b9f92d6] succeeded in 11406.673143764958s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,772: INFO/MainProcess] Task swh.vault.cooking_tasks.SWHBatchCookingTask[64d1ce65-bb21-40cf-bd16-89b44faa352c] succeeded in 11318.264325065073s: None
    Jun 10 06:17:36 worker11 python3[57236]: [2018-06-10 06:17:36,773: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
    ...
    • Jun 13 2018, 3:55 PM
    • 59 Lines
    Author: seirl
  • P264 (An Untitled Masterwork)
    Jun 13 10:07:28 moma python3[26617]: 2018-06-13 10:07:28 [26617] [ERROR] Internal Server Error: /browse/directory/977fc4b98c0e85816348cebd3b12026407c368b6/
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 42, in inner
    response = get_response(request)
    File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 187, in _get_response
    ...
    • Jun 13 2018, 12:08 PM
    • 15 Lines
    Author: olasd
  • P260 (An Untitled Masterwork)
    swh-dedup=# select count(*) from chunk tablesample system (0.0003) left join chunked_content_chunk on chunk_id = chunk.id;
    count
    -------
    4181
    (1 row)
    ...
    • Jun 5 2018, 3:41 PM
    • 15 Lines
    Author: seirl
  • P258 swh-dedup performance stats query planner
    swh-dedup=# explain
    select
    count(distinct chunk.id) as count_chunk,
    sum(chunk.length) as total_size,
    sum(chunk.compressed_length) as total_compressed_size
    ...
    • May 24 2018, 9:13 PM
    • 25 Lines
    • PostgreSQL
    Author: seirl
  • P257 Strange Web-UI error for not found readme.md
    On this directory: https://archive.softwareheritage.org/browse/directory/31b5c8cc985d190b5a7ef4878128ebfdc2358f49/
    The readme didn't show up and when i clicked to see the content i got the error below.
    After I backed to the directory view- i have the error text in the readme placeholder (I have a local screenshot of this)
    ----------------------------------------------------------------------------------------------------------
    ...
    • May 23 2018, 3:51 PM
    • 14 Lines
    Author: moranegg
  • P256 (An Untitled Masterwork)
    swh-dedup=# explain select count(chunk.length)
    from chunk tablesample system(0.001)
    left join chunked_content_chunk on chunk_id = chunk.id;
    QUERY PLAN
    --------------------------------------------------------------------------------------------------
    ...
    • May 7 2018, 6:47 PM
    • 29 Lines
    Author: seirl
  • P255 Sphinx error
    Traceback (most recent call last):
    File "/home/morane/.venv/lib/python3.5/site-packages/sphinx/config.py", line 161, in __init__
    execfile_(filename, config)
    File "/home/morane/.venv/lib/python3.5/site-packages/sphinx/util/pycompat.py", line 150, in execfile_
    exec_(code, _globals)
    ...
    • May 4 2018, 4:28 PM
    • 29 Lines
    Author: moranegg
  • P254 Broken packaging for unstable
    report:
    -
    package: sbuild-build-depends-swh-web-dummy
    version: 0.invalid.0
    architecture: amd64
    ...
    • May 4 2018, 3:10 PM
    • 42 Lines
    Author: anlambert
  • P253 swh-vault test results anlambert psql 10.3
    python3 -m nose -sv --with-doctest ./swh/vault/tests
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-storage » existe déjà
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-vault » existe déjà
    test_available (test_backend.TestBackend) ... ok
    test_cache_expire_oldest (test_backend.TestBackend) ... ok
    ...
    • Apr 26 2018, 2:44 PM
    • 206 Lines
    Author: anlambert
  • P252 swh-vault test results anlambert
    python3 -m nose -sv --with-doctest ./swh/vault/tests
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-vault » existe déjà
    createdb : la création de la base de données a échoué : ERREUR: la base de données « softwareheritage-test-storage » existe déjà
    pg_restore: [programme d'archivage (db)] Erreur pendant le traitement de la TOC (« PROCESSING TOC ») :
    pg_restore: [programme d'archivage (db)] Erreur à partir de l'entrée TOC 240 ; 1259 2506400 SEQUENCE metadata_provider_id_seq ndandrim
    ...
    • Apr 26 2018, 2:36 PM
    • 415 Lines
    Author: anlambert
  • P251 [fixed] swh-vault: traceback `make test` not working locally
    python3 -m nose -sv --with-doctest ./swh/vault/tests
    createdb: database creation failed: ERROR: database "softwareheritage-test-vault" already exists
    Time: 2.739 ms
    Time: 0.344 ms
    Time: 0.454 ms
    ...
    • Apr 26 2018, 10:32 AM
    • 506 Lines
    Author: ardumont
  • P250 (An Untitled Masterwork)
    In [51]: c.batch_progress(4)
    Out[51]:
    {'bundles': [{'id': 3,
    'obj_id': '7d4aecffc20478ea6807b9649b25b71e22ebbcb6',
    'obj_type': 'revision_gitfast',
    ...
    • Apr 24 2018, 6:23 PM
    • 22 Lines
    • JSON
    Author: seirl
  • P249 replication issue?
    softwareheritage=> \conninfo
    You are connected to database "softwareheritage" as user "guest" on host "somerset.internal.softwareheritage.org" at port "5433".
    SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
    softwareheritage=> select * from origin where type='deposit' and url like '%hal-01767263%';
    ┌────┬──────┬─────┬────────┬─────────┐
    ...
    • Apr 17 2018, 9:35 AM
    • 23 Lines
    Author: ardumont
  • P248 .psqlrc
    \pset linestyle unicode
    \pset border 2
    \pset null ¤
    \set PROMPT1 '%[%033[1;;40m%]%`date +%H:%M:%S`%[%033[0m%] %[%033[33;1m%]%x%[%033[0m%]%[%033[1m%]%/%[%033[0m%]@%[%033[33;1m%]%m%[%033[0m%]:%[%033[33;1m%]%>%[%033[0m%]%R%# '
    \timing
    ...
    • Apr 13 2018, 2:41 PM
    • 7 Lines
    • PostgreSQL
    Author: olasd
  • P247 mimetype indexer error
    worker09.euwest.azure: Apr 12 07:45:28 worker09 python3[128430]: [2018-04-12 07:45:27,737: ERROR/Worker-65266] Problem when reading contents metadata.
    worker09.euwest.azure: Traceback (most recent call last):
    worker09.euwest.azure: File "/usr/lib/python3/dist-packages/swh/indexer/indexer.py", line 359, in run
    worker09.euwest.azure: raw_content = self.objstorage.get(sha1)
    worker09.euwest.azure: File "/usr/lib/python3/dist-packages/swh/objstorage/multiplexer/multiplexer_objstorage.py", line 137, in get
    ...
    • Apr 12 2018, 9:47 AM
    • 18 Lines
    • Plain Text
    Author: ardumont
  • P246 (An Untitled Masterwork)
    swh-dedup=# select algo, avg(duration_us) from chunked_content left join chunking_method on chunking_method.id = chunked_content.method_id group by algo;
    algo | avg
    ---------+-----------------------
    buzhash | 976.7281333333333333
    rabin | 1222.3993333333333333
    ...
    • Apr 10 2018, 6:35 PM
    • 6 Lines
    Author: seirl
  • P245 Crossminer first batch 0_1000_origin.csv
    https://github.com/Vishcn/vishWeb,ded9ec3e8ccc8f84d3a0541ed0c5a4a8606038ea
    https://github.com/YxhWife/firstPoj,19f0650345e0ff2cade69f6a105908ea47d0afe5
    https://github.com/ZergyPoo/VolunteerManager,e7cdde6ddc8e4500648de72cfce3aa6bd28d0a12
    https://github.com/adriancosma88/test-java-ocp,f8381b7ea001fe1db9b55c4fbdd6f7bacae74ef6
    https://github.com/ansell/bio2rdf-helpers,9ea852f3a0c73cab0403509f67d4b4164f1b30b2
    ...
    • Apr 10 2018, 4:15 PM
    • 10 Lines
    Author: moranegg
  • P244 (An Untitled Masterwork)
    17:52:45 :
    550 new
    17:54:03 :
    66 done
    ...
    • Mar 29 2018, 5:58 PM
    • 18 Lines
    Author: seirl
  • P243 (An Untitled Masterwork)
    +---------------------------------------------------------------------------------------------+
    | UNIBO NETWORK |
    | |
    | |
    | +-----------------------------------------+ +-------------------------------+ |
    ...
    • Mar 28 2018, 1:36 PM
    • 45 Lines
    Author: seirl
  • P242 [fixed] Hanging 'task reading' function on filtering period without data
    begin;
    create type task_record as (
    task_id bigint,
    task_policy task_policy,
    task_status task_status,
    ...
    • Mar 22 2018, 12:07 PM
    • 64 Lines
    • SQL
    Author: ardumont
  • P241 first indexing which seems to work ;)
    {
    "took": 1,
    "timed_out": false,
    "_shards": {
    "total": 1,
    ...
    • Mar 22 2018, 10:38 AM
    • 43 Lines
    • JSON
    Author: ardumont
  • P240 Index template for swh-scheduler task to archive
    {
    "order": 0,
    "index_patterns": ["swh-tasks-*"],
    "settings": {
    "index": {
    ...
    • Mar 21 2018, 5:09 PM
    • 48 Lines
    • JSON
    Author: ardumont
  • P239 latest mercurial loader issue
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 15:30:21,909: ERROR/Worker-15] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 1001, in store_data\n self.send_batch_revisions(self.get_revisions())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 654, in send_batch_revisions\n send_in_packets(revisions, self.send_revisions, packet_size)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 46, in send_in_packets\n sender(formatted_objects)\n File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f\n return Retrying(*dargs, **dkw).call(f, *args, **kw)\n File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 447, in send_revisions\n self.storage.revision_add(revision_list)\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 80, in revision_add\n return self.post(\'revision/add\', {\'revisions\': revisions})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 62, in post\n return self._decode_response(response)\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 91, in _decode_response\n raise pickle.loads(decode_response(response))\npsycopg2.extensions.TransactionRollbackError: deadlock detected\nDETAIL: Process 6294 waits for ShareLock on transaction 1703510102; blocked by process 6221.\nProcess 6221 waits for ShareLock on transaction 1703510103; blocked by process 6294.\nHINT: See server log for query details.\nCONTEXT: while inserting index tuple (28015479,4) in relation "revision_pkey"\nSQL statement "insert into revision (id, date, date_offset, date_neg_utc_offset, committer_date, committer_date_offset, committer_date_neg_utc_offset, type, directory, message, author, committer, metadata, synthetic)\n select t.id, t.date, t.date_offset, t.date_neg_utc_offset, t.committer_date, t.committer_date_offset, t.committer_date_neg_utc_offset, t.type, t.directory, t.message, a.id, c.id, t.metadata, t.synthetic\n from tmp_revision t\n left join person a on a.fullname = t.author_fullname\n left join person c on c.fullname = t.committer_fullname"\nPL/pgSQL function swh_revision_add() line 5 at SQL statement\n'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 19:26:48,882: ERROR/Worker-54] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 997, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 644, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 41, in send_in_packets\n sender(formatted_objects)\n File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f\n return Retrying(*dargs, **dkw).call(f, *args, **kw)\n File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 397, in send_contents\n self.storage.content_add(content_list)\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 21, in content_add\n return self.post(\'content/add\', {\'content\': content})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 62, in post\n return self._decode_response(response)\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 96, in _decode_response\n response.content,\nswh.core.api.RemoteException: Unexpected status code for API request: 504 (b\'<html>\\r\\n<head><title>504 Gateway Time-out</title></head>\\r\\n<body bgcolor="white">\\r\\n<center><h1>504 Gateway Time-out</h1></center>\\r\\n<hr><center>nginx/1.10.3</center>\\r\\n</body>\\r\\n</html>\\r\\n\')'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 19:29:31,436: ERROR/Worker-84] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 999, in store_data\n self.send_batch_directories(self.get_directories())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 649, in send_batch_directories\n send_in_packets(directories, self.send_directories, packet_size)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 329, in get_directories\n for _, _, new_dirs in self.load_directories():\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 314, in load_directories\n perms_code\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/objects.py", line 157, in add_blob\n node[fbase] = SimpleBlob(file_hash, is_symlink, file_perms)\nTypeError: \'SimpleBlob\' object does not support item assignment'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 20:25:25,808: ERROR/Worker-76] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 999, in store_data\n self.send_batch_directories(self.get_directories())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 649, in send_batch_directories\n send_in_packets(directories, self.send_directories, packet_size)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 329, in get_directories\n for _, _, new_dirs in self.load_directories():\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 307, in load_directories\n tree = tree.remove_tree_node_for_path(path)\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/objects.py", line 124, in remove_tree_node_for_path\n node[first] = node[first].remove_tree_node_for_path(rest)\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/objects.py", line 124, in remove_tree_node_for_path\n node[first] = node[first].remove_tree_node_for_path(rest)\nKeyError: b\'\''}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 22:38:50,016: ERROR/Worker-128] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen\n chunked=chunked)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File "/usr/lib/python3.5/http/client.py", line 1107, in request\n self._send_request(method, url, body, headers)\n File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request\n self.endheaders(body)\n File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders\n self._send_output(message_body)\n File "/usr/lib/python3.5/http/client.py", line 936, in _send_output\n self.send(message_body)\n File "/usr/lib/python3.5/http/client.py", line 908, in send\n self.sock.sendall(data)\nBrokenPipeError: [Errno 32] Broken pipe\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/requests/adapters.py", line 423, in send\n timeout=timeout\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen\n _stacktrace=sys.exc_info()[2])\n File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 334, in increment\n raise six.reraise(type(error), error, _stacktrace)\n File "/usr/lib/python3/dist-packages/six.py", line 685, in reraise\n raise value.with_traceback(tb)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen\n chunked=chunked)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File "/usr/lib/python3.5/http/client.py", line 1107, in request\n self._send_request(method, url, body, headers)\n File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request\n self.endheaders(body)\n File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders\n self._send_output(message_body)\n File "/usr/lib/python3.5/http/client.py", line 936, in _send_output\n self.send(message_body)\n File "/usr/lib/python3.5/http/client.py", line 908, in send\n self.sock.sendall(data)\nrequests.packages.urllib3.exceptions.ProtocolError: (\'Connection aborted.\', BrokenPipeError(32, \'Broken pipe\'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 42, in raw_post\n **opts\n File "/usr/lib/python3/dist-packages/requests/sessions.py", line 535, in post\n return self.request(\'POST\', url, data=data, json=json, **kwargs)\n File "/usr/lib/python3/dist-packages/requests/sessions.py", line 488, in request\n resp = self.send(prep, **send_kwargs)\n File "/usr/lib/python3/dist-packages/requests/sessions.py", line 609, in send\n r = adapter.send(request, **kwargs)\n File "/usr/lib/python3/dist-packages/requests/adapters.py", line 473, in send\n raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: (\'Connection aborted.\', BrokenPipeError(32, \'Broken pipe\'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 997, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 644, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 41, in send_in_packets\n sender(formatted_objects)\n File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f\n return Retrying(*dargs, **dkw).call(f, *args, **kw)\n File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 397, in send_contents\n self.storage.content_add(content_list)\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 21, in content_add\n return self.post(\'content/add\', {\'content\': content})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 61, in post\n headers={\'content-type\': \'application/x-msgpack\'})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 45, in raw_post\n raise self.api_exception(e)\nswh.storage.exc.StorageAPIError: An unexpected error occurred in the api backend: (\'Connection aborted.\', BrokenPipeError(32, \'Broken pipe\'))'}
    ...
    • Mar 21 2018, 2:45 PM
    • 56 Lines
    Author: ardumont
  • P238 How to load and index data
    I usually do something like this:
    ```
    make -C $SWH_ENVIRONMENT_HOME rebuild-testdata
    ...
    • Mar 14 2018, 12:19 PM
    • 65 Lines
    Author: ardumont
  • P237 ~/.config/swh/indexer/fossology_license.yml
    # storage to read sha1's metadata (path)
    # storage:
    # cls: local
    # args:
    # db: "service=swh-dev"
    ...
    • Mar 14 2018, 10:39 AM
    • 36 Lines
    • YAML
    Author: ardumont
  • P236 ~/.config/swh/indexer/mimetype.yml
    # storage to read sha1's metadata (path)
    # storage:
    # cls: local
    # args:
    # db: "service=swh-dev"
    ...
    • Mar 14 2018, 10:37 AM
    • 32 Lines
    • YAML
    Author: ardumont
  • P235 ~/.config/swh/indexer/orchestrator_text.yml
    indexers:
    # language:
    # batch_size: 10
    # check_presence: false
    fossology_license:
    ...
    • Mar 14 2018, 10:36 AM
    • 10 Lines
    • YAML
    Author: ardumont
  • P234 ~/.config/swh/indexer/orchestrator.yml
    indexers:
    mimetype:
    check_presence: false
    batch_size: 100
    • Mar 14 2018, 10:36 AM
    • 4 Lines
    • YAML
    Author: ardumont
  • P233 ~/.config/swh/loader/git-updater.yml
    storage:
    cls: remote
    args:
    url: http://localhost:5002/
    ...
    • Mar 14 2018, 10:12 AM
    • 25 Lines
    • YAML
    Author: ardumont
  • P232 list-sha1.sh
    #!/usr/bin/env bash
    psql service=swh-dev -c "copy (select sha1 from content) to stdin" | sed -e 's/^\\\\x//g'
    • Mar 14 2018, 10:09 AM
    • 3 Lines
    • Bash Scripting
    Author: ardumont
  • P231 scheduler-listener: Issue during update, service repeats trying starting and failing until giving up
    Mar 08 00:39:41 saatchi python3[16096]: Traceback (most recent call last):
    Mar 08 00:39:41 saatchi python3[16096]: File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    Mar 08 00:39:41 saatchi python3[16096]: "__main__", mod_spec)
    Mar 08 00:39:41 saatchi python3[16096]: File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    Mar 08 00:39:41 saatchi python3[16096]: exec(code, run_globals)
    ...
    • Mar 8 2018, 9:52 AM
    • 76 Lines
    Author: ardumont
  • P230 (An Untitled Masterwork)
    CREATE DOMAIN sha1 AS bytea CHECK (length(value) = 20);
    CREATE TABLE content (
    id sha1 PRIMARY KEY, -- SHA1 checksum
    length integer
    ...
    • Feb 28 2018, 7:18 PM
    • 32 Lines
    • SQL
    Author: seirl
  • P229 (An Untitled Masterwork)
    CREATE DOMAIN sha1 AS bytea CHECK (length(value) = 20);
    CREATE TABLE content (
    id sha1 PRIMARY KEY, -- SHA1 checksum
    length integer
    ...
    • Feb 28 2018, 7:14 PM
    • 32 Lines
    • SQL
    Author: seirl
  • P228 sqlitedict (dependency) spawns 2 disk cache whatever the configuration...
    $ python3
    Python 3.6.4 (default, Jan 5 2018, 02:13:53)
    [GCC 7.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> archive_name = 'latest/jackalopetears-wifi-propeller-load-source-archive.zip' # worker lost -> big archive
    ...
    • Feb 19 2018, 6:02 PM
    • 66 Lines
    Author: ardumont
  • P227 (An Untitled Masterwork)
    diff --git a/bin/deploy-on b/bin/deploy-on
    index 46d18e0..8e86207 100755
    --- a/bin/deploy-on
    +++ b/bin/deploy-on
    @@ -29,12 +29,16 @@ die_usage () {
    ...
    • Feb 16 2018, 2:38 PM
    • 33 Lines
    • Diff
    Author: seirl
  • P226 (An Untitled Masterwork)
    diff --git a/bin/deploy-on b/bin/deploy-on
    index 46d18e0..8e86207 100755
    --- a/bin/deploy-on
    +++ b/bin/deploy-on
    @@ -29,12 +29,16 @@ die_usage () {
    ...
    • Feb 16 2018, 2:37 PM
    • 33 Lines
    • Bash Scripting
    Author: seirl
  • P225 stop making me a liar! hglib's erratic behavior on bookmarks call.
    ```
    ardumont@uffizi:~/repo/hg% python3
    Python 3.5.3 (default, Jan 19 2017, 14:11:04)
    [GCC 6.3.0 20170118] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    ...
    • Feb 15 2018, 10:39 AM
    • 37 Lines
    Author: ardumont
  • P224 (An Untitled Masterwork)
    revision = {
    ...
    'parents': []
    }
    ...
    • Feb 15 2018, 9:38 AM
    • 11 Lines
    Author: fiendish
  • P223 (An Untitled Masterwork)
    python3 -m nose -sv --with-doctest ./swh/vault/tests
    pg_restore: [archiver (db)] Error while PROCESSING TOC:
    pg_restore: [archiver (db)] Error from TOC entry 240; 1259 2506400 SEQUENCE metadata_provider_id_seq ndandrim
    pg_restore: [archiver (db)] could not execute query: ERROR: syntax error at or near "AS"
    LINE 2: AS integer
    ...
    • Feb 12 2018, 5:04 PM
    • 19 Lines
    Author: seirl
  • P222 mercurial import: origins in error
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:36,421: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:38,665: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:41,157: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:43,165: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:47,377: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    ...
    • Feb 12 2018, 3:33 PM
    • 397 Lines
    Author: ardumont
  • P221 sample configuration file for snippets/ardumont/kibana_fetch_logs.py
    indexes:
    - logstash-2018.02.09
    - logstash-2018.02.10
    - logstash-2018.02.11
    - logstash-2018.02.12
    ...
    • Feb 12 2018, 3:17 PM
    • 31 Lines
    • YAML
    Author: ardumont
  • P220 ~/.config/swh/loader/hg.yml
    storage:
    cls: local
    args:
    db: 'service=swh-dev'
    objstorage:
    ...
    • Feb 9 2018, 4:55 PM
    • 26 Lines
    Author: ardumont
  • P219 latest svn errors - investigate missing input
    {'args': {}, 'kwargs': {}, 'exception': '[2017-12-20 03:23:24,265: ERROR/Worker-522] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 240, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config[\'revision_packet_size\']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: \'utf-8\' codec can\'t decode byte 0xe7 in position 4: invalid continuation byte'}
    {'args': {}, 'kwargs': {}, 'exception': "[2017-12-20 23:53:54,166: ERROR/Worker-894] Eventful partial visit. Detail: [Errno 2] No such file or directory: b'/tmp/swh.loader.svn.6euhc0_1.tmp/hackabot/trunk/hooks/pubmsg/20-env'"}
    {'args': {}, 'kwargs': {}, 'exception': '[2017-12-20 23:53:54,166: ERROR/Worker-894] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config[\'revision_packet_size\']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 175, in close\n os.chmod(self.fullpath, 0o755)\nFileNotFoundError: [Errno 2] No such file or directory: b\'/tmp/swh.loader.svn.6euhc0_1.tmp/hackabot/trunk/hooks/pubmsg/20-env\'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n \'id\': _id,\nswh.loader.svn.loader.SvnLoaderEventful: [Errno 2] No such file or directory: b\'/tmp/swh.loader.svn.6euhc0_1.tmp/hackabot/trunk/hooks/pubmsg/20-env\''}
    {'args': {}, 'kwargs': {}, 'exception': '[2017-12-21 10:59:38,296: ERROR/Worker-1084] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 240, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config[\'revision_packet_size\']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: \'utf-8\' codec can\'t decode byte 0xb3 in position 10: invalid start byte'}
    {'args': {}, 'kwargs': {}, 'exception': "[2017-12-29 15:17:13,267: ERROR/Worker-1618] Eventful partial visit. Detail: [Errno 2] No such file or directory: b'/tmp/swh.loader.svn.qay2ee8q.tmp/qgcm/trunk/working/QGCM/cases/full/barotropic/with_beta/nk2/plot'"}
    ...
    • Jan 31 2018, 9:28 AM
    • 80 Lines
    Author: ardumont
  • P218 (An Untitled Masterwork)
    diff --git a/data/defaults.yaml b/data/defaults.yaml
    index 7885801..81234a0 100644
    --- a/data/defaults.yaml
    +++ b/data/defaults.yaml
    @@ -1352,12 +1352,7 @@ swh::deploy::vault::backend::http_timeout: 100000
    ...
    • Jan 30 2018, 3:02 PM
    • 45 Lines
    • Diff
    Author: seirl
  • P217 (An Untitled Masterwork)
    root@swh-test:~# cat setup_container.sh
    #!/bin/bash
    set -e
    ...
    • Jan 26 2018, 3:48 PM
    • 35 Lines
    • Bash Scripting
    Author: seirl
  • P216 Make html in swh-deposit/docs/ error
    Configuration error:
    There is a programable error in your configuration file:
    Traceback (most recent call last):
    File "/usr/local/lib/python3.4/dist-packages/sphinx/config.py", line 157, in __init__
    ...
    • Jan 24 2018, 5:08 PM
    • 33 Lines
    Author: moranegg
  • P215 IRC access factorization
    /msg chanserv access #swh-devel del rdicosmo
    /msg chanserv access #swh-devel del anlambert
    /msg chanserv access #swh-devel del grouss
    /msg chanserv access #swh-devel del ftigeot
    /msg chanserv access #swh-devel del moranegg
    ...
    • Jan 22 2018, 2:04 PM
    • 15 Lines
    Author: seirl
  • P213 (An Untitled Masterwork)
    root@swh-test:~# journalctl -u openvpn@softwareheritage.service
    -- Logs begin at Fri 2018-01-19 10:28:46 CET, end at Fri 2018-01-19 12:47:10 CET. --
    Jan 19 11:00:11 swh-test ovpn-softwareheritage[14829]: WARNING: this cipher's block size is less than 128 bit (64 bit). Consider using a --cipher with a larger block size.
    Jan 19 11:00:11 swh-test ovpn-softwareheritage[14829]: WARNING: this cipher's block size is less than 128 bit (64 bit). Consider using a --cipher with a larger block size.
    Jan 19 11:33:17 swh-test ovpn-softwareheritage[14829]: [louvre] Inactivity timeout (--ping-restart), restarting
    ...
    • Jan 19 2018, 12:47 PM
    • 19 Lines
    Author: seirl
  • P212 checking hal deposit archive
    Retrieve the deposit concerned:
    ```
    softwareheritage-deposit=> select dr.archive from deposit d inner join deposit_request dr on d.id=dr.deposit_id where external_id='hal-01243618' and dr.type_id=1;
    archive
    ...
    • Jan 18 2018, 3:46 PM
    • 36 Lines
    • Bash Scripting
    Author: ardumont
  • P211 data for deposit
    - model: deposit.depositclient
    fields:
    user_ptr_id: 1
    domain: archives-ouvertes.fr
    provider_url: https://hal.archives-ouvertes.fr/
    ...
    • Jan 10 2018, 12:10 PM
    • 7 Lines
    Author: moranegg
  • P210 (An Untitled Masterwork)
    File "/home/seirl/swh-environment/.venv/lib/python3.5/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
    File "/home/seirl/swh-environment/swh-web/swh/web/api/apidoc.py", line 108, in documented_view
    doc_data = self.get_doc_data(f)
    File "/home/seirl/swh-environment/swh-web/swh/web/api/apidoc.py", line 187, in get_doc_data
    ...
    • Jan 8 2018, 4:46 PM
    • 15 Lines
    Author: seirl
  • P208 ~/.pg_service.conf
    [swh]
    dbname=softwareheritage
    host=db.internal.softwareheritage.org
    user=guest
    ...
    • Jan 3 2018, 3:03 PM
    • 25 Lines
    Author: zack
  • P207 Convert a bundle from bundle10 to bundle20 - failure
    $ pwd
    /home/storage/hg/repo/not-bundle20/756015-ipv6
    $ hg bundle --rev 'bundle()' \  
    --base 'parents(roots(bundle()))' \
    -R HG20_bundle_none HG20_bundle_none_migrated \
    ...
    • Dec 21 2017, 10:06 AM
    • 37 Lines
    Author: ardumont
  • P206 current mercurial loader release issue 'cause no tag author
    Python 3.5.3 (default, Jan 19 2017, 14:11:04)
    [GCC 6.3.0 20170118] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> # remote repository
    ... origin_url = 'https://mercurial.tuxfamily.org/enchantepic2/enchantepic2hg'
    ...
    • Dec 20 2017, 12:53 PM
    • 82 Lines
    Author: ardumont
  • P205 listener failure on connection issues with the rabbitmq instance
    -- Unit swh-scheduler-listener.service has finished starting up.
    --
    -- The start-up result is done.
    Nov 30 22:23:15 saatchi python3[25458]: Traceback (most recent call last):
    Nov 30 22:23:15 saatchi python3[25458]: File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    ...
    • Dec 15 2017, 9:39 AM
    • 59 Lines
    Author: ardumont
First
Prev
Next
About · Developer information · Legal