Page MenuHomeSoftware Heritage
Create Paste
Paste Active Pastes

Active Pastes
Use Results
Edit Query

  • P246 (An Untitled Masterwork)
    swh-dedup=# select algo, avg(duration_us) from chunked_content left join chunking_method on chunking_method.id = chunked_content.method_id group by algo;
    algo | avg
    ---------+-----------------------
    buzhash | 976.7281333333333333
    rabin | 1222.3993333333333333
    ...
    • Apr 10 2018, 6:35 PM
    • 6 Lines
    Author: seirl
  • P245 Crossminer first batch 0_1000_origin.csv
    https://github.com/Vishcn/vishWeb,ded9ec3e8ccc8f84d3a0541ed0c5a4a8606038ea
    https://github.com/YxhWife/firstPoj,19f0650345e0ff2cade69f6a105908ea47d0afe5
    https://github.com/ZergyPoo/VolunteerManager,e7cdde6ddc8e4500648de72cfce3aa6bd28d0a12
    https://github.com/adriancosma88/test-java-ocp,f8381b7ea001fe1db9b55c4fbdd6f7bacae74ef6
    https://github.com/ansell/bio2rdf-helpers,9ea852f3a0c73cab0403509f67d4b4164f1b30b2
    ...
    • Apr 10 2018, 4:15 PM
    • 10 Lines
    Author: moranegg
  • P244 (An Untitled Masterwork)
    17:52:45 :
    550 new
    17:54:03 :
    66 done
    ...
    • Mar 29 2018, 5:58 PM
    • 18 Lines
    Author: seirl
  • P243 (An Untitled Masterwork)
    +---------------------------------------------------------------------------------------------+
    | UNIBO NETWORK |
    | |
    | |
    | +-----------------------------------------+ +-------------------------------+ |
    ...
    • Mar 28 2018, 1:36 PM
    • 45 Lines
    Author: seirl
  • P242 [fixed] Hanging 'task reading' function on filtering period without data
    begin;
    create type task_record as (
    task_id bigint,
    task_policy task_policy,
    task_status task_status,
    ...
    • Mar 22 2018, 12:07 PM
    • 64 Lines
    • SQL
    Author: ardumont
  • P241 first indexing which seems to work ;)
    {
    "took": 1,
    "timed_out": false,
    "_shards": {
    "total": 1,
    ...
    • Mar 22 2018, 10:38 AM
    • 43 Lines
    • JSON
    Author: ardumont
  • P240 Index template for swh-scheduler task to archive
    {
    "order": 0,
    "index_patterns": ["swh-tasks-*"],
    "settings": {
    "index": {
    ...
    • Mar 21 2018, 5:09 PM
    • 48 Lines
    • JSON
    Author: ardumont
  • P239 latest mercurial loader issue
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 15:30:21,909: ERROR/Worker-15] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 1001, in store_data\n self.send_batch_revisions(self.get_revisions())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 654, in send_batch_revisions\n send_in_packets(revisions, self.send_revisions, packet_size)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 46, in send_in_packets\n sender(formatted_objects)\n File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f\n return Retrying(*dargs, **dkw).call(f, *args, **kw)\n File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 447, in send_revisions\n self.storage.revision_add(revision_list)\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 80, in revision_add\n return self.post(\'revision/add\', {\'revisions\': revisions})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 62, in post\n return self._decode_response(response)\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 91, in _decode_response\n raise pickle.loads(decode_response(response))\npsycopg2.extensions.TransactionRollbackError: deadlock detected\nDETAIL: Process 6294 waits for ShareLock on transaction 1703510102; blocked by process 6221.\nProcess 6221 waits for ShareLock on transaction 1703510103; blocked by process 6294.\nHINT: See server log for query details.\nCONTEXT: while inserting index tuple (28015479,4) in relation "revision_pkey"\nSQL statement "insert into revision (id, date, date_offset, date_neg_utc_offset, committer_date, committer_date_offset, committer_date_neg_utc_offset, type, directory, message, author, committer, metadata, synthetic)\n select t.id, t.date, t.date_offset, t.date_neg_utc_offset, t.committer_date, t.committer_date_offset, t.committer_date_neg_utc_offset, t.type, t.directory, t.message, a.id, c.id, t.metadata, t.synthetic\n from tmp_revision t\n left join person a on a.fullname = t.author_fullname\n left join person c on c.fullname = t.committer_fullname"\nPL/pgSQL function swh_revision_add() line 5 at SQL statement\n'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 19:26:48,882: ERROR/Worker-54] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 997, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 644, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 41, in send_in_packets\n sender(formatted_objects)\n File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f\n return Retrying(*dargs, **dkw).call(f, *args, **kw)\n File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 397, in send_contents\n self.storage.content_add(content_list)\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 21, in content_add\n return self.post(\'content/add\', {\'content\': content})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 62, in post\n return self._decode_response(response)\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 96, in _decode_response\n response.content,\nswh.core.api.RemoteException: Unexpected status code for API request: 504 (b\'<html>\\r\\n<head><title>504 Gateway Time-out</title></head>\\r\\n<body bgcolor="white">\\r\\n<center><h1>504 Gateway Time-out</h1></center>\\r\\n<hr><center>nginx/1.10.3</center>\\r\\n</body>\\r\\n</html>\\r\\n\')'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 19:29:31,436: ERROR/Worker-84] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 999, in store_data\n self.send_batch_directories(self.get_directories())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 649, in send_batch_directories\n send_in_packets(directories, self.send_directories, packet_size)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 329, in get_directories\n for _, _, new_dirs in self.load_directories():\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 314, in load_directories\n perms_code\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/objects.py", line 157, in add_blob\n node[fbase] = SimpleBlob(file_hash, is_symlink, file_perms)\nTypeError: \'SimpleBlob\' object does not support item assignment'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 20:25:25,808: ERROR/Worker-76] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 999, in store_data\n self.send_batch_directories(self.get_directories())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 649, in send_batch_directories\n send_in_packets(directories, self.send_directories, packet_size)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 329, in get_directories\n for _, _, new_dirs in self.load_directories():\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 307, in load_directories\n tree = tree.remove_tree_node_for_path(path)\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/objects.py", line 124, in remove_tree_node_for_path\n node[first] = node[first].remove_tree_node_for_path(rest)\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/objects.py", line 124, in remove_tree_node_for_path\n node[first] = node[first].remove_tree_node_for_path(rest)\nKeyError: b\'\''}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-03-14 22:38:50,016: ERROR/Worker-128] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen\n chunked=chunked)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File "/usr/lib/python3.5/http/client.py", line 1107, in request\n self._send_request(method, url, body, headers)\n File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request\n self.endheaders(body)\n File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders\n self._send_output(message_body)\n File "/usr/lib/python3.5/http/client.py", line 936, in _send_output\n self.send(message_body)\n File "/usr/lib/python3.5/http/client.py", line 908, in send\n self.sock.sendall(data)\nBrokenPipeError: [Errno 32] Broken pipe\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/requests/adapters.py", line 423, in send\n timeout=timeout\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen\n _stacktrace=sys.exc_info()[2])\n File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 334, in increment\n raise six.reraise(type(error), error, _stacktrace)\n File "/usr/lib/python3/dist-packages/six.py", line 685, in reraise\n raise value.with_traceback(tb)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen\n chunked=chunked)\n File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 361, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File "/usr/lib/python3.5/http/client.py", line 1107, in request\n self._send_request(method, url, body, headers)\n File "/usr/lib/python3.5/http/client.py", line 1152, in _send_request\n self.endheaders(body)\n File "/usr/lib/python3.5/http/client.py", line 1103, in endheaders\n self._send_output(message_body)\n File "/usr/lib/python3.5/http/client.py", line 936, in _send_output\n self.send(message_body)\n File "/usr/lib/python3.5/http/client.py", line 908, in send\n self.sock.sendall(data)\nrequests.packages.urllib3.exceptions.ProtocolError: (\'Connection aborted.\', BrokenPipeError(32, \'Broken pipe\'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 42, in raw_post\n **opts\n File "/usr/lib/python3/dist-packages/requests/sessions.py", line 535, in post\n return self.request(\'POST\', url, data=data, json=json, **kwargs)\n File "/usr/lib/python3/dist-packages/requests/sessions.py", line 488, in request\n resp = self.send(prep, **send_kwargs)\n File "/usr/lib/python3/dist-packages/requests/sessions.py", line 609, in send\n r = adapter.send(request, **kwargs)\n File "/usr/lib/python3/dist-packages/requests/adapters.py", line 473, in send\n raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: (\'Connection aborted.\', BrokenPipeError(32, \'Broken pipe\'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 890, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 997, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 644, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 41, in send_in_packets\n sender(formatted_objects)\n File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f\n return Retrying(*dargs, **dkw).call(f, *args, **kw)\n File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call\n return attempt.get(self._wrap_exception)\n File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get\n six.reraise(self.value[0], self.value[1], self.value[2])\n File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call\n attempt = Attempt(fn(*args, **kwargs), attempt_number, False)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 397, in send_contents\n self.storage.content_add(content_list)\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 21, in content_add\n return self.post(\'content/add\', {\'content\': content})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 61, in post\n headers={\'content-type\': \'application/x-msgpack\'})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 45, in raw_post\n raise self.api_exception(e)\nswh.storage.exc.StorageAPIError: An unexpected error occurred in the api backend: (\'Connection aborted.\', BrokenPipeError(32, \'Broken pipe\'))'}
    ...
    • Mar 21 2018, 2:45 PM
    • 56 Lines
    Author: ardumont
  • P238 How to load and index data
    I usually do something like this:
    ```
    make -C $SWH_ENVIRONMENT_HOME rebuild-testdata
    ...
    • Mar 14 2018, 12:19 PM
    • 65 Lines
    Author: ardumont
  • P237 ~/.config/swh/indexer/fossology_license.yml
    # storage to read sha1's metadata (path)
    # storage:
    # cls: local
    # args:
    # db: "service=swh-dev"
    ...
    • Mar 14 2018, 10:39 AM
    • 36 Lines
    • YAML
    Author: ardumont
  • P236 ~/.config/swh/indexer/mimetype.yml
    # storage to read sha1's metadata (path)
    # storage:
    # cls: local
    # args:
    # db: "service=swh-dev"
    ...
    • Mar 14 2018, 10:37 AM
    • 32 Lines
    • YAML
    Author: ardumont
  • P235 ~/.config/swh/indexer/orchestrator_text.yml
    indexers:
    # language:
    # batch_size: 10
    # check_presence: false
    fossology_license:
    ...
    • Mar 14 2018, 10:36 AM
    • 10 Lines
    • YAML
    Author: ardumont
  • P234 ~/.config/swh/indexer/orchestrator.yml
    indexers:
    mimetype:
    check_presence: false
    batch_size: 100
    • Mar 14 2018, 10:36 AM
    • 4 Lines
    • YAML
    Author: ardumont
  • P233 ~/.config/swh/loader/git-updater.yml
    storage:
    cls: remote
    args:
    url: http://localhost:5002/
    ...
    • Mar 14 2018, 10:12 AM
    • 25 Lines
    • YAML
    Author: ardumont
  • P232 list-sha1.sh
    #!/usr/bin/env bash
    psql service=swh-dev -c "copy (select sha1 from content) to stdin" | sed -e 's/^\\\\x//g'
    • Mar 14 2018, 10:09 AM
    • 3 Lines
    • Bash Scripting
    Author: ardumont
  • P231 scheduler-listener: Issue during update, service repeats trying starting and failing until giving up
    Mar 08 00:39:41 saatchi python3[16096]: Traceback (most recent call last):
    Mar 08 00:39:41 saatchi python3[16096]: File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    Mar 08 00:39:41 saatchi python3[16096]: "__main__", mod_spec)
    Mar 08 00:39:41 saatchi python3[16096]: File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    Mar 08 00:39:41 saatchi python3[16096]: exec(code, run_globals)
    ...
    • Mar 8 2018, 9:52 AM
    • 76 Lines
    Author: ardumont
  • P230 (An Untitled Masterwork)
    CREATE DOMAIN sha1 AS bytea CHECK (length(value) = 20);
    CREATE TABLE content (
    id sha1 PRIMARY KEY, -- SHA1 checksum
    length integer
    ...
    • Feb 28 2018, 7:18 PM
    • 32 Lines
    • SQL
    Author: seirl
  • P229 (An Untitled Masterwork)
    CREATE DOMAIN sha1 AS bytea CHECK (length(value) = 20);
    CREATE TABLE content (
    id sha1 PRIMARY KEY, -- SHA1 checksum
    length integer
    ...
    • Feb 28 2018, 7:14 PM
    • 32 Lines
    • SQL
    Author: seirl
  • P228 sqlitedict (dependency) spawns 2 disk cache whatever the configuration...
    $ python3
    Python 3.6.4 (default, Jan 5 2018, 02:13:53)
    [GCC 7.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> archive_name = 'latest/jackalopetears-wifi-propeller-load-source-archive.zip' # worker lost -> big archive
    ...
    • Feb 19 2018, 6:02 PM
    • 66 Lines
    Author: ardumont
  • P227 (An Untitled Masterwork)
    diff --git a/bin/deploy-on b/bin/deploy-on
    index 46d18e0..8e86207 100755
    --- a/bin/deploy-on
    +++ b/bin/deploy-on
    @@ -29,12 +29,16 @@ die_usage () {
    ...
    • Feb 16 2018, 2:38 PM
    • 33 Lines
    • Diff
    Author: seirl
  • P226 (An Untitled Masterwork)
    diff --git a/bin/deploy-on b/bin/deploy-on
    index 46d18e0..8e86207 100755
    --- a/bin/deploy-on
    +++ b/bin/deploy-on
    @@ -29,12 +29,16 @@ die_usage () {
    ...
    • Feb 16 2018, 2:37 PM
    • 33 Lines
    • Bash Scripting
    Author: seirl
  • P225 stop making me a liar! hglib's erratic behavior on bookmarks call.
    ```
    ardumont@uffizi:~/repo/hg% python3
    Python 3.5.3 (default, Jan 19 2017, 14:11:04)
    [GCC 6.3.0 20170118] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    ...
    • Feb 15 2018, 10:39 AM
    • 37 Lines
    Author: ardumont
  • P224 (An Untitled Masterwork)
    revision = {
    ...
    'parents': []
    }
    ...
    • Feb 15 2018, 9:38 AM
    • 11 Lines
    Author: fiendish
  • P223 (An Untitled Masterwork)
    python3 -m nose -sv --with-doctest ./swh/vault/tests
    pg_restore: [archiver (db)] Error while PROCESSING TOC:
    pg_restore: [archiver (db)] Error from TOC entry 240; 1259 2506400 SEQUENCE metadata_provider_id_seq ndandrim
    pg_restore: [archiver (db)] could not execute query: ERROR: syntax error at or near "AS"
    LINE 2: AS integer
    ...
    • Feb 12 2018, 5:04 PM
    • 19 Lines
    Author: seirl
  • P222 mercurial import: origins in error
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:36,421: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:38,665: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:41,157: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:43,165: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:47,377: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    ...
    • Feb 12 2018, 3:33 PM
    • 397 Lines
    Author: ardumont
  • P221 sample configuration file for snippets/ardumont/kibana_fetch_logs.py
    indexes:
    - logstash-2018.02.09
    - logstash-2018.02.10
    - logstash-2018.02.11
    - logstash-2018.02.12
    ...
    • Feb 12 2018, 3:17 PM
    • 31 Lines
    • YAML
    Author: ardumont
  • P220 ~/.config/swh/loader/hg.yml
    storage:
    cls: local
    args:
    db: 'service=swh-dev'
    objstorage:
    ...
    • Feb 9 2018, 4:55 PM
    • 26 Lines
    Author: ardumont
  • P219 latest svn errors - investigate missing input
    {'args': {}, 'kwargs': {}, 'exception': '[2017-12-20 03:23:24,265: ERROR/Worker-522] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 240, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config[\'revision_packet_size\']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: \'utf-8\' codec can\'t decode byte 0xe7 in position 4: invalid continuation byte'}
    {'args': {}, 'kwargs': {}, 'exception': "[2017-12-20 23:53:54,166: ERROR/Worker-894] Eventful partial visit. Detail: [Errno 2] No such file or directory: b'/tmp/swh.loader.svn.6euhc0_1.tmp/hackabot/trunk/hooks/pubmsg/20-env'"}
    {'args': {}, 'kwargs': {}, 'exception': '[2017-12-20 23:53:54,166: ERROR/Worker-894] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config[\'revision_packet_size\']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 175, in close\n os.chmod(self.fullpath, 0o755)\nFileNotFoundError: [Errno 2] No such file or directory: b\'/tmp/swh.loader.svn.6euhc0_1.tmp/hackabot/trunk/hooks/pubmsg/20-env\'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n \'id\': _id,\nswh.loader.svn.loader.SvnLoaderEventful: [Errno 2] No such file or directory: b\'/tmp/swh.loader.svn.6euhc0_1.tmp/hackabot/trunk/hooks/pubmsg/20-env\''}
    {'args': {}, 'kwargs': {}, 'exception': '[2017-12-21 10:59:38,296: ERROR/Worker-1084] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 240, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config[\'revision_packet_size\']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: \'utf-8\' codec can\'t decode byte 0xb3 in position 10: invalid start byte'}
    {'args': {}, 'kwargs': {}, 'exception': "[2017-12-29 15:17:13,267: ERROR/Worker-1618] Eventful partial visit. Detail: [Errno 2] No such file or directory: b'/tmp/swh.loader.svn.qay2ee8q.tmp/qgcm/trunk/working/QGCM/cases/full/barotropic/with_beta/nk2/plot'"}
    ...
    • Jan 31 2018, 9:28 AM
    • 80 Lines
    Author: ardumont
  • P218 (An Untitled Masterwork)
    diff --git a/data/defaults.yaml b/data/defaults.yaml
    index 7885801..81234a0 100644
    --- a/data/defaults.yaml
    +++ b/data/defaults.yaml
    @@ -1352,12 +1352,7 @@ swh::deploy::vault::backend::http_timeout: 100000
    ...
    • Jan 30 2018, 3:02 PM
    • 45 Lines
    • Diff
    Author: seirl
  • P217 (An Untitled Masterwork)
    root@swh-test:~# cat setup_container.sh
    #!/bin/bash
    set -e
    ...
    • Jan 26 2018, 3:48 PM
    • 35 Lines
    • Bash Scripting
    Author: seirl
  • P216 Make html in swh-deposit/docs/ error
    Configuration error:
    There is a programable error in your configuration file:
    Traceback (most recent call last):
    File "/usr/local/lib/python3.4/dist-packages/sphinx/config.py", line 157, in __init__
    ...
    • Jan 24 2018, 5:08 PM
    • 33 Lines
    Author: moranegg
  • P215 IRC access factorization
    /msg chanserv access #swh-devel del rdicosmo
    /msg chanserv access #swh-devel del anlambert
    /msg chanserv access #swh-devel del grouss
    /msg chanserv access #swh-devel del ftigeot
    /msg chanserv access #swh-devel del moranegg
    ...
    • Jan 22 2018, 2:04 PM
    • 15 Lines
    Author: seirl
  • P213 (An Untitled Masterwork)
    root@swh-test:~# journalctl -u openvpn@softwareheritage.service
    -- Logs begin at Fri 2018-01-19 10:28:46 CET, end at Fri 2018-01-19 12:47:10 CET. --
    Jan 19 11:00:11 swh-test ovpn-softwareheritage[14829]: WARNING: this cipher's block size is less than 128 bit (64 bit). Consider using a --cipher with a larger block size.
    Jan 19 11:00:11 swh-test ovpn-softwareheritage[14829]: WARNING: this cipher's block size is less than 128 bit (64 bit). Consider using a --cipher with a larger block size.
    Jan 19 11:33:17 swh-test ovpn-softwareheritage[14829]: [louvre] Inactivity timeout (--ping-restart), restarting
    ...
    • Jan 19 2018, 12:47 PM
    • 19 Lines
    Author: seirl
  • P212 checking hal deposit archive
    Retrieve the deposit concerned:
    ```
    softwareheritage-deposit=> select dr.archive from deposit d inner join deposit_request dr on d.id=dr.deposit_id where external_id='hal-01243618' and dr.type_id=1;
    archive
    ...
    • Jan 18 2018, 3:46 PM
    • 36 Lines
    • Bash Scripting
    Author: ardumont
  • P211 data for deposit
    - model: deposit.depositclient
    fields:
    user_ptr_id: 1
    domain: archives-ouvertes.fr
    provider_url: https://hal.archives-ouvertes.fr/
    ...
    • Jan 10 2018, 12:10 PM
    • 7 Lines
    Author: moranegg
  • P210 (An Untitled Masterwork)
    File "/home/seirl/swh-environment/.venv/lib/python3.5/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
    File "/home/seirl/swh-environment/swh-web/swh/web/api/apidoc.py", line 108, in documented_view
    doc_data = self.get_doc_data(f)
    File "/home/seirl/swh-environment/swh-web/swh/web/api/apidoc.py", line 187, in get_doc_data
    ...
    • Jan 8 2018, 4:46 PM
    • 15 Lines
    Author: seirl
  • P208 ~/.pg_service.conf
    [swh]
    dbname=softwareheritage
    host=db.internal.softwareheritage.org
    user=guest
    ...
    • Jan 3 2018, 3:03 PM
    • 25 Lines
    Author: zack
  • P207 Convert a bundle from bundle10 to bundle20 - failure
    $ pwd
    /home/storage/hg/repo/not-bundle20/756015-ipv6
    $ hg bundle --rev 'bundle()' \  
    --base 'parents(roots(bundle()))' \
    -R HG20_bundle_none HG20_bundle_none_migrated \
    ...
    • Dec 21 2017, 10:06 AM
    • 37 Lines
    Author: ardumont
  • P206 current mercurial loader release issue 'cause no tag author
    Python 3.5.3 (default, Jan 19 2017, 14:11:04)
    [GCC 6.3.0 20170118] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> # remote repository
    ... origin_url = 'https://mercurial.tuxfamily.org/enchantepic2/enchantepic2hg'
    ...
    • Dec 20 2017, 12:53 PM
    • 82 Lines
    Author: ardumont
  • P205 listener failure on connection issues with the rabbitmq instance
    -- Unit swh-scheduler-listener.service has finished starting up.
    --
    -- The start-up result is done.
    Nov 30 22:23:15 saatchi python3[25458]: Traceback (most recent call last):
    Nov 30 22:23:15 saatchi python3[25458]: File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    ...
    • Dec 15 2017, 9:39 AM
    • 59 Lines
    Author: ardumont
  • P204 listener failure on running out of disk in prado (db host vm)
    -- Unit swh-scheduler-listener.service has finished starting up.
    --
    -- The start-up result is done.
    Dec 15 01:04:47 saatchi python3[26433]: Traceback (most recent call last):
    Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    ...
    • Dec 15 2017, 9:37 AM
    • 67 Lines
    Author: ardumont
  • P203 Clean up wrong googlecode origins (googlecode instead of eclipselabs or apache-extras) - T897 related
    ```
    begin;
    select count(*) from origin where type='svn';
    ...
    • Dec 14 2017, 2:55 PM
    • 163 Lines
    Author: ardumont
  • P202 Clean up wrong origins (url is filled with the size instead of the actual origin url) - T896 related
    ```
    begin;
    create or replace function list_wrong_origins()
    returns setof origin.id%type
    ...
    • Dec 13 2017, 6:55 PM
    • 168 Lines
    • SQL
    Author: ardumont
  • P201 api: vault response in production mode
    antoine@guggenheim:~$ curl -i http://localhost:8000/api/1/vault/directory/d4a96ba891017d0d26c15e509b4e6515e40d75ee/raw/
    HTTP/1.1 503 Service Unavailable
    Server: gunicorn/19.6.0
    Date: Fri, 08 Dec 2017 13:57:25 GMT
    Connection: close
    ...
    • Dec 8 2017, 2:58 PM
    • 12 Lines
    Author: anlambert
  • P200 deposit load commands
    commands for running deposit load:
    - run sword server on port 5006: in swh-deposit: make run-dev
    - update deposit db in swh deposit: make db-migrate
    - run deposit with metadata locally in bin/: make new-complete
    ...
    • Dec 5 2017, 4:50 PM
    • 23 Lines
    Author: moranegg
  • P199 Vault task in scheduler
    insert into task_type(
    type,
    description,
    backend_name,
    default_interval, min_interval, max_interval, backoff_factor,
    ...
    • Dec 5 2017, 11:35 AM
    • 12 Lines
    Author: seirl
  • P198 improving indexing using derivative form of materialized view
    -- deal with new content
    create or replace function content_insert() returns trigger
    security definer
    language plpgsql
    as $$
    ...
    • Dec 2 2017, 1:06 PM
    • 62 Lines
    Author: ardumont
  • P197 errors in indexer when uffizi's disk's full -> OSError: [Errno 28] No space left on device
    {
    "_index": "logstash-2017.11.29",
    "_type": "journal",
    "_id": "AWAFo8W9l67zVNRhPjvh",
    "_score": null,
    ...
    • Dec 1 2017, 1:52 PM
    • 61 Lines
    Author: ardumont
  • P196 orangeriedev bin/deploy-on run
    bin/deploy-on --no-apt --no-master orangeriedev.internal.softwareheritage.org    master 
    *** swh-deploy: starting test run on orangeriedev.internal.softwareheritage.org...
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    ...
    • Nov 24 2017, 7:10 PM
    • 21 Lines
    Author: ardumont
  • P195 (An Untitled Masterwork)
    scaramouche ~  psql service=swh-replica
    psql (10.1, server 10.0)
    SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
    Type "help" for help.
    ...
    • Nov 17 2017, 2:16 PM
    • 38 Lines
    Author: zack
  • P194 spam about etckeeper on orangerie not being set
    /etc/cron.daily/etckeeper:
    *** Please tell me who you are.
    Run
    ...
    • Nov 15 2017, 10:03 AM
    • 14 Lines
    Author: ardumont
  • P190 Bulking directory listings with swh_directory_walk_many
    create or replace function swh_directory_walk_many(walked_dirs_id bytea[])
    returns setof directory_entry
    language sql
    stable
    as $$
    ...
    • Nov 9 2017, 2:48 PM
    • 33 Lines
    • SQL
    Author: seirl
  • P189 package build broken...
    On latest swh-environment
    ```
    bin/make-package -b swh-scheduler
    running sdist
    ...
    • Nov 8 2017, 3:57 PM
    • 955 Lines
    • Bash Scripting
    Author: ardumont
  • P188 (An Untitled Masterwork)
    softwareheritage-dev=# select perms, count(perms) from directory_entry_file group by perms;
    perms | count
    -------+-------
    33204 | 157
    40960 | 327
    ...
    • Nov 6 2017, 4:41 PM
    • 8 Lines
    Author: seirl
  • P187 ~/.config/swh/deposit/server.yml
    # dev option
    host: 127.0.0.1
    port: 5006
    # 200 Mib max size
    ...
    • Oct 31 2017, 2:34 PM
    • 14 Lines
    • YAML
    Author: ardumont
  • P186 (An Untitled Masterwork)
    *** swh-deploy: deploying recipes on pergamon.internal.softwareheritage.org...
    /usr/local/sbin/swh-puppet-master-deploy: line 4: /etc/puppet/environments/production/deploy.sh: No such file or directory
    Connection to pergamon.internal.softwareheritage.org closed.
    *** swh-deploy: master ok.
    *** swh-deploy: starting test run on pergamon...
    ...
    • Oct 30 2017, 1:26 PM
    • 9 Lines
    Author: zack
  • P185 load-git-disk.py
    #!/usr/bin/env python3
    import click
    ...
    • Oct 26 2017, 3:47 PM
    • 26 Lines
    Author: ardumont
  • P184 documentation build failure tryout
    ```
    ...swh-docs/docs $ make distclean
    Removing everything under '_build'...
    bin/ln-sphinx-subprojects --remove
    make -C images clean
    ...
    • Oct 21 2017, 5:38 PM
    • 394 Lines
    Author: ardumont
  • P182 scheduler - new task type for loading one-shot task (svn mount and load repository, load deposit archive)...
    insert into task_type(
    type,
    description,
    backend_name,
    default_interval, min_interval, max_interval, backoff_factor,
    ...
    • Oct 10 2017, 7:53 PM
    • 39 Lines
    • SQL
    Author: ardumont
  • P181 TEI sample for HAL
    <TEI><teiHeader><fileDesc><titleStmt><title>HAL TEI export of hal-01587083</title></titleStmt><publicationStmt><distributor>CCSD</distributor><availability status="restricted"><licence target="http://creativecommons.org/licenses/by/4.0/">Distributed under a Creative Commons Attribution 4.0 International License</licence></availability><date when="2017-10-03T17:21:03+02:00"/></publicationStmt><sourceDesc><p part="N">HAL API platform</p></sourceDesc></fileDesc></teiHeader><text><body><listBibl><biblFull><titleStmt><title xml:lang="en">questionnaire software metadata</title><author role="aut"><persName><forename type="first">Morane</forename><surname>Gruenpeter</surname></persName><email type="md5">7de56c632362954fa84172cad80afe4e</email><email type="domain">inria.fr</email><ptr type="url" target="moranegg.github.io"/><idno type="halauthorid">1556733</idno><affiliation ref="#struct-474639"/></author><editor role="depositor"><persName><forename>Morane</forename><surname>Gruenpeter</surname></persName><email...
    • Oct 10 2017, 11:19 AM
    • 1 Line
    Author: moranegg
  • P180 swh.model.from_disk nom nom nom
    In [1]: from swh.model.from_disk import Content, Directory, ignore_named_directories
    In [2]: d = Directory.from_disk(directory=b'swh-model', dir_filter=ignore_named_directories([b'.git', b'.coverage']), data=True)
    In [3]: hash1 = d.hash
    ...
    • Sep 20 2017, 11:33 PM
    • 39 Lines
    • Python
    Author: olasd
  • P179 (An Untitled Masterwork)
    # Generated by iptables-save v1.6.0 on Tue Sep 19 15:45:39 2017
    *mangle
    :PREROUTING ACCEPT [1076309:649025428]
    :INPUT ACCEPT [1065444:647777505]
    :FORWARD ACCEPT [0:0]
    ...
    • Sep 19 2017, 3:45 PM
    • 40 Lines
    Author: seirl
  • P178 (An Untitled Masterwork)
    root@orangerie:~# nc -l 8888 &
    [1] 267
    root@orangerie:~# ss -nltp | grep nc
    LISTEN 0 1 *:44854 *:* users:(("nc",pid=267,fd=3))
    root@orangerie:~# nc localhost 44854
    ...
    • Sep 19 2017, 3:39 PM
    • 7 Lines
    Author: seirl
  • P177 (An Untitled Masterwork)
    Host:
    antoine.pietri@swh-prod:~$ ip a
    ...
    • Sep 19 2017, 3:28 PM
    • 83 Lines
    Author: seirl
  • P176 vault web api factorization using django rest framework
    from django.http import HttpResponse
    from swh.web.api.utils import get_query_params, reverse
    from swh.web.api import apidoc as api_doc
    from swh.web.api.apiurls import api_route
    ...
    • Sep 12 2017, 2:35 PM
    • 116 Lines
    • Python
    Author: anlambert
  • P175 limiter old configuration
    limiter:
    headers_enabled: true
    strategy: moving-window
    storage_uri: "%{hiera('swh::deploy::webapp::redis')}"
    storage_options: {}
    ...
    • Sep 6 2017, 11:14 AM
    • 16 Lines
    • YAML
    Author: ardumont
  • P174 permission divergence for stretch/stable release
    ardumont@pergamon:/srv/softwareheritage/repository% ls -ld /srv/softwareheritage/repository/dists/*/main /srv/softwareheritage/repository/dists/*/main/* | grep "/stretch\|/stable"
    drwxr-sr-x 5 olasd swhdev 4096 Jun 19 17:11 /srv/softwareheritage/repository/dists/stable/main
    drwxr-sr-x 2 olasd swhdev 4096 Jul 27 09:45 /srv/softwareheritage/repository/dists/stable/main/binary-amd64
    drwxr-sr-x 2 olasd swhdev 4096 Jun 30 16:20 /srv/softwareheritage/repository/dists/stable/main/binary-i386
    drwxr-sr-x 2 olasd swhdev 4096 Jul 27 09:45 /srv/softwareheritage/repository/dists/stable/main/source
    ...
    • Aug 1 2017, 1:33 PM
    • 79 Lines
    Author: ardumont
  • P173 local errors in swh-scheduler
    psycopg2.ProgrammingError: relation "task_type" does not exist
    LINE 1: insert into task_type (type, description, backend_name, defa...
    ^
    ...
    • Jul 28 2017, 2:55 PM
    • 52 Lines
    Author: moranegg
  • P172 make rebuild-testdata
    touch filldb-stamp
    make[1]: Leaving directory '/home/morane/Documents/code/swh-environment/swh-storage/sql'
    make -C swh-storage-testdata distclean dumpdb
    make[1]: Entering directory '/home/morane/Documents/code/swh-environment/swh-storage-testdata'
    rm -f dumps/swh.dump dumps/swh-archiver.dump dumps/swh-scheduler.dump dumps/swh.sql dumps/swh-archiver.sql dumps/swh-scheduler.sql
    ...
    • Jul 28 2017, 2:44 PM
    • 14 Lines
    Author: moranegg
  • P171 test_revision_metadata_indexer failure
    ======================================================================
    FAIL: test_revision_metadata_indexer (test_metadata.Metadata)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/home/tony/work/inria/repo/swh/swh-environment/swh-indexer/swh/indexer/tests/test_metadata.py", line 295, in test_revision_metadata_indexer
    ...
    • Jul 28 2017, 1:02 PM
    • 65 Lines
    Author: ardumont
  • P170 gitorious - remaining loader git disk errors
    {'exception': "IsADirectoryError(21, 'Is a directory')", 'args': {'origin_url': 'https://gitorious.org/l2jserver2/l2jserver2.git', 'date': 'Wed, 30 Mar 2016 09:40:04 +0200', 'directory': '/srv/storage/space/mirrors/gitorious.org/mnt/repositories/l2jserver2/l2jserver2.git'}}
    {'exception': "KeyError(b'5btmp_obj_D3f6nk',)", 'args': {'origin_url': 'https://gitorious.org/gcc-ccvm/mainline.git', 'date': 'Wed, 30 Mar 2016 09:40:04 +0200', 'directory': '/srv/storage/space/mirrors/gitorious.org/mnt/repositories/gcc-ccvm/mainline.git'}}
    {'exception': "KeyError(b'b2tmp_obj_SfIbZp',)", 'args': {'origin_url': 'https://gitorious.org/webkit/achelliess-webkit.git', 'date': 'Wed, 30 Mar 2016 09:40:04 +0200', 'directory': '/srv/storage/space/mirrors/gitorious.org/mnt/repositories/webkit/achelliess-webkit.git'}}
    {'exception': "KeyError(b'b2tmp_obj_SfIbZp',)", 'args': {'origin_url': 'https://gitorious.org/webkit/barniz-webkit.git', 'date': 'Wed, 30 Mar 2016 09:40:04 +0200', 'directory': '/srv/storage/space/mirrors/gitorious.org/mnt/repositories/webkit/barniz-webkit.git'}}
    {'exception': "KeyError(b'b2tmp_obj_SfIbZp',)", 'args': {'origin_url': 'https://gitorious.org/webkit/bratsches-webkit.git', 'date': 'Wed, 30 Mar 2016 09:40:04 +0200', 'directory': '/srv/storage/space/mirrors/gitorious.org/mnt/repositories/webkit/bratsches-webkit.git'}}
    ...
    • Jul 28 2017, 11:06 AM
    • 29 Lines
    Author: ardumont
  • P169 One year plan
    Goal: Build semantic web of FOSS and promote SWH in citation and metadata workflows
    1. implementation metadata infrastructure/workflow (all tasks are under #Metadata workflow)
    - [x] strategy and design of metadata component [#T715]
    ...
    • Jul 18 2017, 6:10 PM
    • 107 Lines
    • Plain Text
    Author: moranegg
  • P168 Working on origin_metadata, external_metadata and revision_metadata tables
    -- Discovery of metadata during a listing, loading, deposit or external_catalog of an origin
    -- also provides a translation to a defined json schema using a translation tool (indexer_configuration_id)
    create table origin_metadata(
    id bigserial primary key-- PK object identifier
    origin_id bigint not null references origin(id),
    ...
    • Jul 11 2017, 11:59 AM
    • 15 Lines
    • SQL
    Author: moranegg
  • P167 tar-ing up many objects from the software heritage object storage on uffizi
    $ ssh uffizi.internal.softwareheritage.org
    $ cd /srv/softwareheritage/scratch/zack
    $ time tar -caf content-README-files.tar --files-from <(zcat ../lists/sha1-of-files-named-README.txt.gz | ./id-to-path.pl) -v --xform 's%.*heritage/%%'
    • Jun 26 2017, 3:31 PM
    • 3 Lines
    Author: zack
  • P166 (An Untitled Masterwork)
    begin;
    create temporary table _temp_debtags_sha256
    (
    sha256 sha256
    ) on commit drop;
    ...
    • Jun 26 2017, 2:43 PM
    • 17 Lines
    Author: seirl
  • P165 sha1-of-files-at-root-named-README-multiple-ext.sql
    SELECT distinct c.sha1
    FROM revision rev
    inner join directory dir on rev.directory = dir.id
    inner join directory_entry_file def on def.id = any(dir.file_entries)
    inner join content c on c.sha1_git = def.target
    ...
    • Jun 9 2017, 2:40 PM
    • 38 Lines
    • SQL
    Author: moranegg
  • P164 db upgrade modus operandi
    # tldr
    schema update | pglogical.replicate_ddl_command
    types, functions, index | manual on master and mirrors
    ...
    • Jun 2 2017, 3:48 PM
    • 16 Lines
    Author: ardumont
  • P163 average length, variance, standard deviation on ~42m language indexed contents
    date: Tue May 29 2017
    mean:
    ```
    ...
    • May 29 2017, 7:09 PM
    • 36 Lines
    Author: ardumont
  • P162 This comment currently resides at the top of the mercurial bundle_loader that I'm working on
    #
    # https://www.mercurial-scm.org/wiki/BundleFormat says:
    # "The new bundle format design is described on the BundleFormat2 page."
    #
    # https://www.mercurial-scm.org/wiki/BundleFormat2#Format_of_the_Bundle2_Container says:
    ...
    • May 17 2017, 12:44 PM
    • 23 Lines
    Author: fiendish
  • P161 vault schema
    create table dbversion
    (
    version int primary key,
    release timestamptz not null,
    description text not null
    ...
    • May 16 2017, 3:54 PM
    • 28 Lines
    • SQL
    Author: seirl
  • P160 vault schema
    create table dbversion
    (
    version int primary key,
    release timestamptz not null,
    description text not null
    ...
    • May 16 2017, 3:54 PM
    • 28 Lines
    Author: seirl
  • P159 ~/.config/swh/loader/git-updater.yml
    # storage:
    # cls: local
    # args:
    # db: 'service=swh-dev'
    # objstorage:
    ...
    • May 11 2017, 3:17 PM
    • 26 Lines
    Author: ardumont
  • P158 resize main db disk - problem analysis
    prado partition for the main db is full.
    #+BEGIN_SRC shell
    ardumont@prado:~% df -h /srv/softwareheritage/postgres
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/ssd-prado--postgres-part1 9.0T 9.0T 7.8G 100% /srv/softwareheritage/postgres
    ...
    • Apr 28 2017, 1:55 PM
    • 87 Lines
    Author: ardumont
  • P157 openpreserve/jhove is an origin which should be crawled
    No visit for that origin (according to the mirror swh db, the one that is read by the api):
    You are connected to database "softwareheritage" as user "guest" on host "somerset.internal.softwareheritage.org" at port "5433".
    SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
    softwareheritage=> select * from origin_visit where origin=34927854;
    origin | visit | date | status | metadata
    ...
    • Apr 28 2017, 11:18 AM
    • 24 Lines
    Author: ardumont
  • P155 Systemd unit for gunicorn+aiohttp objstorage deployment
    [Unit]
    Description=Remote Objstorage
    [Service]
    Type=simple
    ...
    • Apr 24 2017, 2:44 PM
    • 14 Lines
    Author: seirl
  • P154 example: filtering origin on latest top file name matching pom.xml
    select o.url, convert_from(def.name, 'utf-8')
    from origin o
    inner join occurrence occ on (occ.origin=o.id and occ.branch='refs/heads/master')
    inner join revision rev on (occ.target_type='revision' and occ.target=rev.id)
    inner join directory dir on rev.directory=dir.id
    ...
    • Apr 18 2017, 3:39 PM
    • 7 Lines
    • SQL
    Author: ardumont
  • P153 hglib vs dulwich finding blobs
    # hgrepo and gitrepo are identical except hg and git.
    # all times are on a fast SSD
    # this takes about 20 seconds
    hgblobs = {}
    ...
    • Apr 4 2017, 4:18 PM
    • 27 Lines
    Author: fiendish
  • P152 (An Untitled Masterwork)
    diff --git a/swh/objstorage/objstorage_pathslicing.py b/swh/objstorage/objstorage_pathslicing.py
    index 897a5f7..5a897e8 100644
    --- a/swh/objstorage/objstorage_pathslicing.py
    +++ b/swh/objstorage/objstorage_pathslicing.py
    @@ -37,12 +38,6 @@ def _write_obj_file(hex_obj_id, objstorage):
    ...
    • Mar 28 2017, 4:56 PM
    • 31 Lines
    • Diff
    Author: seirl
  • P151 `./bin/make-package -b swh.journal` failed to build with trouble about python3-celery deps
    report:
    -
    package: sbuild-build-depends-swh-journal-dummy
    version: 0.invalid.0
    architecture: amd64
    ...
    • Mar 24 2017, 1:16 PM
    • 37 Lines
    Author: ardumont
  • P150 (An Untitled Masterwork)
    ncalls tottime percall cumtime percall filename:lineno(function)
    1289 29.768 0.023 29.768 0.023 {built-in method psycopg2._psycopg._connect}
    4391 29.427 0.007 29.650 0.007 {method 'execute' of 'psycopg2.extensions.cursor' objects}
    1291 7.774 0.006 7.774 0.006 {method 'commit' of 'psycopg2.extensions.connection' objects}
    • Mar 22 2017, 12:25 PM
    • 4 Lines
    Author: seirl
  • P149 (An Untitled Masterwork)
    % git show --pretty=raw b531caa26c10faa10e3b7a727624b98a579aa6a7
    commit b531caa26c10faa10e3b7a727624b98a579aa6a7
    tree 44a773906c835d0d7d14835bce18e809c2fc6c6d
    parent 16686132c12dd22fd664bd117d99a976cd9874f2
    author Ian Cordasco <graffatcolmingov@gmail.com> 1465913233 -0500
    ...
    • Mar 21 2017, 5:38 PM
    • 35 Lines
    Author: seirl
  • P148 (An Untitled Masterwork)
    % git show --pretty=raw 3bf761be5802c726d869702d9fe9592581f4f0f1
    commit 3bf761be5802c726d869702d9fe9592581f4f0f1
    tree 44a773906c835d0d7d14835bce18e809c2fc6c6d
    parent 16686132c12dd22fd664bd117d99a976cd9874f2
    author Ian Cordasco <graffatcolmingov@gmail.com> 1465913233 -0500
    ...
    • Mar 21 2017, 5:38 PM
    • 52 Lines
    Author: seirl
  • P147 (An Untitled Masterwork)
    def _toposort(self, rev_by_id):
    children = collections.defaultdict(list)
    in_degree = collections.defaultdict(int)
    for rev_id, rev in rev_by_id.items():
    for parent in rev['parents']:
    ...
    • Mar 15 2017, 3:28 PM
    • 20 Lines
    • Python
    Author: seirl
  • P146 (An Untitled Masterwork)
    def _toposort(self, rev_by_id):
    children = collections.defaultdict(list)
    in_degree = collections.defaultdict(int)
    for rev_id, rev in rev_by_id.items():
    for parent in rev['parents']:
    ...
    • Mar 15 2017, 3:28 PM
    • 20 Lines
    Author: seirl
  • P145 (An Untitled Masterwork)
    ERROR:root:null value in column "fullname" violates not-null constraint
    DETAIL: Failing row contains (22022, null, null, null).
    CONTEXT: SQL statement "with t as (
    select distinct author_fullname as fullname, author_name as name, author_email as email from tmp_release
    ) insert into person (fullname, name, email)
    ...
    • Mar 14 2017, 3:00 PM
    • 124 Lines
    Author: seirl
  • P144 (An Untitled Masterwork)
    Traceback (most recent call last):
    File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
    File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
    ...
    • Mar 14 2017, 2:55 PM
    • 45 Lines
    Author: seirl
  • P143 (An Untitled Masterwork)
    def filecommands(self, rev, parent=None):
    if not parent:
    parent_dir = []
    else:
    parent_dir = self.dir_by_id[parent['directory']]
    ...
    • Mar 10 2017, 5:19 PM
    • 26 Lines
    • Python
    Author: seirl
  • P142 topo_sort
    def topo_sort(revisions):
    """revisions: dict id -> parents"""
    done = set()
    remaining = set(revisions)
    ...
    • Mar 7 2017, 4:28 PM
    • 18 Lines
    • Python
    Author: olasd
  • P141 (An Untitled Masterwork)
    16:01:29 <seirl> while scraping the metadata of a bitbucket repository, their api told me "Charlie Guse" was one of the contributors
    16:01:33 <seirl> i had never heard of him
    16:01:50 <seirl> so i was wondering how he could be in a repository
    16:01:54 <seirl> apparently it's this guy https://bitbucket.org/none/
    16:02:08 <seirl> and the bitbucket api sometimes returns "none" instead of null when it's missing some data
    ...
    • Mar 5 2017, 10:42 PM
    • 6 Lines
    Author: seirl
  • P140 Monolithic repository
    #!/bin/bash
    set -e
    rm -rf swh-merge
    ...
    • Mar 1 2017, 12:30 PM
    • 28 Lines
    • Bash Scripting
    Author: seirl
First
Prev
Next
About · Developer information · Legal