Page MenuHomeSoftware Heritage
Create Paste
Paste Active Pastes

Active Pastes
Use Results
Edit Query

  • P226 (An Untitled Masterwork)
    diff --git a/bin/deploy-on b/bin/deploy-on
    index 46d18e0..8e86207 100755
    --- a/bin/deploy-on
    +++ b/bin/deploy-on
    @@ -29,12 +29,16 @@ die_usage () {
    ...
    • Feb 16 2018, 2:37 PM
    • 33 Lines
    • Bash Scripting
    Author: seirl
  • P225 stop making me a liar! hglib's erratic behavior on bookmarks call.
    ```
    ardumont@uffizi:~/repo/hg% python3
    Python 3.5.3 (default, Jan 19 2017, 14:11:04)
    [GCC 6.3.0 20170118] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    ...
    • Feb 15 2018, 10:39 AM
    • 37 Lines
    Author: ardumont
  • P224 (An Untitled Masterwork)
    revision = {
    ...
    'parents': []
    }
    ...
    • Feb 15 2018, 9:38 AM
    • 11 Lines
    Author: fiendish
  • P223 (An Untitled Masterwork)
    python3 -m nose -sv --with-doctest ./swh/vault/tests
    pg_restore: [archiver (db)] Error while PROCESSING TOC:
    pg_restore: [archiver (db)] Error from TOC entry 240; 1259 2506400 SEQUENCE metadata_provider_id_seq ndandrim
    pg_restore: [archiver (db)] could not execute query: ERROR: syntax error at or near "AS"
    LINE 2: AS integer
    ...
    • Feb 12 2018, 5:04 PM
    • 19 Lines
    Author: seirl
  • P222 mercurial import: origins in error
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:36,421: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:38,665: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:41,157: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:43,165: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    {'args': {}, 'kwargs': {}, 'exception': '[2018-02-09 15:51:47,377: ERROR/Worker-1] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 969, in store_data\n self.send_batch_contents(self.get_contents())\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 645, in send_batch_contents\n packet_size_bytes=packet_size_bytes)\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 34, in send_in_packets\n for obj in objects:\n File "/usr/lib/python3/dist-packages/swh/loader/mercurial/bundle20_loader.py", line 182, in get_contents\n key_hash=ALGO\n File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in content_missing\n \'key_hash\': key_hash})\n File "/usr/lib/python3/dist-packages/swh/core/api.py", line 58, in post\n data = encode_data(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 18, in encode_data_client\n return msgpack_dumps(data)\n File "/usr/lib/python3/dist-packages/swh/core/serializers.py", line 140, in msgpack_dumps\n return msgpack.packb(data, use_bin_type=True, default=encode_types)\n File "/usr/lib/python3/dist-packages/msgpack/__init__.py", line 47, in packb\n return Packer(**kwargs).pack(o)\n File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3661)\n File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:3503)\n File "msgpack/_packer.pyx", line 192, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:2657)\n File "msgpack/_packer.pyx", line 228, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:3382)\nTypeError: can\'t serialize dict_values([])'}
    ...
    • Feb 12 2018, 3:33 PM
    • 397 Lines
    Author: ardumont
  • P221 sample configuration file for snippets/ardumont/kibana_fetch_logs.py
    indexes:
    - logstash-2018.02.09
    - logstash-2018.02.10
    - logstash-2018.02.11
    - logstash-2018.02.12
    ...
    • Feb 12 2018, 3:17 PM
    • 31 Lines
    • YAML
    Author: ardumont
  • P220 ~/.config/swh/loader/hg.yml
    storage:
    cls: local
    args:
    db: 'service=swh-dev'
    objstorage:
    ...
    • Feb 9 2018, 4:55 PM
    • 26 Lines
    Author: ardumont
  • P219 latest svn errors - investigate missing input
    {'args': {}, 'kwargs': {}, 'exception': '[2017-12-20 03:23:24,265: ERROR/Worker-522] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 240, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config[\'revision_packet_size\']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: \'utf-8\' codec can\'t decode byte 0xe7 in position 4: invalid continuation byte'}
    {'args': {}, 'kwargs': {}, 'exception': "[2017-12-20 23:53:54,166: ERROR/Worker-894] Eventful partial visit. Detail: [Errno 2] No such file or directory: b'/tmp/swh.loader.svn.6euhc0_1.tmp/hackabot/trunk/hooks/pubmsg/20-env'"}
    {'args': {}, 'kwargs': {}, 'exception': '[2017-12-20 23:53:54,166: ERROR/Worker-894] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config[\'revision_packet_size\']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 175, in close\n os.chmod(self.fullpath, 0o755)\nFileNotFoundError: [Errno 2] No such file or directory: b\'/tmp/swh.loader.svn.6euhc0_1.tmp/hackabot/trunk/hooks/pubmsg/20-env\'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 238, in process_swh_revisions\n \'id\': _id,\nswh.loader.svn.loader.SvnLoaderEventful: [Errno 2] No such file or directory: b\'/tmp/swh.loader.svn.6euhc0_1.tmp/hackabot/trunk/hooks/pubmsg/20-env\''}
    {'args': {}, 'kwargs': {}, 'exception': '[2017-12-21 10:59:38,296: ERROR/Worker-1084] Loading failure, updating to `partial` status\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 862, in load\n self.store_data()\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 313, in store_data\n start_from_scratch=self.start_from_scratch)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 503, in process_repository\n svnrepo, revision_start, revision_end, revision_parents)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 240, in process_swh_revisions\n raise e\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 219, in process_swh_revisions\n self.config[\'revision_packet_size\']):\n File "/usr/lib/python3/dist-packages/swh/core/utils.py", line 40, in grouper\n for _data in itertools.zip_longest(*args, fillvalue=None):\n File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 163, in process_svn_revisions\n for rev, nextrev, commit, new_objects, root_directory in gen_revs:\n File "/usr/lib/python3/dist-packages/swh/loader/svn/svn.py", line 267, in swh_hash_data_per_revision\n objects = self.swhreplay.compute_hashes(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 374, in compute_hashes\n self.replay(rev)\n File "/usr/lib/python3/dist-packages/swh/loader/svn/ra.py", line 359, in replay\n self.conn.replay(rev, rev+1, self.editor)\nUnicodeDecodeError: \'utf-8\' codec can\'t decode byte 0xb3 in position 10: invalid start byte'}
    {'args': {}, 'kwargs': {}, 'exception': "[2017-12-29 15:17:13,267: ERROR/Worker-1618] Eventful partial visit. Detail: [Errno 2] No such file or directory: b'/tmp/swh.loader.svn.qay2ee8q.tmp/qgcm/trunk/working/QGCM/cases/full/barotropic/with_beta/nk2/plot'"}
    ...
    • Jan 31 2018, 9:28 AM
    • 80 Lines
    Author: ardumont
  • P218 (An Untitled Masterwork)
    diff --git a/data/defaults.yaml b/data/defaults.yaml
    index 7885801..81234a0 100644
    --- a/data/defaults.yaml
    +++ b/data/defaults.yaml
    @@ -1352,12 +1352,7 @@ swh::deploy::vault::backend::http_timeout: 100000
    ...
    • Jan 30 2018, 3:02 PM
    • 45 Lines
    • Diff
    Author: seirl
  • P217 (An Untitled Masterwork)
    root@swh-test:~# cat setup_container.sh
    #!/bin/bash
    set -e
    ...
    • Jan 26 2018, 3:48 PM
    • 35 Lines
    • Bash Scripting
    Author: seirl
  • P216 Make html in swh-deposit/docs/ error
    Configuration error:
    There is a programable error in your configuration file:
    Traceback (most recent call last):
    File "/usr/local/lib/python3.4/dist-packages/sphinx/config.py", line 157, in __init__
    ...
    • Jan 24 2018, 5:08 PM
    • 33 Lines
    Author: moranegg
  • P215 IRC access factorization
    /msg chanserv access #swh-devel del rdicosmo
    /msg chanserv access #swh-devel del anlambert
    /msg chanserv access #swh-devel del grouss
    /msg chanserv access #swh-devel del ftigeot
    /msg chanserv access #swh-devel del moranegg
    ...
    • Jan 22 2018, 2:04 PM
    • 15 Lines
    Author: seirl
  • P213 (An Untitled Masterwork)
    root@swh-test:~# journalctl -u openvpn@softwareheritage.service
    -- Logs begin at Fri 2018-01-19 10:28:46 CET, end at Fri 2018-01-19 12:47:10 CET. --
    Jan 19 11:00:11 swh-test ovpn-softwareheritage[14829]: WARNING: this cipher's block size is less than 128 bit (64 bit). Consider using a --cipher with a larger block size.
    Jan 19 11:00:11 swh-test ovpn-softwareheritage[14829]: WARNING: this cipher's block size is less than 128 bit (64 bit). Consider using a --cipher with a larger block size.
    Jan 19 11:33:17 swh-test ovpn-softwareheritage[14829]: [louvre] Inactivity timeout (--ping-restart), restarting
    ...
    • Jan 19 2018, 12:47 PM
    • 19 Lines
    Author: seirl
  • P212 checking hal deposit archive
    Retrieve the deposit concerned:
    ```
    softwareheritage-deposit=> select dr.archive from deposit d inner join deposit_request dr on d.id=dr.deposit_id where external_id='hal-01243618' and dr.type_id=1;
    archive
    ...
    • Jan 18 2018, 3:46 PM
    • 36 Lines
    • Bash Scripting
    Author: ardumont
  • P211 data for deposit
    - model: deposit.depositclient
    fields:
    user_ptr_id: 1
    domain: archives-ouvertes.fr
    provider_url: https://hal.archives-ouvertes.fr/
    ...
    • Jan 10 2018, 12:10 PM
    • 7 Lines
    Author: moranegg
  • P210 (An Untitled Masterwork)
    File "/home/seirl/swh-environment/.venv/lib/python3.5/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
    File "/home/seirl/swh-environment/swh-web/swh/web/api/apidoc.py", line 108, in documented_view
    doc_data = self.get_doc_data(f)
    File "/home/seirl/swh-environment/swh-web/swh/web/api/apidoc.py", line 187, in get_doc_data
    ...
    • Jan 8 2018, 4:46 PM
    • 15 Lines
    Author: seirl
  • P208 ~/.pg_service.conf
    [swh]
    dbname=softwareheritage
    host=db.internal.softwareheritage.org
    user=guest
    ...
    • Jan 3 2018, 3:03 PM
    • 25 Lines
    Author: zack
  • P207 Convert a bundle from bundle10 to bundle20 - failure
    $ pwd
    /home/storage/hg/repo/not-bundle20/756015-ipv6
    $ hg bundle --rev 'bundle()' \  
    --base 'parents(roots(bundle()))' \
    -R HG20_bundle_none HG20_bundle_none_migrated \
    ...
    • Dec 21 2017, 10:06 AM
    • 37 Lines
    Author: ardumont
  • P206 current mercurial loader release issue 'cause no tag author
    Python 3.5.3 (default, Jan 19 2017, 14:11:04)
    [GCC 6.3.0 20170118] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> # remote repository
    ... origin_url = 'https://mercurial.tuxfamily.org/enchantepic2/enchantepic2hg'
    ...
    • Dec 20 2017, 12:53 PM
    • 82 Lines
    Author: ardumont
  • P205 listener failure on connection issues with the rabbitmq instance
    -- Unit swh-scheduler-listener.service has finished starting up.
    --
    -- The start-up result is done.
    Nov 30 22:23:15 saatchi python3[25458]: Traceback (most recent call last):
    Nov 30 22:23:15 saatchi python3[25458]: File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    ...
    • Dec 15 2017, 9:39 AM
    • 59 Lines
    Author: ardumont
  • P204 listener failure on running out of disk in prado (db host vm)
    -- Unit swh-scheduler-listener.service has finished starting up.
    --
    -- The start-up result is done.
    Dec 15 01:04:47 saatchi python3[26433]: Traceback (most recent call last):
    Dec 15 01:04:47 saatchi python3[26433]: File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    ...
    • Dec 15 2017, 9:37 AM
    • 67 Lines
    Author: ardumont
  • P203 Clean up wrong googlecode origins (googlecode instead of eclipselabs or apache-extras) - T897 related
    ```
    begin;
    select count(*) from origin where type='svn';
    ...
    • Dec 14 2017, 2:55 PM
    • 163 Lines
    Author: ardumont
  • P202 Clean up wrong origins (url is filled with the size instead of the actual origin url) - T896 related
    ```
    begin;
    create or replace function list_wrong_origins()
    returns setof origin.id%type
    ...
    • Dec 13 2017, 6:55 PM
    • 168 Lines
    • SQL
    Author: ardumont
  • P201 api: vault response in production mode
    antoine@guggenheim:~$ curl -i http://localhost:8000/api/1/vault/directory/d4a96ba891017d0d26c15e509b4e6515e40d75ee/raw/
    HTTP/1.1 503 Service Unavailable
    Server: gunicorn/19.6.0
    Date: Fri, 08 Dec 2017 13:57:25 GMT
    Connection: close
    ...
    • Dec 8 2017, 2:58 PM
    • 12 Lines
    Author: anlambert
  • P200 deposit load commands
    commands for running deposit load:
    - run sword server on port 5006: in swh-deposit: make run-dev
    - update deposit db in swh deposit: make db-migrate
    - run deposit with metadata locally in bin/: make new-complete
    ...
    • Dec 5 2017, 4:50 PM
    • 23 Lines
    Author: moranegg
  • P199 Vault task in scheduler
    insert into task_type(
    type,
    description,
    backend_name,
    default_interval, min_interval, max_interval, backoff_factor,
    ...
    • Dec 5 2017, 11:35 AM
    • 12 Lines
    Author: seirl
  • P198 improving indexing using derivative form of materialized view
    -- deal with new content
    create or replace function content_insert() returns trigger
    security definer
    language plpgsql
    as $$
    ...
    • Dec 2 2017, 1:06 PM
    • 62 Lines
    Author: ardumont
  • P197 errors in indexer when uffizi's disk's full -> OSError: [Errno 28] No space left on device
    {
    "_index": "logstash-2017.11.29",
    "_type": "journal",
    "_id": "AWAFo8W9l67zVNRhPjvh",
    "_score": null,
    ...
    • Dec 1 2017, 1:52 PM
    • 61 Lines
    Author: ardumont
  • P196 orangeriedev bin/deploy-on run
    bin/deploy-on --no-apt --no-master orangeriedev.internal.softwareheritage.org    master 
    *** swh-deploy: starting test run on orangeriedev.internal.softwareheritage.org...
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    ...
    • Nov 24 2017, 7:10 PM
    • 21 Lines
    Author: ardumont
  • P195 (An Untitled Masterwork)
    scaramouche ~  psql service=swh-replica
    psql (10.1, server 10.0)
    SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
    Type "help" for help.
    ...
    • Nov 17 2017, 2:16 PM
    • 38 Lines
    Author: zack
  • P194 spam about etckeeper on orangerie not being set
    /etc/cron.daily/etckeeper:
    *** Please tell me who you are.
    Run
    ...
    • Nov 15 2017, 10:03 AM
    • 14 Lines
    Author: ardumont
  • P190 Bulking directory listings with swh_directory_walk_many
    create or replace function swh_directory_walk_many(walked_dirs_id bytea[])
    returns setof directory_entry
    language sql
    stable
    as $$
    ...
    • Nov 9 2017, 2:48 PM
    • 33 Lines
    • SQL
    Author: seirl
  • P189 package build broken...
    On latest swh-environment
    ```
    bin/make-package -b swh-scheduler
    running sdist
    ...
    • Nov 8 2017, 3:57 PM
    • 955 Lines
    • Bash Scripting
    Author: ardumont
  • P188 (An Untitled Masterwork)
    softwareheritage-dev=# select perms, count(perms) from directory_entry_file group by perms;
    perms | count
    -------+-------
    33204 | 157
    40960 | 327
    ...
    • Nov 6 2017, 4:41 PM
    • 8 Lines
    Author: seirl
  • P187 ~/.config/swh/deposit/server.yml
    # dev option
    host: 127.0.0.1
    port: 5006
    # 200 Mib max size
    ...
    • Oct 31 2017, 2:34 PM
    • 14 Lines
    • YAML
    Author: ardumont
  • P186 (An Untitled Masterwork)
    *** swh-deploy: deploying recipes on pergamon.internal.softwareheritage.org...
    /usr/local/sbin/swh-puppet-master-deploy: line 4: /etc/puppet/environments/production/deploy.sh: No such file or directory
    Connection to pergamon.internal.softwareheritage.org closed.
    *** swh-deploy: master ok.
    *** swh-deploy: starting test run on pergamon...
    ...
    • Oct 30 2017, 1:26 PM
    • 9 Lines
    Author: zack
  • P185 load-git-disk.py
    #!/usr/bin/env python3
    import click
    ...
    • Oct 26 2017, 3:47 PM
    • 26 Lines
    Author: ardumont
  • P184 documentation build failure tryout
    ```
    ...swh-docs/docs $ make distclean
    Removing everything under '_build'...
    bin/ln-sphinx-subprojects --remove
    make -C images clean
    ...
    • Oct 21 2017, 5:38 PM
    • 394 Lines
    Author: ardumont
  • P182 scheduler - new task type for loading one-shot task (svn mount and load repository, load deposit archive)...
    insert into task_type(
    type,
    description,
    backend_name,
    default_interval, min_interval, max_interval, backoff_factor,
    ...
    • Oct 10 2017, 7:53 PM
    • 39 Lines
    • SQL
    Author: ardumont
  • P181 TEI sample for HAL
    <TEI><teiHeader><fileDesc><titleStmt><title>HAL TEI export of hal-01587083</title></titleStmt><publicationStmt><distributor>CCSD</distributor><availability status="restricted"><licence target="http://creativecommons.org/licenses/by/4.0/">Distributed under a Creative Commons Attribution 4.0 International License</licence></availability><date when="2017-10-03T17:21:03+02:00"/></publicationStmt><sourceDesc><p part="N">HAL API platform</p></sourceDesc></fileDesc></teiHeader><text><body><listBibl><biblFull><titleStmt><title xml:lang="en">questionnaire software metadata</title><author role="aut"><persName><forename type="first">Morane</forename><surname>Gruenpeter</surname></persName><email type="md5">7de56c632362954fa84172cad80afe4e</email><email type="domain">inria.fr</email><ptr type="url" target="moranegg.github.io"/><idno type="halauthorid">1556733</idno><affiliation ref="#struct-474639"/></author><editor role="depositor"><persName><forename>Morane</forename><surname>Gruenpeter</surname></persName><email...
    • Oct 10 2017, 11:19 AM
    • 1 Line
    Author: moranegg
  • P180 swh.model.from_disk nom nom nom
    In [1]: from swh.model.from_disk import Content, Directory, ignore_named_directories
    In [2]: d = Directory.from_disk(directory=b'swh-model', dir_filter=ignore_named_directories([b'.git', b'.coverage']), data=True)
    In [3]: hash1 = d.hash
    ...
    • Sep 20 2017, 11:33 PM
    • 39 Lines
    • Python
    Author: olasd
  • P179 (An Untitled Masterwork)
    # Generated by iptables-save v1.6.0 on Tue Sep 19 15:45:39 2017
    *mangle
    :PREROUTING ACCEPT [1076309:649025428]
    :INPUT ACCEPT [1065444:647777505]
    :FORWARD ACCEPT [0:0]
    ...
    • Sep 19 2017, 3:45 PM
    • 40 Lines
    Author: seirl
  • P178 (An Untitled Masterwork)
    root@orangerie:~# nc -l 8888 &
    [1] 267
    root@orangerie:~# ss -nltp | grep nc
    LISTEN 0 1 *:44854 *:* users:(("nc",pid=267,fd=3))
    root@orangerie:~# nc localhost 44854
    ...
    • Sep 19 2017, 3:39 PM
    • 7 Lines
    Author: seirl
  • P177 (An Untitled Masterwork)
    Host:
    antoine.pietri@swh-prod:~$ ip a
    ...
    • Sep 19 2017, 3:28 PM
    • 83 Lines
    Author: seirl
  • P176 vault web api factorization using django rest framework
    from django.http import HttpResponse
    from swh.web.api.utils import get_query_params, reverse
    from swh.web.api import apidoc as api_doc
    from swh.web.api.apiurls import api_route
    ...
    • Sep 12 2017, 2:35 PM
    • 116 Lines
    • Python
    Author: anlambert
  • P175 limiter old configuration
    limiter:
    headers_enabled: true
    strategy: moving-window
    storage_uri: "%{hiera('swh::deploy::webapp::redis')}"
    storage_options: {}
    ...
    • Sep 6 2017, 11:14 AM
    • 16 Lines
    • YAML
    Author: ardumont
  • P174 permission divergence for stretch/stable release
    ardumont@pergamon:/srv/softwareheritage/repository% ls -ld /srv/softwareheritage/repository/dists/*/main /srv/softwareheritage/repository/dists/*/main/* | grep "/stretch\|/stable"
    drwxr-sr-x 5 olasd swhdev 4096 Jun 19 17:11 /srv/softwareheritage/repository/dists/stable/main
    drwxr-sr-x 2 olasd swhdev 4096 Jul 27 09:45 /srv/softwareheritage/repository/dists/stable/main/binary-amd64
    drwxr-sr-x 2 olasd swhdev 4096 Jun 30 16:20 /srv/softwareheritage/repository/dists/stable/main/binary-i386
    drwxr-sr-x 2 olasd swhdev 4096 Jul 27 09:45 /srv/softwareheritage/repository/dists/stable/main/source
    ...
    • Aug 1 2017, 1:33 PM
    • 79 Lines
    Author: ardumont
  • P173 local errors in swh-scheduler
    psycopg2.ProgrammingError: relation "task_type" does not exist
    LINE 1: insert into task_type (type, description, backend_name, defa...
    ^
    ...
    • Jul 28 2017, 2:55 PM
    • 52 Lines
    Author: moranegg
  • P172 make rebuild-testdata
    touch filldb-stamp
    make[1]: Leaving directory '/home/morane/Documents/code/swh-environment/swh-storage/sql'
    make -C swh-storage-testdata distclean dumpdb
    make[1]: Entering directory '/home/morane/Documents/code/swh-environment/swh-storage-testdata'
    rm -f dumps/swh.dump dumps/swh-archiver.dump dumps/swh-scheduler.dump dumps/swh.sql dumps/swh-archiver.sql dumps/swh-scheduler.sql
    ...
    • Jul 28 2017, 2:44 PM
    • 14 Lines
    Author: moranegg
  • P171 test_revision_metadata_indexer failure
    ======================================================================
    FAIL: test_revision_metadata_indexer (test_metadata.Metadata)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/home/tony/work/inria/repo/swh/swh-environment/swh-indexer/swh/indexer/tests/test_metadata.py", line 295, in test_revision_metadata_indexer
    ...
    • Jul 28 2017, 1:02 PM
    • 65 Lines
    Author: ardumont
  • P170 gitorious - remaining loader git disk errors
    {'exception': "IsADirectoryError(21, 'Is a directory')", 'args': {'origin_url': 'https://gitorious.org/l2jserver2/l2jserver2.git', 'date': 'Wed, 30 Mar 2016 09:40:04 +0200', 'directory': '/srv/storage/space/mirrors/gitorious.org/mnt/repositories/l2jserver2/l2jserver2.git'}}
    {'exception': "KeyError(b'5btmp_obj_D3f6nk',)", 'args': {'origin_url': 'https://gitorious.org/gcc-ccvm/mainline.git', 'date': 'Wed, 30 Mar 2016 09:40:04 +0200', 'directory': '/srv/storage/space/mirrors/gitorious.org/mnt/repositories/gcc-ccvm/mainline.git'}}
    {'exception': "KeyError(b'b2tmp_obj_SfIbZp',)", 'args': {'origin_url': 'https://gitorious.org/webkit/achelliess-webkit.git', 'date': 'Wed, 30 Mar 2016 09:40:04 +0200', 'directory': '/srv/storage/space/mirrors/gitorious.org/mnt/repositories/webkit/achelliess-webkit.git'}}
    {'exception': "KeyError(b'b2tmp_obj_SfIbZp',)", 'args': {'origin_url': 'https://gitorious.org/webkit/barniz-webkit.git', 'date': 'Wed, 30 Mar 2016 09:40:04 +0200', 'directory': '/srv/storage/space/mirrors/gitorious.org/mnt/repositories/webkit/barniz-webkit.git'}}
    {'exception': "KeyError(b'b2tmp_obj_SfIbZp',)", 'args': {'origin_url': 'https://gitorious.org/webkit/bratsches-webkit.git', 'date': 'Wed, 30 Mar 2016 09:40:04 +0200', 'directory': '/srv/storage/space/mirrors/gitorious.org/mnt/repositories/webkit/bratsches-webkit.git'}}
    ...
    • Jul 28 2017, 11:06 AM
    • 29 Lines
    Author: ardumont
  • P169 One year plan
    Goal: Build semantic web of FOSS and promote SWH in citation and metadata workflows
    1. implementation metadata infrastructure/workflow (all tasks are under #Metadata workflow)
    - [x] strategy and design of metadata component [#T715]
    ...
    • Jul 18 2017, 6:10 PM
    • 107 Lines
    • Plain Text
    Author: moranegg
  • P168 Working on origin_metadata, external_metadata and revision_metadata tables
    -- Discovery of metadata during a listing, loading, deposit or external_catalog of an origin
    -- also provides a translation to a defined json schema using a translation tool (indexer_configuration_id)
    create table origin_metadata(
    id bigserial primary key-- PK object identifier
    origin_id bigint not null references origin(id),
    ...
    • Jul 11 2017, 11:59 AM
    • 15 Lines
    • SQL
    Author: moranegg
  • P167 tar-ing up many objects from the software heritage object storage on uffizi
    $ ssh uffizi.internal.softwareheritage.org
    $ cd /srv/softwareheritage/scratch/zack
    $ time tar -caf content-README-files.tar --files-from <(zcat ../lists/sha1-of-files-named-README.txt.gz | ./id-to-path.pl) -v --xform 's%.*heritage/%%'
    • Jun 26 2017, 3:31 PM
    • 3 Lines
    Author: zack
  • P166 (An Untitled Masterwork)
    begin;
    create temporary table _temp_debtags_sha256
    (
    sha256 sha256
    ) on commit drop;
    ...
    • Jun 26 2017, 2:43 PM
    • 17 Lines
    Author: seirl
  • P165 sha1-of-files-at-root-named-README-multiple-ext.sql
    SELECT distinct c.sha1
    FROM revision rev
    inner join directory dir on rev.directory = dir.id
    inner join directory_entry_file def on def.id = any(dir.file_entries)
    inner join content c on c.sha1_git = def.target
    ...
    • Jun 9 2017, 2:40 PM
    • 38 Lines
    • SQL
    Author: moranegg
  • P164 db upgrade modus operandi
    # tldr
    schema update | pglogical.replicate_ddl_command
    types, functions, index | manual on master and mirrors
    ...
    • Jun 2 2017, 3:48 PM
    • 16 Lines
    Author: ardumont
  • P163 average length, variance, standard deviation on ~42m language indexed contents
    date: Tue May 29 2017
    mean:
    ```
    ...
    • May 29 2017, 7:09 PM
    • 36 Lines
    Author: ardumont
  • P162 This comment currently resides at the top of the mercurial bundle_loader that I'm working on
    #
    # https://www.mercurial-scm.org/wiki/BundleFormat says:
    # "The new bundle format design is described on the BundleFormat2 page."
    #
    # https://www.mercurial-scm.org/wiki/BundleFormat2#Format_of_the_Bundle2_Container says:
    ...
    • May 17 2017, 12:44 PM
    • 23 Lines
    Author: fiendish
  • P161 vault schema
    create table dbversion
    (
    version int primary key,
    release timestamptz not null,
    description text not null
    ...
    • May 16 2017, 3:54 PM
    • 28 Lines
    • SQL
    Author: seirl
  • P160 vault schema
    create table dbversion
    (
    version int primary key,
    release timestamptz not null,
    description text not null
    ...
    • May 16 2017, 3:54 PM
    • 28 Lines
    Author: seirl
  • P159 ~/.config/swh/loader/git-updater.yml
    # storage:
    # cls: local
    # args:
    # db: 'service=swh-dev'
    # objstorage:
    ...
    • May 11 2017, 3:17 PM
    • 26 Lines
    Author: ardumont
  • P158 resize main db disk - problem analysis
    prado partition for the main db is full.
    #+BEGIN_SRC shell
    ardumont@prado:~% df -h /srv/softwareheritage/postgres
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/ssd-prado--postgres-part1 9.0T 9.0T 7.8G 100% /srv/softwareheritage/postgres
    ...
    • Apr 28 2017, 1:55 PM
    • 87 Lines
    Author: ardumont
  • P157 openpreserve/jhove is an origin which should be crawled
    No visit for that origin (according to the mirror swh db, the one that is read by the api):
    You are connected to database "softwareheritage" as user "guest" on host "somerset.internal.softwareheritage.org" at port "5433".
    SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
    softwareheritage=> select * from origin_visit where origin=34927854;
    origin | visit | date | status | metadata
    ...
    • Apr 28 2017, 11:18 AM
    • 24 Lines
    Author: ardumont
  • P155 Systemd unit for gunicorn+aiohttp objstorage deployment
    [Unit]
    Description=Remote Objstorage
    [Service]
    Type=simple
    ...
    • Apr 24 2017, 2:44 PM
    • 14 Lines
    Author: seirl
  • P154 example: filtering origin on latest top file name matching pom.xml
    select o.url, convert_from(def.name, 'utf-8')
    from origin o
    inner join occurrence occ on (occ.origin=o.id and occ.branch='refs/heads/master')
    inner join revision rev on (occ.target_type='revision' and occ.target=rev.id)
    inner join directory dir on rev.directory=dir.id
    ...
    • Apr 18 2017, 3:39 PM
    • 7 Lines
    • SQL
    Author: ardumont
  • P153 hglib vs dulwich finding blobs
    # hgrepo and gitrepo are identical except hg and git.
    # all times are on a fast SSD
    # this takes about 20 seconds
    hgblobs = {}
    ...
    • Apr 4 2017, 4:18 PM
    • 27 Lines
    Author: fiendish
  • P152 (An Untitled Masterwork)
    diff --git a/swh/objstorage/objstorage_pathslicing.py b/swh/objstorage/objstorage_pathslicing.py
    index 897a5f7..5a897e8 100644
    --- a/swh/objstorage/objstorage_pathslicing.py
    +++ b/swh/objstorage/objstorage_pathslicing.py
    @@ -37,12 +38,6 @@ def _write_obj_file(hex_obj_id, objstorage):
    ...
    • Mar 28 2017, 4:56 PM
    • 31 Lines
    • Diff
    Author: seirl
  • P151 `./bin/make-package -b swh.journal` failed to build with trouble about python3-celery deps
    report:
    -
    package: sbuild-build-depends-swh-journal-dummy
    version: 0.invalid.0
    architecture: amd64
    ...
    • Mar 24 2017, 1:16 PM
    • 37 Lines
    Author: ardumont
  • P150 (An Untitled Masterwork)
    ncalls tottime percall cumtime percall filename:lineno(function)
    1289 29.768 0.023 29.768 0.023 {built-in method psycopg2._psycopg._connect}
    4391 29.427 0.007 29.650 0.007 {method 'execute' of 'psycopg2.extensions.cursor' objects}
    1291 7.774 0.006 7.774 0.006 {method 'commit' of 'psycopg2.extensions.connection' objects}
    • Mar 22 2017, 12:25 PM
    • 4 Lines
    Author: seirl
  • P149 (An Untitled Masterwork)
    % git show --pretty=raw b531caa26c10faa10e3b7a727624b98a579aa6a7
    commit b531caa26c10faa10e3b7a727624b98a579aa6a7
    tree 44a773906c835d0d7d14835bce18e809c2fc6c6d
    parent 16686132c12dd22fd664bd117d99a976cd9874f2
    author Ian Cordasco <graffatcolmingov@gmail.com> 1465913233 -0500
    ...
    • Mar 21 2017, 5:38 PM
    • 35 Lines
    Author: seirl
  • P148 (An Untitled Masterwork)
    % git show --pretty=raw 3bf761be5802c726d869702d9fe9592581f4f0f1
    commit 3bf761be5802c726d869702d9fe9592581f4f0f1
    tree 44a773906c835d0d7d14835bce18e809c2fc6c6d
    parent 16686132c12dd22fd664bd117d99a976cd9874f2
    author Ian Cordasco <graffatcolmingov@gmail.com> 1465913233 -0500
    ...
    • Mar 21 2017, 5:38 PM
    • 52 Lines
    Author: seirl
  • P147 (An Untitled Masterwork)
    def _toposort(self, rev_by_id):
    children = collections.defaultdict(list)
    in_degree = collections.defaultdict(int)
    for rev_id, rev in rev_by_id.items():
    for parent in rev['parents']:
    ...
    • Mar 15 2017, 3:28 PM
    • 20 Lines
    • Python
    Author: seirl
  • P146 (An Untitled Masterwork)
    def _toposort(self, rev_by_id):
    children = collections.defaultdict(list)
    in_degree = collections.defaultdict(int)
    for rev_id, rev in rev_by_id.items():
    for parent in rev['parents']:
    ...
    • Mar 15 2017, 3:28 PM
    • 20 Lines
    Author: seirl
  • P145 (An Untitled Masterwork)
    ERROR:root:null value in column "fullname" violates not-null constraint
    DETAIL: Failing row contains (22022, null, null, null).
    CONTEXT: SQL statement "with t as (
    select distinct author_fullname as fullname, author_name as name, author_email as email from tmp_release
    ) insert into person (fullname, name, email)
    ...
    • Mar 14 2017, 3:00 PM
    • 124 Lines
    Author: seirl
  • P144 (An Untitled Masterwork)
    Traceback (most recent call last):
    File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
    File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
    ...
    • Mar 14 2017, 2:55 PM
    • 45 Lines
    Author: seirl
  • P143 (An Untitled Masterwork)
    def filecommands(self, rev, parent=None):
    if not parent:
    parent_dir = []
    else:
    parent_dir = self.dir_by_id[parent['directory']]
    ...
    • Mar 10 2017, 5:19 PM
    • 26 Lines
    • Python
    Author: seirl
  • P142 topo_sort
    def topo_sort(revisions):
    """revisions: dict id -> parents"""
    done = set()
    remaining = set(revisions)
    ...
    • Mar 7 2017, 4:28 PM
    • 18 Lines
    • Python
    Author: olasd
  • P141 (An Untitled Masterwork)
    16:01:29 <seirl> while scraping the metadata of a bitbucket repository, their api told me "Charlie Guse" was one of the contributors
    16:01:33 <seirl> i had never heard of him
    16:01:50 <seirl> so i was wondering how he could be in a repository
    16:01:54 <seirl> apparently it's this guy https://bitbucket.org/none/
    16:02:08 <seirl> and the bitbucket api sometimes returns "none" instead of null when it's missing some data
    ...
    • Mar 5 2017, 10:42 PM
    • 6 Lines
    Author: seirl
  • P140 Monolithic repository
    #!/bin/bash
    set -e
    rm -rf swh-merge
    ...
    • Mar 1 2017, 12:30 PM
    • 28 Lines
    • Bash Scripting
    Author: seirl
  • P139 (An Untitled Masterwork)
    seirl@grand-palais ~/prologin/sadm (git)-[master] % du -sh .git
    9.6M .git
    seirl@grand-palais ~/prologin/sadm (git)-[master] % git fast-export --all --signed-tags=strip | wc -c | numfmt --to=iec-i
    25Mi
    seirl@grand-palais ~/prologin/sadm (git)-[master] % git fast-export --all --signed-tags=strip | gzip | wc -c | numfmt --to=iec-i
    ...
    • Feb 27 2017, 4:08 PM
    • 6 Lines
    Author: seirl
  • P138 (An Untitled Masterwork)
    antoine@elune /tmp % tar xvf lol.tar
    675b07e73a367e3e9c927fabad82b03c86be0e03/
    675b07e73a367e3e9c927fabad82b03c86be0e03/symlink
    675b07e73a367e3e9c927fabad82b03c86be0e03/realfile
    antoine@elune /tmp % cd 675b07e73a367e3e9c927fabad82b03c86be0e03
    ...
    • Feb 17 2017, 2:56 PM
    • 13 Lines
    Author: seirl
  • P137 (An Untitled Masterwork)
    antoine@elune /tmp % tar xvf lol.tar
    675b07e73a367e3e9c927fabad82b03c86be0e03/
    675b07e73a367e3e9c927fabad82b03c86be0e03/symlink
    675b07e73a367e3e9c927fabad82b03c86be0e03/realfile
    antoine@elune /tmp % cd 675b07e73a367e3e9c927fabad82b03c86be0e03
    ...
    • Feb 17 2017, 2:56 PM
    • 11 Lines
    Author: seirl
  • P135 u wot m8
    % curl -X POST http://0.0.0.0:5000/vault/directory/f5ee4ee472893773863e14831f9f1e0bb682a04c/
    Äacbuiltins
    OSError
    qcbuiltins
    ConnectionRefusedError
    ...
    • Feb 16 2017, 2:38 PM
    • 6 Lines
    Author: seirl
  • P134 (An Untitled Masterwork)
    ======================================================================
    ERROR: Run archiver on a missing content should archive it.
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/home/antoine/softwareheritage/swh-environment/swh-storage/swh/storage/tests/test_archiver.py", line 195, in archive_missing_content
    ...
    • Feb 15 2017, 2:32 PM
    • 22 Lines
    Author: seirl
  • P133 Lingering IRC session
    # Start it
    screen -S irc
    weechat
    # Inside weechat
    ...
    • Feb 14 2017, 4:02 PM
    • 10 Lines
    • Bash Scripting
    Author: seirl
  • P132 error when posting to forge through api calls
    From either swh-mirror-forge or python3 toplevel:
    >>> r = requests.post('https://forge.softwareheritage.org/api/diffusion.repository.search', data={'api.token': 'api-token-redacted', 'attachments': {'uris': True}, 'constraints': {'ids': [78]}})
    >>> r.json()
    {'error_code': 'ERR-CONDUIT-CORE', 'error_info': 'Argument 1 passed to PhabricatorSearchField::getValueExistsInConduitRequest() must be of the type array, string given, called in /srv/phabricator/phabricator/src/applications/search/engine/PhabricatorApplicationSearchEngine.php on line 1118 and defined', 'result': None}
    ...
    • Feb 8 2017, 5:12 PM
    • 27 Lines
    Author: ardumont
  • P128 Mirroring forge from phab to github
    #+title: Mirroring forge from phab to github
    #+author: ardumont, olasd
    * Need
    ...
    • Jan 27 2017, 12:29 PM
    • 102 Lines
    • Plain Text
    Author: ardumont
  • P125 pg_logical issue?
    Jan 10 16:53:27 worker01 python3[14888]: [2017-01-10 16:53:27,252: INFO/MainProcess] Received task: swh.loader.svn.tasks.MountAndLoadSvnRepositoryTsk[441d07a9-014b-4925-b81d-00f97a31d730]
    Jan 10 16:53:27 worker01 python3[14888]: [2017-01-10 16:53:27,476: INFO/Worker-1] Archive to mount and load /srv/storage/space/mirrors/code.google.com/sources/v2/apache-extras.org/c/cassandra-gui/cassandra-gui-repo.svndump.gz
    Jan 10 16:53:34 worker01 python3[14888]: [2017-01-10 16:53:34,163: INFO/Worker-1] [revision_start-revision_end]: [1-87]
    Jan 10 16:53:34 worker01 python3[14888]: [2017-01-10 16:53:34,186: INFO/Worker-1] Processing {'uuid': b'd8744a28-bcd7-4428-a461-397a9a970a4c', 'remote_url': 'file:///tmp/tmp.ocn5fi75.swh.loader.svn/cassandra-gui', 'swh-origin': 49908314, 'local_url': b'/tmp/tmp.6rze6i8h.swh.loader/cassandra-gui'}.
    Jan 10 16:53:34 worker01 python3[14888]: [2017-01-10 16:53:34,489: ERROR/MainProcess] Task swh.loader.svn.tasks.MountAndLoadSvnRepositoryTsk[441d07a9-014b-4925-b81d-00f97a31d730] raised unexpected: ProgrammingError('permission denied for schema pglogical\nCONTEXT: SQL statement "\n\tcreate temporary table tmp_content\n\t (like content including defaults)\n\t on commit drop;\n alter table tmp_content drop column if exists object_id;\n\t"\nPL/pgSQL function swh_mktemp(regclass) line 3 at EXECUTE\n',)
    ...
    • Jan 10 2017, 5:59 PM
    • 2,500 Lines
    Author: ardumont
  • P124 (An Untitled Masterwork)
    *** swh-deploy: starting test run on pergamon...
    Warning: Unable to fetch my node definition, but the agent run will continue:
    Warning: Error 400 on SERVER: Could not retrieve facts for pergamon.softwareheritage.org: could not connect to server: Connection refused
    Is the server running on host "localhost" (::1) and accepting
    TCP/IP connections on port 5432?
    ...
    • Jan 1 2017, 7:44 PM
    • 24 Lines
    Author: zack
  • P123 (An Untitled Masterwork)
    ls -al *azure*
    -rw-r--r-- 1 ardumont swhdev 146450305 set 23 14:11 contents-sha1-to-azure-2.txt.gz
    -rw-r--r-- 1 ardumont swhdev 99 set 23 14:10 contents-sha1-to-azure-2.txt.gz.sha1sum
    -rw-r--r-- 1 ardumont swhdev 175336017 ott 13 15:24 contents-sha1-to-azure-3.txt.gz
    -rw-r--r-- 1 ardumont swhdev 99 ott 13 16:42 contents-sha1-to-azure-3.txt.gz.sha1sum
    ...
    • Dec 19 2016, 3:24 PM
    • 13 Lines
    Author: zack
  • P122 configuration's new potential design for storage
    # local storage with a pathslicing objstorage
    storage:
    cls: local
    args:
    db: 'service=swh-dev'
    ...
    • Dec 15 2016, 3:40 PM
    • 16 Lines
    • YAML
    Author: ardumont
  • P121 number of objects per archive
    select copies.key as archive, count(content_id)
    from content_archive, jsonb_each(copies) as copies
    where copies.value->>'status' = 'present'
    group by copies.key;
    • Dec 15 2016, 2:32 PM
    • 4 Lines
    • PostgreSQL
    Author: zack
  • P120 check reader git did its job
    #+title: Check reader git's data
    * Update swh.loader.git.reader to list sha1s from origin
    https://forge.softwareheritage.org/rDLDGae4606dbb59b0c588e81191f6356c4cba12c64e3
    ...
    • Nov 4 2016, 3:14 PM
    • 108 Lines
    Author: ardumont
  • P119 Problem when running reader-git in uffizi
    Oct 26 14:09:31 uffizi python3[16259]: [2016-10-26 14:09:31,108: ERROR/MainProcess] Task swh.loader.git.tasks.ReaderGitRepository[a9df82ad-802f-4271-8c49-73d8ed4e4746] raised unexpected: UnboundLocalError("local variable 'err' referenced before assignment",)
    Oct 26 14:09:31 uffizi python3[16259]: Traceback (most recent call last):
    Oct 26 14:09:31 uffizi python3[16259]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 240, in trace_task
    Oct 26 14:09:31 uffizi python3[16259]: R = retval = fun(*args, **kwargs)
    Oct 26 14:09:31 uffizi python3[16259]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 437, in __protected_call__
    ...
    • Oct 26 2016, 4:18 PM
    • 52 Lines
    Author: ardumont
  • P118 Language unknown repartition
    softwareheritage=> select convert_from(mimetype, 'utf-8') as mimetype, count(*) as count from content_language cl inner join content_mimetype using(id) where lang='unknown' group by mimetype order by count desc;
    mimetype | count
    -------------------------------+--------
    text/plain | 491510
    text/x-ruby | 93716
    ...
    • Oct 24 2016, 1:55 PM
    • 51 Lines
    • SQL
    Author: ardumont
  • P117 swh mimetype recognized repartition - updated
    softwareheritage=> select mimetype, count, percent from swh_content_mimetype_text_repartition();
    mimetype | count | percent
    ------------------------------------+---------+-----------
    text/plain | 2764656 | 37.3915
    text/x-c | 1603002 | 21.6803
    ...
    • Oct 21 2016, 11:36 AM
    • 112 Lines
    • SQL
    Author: ardumont
  • P116 swh language recognized repartition
    softwareheritage=> select lang, count, percent from swh_content_language_repartition();
    | lang | count | percent |
    |---------------------+---------+---------|
    | python | 1130544 | 16.1095 |
    | javascript+lasso | 1078207 | 15.3637 |
    ...
    • Oct 21 2016, 11:29 AM
    • 117 Lines
    • SQL
    Author: ardumont
  • P115 ctags schema + output sample
    # schema
    -- SWH DB schema upgrade
    -- from_version: 88
    -- to_version: 89
    ...
    • Oct 20 2016, 9:33 AM
    • 316 Lines
    Author: ardumont
  • P114 some erratic errors on indexer computations (orchestrator, mimetype, language)
    Oct 11 23:03:26 worker01.euwest.azure python3[49773]: [2016-10-11 23:03:26,156: ERROR/MainProcess] Task swh.indexer.tasks.SWHOrchestratorTask[2561d4d2-1152-46b9-ba81-a3756d8b49be] raised unexpected: StorageAPIError(ConnectionError(Protocol
    Error('Connection aborted.', gaierror(-2, 'Name or service not known')),),)
    Oct 11 23:03:26 worker01.euwest.azure python3[49773]: Traceback (most recent call last):
    Oct 11 23:03:26 worker01.euwest.azure python3[49773]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 240, in trace_task
    Oct 11 23:03:26 worker01.euwest.azure python3[49773]: R = retval = fun(*args, **kwargs)
    ...
    • Oct 12 2016, 1:04 AM
    • 60 Lines
    Author: ardumont
First
Prev
Next
About · Developer information · Legal