Page MenuHomeSoftware Heritage
Create Paste
Paste Active Pastes

Active Pastes
Use Results
Edit Query

  • P132 error when posting to forge through api calls
    From either swh-mirror-forge or python3 toplevel:
    >>> r = requests.post('https://forge.softwareheritage.org/api/diffusion.repository.search', data={'api.token': 'api-token-redacted', 'attachments': {'uris': True}, 'constraints': {'ids': [78]}})
    >>> r.json()
    {'error_code': 'ERR-CONDUIT-CORE', 'error_info': 'Argument 1 passed to PhabricatorSearchField::getValueExistsInConduitRequest() must be of the type array, string given, called in /srv/phabricator/phabricator/src/applications/search/engine/PhabricatorApplicationSearchEngine.php on line 1118 and defined', 'result': None}
    ...
    • Feb 8 2017, 5:12 PM
    • 27 Lines
    Author: ardumont
  • P128 Mirroring forge from phab to github
    #+title: Mirroring forge from phab to github
    #+author: ardumont, olasd
    * Need
    ...
    • Jan 27 2017, 12:29 PM
    • 102 Lines
    • Plain Text
    Author: ardumont
  • P125 pg_logical issue?
    Jan 10 16:53:27 worker01 python3[14888]: [2017-01-10 16:53:27,252: INFO/MainProcess] Received task: swh.loader.svn.tasks.MountAndLoadSvnRepositoryTsk[441d07a9-014b-4925-b81d-00f97a31d730]
    Jan 10 16:53:27 worker01 python3[14888]: [2017-01-10 16:53:27,476: INFO/Worker-1] Archive to mount and load /srv/storage/space/mirrors/code.google.com/sources/v2/apache-extras.org/c/cassandra-gui/cassandra-gui-repo.svndump.gz
    Jan 10 16:53:34 worker01 python3[14888]: [2017-01-10 16:53:34,163: INFO/Worker-1] [revision_start-revision_end]: [1-87]
    Jan 10 16:53:34 worker01 python3[14888]: [2017-01-10 16:53:34,186: INFO/Worker-1] Processing {'uuid': b'd8744a28-bcd7-4428-a461-397a9a970a4c', 'remote_url': 'file:///tmp/tmp.ocn5fi75.swh.loader.svn/cassandra-gui', 'swh-origin': 49908314, 'local_url': b'/tmp/tmp.6rze6i8h.swh.loader/cassandra-gui'}.
    Jan 10 16:53:34 worker01 python3[14888]: [2017-01-10 16:53:34,489: ERROR/MainProcess] Task swh.loader.svn.tasks.MountAndLoadSvnRepositoryTsk[441d07a9-014b-4925-b81d-00f97a31d730] raised unexpected: ProgrammingError('permission denied for schema pglogical\nCONTEXT: SQL statement "\n\tcreate temporary table tmp_content\n\t (like content including defaults)\n\t on commit drop;\n alter table tmp_content drop column if exists object_id;\n\t"\nPL/pgSQL function swh_mktemp(regclass) line 3 at EXECUTE\n',)
    ...
    • Jan 10 2017, 5:59 PM
    • 2,500 Lines
    Author: ardumont
  • P124 (An Untitled Masterwork)
    *** swh-deploy: starting test run on pergamon...
    Warning: Unable to fetch my node definition, but the agent run will continue:
    Warning: Error 400 on SERVER: Could not retrieve facts for pergamon.softwareheritage.org: could not connect to server: Connection refused
    Is the server running on host "localhost" (::1) and accepting
    TCP/IP connections on port 5432?
    ...
    • Jan 1 2017, 7:44 PM
    • 24 Lines
    Author: zack
  • P123 (An Untitled Masterwork)
    ls -al *azure*
    -rw-r--r-- 1 ardumont swhdev 146450305 set 23 14:11 contents-sha1-to-azure-2.txt.gz
    -rw-r--r-- 1 ardumont swhdev 99 set 23 14:10 contents-sha1-to-azure-2.txt.gz.sha1sum
    -rw-r--r-- 1 ardumont swhdev 175336017 ott 13 15:24 contents-sha1-to-azure-3.txt.gz
    -rw-r--r-- 1 ardumont swhdev 99 ott 13 16:42 contents-sha1-to-azure-3.txt.gz.sha1sum
    ...
    • Dec 19 2016, 3:24 PM
    • 13 Lines
    Author: zack
  • P122 configuration's new potential design for storage
    # local storage with a pathslicing objstorage
    storage:
    cls: local
    args:
    db: 'service=swh-dev'
    ...
    • Dec 15 2016, 3:40 PM
    • 16 Lines
    • YAML
    Author: ardumont
  • P121 number of objects per archive
    select copies.key as archive, count(content_id)
    from content_archive, jsonb_each(copies) as copies
    where copies.value->>'status' = 'present'
    group by copies.key;
    • Dec 15 2016, 2:32 PM
    • 4 Lines
    • PostgreSQL
    Author: zack
  • P120 check reader git did its job
    #+title: Check reader git's data
    * Update swh.loader.git.reader to list sha1s from origin
    https://forge.softwareheritage.org/rDLDGae4606dbb59b0c588e81191f6356c4cba12c64e3
    ...
    • Nov 4 2016, 3:14 PM
    • 108 Lines
    Author: ardumont
  • P119 Problem when running reader-git in uffizi
    Oct 26 14:09:31 uffizi python3[16259]: [2016-10-26 14:09:31,108: ERROR/MainProcess] Task swh.loader.git.tasks.ReaderGitRepository[a9df82ad-802f-4271-8c49-73d8ed4e4746] raised unexpected: UnboundLocalError("local variable 'err' referenced before assignment",)
    Oct 26 14:09:31 uffizi python3[16259]: Traceback (most recent call last):
    Oct 26 14:09:31 uffizi python3[16259]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 240, in trace_task
    Oct 26 14:09:31 uffizi python3[16259]: R = retval = fun(*args, **kwargs)
    Oct 26 14:09:31 uffizi python3[16259]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 437, in __protected_call__
    ...
    • Oct 26 2016, 4:18 PM
    • 52 Lines
    Author: ardumont
  • P118 Language unknown repartition
    softwareheritage=> select convert_from(mimetype, 'utf-8') as mimetype, count(*) as count from content_language cl inner join content_mimetype using(id) where lang='unknown' group by mimetype order by count desc;
    mimetype | count
    -------------------------------+--------
    text/plain | 491510
    text/x-ruby | 93716
    ...
    • Oct 24 2016, 1:55 PM
    • 51 Lines
    • SQL
    Author: ardumont
  • P117 swh mimetype recognized repartition - updated
    softwareheritage=> select mimetype, count, percent from swh_content_mimetype_text_repartition();
    mimetype | count | percent
    ------------------------------------+---------+-----------
    text/plain | 2764656 | 37.3915
    text/x-c | 1603002 | 21.6803
    ...
    • Oct 21 2016, 11:36 AM
    • 112 Lines
    • SQL
    Author: ardumont
  • P116 swh language recognized repartition
    softwareheritage=> select lang, count, percent from swh_content_language_repartition();
    | lang | count | percent |
    |---------------------+---------+---------|
    | python | 1130544 | 16.1095 |
    | javascript+lasso | 1078207 | 15.3637 |
    ...
    • Oct 21 2016, 11:29 AM
    • 117 Lines
    • SQL
    Author: ardumont
  • P115 ctags schema + output sample
    # schema
    -- SWH DB schema upgrade
    -- from_version: 88
    -- to_version: 89
    ...
    • Oct 20 2016, 9:33 AM
    • 316 Lines
    Author: ardumont
  • P114 some erratic errors on indexer computations (orchestrator, mimetype, language)
    Oct 11 23:03:26 worker01.euwest.azure python3[49773]: [2016-10-11 23:03:26,156: ERROR/MainProcess] Task swh.indexer.tasks.SWHOrchestratorTask[2561d4d2-1152-46b9-ba81-a3756d8b49be] raised unexpected: StorageAPIError(ConnectionError(Protocol
    Error('Connection aborted.', gaierror(-2, 'Name or service not known')),),)
    Oct 11 23:03:26 worker01.euwest.azure python3[49773]: Traceback (most recent call last):
    Oct 11 23:03:26 worker01.euwest.azure python3[49773]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 240, in trace_task
    Oct 11 23:03:26 worker01.euwest.azure python3[49773]: R = retval = fun(*args, **kwargs)
    ...
    • Oct 12 2016, 1:04 AM
    • 60 Lines
    Author: ardumont
  • P113 error on cache population
    Oct 11 22:55:39 uffizi python3[5012]: [2016-10-11 22:55:39,806: ERROR/MainProcess] Task swh.storage.provenance.tasks.PopulateCacheContentRevision[0be5d71e-6a3f-49db-8c45-03c5f11e7326] raised unexpected: StorageAPIError(ConnectionError(ProtocolError('Connection aborted.', ConnectionRefusedError(111, 'Connection refused')),),)
    Oct 11 22:55:39 uffizi python3[5012]: Traceback (most recent call last):
    Oct 11 22:55:39 uffizi python3[5012]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 240, in trace_task
    Oct 11 22:55:39 uffizi python3[5012]: R = retval = fun(*args, **kwargs)
    Oct 11 22:55:39 uffizi python3[5012]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 437, in __protected_call__
    ...
    • Oct 12 2016, 1:00 AM
    • 13 Lines
    Author: ardumont
  • P112 [invalid] connection error when producing message for orchestrator - ardumont not in swhworker group!
    ardumont@worker01:~% pigz -dc /srv/storage/space/lists/contents-sha1-to-azure.txt.gz | head | SWH_WORKER_INSTANCE=swh_indexer_orchestrator python3 -m swh.indexer.producer --batch 100 --limit 1000
    10 sent - [b'=\x83=\x02\x14=\x1c\xad\xb8UjL\x91\xcc\x15C$B\xdel', ...]
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/kombu/utils/__init__.py", line 420, in __call__
    return self.__value__
    ...
    • Oct 10 2016, 4:07 PM
    • 88 Lines
    Author: ardumont
  • P111 poc azure - Notes on technology
    #+title: POC azure
    #+author: ardumont
    * Goal
    ...
    • Sep 27 2016, 12:41 AM
    • 131 Lines
    Author: ardumont
  • P110 remains some noise left in archiver db
    Sep 23 16:33:28 uffizi python3[6770]: [2016-09-23 16:33:28,159: ERROR/MainProcess] Task swh.storage.archiver.tasks.SWHArchiverToBackendTask[f3d55225-cd8b-4bae-bba8-5dc43a05e6cb] raised unexpected: KeyError('<swh.objstorage.api.client.RemoteObjStorage object at 0x7fb0b3f7bfd0>',)
    Sep 23 16:33:28 uffizi python3[6770]: Traceback (most recent call last):
    Sep 23 16:33:28 uffizi python3[6770]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 240, in trace_task
    Sep 23 16:33:28 uffizi python3[6770]: R = retval = fun(*args, **kwargs)
    ...
    • Sep 23 2016, 4:39 PM
    • 31 Lines
    Author: ardumont
  • P109 query wait timeout when too many worker for copy to azure
    Sep 22 20:41:47 uffizi python3[28445]: [2016-09-22 20:41:47,054: ERROR/MainProcess] Task swh.storage.archiver.tasks.SWHArchiverToBackendTask[cdfe6d71-704d-4c8a-b664-e9e8fe9d3308] raised unexpected: DatabaseError('query_wait_timeout\nSSL co
    nnection has been closed unexpectedly\n',)
    Sep 22 20:41:47 uffizi python3[28445]: Traceback (most recent call last):
    Sep 22 20:41:47 uffizi python3[28445]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 240, in trace_task
    Sep 22 20:41:47 uffizi python3[28445]: R = retval = fun(*args, **kwargs)
    ...
    • Sep 23 2016, 1:51 PM
    • 21 Lines
    Author: ardumont
  • P108 pip3 completion for zsh
    # somewhere in .zshrc
    . <(pip3 completion --zsh | sed -e 's/pip/pip3/g')
    • Sep 9 2016, 10:39 AM
    • 3 Lines
    Author: ardumont
  • P104 provenance - populate caches (dev) - compatible schema v085
    create or replace function populate_provenance_caches()
    returns void
    language plpgsql
    as $$
    declare
    ...
    • Sep 1 2016, 6:40 PM
    • 23 Lines
    Author: ardumont
  • P103 [Fixed] packaging swh-storage: problem with transitive runtime dependency from swh.core on python3-yaml
    TL;DR Problem with swh-core/requirements.txt with the wrong package name
    Fixed in rDCOREfd7f8845a6a710fe80cb91dd85deab3826f86907
    creating /<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/swh
    creating /<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build/swh/storage
    ...
    • Aug 23 2016, 4:16 PM
    • 194 Lines
    Author: ardumont
  • P101 vault-blueprint.md
    Software Heritage Vault
    =======================
    Software source code **objects**---e.g., individual source code files,
    tarballs, commits, tagged releases, etc.---are stored in the Software Heritage
    ...
    • Aug 12 2016, 11:34 AM
    • 139 Lines
    • Plain Text
    Author: zack
  • P98 puppet deployment on moma failed due to possible python3-swh.objstorage broken dependency
     tony  ⋯  repo  swh  puppet-environment  ./bin/deploy-on moma.internal.softwareheritage.org    master 
    *** swh-deploy: deploying recipes on pergamon.internal.softwareheritage.org...
    [R10K::Task::Deployment::DeployEnvironments - INFO] Loading environments from all sources
    [R10K::Task::Environment::Deploy - NOTICE] Deploying environment production
    [R10K::Task::Puppetfile::Sync - INFO] Loading modules from Puppetfile into queue
    ...
    • Jul 18 2016, 12:11 PM
    • 354 Lines
    Author: ardumont
  • P97 Pb in debian packaging on latest testing...
     tony  ⋯  repo  swh  swh-environment  ./bin/make-package -b swh-loader-debian
    running sdist
    running egg_info
    creating swh.loader.debian.egg-info
    writing swh.loader.debian.egg-info/PKG-INFO
    ...
    • Jul 6 2016, 4:33 PM
    • 900 Lines
    Author: ardumont
  • P96 Lazarus Extended
    Extended Components are LAZARUS and DELPHI components.
    The Extended Components are part of a Very Rapid Applications Development framework.
    There are most visual components in the package. The color change when we focus on them. They are :
    ...
    • Jul 4 2016, 4:49 PM
    • 23 Lines
    Author: matthius
  • P94 Get a route's usage count over 24hrs from uswgi logs
    # Adapt route, time and date accordingly
    grep -cE "((Thu Jun 30 (09|1[0-9]|2[0-4]):[0-9]{2}:[0-9]{2})|(Fri Jul 1 (0[0-8]):[0-9]{2}:[0-9]{2})).*/api/1/search/"
    • Jul 1 2016, 3:26 PM
    • 3 Lines
    • Bash Scripting
    Author: jbertran
  • P90 [Answer: No] hash revision computation difference?
    swh-revhash computes the raw commit hash (as in git) using our swh.model api.
    ```
    $ swh-revhash 'tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904\nparent 22c0fa5195a53f2e733ec75a9b6e9d1624a8b771\nauthor seanius <seanius@3187e211-bb14-4c82-9596-0b59d67cd7f4> 1138341044 +0000\ncommitter seanius <seanius@3187e211-bb14-4c82-9596-0b59d67cd7f4> 1138341044 +0000\n\nmaking dir structure...\n'
    17a631d474f49bbebfdf3d885dcde470d7faafd7
    ...
    • Jun 23 2016, 11:57 AM
    • 28 Lines
    Author: ardumont
  • P89 .ssh/config
    Host swh-gandi
    User 4215210
    Port 22
    Hostname sftp.dc0.gpaas.net
    Identityfile ~/.ssh/id-rsa-swh-gandi
    • Jun 22 2016, 2:58 PM
    • 5 Lines
    Author: ardumont
  • P88 git-svn - empty author sample
    asf repository, svn revision 10064:
    ```
    commit 5c8e2dc28e695178fda863b8a4bc3e1b24911c42
    tree 0db7283ffbd534bc767052f06529210e647005df
    ...
    • Jun 21 2016, 3:19 PM
    • 11 Lines
    Author: ardumont
  • P87 Storage client error
    ======================================================================
    ERROR: revision_log_by (ui.tests.test_backend.BackendTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/home/jbertran/swh-environment/swh-web-ui/swh/web/ui/tests/test_backend.py", line 490, in revision_log_by
    ...
    • Jun 21 2016, 12:02 PM
    • 13 Lines
    Author: jbertran
  • P84 Empty author?
    softwareheritage=> select convert_from(name, 'utf-8') as name, convert_from(email, 'utf-8') as email, convert_from(fullname, 'utf-8') as fullname from person where (name = '' or name is null) and (email = '' or email is null) and fullname
    is not null;
    name | email | fullname
    ------+-------+----------
    ...
    • Jun 14 2016, 3:48 PM
    • 8 Lines
    Author: ardumont
  • P83 swh-loader-svn import from worker01 (with storage)
    Jun 11 22:31:34 worker01 python3[13656]: [2016-06-11 22:31:34,315: INFO/MainProcess] Received task: swh.loader.svn.tasks.LoadSvnRepositoryTsk[9b6b05d0-f7fe-4799-86a0-f9ec7bd67ead]
    Jun 11 22:31:34 worker01 python3[13656]: [2016-06-11 22:31:34,657: INFO/Worker-10] [revision_start-revision_end]: [1-145]
    Jun 11 22:31:34 worker01 python3[13656]: [2016-06-11 22:31:34,674: INFO/Worker-10] Processing {'remote_url': 'svn://svn.debian.org/svn/pkg-fox', 'local_url': b'/tmp/tmp.lod3o5u5.swh.loader/pkg-fox', 'uuid': b'd908f651-7add-0310-a5d1-c7ac9dfebe41', 'swh-origin': 4}.
    Jun 11 22:33:27 worker01 python3[13656]: [2016-06-11 22:33:27,415: INFO/Worker-10] Processed 145 revisions: [1dda85506a12af80c5a701a02aba5a02c703642f, ...]
    Jun 11 22:33:31 worker01 python3[13656]: [2016-06-11 22:33:31,508: INFO/MainProcess] Task swh.loader.svn.tasks.LoadSvnRepositoryTsk[9b6b05d0-f7fe-4799-86a0-f9ec7bd67ead] succeeded in 117.17479287087917s: None
    ...
    • Jun 12 2016, 9:57 PM
    • 127 Lines
    Author: ardumont
  • P82 swh-loader-svn import from worker01 (without storage)
    Jun 11 20:13:30 worker01 python3[27823]: [tasks]
    Jun 11 20:13:30 worker01 python3[27823]: . swh.loader.core.tasks.LoaderCoreTask
    Jun 11 20:13:30 worker01 python3[27823]: . swh.loader.svn.tasks.LoadSvnRepositoryTsk
    Jun 11 20:13:30 worker01 python3[27823]: [2016-06-11 20:13:30,213: INFO/MainProcess] Connected to amqp://swhconsumer:**@moma:5672//
    Jun 11 20:13:30 worker01 python3[27823]: [2016-06-11 20:13:30,259: INFO/MainProcess] mingle: searching for neighbors
    ...
    • Jun 12 2016, 9:56 PM
    • 143 Lines
    Author: ardumont
  • P81 T389 related actions - Logo for testimonials
    #+title: Images about testimonials
    * Organization
    * DONE Organization
    CLOSED: [2016-06-09 Thu 15:17]
    ...
    • Jun 9 2016, 3:39 PM
    • 81 Lines
    Author: ardumont
  • P79 magic svn keyword thingy diff
     tony  ⋯  swh-environment  swh-loader-svn  repo  diff -r glibc-bsd-git-r8 glibc-bsd-svn-r8    master Only in glibc-bsd-svn-r8: branches
    Only in glibc-bsd-svn-r8: tags
    Only in glibc-bsd-svn-r8/trunk/glibc-2.3/sysdeps/unix/bsd/bsd4.4/kfreebsd: net
    Only in glibc-bsd-svn-r8/trunk/glibc-2.3/sysdeps/unix/bsd/bsd4.4/kfreebsd: nfs
    diff -r glibc-bsd-git-r8/trunk/web/kfreebsd/livecd/init.c glibc-bsd-svn-r8/trunk/web/kfreebsd/livecd/init.c
    ...
    • May 26 2016, 3:22 PM
    • 19 Lines
    Author: ardumont
  • P76 Problem tests swh.storage during debian packaging
    content_add (test_objstorage_api.TestRemoteObjStorage) ... ERROR
    content_check_invalid (test_objstorage_api.TestRemoteObjStorage) ... HTTPConnectionPool(host='127.0.0.1', port=9): Max retries exceeded with url: http://127.0.0.1:11567/content/add (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f3e6fed9b00>: Failed to establish a new connection: [Errno 111] Connection refused',)))
    ERROR
    content_check_missing (test_objstorage_api.TestRemoteObjStorage) ... HTTPConnectionPool(host='127.0.0.1', port=9): Max retries exceeded with url: http://127.0.0.1:39245/content/add (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f3e6ff477b8>: Failed to establish a new connection: [Errno 111] Connection refused',)))
    ERROR
    ...
    • May 17 2016, 5:23 PM
    • 152 Lines
    Author: qcampos
  • P74 (An Untitled Masterwork)
    https://forge.softwareheritage.org/diffusion/60/edit/policy/
    https://forge.softwareheritage.org/diffusion/61/edit/policy/
    https://forge.softwareheritage.org/diffusion/65/edit/policy/
    https://forge.softwareheritage.org/diffusion/DCLG/edit/policy/
    https://forge.softwareheritage.org/diffusion/DCORE/edit/policy/
    ...
    • May 13 2016, 5:43 PM
    • 58 Lines
    Author: olasd
First
Prev
About · Developer information · Legal