Page MenuHomeSoftware Heritage
Paste Active Pastes
  • with swh_count_origins as (
    select value
    from object_counts
    where object_type='origin'
    ),
    ...
    • Fri, Dec 6, 12:14 PM
    • 9 Lines
    • SQL
  • * scheduler
    Following module migration to their own namespace (D2395):
    #+BEGIN_SRC sh
    ...
    • Thu, Dec 5, 8:38 AM
    • 16 Lines
  • self.search.origin_update([
    {'url': 'https://bitbucket.org/bitbucket0145/bitbucket_repo.git'},
    {'url': 'https://gitorious.org/railstutorial/railstutorial.git'},
    {'url': 'https://bitbucket.org/bittelc/railstutorial.git'},
    ])
    ...
    • Wed, Dec 4, 4:40 PM
    • 11 Lines
    • Python
  • [b'3\xbe\x07S\xcf(~j\xc0\xc0C\xd6\xea\xe6-\x1f\xd3&7;',
    b'\xa2\x82b\xfcGdS\x82O\xe0\x00\xf9\xda{\x85!\x1b\x82\x9d\x07',
    b'\x1c\xff\x7f\xb26\xdb[\xda3\xde\x11\xe5H\xa04\x02\x12\x9b\x8c\xf4',
    b'?J\x9c]\xc1\x1c\x13|\xb9\xd3.\x0eO\xf0\x9e2\xaf\x15M\x15',
    b'-q\x82E\x03\xd1\xf6\xa0G\x14S\xf8\xa0\t\xfdu?\x9e\xb02',
    ...
    • Thu, Nov 28, 4:04 PM
    • 20 Lines
  • import json
    from pprint import pprint
    import re
    import elasticsearch
    ...
    • Mon, Nov 25, 2:07 PM
    • 46 Lines
    • Python
  • default: &default_settings
    memory: 200G
    java_tool_options: -XXlol
    compress:
    <<: *default_settings
    ...
    • Fri, Nov 8, 2:49 PM
    • 6 Lines
  • default:
    memory: 200G
    java_tool_options: -XXlol
    compress:
    memory: 1000G
    • Fri, Nov 8, 2:47 PM
    • 5 Lines
    • YAML
  • 11:18 <+ardumont> douardda: D2237 draft to check for missing task types
    11:18 -- Notice(swhbot): D2237 (author: ardumont, Needs Review) on swh-lister: lister: Add checks on expected scheduler's output tasks <https://forge.softwareheritage.org/D2237>
    11:18 <+ardumont> i'm not sure when/where to plug that check though
    11:21 <+olasd> if we're doing that, we might just as well create the task type with the proper settings
    11:25 <+ardumont> mmm, unsure
    ...
    • Fri, Nov 8, 12:00 PM
    • 33 Lines
  • swh/graph/tests/test_cli.py::TestCompress::test_pipeline
    -------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------
    webgraph.py 233 INFO starting compression
    webgraph.py 242 INFO starting compression step MPH (1/11)
    webgraph.py 153 INFO running: java it.unimi.dsi.sux4j.mph.GOVMinimalPerfectHashFunction --zipped /tmp/tmp465gdf9e.swh-graph-test/example.mph --temp-dir /tmp/tmp465gdf9e.swh-graph-test/tmp /home/antoine/swh/swh-environment/swh-graph/swh/graph/tests/dataset/example.nodes.csv.gz
    ...
    • Nov 4 2019, 12:01 PM
    • 49 Lines
  • url
    ------------------------------------------------------------------------------------------------
    https://github.com/rootpy/root_numpy
    https://github.com/stevengj/mpb
    https://github.com/barbagroup/pygbe
    ...
    • Oct 15 2019, 2:49 PM
    • 263 Lines
  • package org.softwareheritage.graph;
    import org.softwareheritage.graph.algo.Traversal;
    import java.io.OutputStream;
    ...
    • Oct 14 2019, 1:52 PM
    • 67 Lines
    • Java
  • ERR-CONDUIT-CORE: Graph cycle detected (type=5, cycle=PHID-DREV-vvlfxmyjkkrcdnbfzb5l, PHID-DREV-sz2tjc63iowtoyay6mey, PHID-DREV-ni5tcyma6542fypzaalx, PHID-DREV-q6y4zi7eciqddatexzxl, PHID-DREV-tsal5vzfklovorhkqvzn, PHID-DREV-rtmmo2kgo7wacl4warhv, PHID-DREV-5muna6xupz7kvfkz2yuo, PHID-DREV-ee7627jtcof4i73qqrfm, PHID-DREV-kf4oob54dw4sttlkv2az, PHID-DREV-vvlfxmyjkkrcdnbfzb5l).
    • Oct 14 2019, 12:28 PM
    • 1 Line
  • git pull --rebase
    remote: Enumerating objects: 21, done.
    remote: Counting objects: 100% (21/21), done.
    remote: Compressing objects: 100% (11/11), done.
    remote: Total 12 (delta 6), reused 0 (delta 0)
    ...
    • Oct 9 2019, 2:58 PM
    • 15 Lines
  • P545 Data
    {'_id': ObjectId('5ad99f9fbd95630dfc4b9a4e'),
    'graphql': {'user': {'biography': 'International educator, traveler, music '
    'lover, photographer, Californian.',
    'blocked_by_viewer': False,
    'connected_fb_page': None,
    ...
    • Oct 8 2019, 6:30 PM
    • 640 Lines
  • 17:09 <+douardda> and the pb should not happen if we put the plugin out of the swh package (i.e. ar the root of swh-core directory)
    17:10 <+douardda> (I've reproduced the issue in a minimal src repo)
    17:12 <+douardda> the problem seems to be that when the plugin lives under our package's hat, it's loaded very soon, thus swh.core is loaded but from the installed location (in the tox venv here)
    17:13 <+douardda> so when the subsequent import statement for a conftest or a testfile occurs, it's looked under this package's root directory first
    17:14 <+douardda> so the simple solution is to put this plugin in a dedicated python module not under the swh's (and especially the swh.core one I think) package
    ...
    • Oct 8 2019, 11:49 AM
    • 6 Lines
  • Bad:
    Returns
    List of tarball urls and their associated metadata (time, length).
    For example:
    ...
    • Oct 8 2019, 11:33 AM
    • 21 Lines
  • ```
    $ sudo apt install r-base
    ```
    ```
    ...
    • Oct 6 2019, 11:49 AM
    • 34 Lines
  • Hello,
    It's the time of the year where I ask you (again!) for your help to better archive GNU source code in the Software Heritage archive.
    Would it be possible to change the format of the GNU file listing [1] to also include SHA256 checksums?
    ...
    • Oct 1 2019, 12:10 PM
    • 16 Lines
  • indexes:
    - swh_workers-2018.03.*
    size: 100
    from: 0
    ...
    • Sep 30 2019, 7:47 PM
    • 27 Lines
  • diff --git a/Makefile b/Makefile
    index 524175c..be63f09 100644
    --- a/Makefile
    +++ b/Makefile
    @@ -3,3 +3,12 @@
    ...
    • Sep 27 2019, 7:10 PM
    • 29 Lines
    • Diff
  • ✘ ⚙ dev@desktop5  ~/swh-environment/swh-journal   bencode-key ●  pytest
    ========================================================================================================= test session starts =========================================================================================================
    platform linux -- Python 3.5.3, pytest-5.0.1, py-1.7.0, pluggy-0.12.0
    hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/dev/swh-environment/swh-journal/.hypothesis/examples')
    rootdir: /home/dev/swh-environment/swh-journal, inifile: pytest.ini
    ...
    • Sep 23 2019, 3:00 PM
    • 51 Lines
  • -- gitlab (renamed key "api_baseurl" to "url")
    update task set arguments='{"args": [], "kwargs": {"instance": "inria", "url": "https://gitlab.inria.fr/api/v4"}}' where arguments#>>'{kwargs,instance}' = 'inria' and type in ('list-gitlab-full', 'list-gitlab-incremental');
    update task set arguments='{"args": [], "kwargs": {"instance": "framagit", "url": "https://framagit.org/api/v4"}}' where arguments#>>'{kwargs,instance}' = 'framagit' and type in ('list-gitlab-full', 'list-gitlab-incremental');
    update task set arguments='{"args": [], "kwargs": {"instance": "riseup", "url": "https://0xacab.org/api/v4"}}' where arguments#>>'{kwargs,instance}' = 'riseup' and type in ('list-gitlab-full', 'list-gitlab-incremental');
    update task set arguments='{"args": [], "kwargs": {"instance": "gitlab", "url": "https://gitlab.com/api/v4"}}' where arguments#>>'{kwargs,instance}' = 'gitlab' and type in ('list-gitlab-full', 'list-gitlab-incremental');
    ...
    • Sep 11 2019, 3:54 PM
    • 35 Lines
    • SQL
  • messages: 168300
    messages: 168400
    messages: 168500
    messages: 168600
    messages: 168700
    ...
    • Sep 11 2019, 3:18 PM
    • 23 Lines
  • swh_graph-replayer.1.khy6rzsfohdz@mirror-node-3 | INFO:swh.journal.cli:Processed 35000 messages.
    swh_graph-replayer.1.khy6rzsfohdz@mirror-node-3 | Traceback (most recent call last):
    swh_graph-replayer.1.khy6rzsfohdz@mirror-node-3 | File "/usr/bin/swh", line 11, in <module>
    swh_graph-replayer.1.khy6rzsfohdz@mirror-node-3 | load_entry_point('swh.core==0.0.67', 'console_scripts', 'swh')()
    swh_graph-replayer.1.khy6rzsfohdz@mirror-node-3 | File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 56, in main
    ...
    • Sep 10 2019, 5:14 PM
    • 36 Lines
  • swh_graph-replayer.1.p18dcgg4lwq5@mirror-node-3 | INFO:swh.journal.cli:Processed 124000 messages.
    swh_graph-replayer.1.p18dcgg4lwq5@mirror-node-3 | INFO:kafka.client:Closing idle connection 1, last active 540038 ms ago
    swh_graph-replayer.1.p18dcgg4lwq5@mirror-node-3 | INFO:kafka.conn:<BrokerConnection node_id=1 host=kafka01.euwest.azure.internal.softwareheritage.org:9092 <connected> [IPv4 ('192.168.200.24', 9092)]>: Closing connection.
    swh_graph-replayer.1.p18dcgg4lwq5@mirror-node-3 | INFO:kafka.client:Closing idle connection 3, last active 540139 ms ago
    swh_graph-replayer.1.p18dcgg4lwq5@mirror-node-3 | INFO:kafka.conn:<BrokerConnection node_id=3 host=kafka03.euwest.azure.internal.softwareheritage.org:9092 <connected> [IPv4 ('192.168.200.31', 9092)]>: Closing connection.
    ...
    • Sep 10 2019, 1:50 PM
    • 9 Lines
  • Sep 10 11:59:01 desktop5 replayer-12301[2746]: INFO:kafka.conn:<BrokerConnection node_id=11 host=esnode1.internal.softwareheritage.org:9092 <connecting> [IPv4 ('192.168.100.61', 9092)]>: connecting to esnode1.internal.softwareheritage.org:9092 [('192.168.100.61', 9092) IPv4]
    Sep 10 11:59:06 desktop5 replayer-12301[2746]: INFO:kafka.conn:<BrokerConnection node_id=12 host=esnode2.internal.softwareheritage.org:9092 <connecting> [IPv4 ('192.168.100.62', 9092)]>: connecting to esnode2.internal.softwareheritage.org:9092 [('192.168.100.62', 9092) IPv4]
    Sep 10 11:59:06 desktop5 replayer-12301[2746]: INFO:kafka.conn:<BrokerConnection node_id=11 host=esnode1.internal.softwareheritage.org:9092 <connecting> [IPv4 ('192.168.100.61', 9092)]>: Connection complete.
    Sep 10 11:59:06 desktop5 replayer-12301[2746]: INFO:kafka.conn:<BrokerConnection node_id=12 host=esnode2.internal.softwareheritage.org:9092 <connecting> [IPv4 ('192.168.100.62', 9092)]>: Connection complete.
    Sep 10 11:59:19 desktop5 replayer-12301[2746]: WARNING:kafka.coordinator:Heartbeat session expired, marking coordinator dead
    ...
    • Sep 10 2019, 12:00 PM
    • 43 Lines
  • ERROR:root:An error occurred while calling o0.visit.
    : java.lang.ArrayIndexOutOfBoundsException: Index 272313807 out of bounds for length 121824
    at it.unimi.dsi.bits.LongArrayBitVector.getBoolean(LongArrayBitVector.java:374)
    at org.softwareheritage.graph.algo.Traversal.visitNodesVisitor(Traversal.java:160)
    at org.softwareheritage.graph.Entry.visit(Entry.java:28)
    ...
    • Sep 9 2019, 7:33 PM
    • 108 Lines
  • /browse/origin/26984/latest_snapshot/
    /browse/origin/34423/latest_snapshot/
    /browse/origin/21387/latest_snapshot/
    /browse/origin/48567/latest_snapshot/
    /browse/origin/29526/latest_snapshot/
    ...
    • Sep 9 2019, 2:59 PM
    • 851 Lines
  • import json
    import requests
    from pprint import pprint
    ...
    • Sep 9 2019, 2:53 PM
    • 74 Lines
    • Python
  • swh/indexer/tests/test_metadata.py .................................F
    ================================================================================================================== FAILURES ===================================================================================================================
    ___________________________________________________________________________________________________ Metadata.test_revision_metadata_indexer ___________________________________________________________________________________________________
    ...
    • Sep 5 2019, 2:55 PM
    • 92 Lines
  • # Graph compression output
    Compression script and environment used: https://forge.softwareheritage.org/source/swh-graph/browse/master/dockerfiles/
    - Direct compressed graph: `all.{graph,obl,offsets,properties}`
    ...
    • Aug 27 2019, 9:20 PM
    • 8 Lines
  • https://forge.softwareheritage.org/rDGRPHb31d2e86a80cf8b85d4bf51f30be8e463fe994e4
    https://forge.softwareheritage.org/rDGRPH0b46253799f43a25a8528926052340f93a1a911b
    https://forge.softwareheritage.org/rDGRPHc7363b064ae1ed52c271b9831b934cd196589c8e
    https://forge.softwareheritage.org/rDGRPHb6c6e1eec131a002a44e01cef17abb81ec958421
    https://forge.softwareheritage.org/rDGRPHd5dcbfcdf245777a8753ccc6ac5414e762605abe
    ...
    • Aug 27 2019, 2:02 PM
    • 186 Lines
  • Command : $ django-admin shell --settings=swh.web.settings.tests
    /home/kalpitk/.virtualenvs/swh/lib/python3.6/site-packages/swh/scheduler/__init__.py:69: DeprecationWarning: Call to deprecated class SWHRemoteAPI. (Use the RPCClient instead) -- Deprecated since version 0.0.64.
    return SchedulerBackend(**args)
    Traceback (most recent call last):
    ...
    • Aug 22 2019, 11:45 AM
    • 38 Lines
    • Bash Scripting
  • #!/usr/bin/env python3
    import sys
    import dateutil.parser
    ...
    • Aug 20 2019, 2:40 PM
    • 36 Lines
    • Python
  • On rioc:
    (base) [zacchiro@rioc graph]$ cat all+ori.*.count
    164513699014
    11683687950
    ...
    • Aug 19 2019, 10:44 AM
    • 10 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-lister/swh/lister$ python jjj.py
    DEBUG:swh.lister.core.lister_base:Loading config from lister_gnu
    INFO:swh.core.config:Loading config file /home/archit/.config/swh/lister_gnu.yml
    DEBUG:swh.lister.core.lister_base:<swh.lister.gnu.lister.GNULister object at 0x7ffacb9f7a20> CONFIG={'content_size_limit': 104857600, 'log_db': 'dbname=softwareheritage-log', 'storage': {'cls': 'remote', 'args': {'url': 'http://localhost:5002/'}}, 'scheduler': {'cls': 'remote', 'args': {'url': 'http://localhost:5008/'}}, 'lister': {'cls': 'local', 'args': {'db': 'postgresql:///lister-gnu'}}, 'credentials': [], 'cache_responses': True, 'cache_dir': '/home/archit/.cache/swh/lister/gnu/'}
    DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
    ...
    • Aug 18 2019, 2:35 PM
    • 91 Lines
  • Done on rioc:
    Used 1000 random nodes (results are in seconds):
    'git bundle' use-case
    ...
    • Aug 15 2019, 9:46 PM
    • 15 Lines
  • swh-storage=# select * from origin_visit where origin=424 ;
    origin | visit | date | type | status | metadata | snapshot
    --------+-------+-------------------------------+------+---------+----------+--------------------------------------------
    424 | 1 | 2019-08-14 10:02:03.190532+00 | gnu | partial | |
    424 | 2 | 2019-08-14 10:02:46.964868+00 | gnu | full | | \x1f3305edbd687c27ca005f7cafea3d6f809c38d1
    ...
    • Aug 14 2019, 1:32 PM
    • 16 Lines
  • Done on rioc:
    Used 100000 random nodes (results are in seconds):
    'ls' use-case
    ...
    • Aug 14 2019, 11:22 AM
    • 27 Lines
  • Print of `revisions` at this line
    https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/storage.py$692
    [{'author': {'email': 'robot@softwareheritage.org',
    'fullname': 'Software Heritage',
    ...
    • Aug 14 2019, 10:32 AM
    • 182 Lines
  • swh-storage_1 | [2019-08-13 20:10:39 +0000] [41] [DEBUG] POST /content/missing
    swh-storage_1 | [2019-08-13 20:10:39 +0000] [41] [DEBUG] POST /directory/missing
    swh-storage_1 | [2019-08-13 20:10:39 +0000] [41] [DEBUG] POST /revision/missing
    swh-storage_1 | [2019-08-13 20:10:39 +0000] [41] [DEBUG] POST /revision/add
    swh-storage_1 | ERROR:root:Object of type bytes is not JSON serializable
    ...
    • Aug 13 2019, 10:12 PM
    • 39 Lines
  • utkarsh@G3:~$ workon swh
    Usage:: command not found
    Command: command not found
    Options:: command not found
    INFO: command not found
    ...
    • Aug 12 2019, 3:26 PM
    • 12 Lines
  • (swh) utkarsh@G3:~/swh-environment$ swh scheduler task-type add list-packagist-full2 "swh.lister.packagist.tasks.PackagistListerTask" "Full PACKAGIST lister" --default-interval '1 day' --backoff-factor 1
    /home/utkarsh/.virtualenvs/swh/lib/python3.7/site-packages/swh/scheduler/__init__.py:69: DeprecationWarning: Call to deprecated class SWHRemoteAPI. (Use the RPCClient instead) -- Deprecated since version 0.0.64.
    return SchedulerBackend(**args)
    OK
    • Aug 12 2019, 2:48 PM
    • 4 Lines
  • There were two revisions for this package
    {'author': {'email': b'robot@softwareheritage.org',
    'fullname': b'Software Heritage',
    'name': b'Software Heritage'},
    ...
    • Aug 11 2019, 9:10 PM
    • 104 Lines
  • swh-loader_1 | [2019-08-11 19:02:59,149: INFO/MainProcess] Received task: swh.loader.package.tasks.LoadGNU[11fa45c7-ffc8-47ad-9e18-dcff159f1bd2]
    swh-loader_1 | [2019-08-11 19:02:59,152: INFO/ForkPoolWorker-1] Loading config file /loader.yml
    swh-loader_1 | [2019-08-11 19:02:59,169: DEBUG/ForkPoolWorker-1] Creating gnu origin for https://ftp.gnu.org/gnu/hello/
    swh-loader_1 | [2019-08-11 19:02:59,304: DEBUG/ForkPoolWorker-1] Done creating gnu origin for https://ftp.gnu.org/gnu/hello/
    swh-loader_1 | [2019-08-11 19:02:59,304: DEBUG/ForkPoolWorker-1] Creating origin_visit for origin https://ftp.gnu.org/gnu/hello/ at time 2019-08-11 19:02:59.304333+00:00
    ...
    • Aug 11 2019, 9:06 PM
    • 82 Lines
  • iled'}
    swh-loader_1 | [2019-08-11 18:50:23,779: INFO/MainProcess] Received task: swh.loader.package.tasks.LoadGNU[7a7e43e1-dd5b-4ebe-be5b-2ea10fd0bdbc]
    swh-loader_1 | [2019-08-11 18:50:23,871: INFO/ForkPoolWorker-1] Loading config file /loader.yml
    swh-loader_1 | [2019-08-11 18:50:23,915: DEBUG/ForkPoolWorker-1] Creating gnu origin for https://ftp.gnu.org/gnu/dap/
    swh-loader_1 | [2019-08-11 18:50:23,924: DEBUG/ForkPoolWorker-1] Done creating gnu origin for https://ftp.gnu.org/gnu/dap/
    ...
    • Aug 11 2019, 8:58 PM
    • 91 Lines
  • P497 logs
    wh-loader_1 | Using pip from /srv/softwareheritage/venv/bin/pip
    swh-loader_1 | Processing /src/swh-loader
    swh-loader_1 | Requirement already satisfied: vcversioner in /srv/softwareheritage/venv/lib/python3.7/site-packages (from swh.loader.core==0.0.44.post3) (2.16.0.0)
    swh-loader_1 | Requirement already satisfied: retrying in /srv/softwareheritage/venv/lib/python3.7/site-packages (from swh.loader.core==0.0.44.post3) (1.3.3)
    swh-loader_1 | Requirement already satisfied: psutil in /srv/softwareheritage/venv/lib/python3.7/site-packages (from swh.loader.core==0.0.44.post3) (5.6.3)
    ...
    • Aug 9 2019, 4:35 PM
    • 295 Lines
  • NPM Revision
    {'synthetic': True,
    'metadata': {'package_source': {'name': 'ja','version': '0.0.1', 'filename': 'ja-0.0.1.tgz', 'sha1': '31399c51d3024f6eb91c626a31a175dc30f343e5', 'date': '2014-04-07T16:02:07.453Z', 'url': 'https://registry.npmjs.org/ja/-/ja-0.0.1.tgz', 'sha256': '8101e284d5846e77f9698628da660e92f8627535c266a3aef28549b25e63a597', 'blake2s256': 'fcf5438822e12c8a6973b9ecb40df4978947edd675c4269d728d5279dc82b203'},
    'package': {'author': 'Goldeneye Solutions', 'name': 'ja', 'description': 'Compose a cross-platform application from various things.',
    ...
    • Aug 9 2019, 3:56 PM
    • 95 Lines
  • id | type | arguments | next_run | current_interval | status | policy | retries_left | priority
    --------+---------------+----------------------------+-------------------------------+------------------+--------------------+---------+--------------+----------
    225784 | list-gnu-full | {"args": [], "kwargs": {}} | 2019-08-08 19:22:27.338807+00 | 90 days | next_run_scheduled | oneshot | 0 |
    225786 | list-gnu-full | {"args": [], "kwargs": {}} | 2019-08-08 19:25:28.450559+00 | 90 days | next_run_scheduled | oneshot | 0 |
    225787 | list-gnu-full | {"args": [], "kwargs": {}} | 2019-08-08 19:29:55.666438+00 | 90 days | disabled | oneshot | 0 |
    ...
    • Aug 9 2019, 7:47 AM
    • 8 Lines
  • swh-lister_1 | [2019-08-09 05:32:01,482: ERROR/ForkPoolWorker-1] Task swh.lister.gnu.tasks.GNUListerTask[e997c3cf-0dfc-423e-9460-3e8d685321b7] raised unexpected: NotNullViolation('null value in column "retries_left" violates not-null constraint\nDETAIL: Failing row contains (225791, load-tar, {"args": ["gcal", "https://ftp.gnu.org/gnu/gcal/"], "kwargs": {"..., 2019-08-09 05:32:01.13707+00, null, next_run_not_scheduled, recurring, null, null).\nCONTEXT: SQL statement "insert into task (type, arguments, next_run, status, current_interval, policy,\n retries_left, priority)\n select type, arguments, next_run, status, current_interval, policy,\n retries_left, priority\n from tmp_task t\n where not exists(select 1\n from task\n where type = t.type and\n arguments->\'args\' = t.arguments->\'args\' and\n arguments->\'kwargs\' = t.arguments->\'kwargs\' and\n policy = t.policy and\n priority is not distinct from t.priority and\n status = t.status)"\nPL/pgSQL function swh_scheduler_create_tasks_from_temp() line 12 at SQL statement\n')
    swh-lister_1 | Traceback (most recent call last):
    swh-lister_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
    swh-lister_1 | R = retval = fun(*args, **kwargs)
    swh-lister_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/task.py", line 45, in __call__
    ...
    • Aug 9 2019, 7:35 AM
    • 41 Lines
  • Notice: /Stage[main]/Profile::Ssh::Server/Sshkey[ssh-worker1.internal.staging.swh.network-ecdsa-sha2-nistp256]/ensure: current_value absent, should be present (noop)
    Notice: /Stage[main]/Profile::Ssh::Server/Sshkey[ssh-worker0.internal.staging.swh.network-rsa]/ensure: current_value absent, should be present (noop)
    Notice: /Stage[main]/Profile::Ssh::Server/Sshkey[ssh-worker0.internal.staging.swh.network-ecdsa-sha2-nistp256]/ensure: current_value absent, should be present (noop)
    Notice: /Stage[main]/Profile::Ssh::Server/Sshkey[ssh-worker0.internal.staging.swh.network-ed25519]/ensure: current_value absent, should be present (noop)
    Notice: /Stage[main]/Profile::Ssh::Server/Sshkey[ssh-worker0.internal.staging.swh.network-dsa]/ensure: current_value absent, should be present (noop)
    ...
    • Aug 8 2019, 7:28 PM
    • 172 Lines
  • version: '2'
    services:
    swh-objstorage:
    volumes:
    ...
    • Aug 8 2019, 3:41 PM
    • 15 Lines
  • Benchmark results for content_find:
    hash_algo = sha1 (sample size=263):
    cassandra: avg = 9 ms, stdev = 2.5 ms
    postgres: avg = 14 ms, stdev = 14.9 ms
    ...
    • Aug 8 2019, 1:47 PM
    • 72 Lines
  • from collections import defaultdict
    import csv
    import itertools
    import os
    from pprint import pprint
    ...
    • Aug 8 2019, 12:21 PM
    • 207 Lines
    • Python
  • Benchmark results for content_find:
    hash_algo = sha1 (sample size=248):
    avg cassandra = 5 ms
    avg postgres = 14 ms
    hash_algo = sha1_git (sample size=242):
    ...
    • Aug 8 2019, 12:16 PM
    • 25 Lines
  • DEBUG:swh.loader.package.GNULoader:Sending 2 revisions
    DEBUG:urllib3.connectionpool:Resetting dropped connection: localhost
    DEBUG:urllib3.connectionpool:http://localhost:5002 "POST /revision/add HTTP/1.1" 400 85
    ERROR:swh.loader.package.GNULoader:Loading failure, updating to `partial` status
    Traceback (most recent call last):
    ...
    • Aug 7 2019, 1:12 PM
    • 38 Lines
  • pip install $( ./bin/pip-swh-packages --with-testing )
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-core
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-model
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-core
    Obtaining file:///home/tony/work/inria/repo/swh/swh-environment/swh-objstorage
    ...
    • Aug 2 2019, 12:03 PM
    • 52 Lines
  • variable "region" {
    type = "string"
    default = "northeurope"
    }
    ...
    • Aug 1 2019, 5:40 PM
    • 68 Lines
  • Bonjour,
    ci-dessous mes reponses.
    Toutefois, seriez-vous disponible demain pour en discuter par telephone?
    ...
    • Aug 1 2019, 11:58 AM
    • 91 Lines
  • # Get an available port number
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind(('127.0.0.1', 0))
    self.port = sock.getsockname()[1]
    sock.close()
    • Aug 1 2019, 10:05 AM
    • 5 Lines
    • Python
  • Info: Using configured environment 'new_staging'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Loading facts
    Info: Applying configuration version '1564591379'
    ...
    • Jul 31 2019, 6:45 PM
    • 59 Lines
  • Package: DZEXPM
    Type: Package
    Title: Estimation and Prediction of Skewed Spatial Processes
    Version: 1.0
    Date: 2017-06-24
    ...
    • Jul 25 2019, 6:56 PM
    • 14 Lines
  • root@pergamon:~# puppet agent --test --noop
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Loading facts
    ...
    • Jul 24 2019, 4:27 PM
    • 68 Lines
  • $ git commit --date '1999-01-01' -m "foo" --allow-empty | grep Date
    Date: Fri Jan 1 12:59:34 1999 +0100
    $ git commit --date '1973-03-03' -m "foo" --allow-empty | grep Date
    Date: Sat Mar 3 13:00:21 1973 +0100
    $ git commit --date '1973-03-02' -m "foo" --allow-empty | grep Date
    ...
    • Jul 23 2019, 2:02 PM
    • 10 Lines
  • ardumont@pergamon:~% cat /etc/fstab
    # /etc/fstab: static file system information.
    #
    # Use 'blkid' to print the universally unique identifier for a
    # device; this may be used with UUID= as a more robust way to name devices
    ...
    • Jul 22 2019, 11:41 AM
    • 32 Lines
  • /
    -----master----. b
    \ /
    ...
    • Jul 19 2019, 2:10 PM
    • 18 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-loader-core/swh/loader/gnu$ tox .
    GLOB sdist-make: /home/archit/swh-environment/swh-loader-core/setup.py
    flake8 installed: entrypoints==0.3,flake8==3.7.8,mccabe==0.6.1,pycodestyle==2.5.0,pyflakes==2.1.1,swh.loader.core==0.0.44.post2
    flake8 run-test-pre: PYTHONHASHSEED='1804426209'
    flake8 run-test: commands[0] | /home/archit/swh-environment/swh-loader-core/.tox/flake8/bin/python -m flake8
    ...
    • Jul 19 2019, 9:48 AM
    • 65 Lines
  • swh_content-replayer.1.bgt2xzycn7wo@desktop6 | Starting the SWH mirror content replayer
    swh_content-replayer.1.bgt2xzycn7wo@desktop6 | /usr/lib/python3/dist-packages/swh/storage/api/server.py:20: DeprecationWarning: Call to deprecated class SWHServerAPIApp. (Use the RPCServerApp instead) -- Deprecated since version 0.0.64.
    swh_content-replayer.1.bgt2xzycn7wo@desktop6 | app = SWHServerAPIApp(__name__)
    swh_content-replayer.1.bgt2xzycn7wo@desktop6 | Traceback (most recent call last):
    swh_content-replayer.1.bgt2xzycn7wo@desktop6 | File "/usr/bin/swh", line 11, in <module>
    ...
    • Jul 18 2019, 3:42 PM
    • 15 Lines
  • kafka:
    image: wurstmeister/kafka
    environment:
    KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
    ports:
    ...
    • Jul 14 2019, 1:30 PM
    • 9 Lines
  • (swh) archit@work-pc:~/swh-environment$ doco logs kafka
    kafka_1 | log.cleaner.min.cleanable.ratio = 0.5
    kafka_1 | log.cleaner.min.compaction.lag.ms = 0
    ...
    • Jul 14 2019, 1:23 PM
    • 177 Lines
  • (swh) archit@work-pc:~/swh-environment$ doco logs swh-storage
    ...
    • Jul 14 2019, 1:21 PM
    • 47 Lines
  • INFO:swh.core.config:Loading config file /home/archit/.config/swh/loader/cran.yml
    /home/archit/swh-environment/swh-storage/swh/storage/api/client.py:13: DeprecationWarning: Call to deprecated class MetaRPCClient. (Use the MetaRPCClient instead) -- Deprecated since version 0.0.64.
    class RemoteStorage(SWHRemoteAPI):
    /home/archit/swh-environment/swh-storage/swh/storage/__init__.py:43: DeprecationWarning: Call to deprecated class RPCClient. (Use the RPCClient instead) -- Deprecated since version 0.0.64.
    return Storage(**args)
    ...
    • Jul 14 2019, 12:55 PM
    • 48 Lines
  • Traceback (most recent call last):
    File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/browse/views/utils/snapshot_context.py", line 239, in browse_snapshot_directory
    browse_context='directory') # noqa
    File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/browse/views/utils/snapshot_context.py", line 135, in _process_snapshot_request
    origin_url, timestamp, visit_id)
    ...
    • Jul 13 2019, 9:00 PM
    • 10 Lines
  • ttaching to swh-docker-dev_swh-storage_1
    swh-storage_1 | Using pip from /srv/softwareheritage/venv/bin/pip
    swh-storage_1 | Installed Python packages:
    swh-storage_1 | Package Version
    swh-storage_1 | ----------------------------- ---------
    ...
    • Jul 11 2019, 5:09 PM
    • 107 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-loader-core/swh/loader$ pip list
    Package Version Location
    ----------------------------- -------------- -------------------------------------------------
    aiohttp 3.5.4
    alabaster 0.7.12
    ...
    • Jul 11 2019, 5:05 PM
    • 164 Lines
  • DEBUG:urllib3.connectionpool:http://localhost:5002 "POST /origin/visit/get_latest HTTP/1.1" 400 74
    ERROR:swh.loader.pypi.PyPILoader:Loading failure, updating to `partial` status
    Traceback (most recent call last):
    File "/home/archit/swh-environment/swh-loader-core/swh/loader/core/loader.py", line 868, in load
    self.prepare(*args, **kwargs)
    ...
    • Jul 11 2019, 3:21 PM
    • 71 Lines
  • FROM debian:stretch
    RUN apt-get update
    RUN apt-get install -y python3-pip tox git make
    RUN git clone https://github.com/t-b/sphinxcontrib-images.git
    ...
    • Jul 11 2019, 2:57 PM
    • 18 Lines
  • Attaching to swh-docker-dev_swh-web_1
    swh-web_1 | Using pip from /srv/softwareheritage/venv/bin/pip
    swh-web_1 | Installed Python packages:
    swh-web_1 | Package Version
    swh-web_1 | ----------------------------- ---------
    ...
    • Jul 11 2019, 11:09 AM
    • 491 Lines
  • (swh) archit@work-pc:~/swh-environment$ pip uninstall sphinx
    (swh) archit@work-pc:~/swh-environment$ ./bin/update
    (swh) archit@work-pc:~/swh-environment$ pip install $(./bin/pip-swh-packages --with-testing)
    Stored in directory: /home/archit/.cache/pip/wheels/7b/6c/23/eb26369b77904c8963fae9e64338b0f0b948b4d59710760834
    ...
    • Jul 10 2019, 10:58 PM
    • 9 Lines
  • (swh) ~/swh-environment$ pip install $(./bin/pip-swh-packages --with-testing)
    Successfully built PyYAML systemd-python psycopg2 vcversioner python-cephlibs retrying pyld fastimport dulwich SQLAlchemy python-hglib sqlitedict subvertpy htmlmin pypandoc sphinxcontrib-images wrapt yarl billiard snowballstemmer future
    ERROR: sphinxcontrib-images 0.8.0 has requirement sphinx<2,>=1.1.3, but you'll have sphinx 2.1.2 which is incompatible.
    ERROR: swh-docs 0.0.0.post156 has requirement sphinx<2,>=1.3, but you'll have sphinx 2.1.2 which is incompatible.
    ...
    • Jul 10 2019, 9:08 PM
    • 6 Lines
  • Traceback (most recent call last):
    File "/home/twitu/Code/swh-environment/swh-indexer/.tox/py3/bin/pytest", line 10, in <module>
    sys.exit(main())
    File "/home/twitu/Code/swh-environment/swh-indexer/.tox/py3/lib/python3.5/site-packages/_pytest/config/__init__.py", line 58, in main
    config = _prepareconfig(args, plugins)
    ...
    • Jul 10 2019, 6:11 PM
    • 77 Lines
  • This document lists use cases and benchmark scenarii for the Software Heritage
    graph service.
    Conventions
    ...
    • Jul 7 2019, 1:20 PM
    • 170 Lines
    • reStructuredText
  • ```
    swh-lister=> select * from bitbucket_repo order by indexable desc limit 10;
    name | full_name | html_url | origin_url | origin_type | last_seen | task_id | uid | indexable
    ------------------------------+----------------------------------------------+--------------------------------------------------------------------+------------------------------------------------------------------------+-------------+----------------------------+-----------+----------------------------------------+-------------------------------
    issues_crud_hg | bb-synthetic_2-bot/issues_crud_hg | https://bitbucket.org/bb-synthetic_2-bot/issues_crud_hg | https://bitbucket.org/bb-synthetic_2-bot/issues_crud_hg | hg | 2019-07-06 07:23:56.735599 | 169799503 | {bf9c4d33-712b-4da3-b420-a959f53db27f} | 2019-07-06 07:23:52.331212+00
    ...
    • Jul 6 2019, 9:31 AM
    • 17 Lines
  • on worker09:
    ```
    @app.task(name=__name__ + '.IncrementalBitBucketLister')
    def incremental_bitbucket_lister(**lister_args):
    lister = new_lister(**lister_args)
    ...
    • Jul 4 2019, 4:53 PM
    • 13 Lines
  • swhstorage@getty:~$ swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
    INFO:swh.core.config:Loading config file /etc/softwareheritage/indexer/journal_client.yml
    Usage: swh indexer journal-client [OPTIONS]
    Error: You must specify at least one kafka broker.
    • Jul 3 2019, 4:20 PM
    • 5 Lines
  • ========== ============== ================================== ====================================== ==========
    type name url format Example
    ========== ============== ================================= ====================================== ===========
    deposit hal https://hal.archives-ouvertes.fr/ codemeta-ish [1]
    deposit intel https://software.intel.com/ codemeta-ish [2]
    ...
    • Jul 3 2019, 2:24 PM
    • 17 Lines
  • Done in 5.97s.
    python3 swh/web/manage.py migrate -v0 2>/dev/null
    python3 swh/web/manage.py createcachetable -v0 2>/dev/null
    bash -c "trap 'trap - SIGINT SIGTERM ERR EXIT; jobs -p | head -1 | xargs pkill -P' SIGINT SIGTERM ERR EXIT; python3 swh/web/manage.py runserver --nostatic --settings=swh.web.settings.tests & sleep 10 && yarn run cypress open"
    Loading failure, updating to `partial` status
    ...
    • Jul 3 2019, 12:23 PM
    • 88 Lines
  • {"exception":"BadInputExc","reason":"Error when parsing identifier. Wrong format: There should be 4 mandatory values","traceback":"Traceback (most recent call last):\n File \"/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/common/utils.py\", line 235, in resolve_swh_persistent_id\n swh_id_parsed = parse_persistent_identifier(swh_id)\n File \"/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/model/identifiers.py\", line 747, in parse_persistent_identifier\n 'Wrong format: There should be 4 mandatory values')\nswh.model.exceptions.ValidationError: ['Wrong format: There should be 4 mandatory values']\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/api/apidoc.py\", line 307, in documented_view\n response = f(request, **kwargs)\n File \"/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/web/api/views/identifiers.py\", line 57, in...
    • Jun 26 2019, 3:33 PM
    • 1 Line
  • {
    "data": {
    "origins": [
    {
    "id": 53607151,
    ...
    • Jun 25 2019, 4:42 PM
    • 142 Lines
    • JSON
  • query {
    origins(urls: ["https://github.com/SoftwareHeritage/swh-model"]) {
    id
    type
    url
    ...
    • Jun 25 2019, 4:42 PM
    • 24 Lines
    • Plain Text
  • swh-environment/swh-web/.tox/py3/lib/python3.5/site-packages/swh/web/tests/api/views/test_origin.py(330)test_api_origin_intrinsic_metadata_get()
    -> def test_api_origin_intrinsic_metadata_get(self, origin):
    swh-environment/swh-web/.tox/py3/lib/python3.5/site-packages/hypothesis/core.py(1023)wrapped_test()
    -> raise the_error_hypothesis_found
    • Jun 23 2019, 2:55 PM
    • 4 Lines
    • Bash Scripting
  • /swh-environment/swh-web/.tox/py3/bin/pytest --hypothesis-profile=swh-web-fast --cov .tox/py3/lib/python3.5/site-packages/swh/web --cov-branch .tox/py3/lib/python3.5/site-packages/swh/web/tests/api/views/test_origin.py
    • Jun 22 2019, 12:26 PM
    • 1 Line
    • Bash Scripting
  • with (
    patch('swh.web.common.service.idx_storage') as mock_idx_storage,
    patch('swh.web.common.service.storage' as mock_storage
    ):
    • Jun 22 2019, 9:50 AM
    • 4 Lines
    • Python
  • LinkNode([('', LinkNode()),
    ('1',
    LinkNode([('content',
    LinkNode([('ctags', LinkNode()),
    ('filetype', LinkNode()),
    ...
    • Jun 20 2019, 1:57 PM
    • 49 Lines
  • info:
    description: ''
    title: Bookings API
    version: ''
    openapi: 3.0.0
    ...
    • Jun 20 2019, 1:38 PM
    • 639 Lines
    • YAML
  • diff --git a/swh/web/api/urls.py b/swh/web/api/urls.py
    index afbcaf44..4267c73d 100644
    --- a/swh/web/api/urls.py
    +++ b/swh/web/api/urls.py
    @@ -15,6 +15,25 @@ import swh.web.api.views.snapshot # noqa
    ...
    • Jun 20 2019, 1:37 PM
    • 30 Lines
    • Diff
  • P442 code
    from bs4 import BeautifulSoup
    from collections import defaultdict
    import requests
    base_url = "https://git.savannah.gnu.org/cgit/"
    ...
    • Jun 19 2019, 11:27 PM
    • 81 Lines
  • (swh) ddouard@desktop6:~/src/swh-environment/swh-journal$ pyarc diff -c
    Accepted D1608
    Repo: swh-journal (DJNL)
    Author: douardda
    Created: 3 hours ago
    ...
    • Jun 19 2019, 2:38 PM
    • 26 Lines
  • alter table gnu_repo drop column origin_id;
    alter table gnu_repo drop column description;
    alter table phabricator_repo drop column origin_id;
    alter table phabricator_repo drop column description;
    ...
    • Jun 19 2019, 10:34 AM
    • 24 Lines
    • SQL
  • swh-lister git:(master) git push
    Counting objects: 5, done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (5/5), done.
    Writing objects: 100% (5/5), 629 bytes | 0 bytes/s, done.
    ...
    • Jun 19 2019, 7:59 AM
    • 17 Lines