Page MenuHomeSoftware Heritage
Paste Active Pastes
  • =================================================================================== short test summary info ====================================================================================
    FAILED swh/web/tests/browse/test_snapshot_context.py::test_get_snapshot_context_no_origin - AssertionError: assert {'branch': 'H...': False, ...} == {'branch': 'H...': False, ...}
    FAILED swh/web/tests/browse/test_snapshot_context.py::test_get_snapshot_context_with_origin - AssertionError: assert {'branch': 'H...': False, ...} == {'branch': 'H...': False, ...}
    FAILED swh/web/tests/browse/views/test_content.py::test_content_origin_snapshot_branch_browse - AssertionError: False is not true : Couldn't find 'Branches (12)' in response
    FAILED swh/web/tests/browse/views/test_content.py::test_content_origin_snapshot_release_browse - AssertionError: False is not true : Couldn't find 'Branches (12)' in response
    ...
    • Jul 10 2020, 12:08 PM
    • 13 Lines
  • GLOB sdist-make: /home/tony/work/inria/repo/swh/swh-environment/swh-loader-core/setup.py
    black recreate: /home/tony/work/inria/repo/swh/swh-environment/swh-loader-core/.tox/black
    black installdeps: black
    black installed: appdirs==1.4.4,attrs==19.3.0,black==19.10b0,click==7.1.2,pathspec==0.8.0,regex==2020.6.8,toml==0.10.1,typed-ast==1.4.1
    black run-test-pre: PYTHONHASHSEED='2865243897'
    ...
    • Jul 9 2020, 4:13 PM
    • 50 Lines
  • Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/home/antoine/.virtualenvs/swh/lib/python3.7/site-packages/dulwich/porcelain.py", line 404, in clone
    depth=depth, **kwargs)
    File "/home/antoine/.virtualenvs/swh/lib/python3.7/site-packages/dulwich/porcelain.py", line 1393, in fetch
    ...
    • Jul 8 2020, 6:58 PM
    • 15 Lines
  • [2020-07-07 04:12:55,275: ERROR/ForkPoolWorker-534] Loading failure, updating to `partial` status
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 310, in load
    more_data_to_fetch = self.fetch_data()
    File "/usr/lib/python3/dist-packages/swh/loader/git/loader.py", line 245, in fetch_data
    ...
    • Jul 8 2020, 6:53 PM
    • 15 Lines
  • self = <swh.core.api.tests.test_serializers.Serializers testMethod=test_encode_native_datetime>
    def test_encode_native_datetime(self):
    dt = datetime.datetime(2015, 1, 1, 12, 4, 42, 231455)
    ...
    • Jul 8 2020, 12:56 PM
    • 9 Lines
  • (Pdb) p revision.extra_headers
    ((b'encoding', b'ISO-8859-15'), (b'mergetag', b'object 9768d0b576dbaaecd80abedad6dfd0d72f1476da\ntype commit\ntag v0.0.1\ntagger David Douard <david.douard@sdfa3.org> 1594138133 +0200\n\nv0.0.1\n-----BEGIN PGP SIGNATURE-----\n\niQJLBAABCAA1FiEEAOWDevQbOk/9ITMF6ImSleOlnUcFAl8EnhkXHGRhdmlkLmRv\ndWFyZEBzZGZhMy5vcmcACgkQ6ImSleOlnUcdzg//ZW9y2xU5JFQuUsBe/LfKrs+m\n0ohVInPKXwAfpB3+gn/XtTSLe+Nnr8+QEZyVRCUz2gpGZ2tNqRjhYLIX4x5KKlaV\nrfl/6Cy7zibsxxuzA1h7HylCs3IPsueQpznVHUwD9jQ5baGJSc2Lt1LufXTueHZJ\nOc0oLiP5xCZcPqeX8R/4zUUImJZ1QrPeKmQ/3F+Iq62iWp7nWDp8PtwpykSiYlNf\nKrJM8omGvrlrWLtfPNUaQFClXwnwK1/HyNY2kYan6K5NtsIl2UX0LZ42GkRjJIrb\nq4TFIZWZ6xndtEhHEX6B8Q5TZV6sqPgNnfGpbhj8BDoZgjD0Y43fzfDiZ0Bl2tph\ntXaLg3SX/UUjFVzC1zkoQ2MR7+j8NVKauAsBINpKF4pMGsrsVRk8764pgO49iQ+S\n8JVCVV76dNNm1gd7BbhFAdIAiegBtsEF69niJBoHKYLlrT8E8hDkF/gk4IkimPqf\nUHtw/fPhVW3B4G2skd013NJGcnRj5oKtaM99d2Roxc3vhSRiTsoaM8BM9NDvLmJg\n35rWEOnet39iJIMCHk3AYaJl8QmUhllDdr6vygaBVeVEf27m2c3NzONmIKpWqa2J\nkTpF4cmzHYro34G7WuJ1bYvmLb6qWNQt9wd8RW+...
    • Jul 7 2020, 6:39 PM
    • 2 Lines
  • # Copyright (C) 2020 The Software Heritage developers
    # See the AUTHORS file at the top-level directory of this distribution
    # License: GNU General Public License version 3, or any later version
    # See top-level LICENSE file for more information
    ...
    • Jul 7 2020, 2:21 PM
    • 29 Lines
    • Python
  • 2020-07-03 17:23:20 civodul zack: hey! could you clarify the policy wrt. tarballs?
    2020-07-03 17:23:43 civodul i understand it's undesirable, but still it happens, so i'm trying to see if we can have our cake and eat it too :-)
    2020-07-03 17:27:30 +zack civodul: I understand it's annoying for your needs, but for now I confirm what both I and rdicosmo have said in the ticket(s), i.e., no tarball archival for now
    2020-07-03 17:28:01 +zack the lookup service is more likely to happen, but I (now) understand your point about verifiability of them
    2020-07-03 17:28:49 +zack I confess I don't entirely buy it, especially because in the medium term the hashes you have today are going to be broken, and with them your verifiability will become also moot
    ...
    • Jul 6 2020, 6:39 PM
    • 69 Lines
  • swh_storage_backend_config["journal_writer"] = journal1
    storage = get_storage(**swh_storage_backend_config)
    # fill the storage and the journal (under prefix1)
    for object_type, objects in TEST_OBJECTS.items():
    ...
    • Jul 3 2020, 5:35 PM
    • 50 Lines
  • KT = TypeVar("KT")
    VT = TypeVar("VT")
    def freeze_optional_dict(
    ...
    • Jul 3 2020, 4:14 PM
    • 17 Lines
    • Python
  • ERROR swh.loader.svn.SvnLoader:loader.py:493 History of svn file:///tmp/pytest-of-tony/pytest-138/test_loader_tampered_repositor0/pkg-gourmet@6 altered. Skipping...
    DEBUG swh.core.statsd:statsd.py:418 Error submitting statsd packet. Dropping the packet and closing the socket.
    DEBUG swh.core.statsd:statsd.py:418 Error submitting statsd packet. Dropping the packet and closing the socket.
    ERROR swh.loader.svn.SvnLoader:loader.py:328 Loading failure, updating to `partial` status
    Traceback (most recent call last):
    ...
    • Jul 3 2020, 1:41 PM
    • 11 Lines
  • 2020-07-01 22:28:58,520 | ERROR: ('repo /home/danseraf/gitsample/zope FAILED',)
    2020-07-01 22:28:58,520 | ERROR: ('"<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>502 Proxy Error</title>\n</head><body>\n<h1>Proxy Error</h1>\n<p>The proxy server received an invalid\r\nresponse from an upstream server.<br />\r\nThe proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>\n</body></html>\n with given values [\'swh:1:cnt:756d315be4dbe8a982c569f0b62566729eee8f44\', \'swh:1:cnt:1ead24b6bb49747d0caf2e7abc5fd20151625de8\', \'swh:1:cnt:15259c1c8b5844e5820671419583c6f2556ce5d8\']"',)
    swh.scanner.exceptions.APIError: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html><head>
    ...
    • Jul 3 2020, 12:43 PM
    • 27 Lines
  • def test_loader_svn_new_visit(swh_storage, datadir, tmp_path):
    repo_url = prepare_repository("pkg-gourmet.tgz", resources_path=datadir)
    loader = SvnLoader(
    repo_url,
    ...
    • Jul 2 2020, 6:06 PM
    • 19 Lines
  • def escape(data, toplevel=True):
    if data is None:
    return ""
    if isinstance(data, bytes):
    ...
    • Jul 2 2020, 12:31 PM
    • 36 Lines
    • Python
  • swh  swh  tony  yavin4  ~  work  …  swh  swh-environment  swh-core  master  %  pytest -x -s
    ============================================================================================================= test session starts =============================================================================================================
    platform linux -- Python 3.7.3, pytest-5.4.1, py-1.8.0, pluggy-0.12.0
    rootdir: /home/tony/work/inria/repo/swh/swh-environment/swh-core, inifile: pytest.ini
    plugins: mock-1.10.4, aiohttp-0.3.0, xdist-1.31.0, forked-1.1.3, postgresql-2.2.0, dash-1.13.1, testinfra-5.1.0, requests-mock-1.6.0, flask-1.0.0, hypothesis-5.8.0, asyncio-0.10.0, celery-4.3.0, django-3.6.0, kafka-0.3.1, swh.core-0.0.95.post4, swh.journal-0.3.4.post2
    ...
    • Jul 2 2020, 12:25 PM
    • 45 Lines
  • ```
    lang=python
    @pytest.mark.parametrize(
    "value", [pytest.param("foo", id="str"), pytest.param(1, id="int")]
    )
    ...
    • Jul 1 2020, 5:12 PM
    • 203 Lines
    • Remarkup
  • -- SWH DB schema upgrade
    -- from_version: 156
    -- to_version: 157
    -- description: Add extrinsic artifact metadata
    ...
    • Jul 1 2020, 2:46 PM
    • 68 Lines
  • 18:07:42 .tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_storage.py::TestStorage::test_origin_visit_get_latest_none Fatal Python error: Segmentation fault
    18:07:42
    18:07:42 Current thread 0x00007fc83f656740 (most recent call first):
    18:07:42 File "/usr/lib/python3.7/inspect.py", line 687 in getsourcefile
    18:07:42 File "/usr/lib/python3.7/inspect.py", line 1460 in getframeinfo
    ...
    • Jun 30 2020, 10:14 AM
    • 73 Lines
  • root@pergamon:~# clush -b -w @swh-workers "ps aux | grep 'python3 -m celery worker -n loader_' "
    ---------------
    worker01
    ---------------
    root 2907321 0.0 0.0 8332 2792 ? Ss 15:46 0:00 bash -c ps aux | grep 'python3 -m celery worker -n loader_'
    ...
    • Jun 29 2020, 6:15 PM
    • 98 Lines
  • def test_release_add_twice(self, swh_storage):
    actual_result = swh_storage.release_add([data.release])
    assert actual_result == {"release:add": 1}
    assert list(swh_storage.journal_writer.journal.objects) == [
    ...
    • Jun 29 2020, 4:01 PM
    • 16 Lines
    • Python
  • softwareheritage=> SELECT attname, inherited, n_distinct FROM pg_stats WHERE tablename = 'snapshot_branch';
    attname | inherited | n_distinct
    -------------+-----------+-------------
    object_id | f | -1
    name | f | 15056
    ...
    • Jun 19 2020, 5:12 PM
    • 7 Lines
  • softwareheritage=> explain select * from snapshot_branches inner join snapshot_branch on (snapshot_branches.branch_id=snapshot_branch.object_id) where name >= 'gggg' and snapshot_id=110650857 order by name limit 1000;
    QUERY PLAN
    ----------------------------------------------------------------------------------------------------------------------------------------------
    Limit (cost=93915.87..94032.54 rows=1000 width=67)
    -> Gather Merge (cost=93915.87..97631.26 rows=31844 width=67)
    ...
    • Jun 19 2020, 4:50 PM
    • 14 Lines
  • Traceback (most recent call last):
    File "/home/antoine/swh/swh-environment/swh-web/swh/web/browse/snapshot_context.py", line 1070, in browse_snapshot_log
    revision_id=request.GET.get("revision"),
    File "/home/antoine/swh/swh-environment/swh-web/swh/web/browse/snapshot_context.py", line 497, in get_snapshot_context
    snapshot_sizes = service.lookup_snapshot_sizes(snapshot_id)
    ...
    • Jun 19 2020, 3:56 PM
    • 20 Lines
  • swh_graph-replayer-directory.1.uyfyr8j3012g@bespin | swh.core.api.RemoteException: <RemoteException 500 ProgramLimitExceeded: ['index row size 7560 exceeds maximum 2712 for index "directory_entry_dir_target_name_perms_idx"\nHINT: Values larger than 1/3 of a buffer page cannot be indexed.\nConsider a function index of an MD5 hash of the value, or use full text indexing.\nCONTEXT: SQL statement "\n insert into directory_entry_dir (target, name, perms)\n select distinct t.target, t.name, t.perms\n from tmp_directory_entry_dir t\n where not exists (\n select 1\n from directory_entry_dir i\n where t.target = i.target and t.name = i.name and t.perms = i.perms)\n "\nPL/pgSQL function swh_directory_entry_add(directory_entry_type) line 3 at EXECUTE\nSQL statement "SELECT swh_directory_entry_add(\'dir\')"\nPL/pgSQL function swh_directory_add() line 4 at PERFORM\n']>
    • Jun 17 2020, 2:16 PM
    • 1 Line
  • query_parts = [
    "SELECT * FROM origin_visit_status ",
    "WHERE origin = %s AND visit = %s ",
    ]
    query_params: List[Any] = [origin, visit]
    ...
    • Jun 16 2020, 7:34 PM
    • 23 Lines
  • Release {'author': {'fullname': None, 'name': None, 'email': None}, 'date': None, 'id': b'\x03\xcf\xc9\xad\x03 \x95\x10\x00\xfd\xfe\xb0\x9e\xf2\x1d\xa7\xd2Rr<', 'name': b'klibc-0.59', 'message': None, 'synthetic': False, 'target': b'\xa8^\x07t:\r\xd0\xc1\xde\x0eVP\xc2Y\xb4\xd3N\x8b\x926', 'target_type': 'revision'}
    • Jun 15 2020, 5:37 PM
    • 1 Line
  • diff --git a/swh/loader/tests/test_cli.py b/swh/loader/tests/test_cli.py
    index bdc0a76..c9968f6 100644
    --- a/swh/loader/tests/test_cli.py
    +++ b/swh/loader/tests/test_cli.py
    @@ -87,7 +87,9 @@ def test_run_with_visit_date(mocker, swh_config):
    ...
    • Jun 12 2020, 4:20 PM
    • 15 Lines
  • aiohttp==3.6.2
    aiohttp-utils==3.1.1
    aioresponses==0.6.3
    alabaster==0.7.12
    amqp==2.5.1
    ...
    • Jun 12 2020, 4:06 PM
    • 281 Lines
  • modified swh/model/model.py
    @@ -175,7 +175,7 @@ class TimestampWithTimezone(BaseModel):
    timestamp = attr.ib(type=Timestamp, validator=type_validator())
    offset = attr.ib(type=int, validator=type_validator())
    ...
    • Jun 12 2020, 12:42 PM
    • 22 Lines
    • Diff
  • >>> class Foo:
    ... def __bool__(self):
    ... return False
    ...
    >>> foo = Foo()
    ...
    • Jun 11 2020, 10:49 AM
    • 7 Lines
  • % join old.txt new.txt | awk '{print $1,"+"($3/$2-1)*100" %"}'
    cnt +49.3072 %
    dir +40.5945 %
    ori +22.4553 %
    rel +45.207 %
    ...
    • Jun 9 2020, 2:55 PM
    • 21 Lines
  • Size
    ----
    - nodes: 17075708289 (17 B)
    - edges: 203351589619 (203 B)
    ...
    • Jun 9 2020, 2:22 PM
    • 35 Lines
    • reStructuredText
  • #!/usr/bin/env python3
    """Generate a feed of random (hence bogus, but syntactically correct!) SWHIDs
    """
    ...
    • Jun 9 2020, 10:31 AM
    • 41 Lines
  • ______________________________________________________________________________________________________________ test_kafka_writer ______________________________________________________________________________________________________________
    kafka_prefix = 'abcxwtgptr', kafka_server = '127.0.0.1:34923', consumer = <cimpl.Consumer object at 0x7f50d122ea60>, privileged_object_types = {'release', 'revision'}
    def test_kafka_writer(
    ...
    • Jun 8 2020, 6:08 PM
    • 70 Lines
  • cnt 8181993787
    dir 6914748995
    ori 108109058
    rel 14386337
    rev 1734773279
    ...
    • Jun 8 2020, 3:04 PM
    • 6 Lines
  • ==================================== ERRORS ====================================
    ERROR collecting .pybuild/cpython3_3.7_swh.model/build/swh/model/tests/test_cli.py
    swh/model/tests/test_cli.py:14: in <module>
    from swh.model import cli
    swh/model/cli.py:17: in <module>
    ...
    • Jun 3 2020, 3:40 PM
    • 52 Lines
  • .tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_api_client.py::TestStorage::test_revision_get_order <- .tox/py3/lib/python3.7/site-packages/swh/storage/tests/test_storage.py
    >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
    > /home/tony/work/inria/repo/swh/swh-environment/swh-storage/.tox/py3/lib/python3.7/site-packages/swh/storage/in_memory.py(482)revision_get()
    -> yield rev
    ...
    • Jun 3 2020, 2:16 PM
    • 25 Lines
  • Traceback (most recent call last):
    File "/home/antoine/swh/swh-environment/swh-loader-git/.tox/py3/lib/python3.7/site-packages/_pytest/main.py", line 191, in wrap_session
    session.exitstatus = doit(config, session) or 0
    File "/home/antoine/swh/swh-environment/swh-loader-git/.tox/py3/lib/python3.7/site-packages/_pytest/main.py", line 247, in _main
    config.hook.pytest_runtestloop(session=session)
    ...
    • Jun 3 2020, 11:21 AM
    • 106 Lines
  • ERROR:root:canceling statement due to statement timeout
    Traceback (most recent call last):
    File "/home/antoine/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
    File "/home/antoine/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    ...
    • Jun 2 2020, 11:34 AM
    • 24 Lines
  • ERROR:root:canceling statement due to statement timeout
    Traceback (most recent call last):
    File "/home/antoine/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
    File "/home/antoine/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    ...
    • Jun 2 2020, 11:33 AM
    • 34 Lines
  • diff --git java/pom.xml java/pom.xml
    index 9f1e4fd..707fa73 100644
    --- java/pom.xml
    +++ java/pom.xml
    @@ -52,7 +52,10 @@
    ...
    • May 28 2020, 4:12 PM
    • 16 Lines
    • Diff
  • ~/dati/src/webgraph/svn  ant ivy-setupjars
    Buildfile: /srv/src/webgraph/svn/build.xml
    [taskdef] Could not load definitions from resource org/apache/ivy/ant/antlib.xml. It could not be found.
    ivy-setupjars:
    ...
    • May 27 2020, 3:52 PM
    • 34 Lines
  • https://archive.softwareheritage.org/swh:1:dir:1cdac93986d0ee72687d51893f9b8f9a5fa61f92;origin=https://doi.org/10.5201/ipol.2018.236;visit=swh:1:snp:3935a5d32ed955d4e30267c6718051a2391c5aec;anchor=swh:1:rev:672487d8be02ed1bc026297dabf2d7a0cea00d2d;path=//
    {
    "extrinsic": {
    "provider": "https://deposit.softwareheritage.org/1/private/630/meta/",
    ...
    • May 26 2020, 3:13 PM
    • 95 Lines
  • softwareheritage=> select now() at time zone interval '00:00';
    timezone
    ----------------------------
    2020-05-24 13:31:01.133371
    (1 row)
    ...
    • May 24 2020, 3:31 PM
    • 11 Lines
  • /**
    * Copyright (C) 2020 The Software Heritage developers
    * See the AUTHORS file at the top-level directory of this distribution
    * License: GNU Affero General Public License version 3, or any later version
    * See top-level LICENSE file for more information
    ...
    • May 22 2020, 7:09 PM
    • 320 Lines
    • May 19 2020, 3:07 PM
    • 38 Lines
  • describe('Test admin deposit page', function() {
    it('Does not do much!', function () {
    expect(true).to.equal(true);
    });
    ...
    • May 19 2020, 9:38 AM
    • 98 Lines
    • Javascript
  • -- SWH DB schema upgrade
    -- from_version: 148
    -- to_version: 149
    -- description: Implement extrinsic origin-metadata specification
    ...
    • May 18 2020, 4:12 PM
    • 104 Lines
  • root@moma:~# SWH_CONFIG_FILENAME=/etc/softwareheritage/deposit/server.yml django-admin migrate --settings=swh.deposit.settings.production --verbosity 3
    Operations to perform:
    Apply all migrations: auth, contenttypes, deposit, sessions
    Running pre-migrate handlers for application auth
    Running pre-migrate handlers for application contenttypes
    ...
    • May 15 2020, 2:19 PM
    • 47 Lines
  • diff --git swh/dataset/exporter.py swh/dataset/exporter.py
    index 741de3f..02361c5 100644
    --- swh/dataset/exporter.py
    +++ swh/dataset/exporter.py
    @@ -216,6 +216,7 @@ class ParallelExporter:
    ...
    • May 15 2020, 12:13 PM
    • 12 Lines
    • Diff
  • {
    "extrinsic": {
    "provider": "https://deposit.softwareheritage.org/1/private/606/meta/",
    "raw": {
    "origin": {
    ...
    • May 14 2020, 5:16 PM
    • 79 Lines
  • 1 git … commit --
    Trim Trailing Whitespace.................................................Passed
    Flake8...................................................................Passed
    Check JSON...........................................(no files to check)Skipped
    Check Yaml...........................................(no files to check)Skipped
    ...
    • May 14 2020, 11:37 AM
    • 34 Lines
  • refs/heads/master 69394173
    refs/pull/@@NUM@@/head 58297691
    HEAD 51612028
    releases/@@NUM@@.@@NUM@@.@@NUM@@ 11783124
    refs/changes/@@NUM@@/@@NUM@@/@@NUM@@ 6991689
    ...
    • May 13 2020, 7:52 PM
    • 3,613 Lines
  • tmpdir_factory = TempdirFactory(_tmppath_factory=TempPathFactory(_given_basetemp=None, _trace=<pluggy._tracing.TagTracerSub object at 0x7f3d11bfc710>, _basetemp=PosixPath('/tmp/pytest-of-tony/pytest-1')))
    @pytest.fixture(scope="session")
    def elasticsearch_session(tmpdir_factory):
    tmpdir = tmpdir_factory.mktemp("elasticsearch")
    ...
    • May 13 2020, 11:21 AM
    • 175 Lines
  • {
    "extrinsic": {
    "provider": "https://deposit.softwareheritage.org/1/private/602/meta/",
    "raw": {
    "origin": {
    ...
    • May 12 2020, 4:25 PM
    • 79 Lines
  • Staged
    modified swh/search/tests/test_cli.py
    @@ -9,6 +9,7 @@ import yaml
    import pytest
    ...
    • May 12 2020, 2:01 PM
    • 39 Lines
  • ```
    $ curl -i -u user:pass https://deposit.softwareheritage.org/1/servicedocument/
    HTTP/1.1 503 Service Unavailable
    Date: Mon, 11 May 2020 14:03:49 GMT
    Server: gunicorn/19.9.0
    ...
    • May 11 2020, 4:05 PM
    • 25 Lines
  • {
    "extrinsic": {
    "provider": "https://deposit.softwareheritage.org/1/private/596/meta/",
    "raw": {
    "origin": {
    ...
    • May 7 2020, 11:26 AM
    • 79 Lines
  • -- Crawling history of software origin visits by Software Heritage. Each
    -- visit see its history change through new origin visit status updates
    create table origin_visit_status
    (
    origin bigint not null,
    ...
    • May 6 2020, 4:19 PM
    • 516 Lines
  • swh scheduler task list
    WARNING:swh.core.cli:Could not load subcommand search: cannot import name 'get_journal_client' from 'swh.journal.cli' (/home/ddouard/src/swh-environment/swh-journal/swh/journal/cli.py)
    INFO:swh.core.config:Loading config file /etc/softwareheritage/global.ini
    Found 3 tasks
    ...
    • May 5 2020, 12:17 PM
    • 15 Lines
  • softwareheritage-deposit=> select * from deposit_client ;
    user_ptr_id | collections | provider_url | domain
    -------------+-------------+------------------------------------------------+---------------------------------
    2 | {1} | https://hal.archives-ouvertes.fr/ | archives-ouvertes.fr
    3 | {2} | https://www.softwareheritage.org | softwareheritage.org
    ...
    • May 4 2020, 4:31 PM
    • 10 Lines
  • swh-loader_1 | Installing collected packages: swh.deposit
    swh-loader_1 | Attempting uninstall: swh.deposit
    swh-loader_1 | Found existing installation: swh.deposit 0.0.82
    swh-loader_1 | Uninstalling swh.deposit-0.0.82:
    swh-loader_1 | Successfully uninstalled swh.deposit-0.0.82
    ...
    • May 4 2020, 11:54 AM
    • 16 Lines
  • docker-compose up swh-loader
    docker_swh-scheduler-db_1 is up-to-date
    docker_swh-deposit-db_1 is up-to-date
    docker_swh-storage-db_1 is up-to-date
    Recreating docker_swh-objstorage_1 ...
    ...
    • May 4 2020, 11:20 AM
    • 54 Lines
  • docker-compose up swh-loader
    docker_swh-storage-db_1 is up-to-date
    docker_swh-objstorage_1 is up-to-date
    docker_amqp_1 is up-to-date
    docker_zookeeper_1 is up-to-date
    ...
    • May 4 2020, 11:01 AM
    • 55 Lines
  • dockeruser@desktop5  /home/dev/swh-environment/docker   master  docker-compose up swh-loader
    docker_amqp_1 is up-to-date
    docker_zookeeper_1 is up-to-date
    docker_swh-storage-db_1 is up-to-date
    docker_swh-objstorage_1 is up-to-date
    ...
    • Apr 30 2020, 1:06 PM
    • 62 Lines
  • version: '2'
    services:
    swh-loader:
    volumes:
    ...
    • Apr 30 2020, 1:06 PM
    • 8 Lines
  • 12:25 <+olasd> ardumont: you should even do this migration the other way around. 1/ create the new table, empty; 2/ deploy new storage that fills both tables in parallel; 3/ run the migration process for origin visits that don't have any state information
    12:26 <+olasd> 4/ switch over the read queries to the new tables; 5/ drop fields from the old table
    12:26 <+olasd> (#showerthoughts)
    12:27 <+olasd> that way, the only downtime needed is during #2 which should be minimal; it should even be doable on the fly as the RPC api is not changing at all
    12:28 <+ardumont> for 2. i need to check the code, i think we stop the writing in origin visit to only write to origin visit status
    ...
    • Apr 30 2020, 12:02 PM
    • 64 Lines
  • ERROR:root:relation "origin_visit_status" does not exist
    LIGNE 4 : INNER JOIN origin_visit_status ovs
    ^
    Traceback (most recent call last):
    File "/home/antoine/.virtualenvs/swh/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    ...
    • Apr 29 2020, 12:17 PM
    • 35 Lines
  • (swh) david@pavo:~/swh/swh-environment/swh-storage$ /usr/bin/python3 -m tox -e py3 -- -v -m 'not cassandra' -k test_rpc_serve_bwcompat -s
    GLOB sdist-make: /home/david/swh/swh-environment/swh-storage/setup.py
    py3 inst-nodeps: /home/david/swh/swh-environment/swh-storage/.tox/.tmp/package/1/swh.storage-0.0.187.post9.zip
    py3 installed: aiohttp==3.6.2,aiohttp-utils==3.1.1,apipkg==1.5,arrow==0.15.5,async-timeout==3.0.1,attrs==19.3.0,attrs-strict==0.0.6,blinker==1.4,cassandra-driver==3.23.0,certifi==2020.4.5.1,chardet==3.0.4,click==7.1.1,confluent-kafka==1.4.1,coverage==5.1,decorator==4.4.2,Deprecated==1.2.9,execnet==1.7.1,Flask==1.1.2,geomet==0.1.2,gunicorn==20.0.4,hypothesis==5.10.4,idna==2.9,importlib-metadata==1.6.0,iso8601==0.1.12,itsdangerous==1.1.0,Jinja2==2.11.2,MarkupSafe==1.1.1,mirakuru==2.2.0,more-itertools==8.2.0,msgpack==1.0.0,multidict==4.7.5,mypy==0.770,mypy-extensions==0.4.3,packaging==20.3,pkg-resources==0.0.0,pluggy==0.13.1,port-for==0.4,psutil==5.7.0,psycopg2==2.8.5,py==1.8.1,pyparsing==2.4.7,pytest==5.4.1,pytest-cov==2.8.1,pytest-forked==1.1.3,pytest-mock==3.1.0,pytest-postgresql==2.3.0,pytest-xdist==1.31.0,python-dateutil==2.8.1,python-mimeparse==1.6.0,pytz==2019.3,PyYAML==5.3.1,requests==2.23.0,sentry-sdk==0.14.3,six==1.14.0,sortedcontainers==2.1.0,sqlalchemy-stubs==0.3,swh.core==0.0.95,swh.journal==0.0.31,swh.model==0.0.68,swh.objstorage==0.0.42,swh.storage==0.0.187.post9,tenacity==6.1.0,typed-ast==1.4.1,typing-extensions==3.7.4.2,urllib3==1.25.9,vcversioner==2.16.0.0,wcwidth==0.1.9,Werkzeug==1.0.1,wrapt==1.12.1,yarl==1.4.2,zipp==3.1.0
    py3 run-test-pre: PYTHONHASHSEED='2792914223'
    ...
    • Apr 24 2020, 2:38 PM
    • 37 Lines
  • swh-loader_1 | Total 6363 (delta 4101), reused 1867 (delta 1059)
    swh-loader_1 | [2020-04-23 12:04:33,713: INFO/ForkPoolWorker-1] Listed 307 refs for repo https://forge.softwareheritage.org/source/swh-loader-core.git
    swh-loader_1 | [2020-04-23 12:05:06,623: ERROR/ForkPoolWorker-1] Loading failure, updating to `partial` status
    swh-loader_1 | Traceback (most recent call last):
    swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 304, in load
    ...
    • Apr 23 2020, 3:03 PM
    • 25 Lines
  • _________________________________________________________________________ ERROR at setup of test_origin_save_metrics __________________________________________________________________________
    file /home/antoine/swh/swh-environment/swh-web/swh/web/tests/misc/test_metrics.py, line 34
    @pytest.mark.django_db
    def test_origin_save_metrics(client):
    file /home/antoine/.virtualenvs/swh/lib/python3.7/site-packages/pytest_flask/plugin.py, line 132
    ...
    • Apr 23 2020, 2:21 PM
    • 12 Lines
  • $ psql service=staging-swh -c "select convert_from(name, 'utf-8') as bname, rev.metadata from snapshot s inner join snapshot_branches sbs on sbs.snapshot_id=s.object_id inner join snapshot_branch sb on sbs.branch_id=sb.object_id inner join revision rev on (rev.id=sb.target and sb.target_type='revision') where s.id='\x757b6422d07b0f50f983d86b0eef192ef248ca0b'" > output.txt
    | metadata
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    http://kokkinizita.linuxaudio.org/linuxaudio/downloads/AMB-plugins-0.8.1.tar.bz2 | {"extrinsic": {"raw": {"url": "http://kokkinizita.linuxaudio.org/linuxaudio/downloads/AMB-plugins-0.8.1.tar.bz2"}, "when": "2020-03-18T11:40:20.875888+00:00", "provider": "https://nix-community.github.io/nixpkgs-swh/sources.json"}, "original_artifact": [{"length": 28988, "filename": "AMB-plugins-0.8.1.tar.bz2", "checksums": {"sha1": "fda55d11342d9a59ead64e30e037d92114637c87", "sha256": "f44a60b782948662537c0cb14befa6678d6dce790c64dc2c9058eab849a58b74"}}]}
    ...
    • Apr 22 2020, 10:47 AM
    • 12 Lines
  • diff --git a/swh/web/common/origin_visits.py b/swh/web/common/origin_visits.py
    index 4adaed89..e37eedaa 100644
    --- a/swh/web/common/origin_visits.py
    +++ b/swh/web/common/origin_visits.py
    @@ -125,9 +125,9 @@ def get_origin_visit(
    ...
    • Apr 21 2020, 11:44 AM
    • 16 Lines
    • Diff
  • subprocess.run(
    ("pv {export_path}/*/*.edges.csv.zst | "
    "tee graph.edges.csv.zst |"
    "zstdcat |"
    "tee >( wc -l > graph.edges.count.txt ) |"
    ...
    • Apr 17 2020, 4:34 PM
    • 18 Lines
    • Python
  • GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
    swh-dataset-export-seirl-test-20200323 swh.journal.objects.snapshot 234 647473 658902 11429 - - -
    swh-dataset-export-seirl-test-20200323 swh.journal.objects.release 196 98984 99633 649 - - -
    swh-dataset-export-seirl-test-20200323 swh.journal.objects.directory 229 28637831 29006612 368781 - - -
    ...
    • Apr 15 2020, 3:57 PM
    • 1,282 Lines
  • $ swh graph memcache --graph ./swh-graph --cache /dev/shm/graph-small
    $ ls -l /dev/shm/graph-small
    total 16464
    lrwxrwxrwx 1 antoine antoine 47 Apr 10 11:47 stderr -> /home/antoine/swh/graph/small/compressed/stderr
    lrwxrwxrwx 1 antoine antoine 47 Apr 10 11:47 stdout -> /home/antoine/swh/graph/small/compressed/stdout
    ...
    • Apr 10 2020, 11:49 AM
    • 26 Lines
    • Bash Scripting
  • softwareheritage=> \copy metadata_provider to stdout with (format csv, header);
    id,provider_name,provider_type,provider_url,metadata
    1,"",deposit_client,https://hal.archives-ouvertes.fr/,{}
    2,"",deposit_client,https://www.softwareheritage.org,{}
    3,hal,deposit_client,https://hal.archives-ouvertes.fr/,{}
    ...
    • Apr 9 2020, 1:09 PM
    • 181 Lines
  • diff --cc swh/storage/tests/test_converters.py
    index 72443ec4,f3ee1904..00000000
    --- a/swh/storage/tests/test_converters.py
    +++ b/swh/storage/tests/test_converters.py
    @@@ -144,7 -144,7 +144,11 @@@ def test_db_to_release()
    ...
    • Apr 9 2020, 12:05 PM
    • 48 Lines
  • psql service=staging-swh
    psql (11.7, server 11.6 (Debian 11.6-1.pgdg100+1))
    SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
    Type "help" for help.
    ...
    • Apr 9 2020, 10:10 AM
    • 15 Lines
  • 17:00 $ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging moma
    Found host moma.softwareheritage.org
    WARN -> Environment "add-keycloak-realm-and-client" contained non-word characters, correcting name to add_keycloak_realm_and_client
    WARN -> Environment "api-remove-rl-for-m" contained non-word characters, correcting name to api_remove_rl_for_m
    WARN -> Environment "change-swh-web-static-dir" contained non-word characters, correcting name to change_swh_web_static_dir
    ...
    • Apr 8 2020, 5:02 PM
    • 23 Lines
  • diff --git a/swh/storage/storage.py b/swh/storage/storage.py
    index 6b52485..44f3df9 100644
    --- a/swh/storage/storage.py
    +++ b/swh/storage/storage.py
    @@ -894,10 +894,9 @@ class Storage:
    ...
    • Apr 8 2020, 3:11 PM
    • 16 Lines
  • sbuild (Debian sbuild) 0.79.0 (05 February 2020) on mirzakhani.olasd.eu
    +==============================================================================+
    | swh-web 0.0.226-1~swh2 (amd64) Tue, 07 Apr 2020 12:37:33 +0000 |
    +==============================================================================+
    ...
    • Apr 7 2020, 2:39 PM
    • 2,871 Lines
  • Apr 07 03:33:13 worker0 python3[16392]: [2020-04-07 03:33:13,434: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).')
    Apr 07 03:33:13 worker0 python3[16392]: Traceback (most recent call last):
    Apr 07 03:33:13 worker0 python3[16392]: File "/usr/lib/python3/dist-packages/billiard/pool.py", line 1267, in mark_as_worker_lost
    Apr 07 03:33:13 worker0 python3[16392]: human_status(exitcode)),
    Apr 07 03:33:13 worker0 python3[16392]: billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).
    ...
    • Apr 7 2020, 9:58 AM
    • 10 Lines
  • dir_to_dir
    old: 48341950415
    new: 49543338345
    +2.48%
    ...
    • Apr 5 2020, 2:12 PM
    • 42 Lines
  • origin_visit
    old: 194970670
    new: 1009322644
    snapshot
    ...
    • Apr 4 2020, 12:20 PM
    • 19 Lines
  • $ cat foo.py
    from tenacity import retry
    @retry
    def foo(n):
    ...
    • Apr 2 2020, 6:59 PM
    • 18 Lines
  • @overload
    def _get_key(self, object_type: str, object_: Union[Revision, Release, Directory, Snapshot]) -> bytes:
    ...
    ...
    • Apr 2 2020, 4:22 PM
    • 37 Lines
    • Python
  • def _get_key(
    self,
    object_type: str,
    object_: BaseModel) -> Union[bytes, Dict]:
    if object_type in ('revision', 'release', 'directory', 'snapshot'):
    ...
    • Apr 2 2020, 4:03 PM
    • 28 Lines
    • Python
  • Info: Loading facts
    Info: Caching catalog for 9fb17b6df4b3.test
    Info: Applying configuration version '1585756962'
    Notice: /Stage[main]/Apt/File[preferences]/ensure: created
    Info: /Stage[main]/Apt/File[preferences]: Scheduling refresh of Class[Apt::Update]
    ...
    • Apr 1 2020, 6:08 PM
    • 279 Lines
  • 14:59 $ doco exec puppet-agent puppet agent -t
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Apr 1 2020, 5:06 PM
    • 307 Lines
  • WARNING:swh.core.cli:Could not load subcommand journal: cannot import name 'HashCollision' from 'swh.storage' (/home/danseraf/swh-environment/swh-storage/swh/storage/__init__.py)
    WARNING:swh.core.cli:Could not load subcommand indexer: cannot import name 'HashCollision' from 'swh.storage' (/home/danseraf/swh-environment/swh-storage/swh/storage/__init__.py)
    WARNING:swh.core.cli:Could not load subcommand search: cannot import name 'HashCollision' from 'swh.storage' (/home/danseraf/swh-environment/swh-storage/swh/storage/__init__.py)
    • Mar 26 2020, 2:32 PM
    • 3 Lines
  • (swh) ~/swh-environment/swh-scanner   master  make check
    python3 -m flake8 swh
    swh/scanner/model.py:70:47: E226 missing whitespace around arithmetic operator
    swh/scanner/scanner.py:18:9: E126 continuation line over-indented for hanging indent
    swh/scanner/scanner.py:25:9: E123 closing bracket does not match indentation of opening bracket's line
    ...
    • Mar 25 2020, 11:00 AM
    • 14 Lines
  • ```
    id | type | arguments | next_run | current_interval | status | policy | retries_left | priorit$
    ---------+-----------------+------------------------------------------------------------------------------------------------------+-------------------------------+------------------+--------------------+-----------+--------------+--------$
    1186662 | load-functional | {"args": [], "kwargs": {"url": "https://nix-community.github.io/nixpkgs-swh/sources.json"}} | 2020-03-25 02:40:59.571373+00 | 1 day | disabled | recurring | 0 |
    1186665 | load-functional | {"args": [], "kwargs": {"url": "https://nix-community.github.io/nixpkgs-swh/sources-unstable.json"}} | 2020-03-25 08:30:08.666509+00 | 1 day | next_run_scheduled | recurring | 0 |
    ...
    • Mar 25 2020, 9:32 AM
    • 8 Lines
  • { config, lib, pkgs, unstable-pkgs, mypkgs, ... }:
    with lib;
    let xsession-enable = config.my.xsession.enable;
    optional-dependencies = config.my.emacs.optional-dependencies;
    ...
    • Mar 24 2020, 8:02 PM
    • 242 Lines
  • from typing import Callable
    class Foo:
    def __init__(self, f: Callable[[int], None]):
    self.f: Callable[[int], None] = f
    ...
    • Mar 24 2020, 5:15 PM
    • 8 Lines
  • from typing import Callable
    class Foo:
    f: Callable[[int], None]
    ...
    • Mar 24 2020, 5:06 PM
    • 15 Lines
  • from typing import Callable
    class Foo:
    f: Callable
    ...
    • Mar 24 2020, 5:02 PM
    • 7 Lines
  • *** rdkafka_cgrp.c:3037:rd_kafka_cgrp_op_serve: assert: rktp->rktp_assigned ***
    rd_kafka_t 0x2121b30: rdkafka#consumer-2
    producer.msg_cnt 0 (0 bytes)
    rk_rep reply queue: 2259 ops
    brokers:
    ...
    • Mar 23 2020, 9:45 PM
    • 12 Lines
  • tony  yavin4  ~  %  my-hm build
    ~/repo/private/home ~
    querying info about '/nix/store/34ckk0vslnar8xj70sa0cgpiyg1r96r5-emacs-with-packages-26.3' on 'https://cache.nixos.org'...
    downloading 'https://cache.nixos.org/34ckk0vslnar8xj70sa0cgpiyg1r96r5.narinfo'...
    querying info about '/nix/store/dmjkw4hlf56x4wg2g5kgw0cj96qjqdp3-emacs-rust-mode-20191208.1654' on 'https://cache.nixos.org'...
    ...
    • Mar 23 2020, 6:28 PM
    • 80 Lines
  • {
    "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
    "@type": "SoftwareSourceCode",
    "name": "Foo Software",
    "author": [
    ...
    • Mar 21 2020, 12:06 PM
    • 25 Lines
    • JSON