Page MenuHomeSoftware Heritage
Paste Active Pastes
  • Jun 18 11:51:45 worker01 python3[29020]: [2019-06-18 11:51:45,602: INFO/MainProcess] Received task: swh.lister.phabricator.tasks.IncrementalPhabricatorLister[6511673e-9881-494d-bc32-e2d5631809df]
    Jun 18 11:51:45 worker01 python3[29026]: [2019-06-18 11:51:45,820: ERROR/ForkPoolWorker-3] Task swh.lister.phabricator.tasks.IncrementalPhabricatorLister[6511673e-9881-494d-bc32-e2d5631809df] raised unexpected: ValueError('Problem during information fetch: ERR-CONDUIT-CORE',)
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
    ...
    • Jun 18 2019, 1:57 PM
    • 23 Lines
  • Jun 18 11:24:42 worker01 python3[12919]: [2019-06-18 11:24:42,238: INFO/MainProcess] Received task: swh.lister.phabricator.tasks.FullPhabricatorLister[551ff6d5-473b-4e6f-8e70-9d8041ab5fce]
    Jun 18 11:27:32 worker01 python3[12934]: [2019-06-18 11:27:32,544: ERROR/ForkPoolWorker-5] Task swh.lister.phabricator.tasks.FullPhabricatorLister[551ff6d5-473b-4e6f-8e70-9d8041ab5fce] raised unexpected: ChunkedEncodingError(ProtocolError('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)),)
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/urllib3/response.py", line 601, in _update_chunk_length
    self.chunk_left = int(line, 16)
    ...
    • Jun 18 2019, 1:34 PM
    • 69 Lines
  • Jun 18 10:25:08 worker06 python3[14188]: [2019-06-18 10:25:08,661: WARNING/ForkPoolWorker-5] Failed to fetch https://developers.blender.org/api/diffusion.repository.search?order=oldest&attachments[uris]=1&after=&order=oldest&limit=1: HTTPSConnectionPool(host='developers.blender.org', port=443): Max retries exceeded with url: /api/diffusion.repository.search?order=oldest&attachment
    s[uris]=1&after=&order=oldest&limit=1&api.token=api-e45yly42hzfvmxi62nssekngtl5w (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8109bbbbe0>: Failed to establish a new connection: [Errno -2] Name or service not known',))
    Jun 18 10:25:08 worker06 python3[14188]: [2019-06-18 10:25:08,661: WARNING/ForkPoolWorker-5] connection error on &order=oldest&limit=1: sleep for 10 seconds
    Jun 18 10:25:18 worker06 python3[14188]: [2019-06-18 10:25:18,671: WARNING/ForkPoolWorker-5] Failed to fetch https://developers.blender.org/api/diffusion.repository.search?order=oldest&attachments[uris]=1&after=&order=oldest&limit=1: HTTPSConnectionPool(host='developers.blender.org', port=443): Max retries exceeded with url: /api/diffusion.repository.search?order=oldest&attachment
    s[uris]=1&after=&order=oldest&limit=1&api.token=api-e45yly42hzfvmxi62nssekngtl5w (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8109bbba20>: Failed to establish a new connection: [Errno -2] Name or service not known',))
    ...
    • Jun 18 2019, 1:11 PM
    • 31 Lines
  • (swh) archit@work-pc:~/swh-environment$ doco up swh-lister
    ...
    swh-lister_1 | swh.model 0.0.36
    swh-lister_1 | swh.objstorage 0.0.33
    swh-lister_1 | swh.scheduler 0.0.56
    ...
    • Jun 18 2019, 12:39 PM
    • 97 Lines
  • -- Align table names to singular (phabricator, bitbucket, etc...) ~> same convention as in the main archive for instance
    alter table phabricator_repos rename to phabricator_repo;
    alter table bitbucket_repos rename to bitbucket_repo;
    alter table github_repos rename to github_repo;
    -- Add the missing instance column
    ...
    • Jun 18 2019, 7:05 AM
    • 10 Lines
  • P433 error
    (swh) archit@work-pc:~/swh-environment/swh-lister$ python3 setup.py develop
    running develop
    running egg_info
    writing swh.lister.egg-info/PKG-INFO
    writing dependency_links to swh.lister.egg-info/dependency_links.txt
    ...
    • Jun 17 2019, 3:50 PM
    • 284 Lines
  • rubygems_development=# SELECT count(*) FROM linksets where code != '' ;
    count
    -------
    17306
    ...
    • Jun 15 2019, 10:21 PM
    • 19 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-lister/swh/lister/packagist$ tox --recreate .
    GLOB sdist-make: /home/archit/swh-environment/swh-lister/setup.py
    flake8 recreate: /home/archit/swh-environment/swh-lister/.tox/flake8
    flake8 installdeps: flake8
    flake8 installed: entrypoints==0.3,flake8==3.7.7,mccabe==0.6.1,pycodestyle==2.5.0,pyflakes==2.1.1,swh.lister==0.0.25.post1
    ...
    • Jun 14 2019, 1:09 PM
    • 130 Lines
  • ______________________________________________ TestGitLoader.test_load_unchanged ______________________________________________
    self = <swh.loader.git.tests.test_loader.TestGitLoader testMethod=test_load_unchanged>
    def test_load_unchanged(self):
    ...
    • Jun 12 2019, 6:27 PM
    • 48 Lines
  • Here is a short report of my work on listers:
    Listers I proposed to implement in my proposal -
    GNU Lister
    Completed and merged
    ...
    • Jun 11 2019, 12:41 PM
    • 24 Lines
  • [haltode@sexus ~]$ jmap -histo 27604 | head -n 20
    num #instances #bytes class name (module)
    -------------------------------------------------------
    1: 556765035 583484484400 [B (java.base@12.0.1)
    2: 2 5047568000 [[B (java.base@12.0.1)
    ...
    • Jun 9 2019, 7:05 AM
    • 24 Lines
  • P426 cli
    # Copyright (C) 2018 The Software Heritage developers
    # See the AUTHORS file at the top-level directory of this distribution
    # License: GNU General Public License version 3, or any later version
    # See top-level LICENSE file for more information
    ...
    • Jun 8 2019, 7:11 PM
    • 155 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-lister$ swh scheduler task list
    Traceback (most recent call last):
    File "/home/archit/.virtualenvs/swh/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2419, in resolve
    return functools.reduce(getattr, self.attrs, module)
    AttributeError: module 'swh.lister.cli' has no attribute 'lister'
    ...
    • Jun 8 2019, 5:48 PM
    • 18 Lines
  • Task 1772
    Next run: in 3 months (2019-09-06 13:00:14+00:00)
    Interval: 90 days, 0:00:00
    Type: load-cran
    Policy: recurring
    ...
    • Jun 8 2019, 3:05 PM
    • 13 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-lister$ python ttesting.py
    DEBUG:swh.lister.core.lister_base:Loading config from lister_gnu
    INFO:swh.core.config:Loading config file /home/archit/.config/swh/lister_gnu.yml
    DEBUG:swh.lister.core.lister_base:<swh.lister.gnu.lister.GNULister object at 0x7fb7397e49e8> CONFIG={'content_size_limit': 104857600, 'log_db': 'dbname=softwareheritage-log', 'storage': {'cls': 'remote', 'args': {'url': 'http://localhost:5002/'}}, 'scheduler': {'cls': 'remote', 'args': {'url': 'http://localhost:5008/'}}, 'lister': {'cls': 'local', 'args': {'db': 'postgresql:///lister-gnu'}}, 'credentials': [], 'cache_responses': True, 'cache_dir': '/home/archit/.cache/swh/lister/gnu/'}
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ftp.gnu.org:443
    ...
    • Jun 7 2019, 5:53 PM
    • 40 Lines
  • wh-scheduler-api_1 | Using pip from /srv/softwareheritage/venv/bin/pip
    swh-scheduler-api_1 | Installed Python packages:
    swh-scheduler-api_1 | Package Version
    swh-scheduler-api_1 | ----------------------------- --------
    swh-scheduler-api_1 | aiohttp 3.5.4
    ...
    • Jun 6 2019, 10:42 PM
    • 505 Lines
  • Name Command State Ports
    --------------------------------------------------------------------------------
    swh-docker- docker-entrypoint.sh Up 15671/tcp,
    dev_amqp_1 rabbi ... 15672/tcp,
    25672/tcp,
    ...
    • Jun 6 2019, 10:39 PM
    • 65 Lines
  • python ttesting.py
    DEBUG:swh.lister.core.lister_base:Loading config from lister_gnu
    INFO:swh.core.config:Loading config file /home/archit/.config/swh/lister_gnu.yml
    DEBUG:swh.lister.core.lister_base:<swh.lister.gnu.lister.GNULister object at 0x7f526d50dac8> CONFIG={'content_size_limit': 104857600, 'log_db': 'dbname=softwareheritage-log', 'storage': {'cls': 'remote', 'args': {'url': 'http://localhost:5002/'}}, 'scheduler': {'cls': 'remote', 'args': {'url': 'http://localhost:5008/'}}, 'lister': {'cls': 'local', 'args': {'db': 'postgresql:///lister-gnu'}}, 'credentials': [], 'cache_responses': True, 'cache_dir': '/home/archit/.cache/swh/lister/gnu/'}
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ftp.gnu.org:443
    ...
    • Jun 6 2019, 9:45 PM
    • 35 Lines
  • import java.math.BigInteger;
    import static java.math.BigInteger.ZERO;
    import static java.nio.charset.StandardCharsets.US_ASCII;
    import static java.util.Arrays.copyOf;
    import static java.lang.Math.max;
    ...
    • Jun 6 2019, 2:29 PM
    • 48 Lines
    • Java
  • import java.math.BigInteger;
    import static java.math.BigInteger.ZERO;
    import static java.nio.charset.StandardCharsets.US_ASCII;
    import static java.util.Arrays.copyOf;
    import static java.lang.Math.max;
    ...
    • Jun 6 2019, 12:03 PM
    • 48 Lines
    • Java
  • (swh) archit@work-pc:~/swh-environment/swh-lister$ python3 -m swh.lister.cli --db-url postgres:///lister-gnu gnu
    Traceback (most recent call last):
    File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
    File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    ...
    • Jun 5 2019, 10:42 AM
    • 36 Lines
  • version: '2'
    services:
    swh-lister:
    volumes:
    ...
    • Jun 5 2019, 10:19 AM
    • 6 Lines
  • $ gem list -r --all
    ...
    zulip (0.0.2, 0.0.1)
    zulip-client (0.2.4, 0.2.3, 0.2.2, 0.2.1, 0.2.0, 0.1.0)
    zulip_machine (0.0.1)
    ...
    • Jun 2 2019, 8:23 PM
    • 82 Lines
  • [haltode@sexus software-heritage]$ zgrep "f39d7d78b70e0f39facb1e4fab77ad3df5c52a35" rev_to_rev.edges.csv.gz
    f39d7d78b70e0f39facb1e4fab77ad3df5c52a35 3150ce629cf66d313453fd9a5e20b3f4884911eb
    f39d7d78b70e0f39facb1e4fab77ad3df5c52a35 30a7acd573899fd8b8ac39236eff6468b195ac7d
    52c90f2d32bfa7d6eccd66a56c44ace1f78fbadd f39d7d78b70e0f39facb1e4fab77ad3df5c52a35
    a31e58e129f73ab5b04016330b13ed51fde7a961 f39d7d78b70e0f39facb1e4fab77ad3df5c52a35
    • May 31 2019, 9:35 AM
    • 5 Lines
  • P411 Error
    arc diff --update D1482 --less-context origin/master
    You have untracked files in this working copy.
    Working copy: /home/archit/swh-environment/swh-lister/
    ...
    • May 28 2019, 4:47 PM
    • 35 Lines
  • arc diff --update D1482 origin/master
    You have untracked files in this working copy.
    Working copy: /home/archit/swh-environment/swh-lister/
    ...
    • May 28 2019, 4:43 PM
    • 36 Lines
  • 3dldf
    https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.3.tar.gz
    https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.4.tar.gz
    https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.5.1.tar.gz
    https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.5.tar.gz
    ...
    • May 27 2019, 11:42 PM
    • 8,990 Lines
  • {'name': '3dldf', 'url': 'https://ftp.gnu.org/gnu/3dldf/', 'list_of_tarballs': [{'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.3.tar.gz', 'date': '1071002600'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.4.tar.gz', 'date': '1071078759'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.5.1.tar.gz', 'date': '1074278633'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-1.1.5.tar.gz', 'date': '1074281010'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-2.0.1.tar.gz', 'date': '1384100514'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-2.0.2.tar.gz', 'date': '1384108703'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-2.0.3-examples.tar.gz', 'date': '1386961043'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-2.0.3.tar.gz', 'date': '1386954662'}, {'archive': 'https://ftp.gnu.org/gnu/3dldf/3DLDF-2.0.tar.gz', 'date': '1383935767'}]}
    {'name': '8sync', 'url': 'https://ftp.gnu.org/gnu/8sync/', 'list_of_tarballs': [{'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.1.0.tar.gz', 'date': '1461357336'}, {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.2.0.tar.gz', 'date': '1480991830'}, {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.3.0.tar.gz', 'date': '1482181784'}, {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.4.0.tar.gz', 'date': '1483894300'}, {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.4.1.tar.gz', 'date': '1483911251'}, {'archive': 'https://ftp.gnu.org/gnu/8sync/8sync-0.4.2.tar.gz', 'date': '1489817041'}]}
    {'name': 'a2ps', 'url': 'https://ftp.gnu.org/gnu/a2ps/', 'list_of_tarballs': [{'archive': 'https://ftp.gnu.org/gnu/a2ps/a2ps-4.10.4.tar.gz', 'date': '907035300'}, {'archive': 'https://ftp.gnu.org/gnu/a2ps/a2ps-4.12.tar.gz', 'date': '919729234'}, {'archive': 'https://ftp.gnu.org/gnu/a2ps/a2ps-4.13b.tar.gz', 'date': '1015264004'}, {'archive': 'https://ftp.gnu.org/gnu/a2ps/a2ps-4.14.tar.gz', 'date': '1198900401'}]}
    {'name': 'acct', 'url': 'https://ftp.gnu.org/gnu/acct/', 'list_of_tarballs': [{'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.3.2.tar.gz', 'date': '889516800'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.3.tar.gz', 'date': '872060400'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.5.1.tar.gz', 'date': '1252260451'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.5.3.tar.gz', 'date': '1264282921'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.5.4.tar.gz', 'date': '1265992809'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.5.5.tar.gz', 'date': '1288986081'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.5.tar.gz', 'date': '1251999322'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.6.1.tar.gz', 'date': '1362788330'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.6.2.tar.gz', 'date': '1415424027'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.6.3.tar.gz', 'date': '1492450287'}, {'archive': 'https://ftp.gnu.org/gnu/acct/acct-6.6.4.tar.gz', 'date': '1499010720'}]}
    {'name': 'acm', 'url': 'https://ftp.gnu.org/gnu/acm/', 'list_of_tarballs': [{'archive': 'https://ftp.gnu.org/gnu/acm/acm-5.1.tar.gz', 'date': '1291834535'}]}
    ...
    • May 27 2019, 11:12 PM
    • 378 Lines
  • [
    {
    "Package": "A3",
    "Version": "1.0.0",
    "Depends": "R (>= 2.15.0), xtable, pbapply",
    ...
    • May 26 2019, 7:44 PM
    • 99 Lines
  • #+title: dbs
    Dbs inventory so far.
    Order by node name/cluster:port
    ...
    • May 23 2019, 3:46 PM
    • 94 Lines
    • Plain Text
  • <?xml version='1.0' encoding='UTF-8'?>
    <index>
    <info>
    <indexPath>/home/archit/Downloads</indexPath>
    <fields count='31'>
    ...
    • May 23 2019, 1:02 PM
    • 1,598 Lines
  • Found host vangogh.euwest.azure.internal.softwareheritage.org
    *** Running octocatalog-diff on host vangogh.euwest.azure.internal.softwareheritage.org
    diff origin/production/vangogh.euwest.azure.internal.softwareheritage.org current/vangogh.euwest.azure.internal.softwareheritage.org
    *******************************************
    + Concat::Fragment[nginx-swh-objstorage-500-6666cd76f96956469e7be39d750cc7d9] =>
    ...
    • May 22 2019, 10:05 AM
    • 1,994 Lines
    • Bash Scripting
  • #!/bin/bash
    cat log_mph log_bv log_bfs log_transform > log
    rm log_*
    cat timings_mph timings_bv timings_obl timings_bfs timings_transform timings_obl2 timings_stats > timings
    ...
    • May 19 2019, 6:45 AM
    • 6 Lines
    • Bash Scripting
  • Graph compression output
    ========================
    These are the output directories from experiments running the WebGraph framework
    to compress the Software Heritage graph datasets. Each directory is the output
    ...
    • May 18 2019, 7:49 AM
    • 16 Lines
  • (swh) morane@hplaptopft0:~/Documents/code/swh-environment/swh-docs$ tox -e sphinx-dev
    GLOB sdist-make: /home/morane/Documents/code/swh-environment/swh-docs/setup.py
    sphinx-dev create: /home/morane/Documents/code/swh-environment/swh-docs/.tox/sphinx-dev
    sphinx-dev installdeps: django < 2, -rrequirements-swh-dev.txt, pifpaf
    ...
    • May 15 2019, 2:30 PM
    • 23 Lines
  • Stats
    -----
    Returns statistics on the compressed graph.
    ...
    • May 13 2019, 8:00 AM
    • 74 Lines
  • # Edge dataset
    The dataset in this folder only contains informations about the **edges** of the
    Software Heritage Graph (and none of the associated metadata). This is useful
    for studying the **topology** of the graph.
    ...
    • May 12 2019, 1:56 PM
    • 39 Lines
  • #!/bin/bash
    for dataset in dir_to_dir dir_to_file dir_to_rev origin_to_snapshot \
    release_to_obj rev_to_dir rev_to_rev snapshot_to_obj; do
    mv $dataset.csv.gz $dataset.edges.csv.gz
    ...
    • May 12 2019, 4:12 AM
    • 9 Lines
    • Bash Scripting
  • /srv/ftp/pub/R/src/contrib/00Archive/A3/A3_0.9.1.tar.gz 45252 FALSE 664 2013-02-07 14:30:29 2019-05-05 09:40:28 2019-05-05 02:56:44 1001 1001 hornik cranadmin
    ...
    • May 10 2019, 7:55 PM
    • 30 Lines
  • update deposit set check_task_id='167830447', load_task_id='167830455' where id = 264;
    update deposit set check_task_id='164895648', load_task_id='167830454' where id = 263;
    update deposit set check_task_id='160036918', load_task_id='160037202' where id = 262;
    update deposit set check_task_id='159935272', load_task_id='159936134' where id = 261;
    ...
    • May 9 2019, 11:23 AM
    • 15 Lines
    • SQL
  • ~/annex/dataset/swh-graph-2019-01-28/edges $ cat *.csv.count | paste -d+ -s | bc
    164_513_703_039 # 160 B
    ~/annex/dataset/swh-graph-2019-01-28/edges $ cat *.nodes.count | paste -d+ -s | bc
    17_537_088_222 # 17 B
    • May 7 2019, 1:28 PM
    • 4 Lines
  • Hello,
    sorry for the late reply.
    So, as zack mentioned, we did the changes.
    ...
    • May 6 2019, 3:30 PM
    • 20 Lines
  • swh_app = <Celery celery.tests at 0x7ff6a13d0048>
    celery_session_worker = <Worker: gen658@b84cee4ca65b (running)>
    def test_ping(swh_app, celery_session_worker):
    res = swh_app.send_task(
    ...
    • Apr 28 2019, 12:02 AM
    • 69 Lines
  • swh-docker-dev_zookeeper_1 is up-to-date
    Starting swh-docker-dev_swh-objstorage_1 ...
    swh-docker-dev_swh-storage-db_1 is up-to-date
    swh-docker-dev_swh-idx-storage-db_1 is up-to-date
    Starting swh-docker-dev_swh-objstorage_1 ... done
    ...
    • Apr 23 2019, 3:08 PM
    • 11 Lines
  • Name Command State Ports
    -------------------------------------------------------------------------------------------------------------------------------------------------------------------
    swh-docker-dev_amqp_1 docker-entrypoint.sh rabbi ... Up 15671/tcp, 15672/tcp, 25672/tcp, 4369/tcp, 5671/tcp, 0.0.0.0:5072->5672/tcp
    swh-docker-dev_grafana_1 /run.sh Up 3000/tcp
    swh-docker-dev_kafka-manager_1 /kafka-manager/bin/kafka-m ... Up 0.0.0.0:5093->9000/tcp
    ...
    • Apr 23 2019, 2:47 PM
    • 30 Lines
  • azure@desktop5~/ansible   master ✚  cat plugins/inventory/terraform.py
    import json
    import os
    from subprocess import check_output
    ...
    • Apr 17 2019, 2:33 PM
    • 61 Lines
    • Python
  • P382 test.tf
    provider "azurerm" {
    }
    data "azurerm_network_security_group" "worker-nsg" {
    name = "worker-nsg"
    ...
    • Apr 17 2019, 1:37 PM
    • 74 Lines
  • $ ipython
    Python 3.7.2+ (default, Feb 2 2019, 14:31:48)
    Type 'copyright', 'credits' or 'license' for more information
    IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help.
    ...
    • Apr 16 2019, 5:06 PM
    • 23 Lines
    • Python
  • $ parallel
    Academic tradition requires you to cite works you base your article on.
    When using programs that use GNU Parallel to process data for publication
    please cite:
    ...
    • Apr 16 2019, 11:24 AM
    • 18 Lines
  • def test_content_add_collision_sha256(self):
    cont1 = self.cont
    # create (corrupted) content with same sha256 but != sha1{,_git}
    cont1b = cont1.copy()
    ...
    • Apr 2 2019, 4:14 PM
    • 13 Lines
    • Python
  • diff --git a/swh-team/swh-weekly-report b/swh-team/swh-weekly-report
    index 10b53e1..ec8f64d 100755
    --- a/swh-team/swh-weekly-report
    +++ b/swh-team/swh-weekly-report
    @@ -9,7 +9,7 @@ from dateutil.relativedelta import relativedelta
    ...
    • Apr 1 2019, 6:02 PM
    • 93 Lines
    • Diff
  • Delivered-To: antoine.romain.dumont@gmail.com
    Received: by 2002:a02:c722:0:0:0:0:0 with SMTP id h2csp1100444jao;
    Wed, 27 Mar 2019 11:20:52 -0700 (PDT)
    X-Google-Smtp-Source: APXvYqy7L3Rh5mg/8x1/ZMWHV2gDO+U7xH/Ps0EYSqvWFS2HHupe3xHr8/Y0jPrUmtAJR/vaAWu9
    X-Received: by 2002:adf:8367:: with SMTP id 94mr25834115wrd.46.1553710852555;
    ...
    • Mar 28 2019, 10:30 AM
    • 141 Lines
  • as of now, swh-environment within nix:
    ```
    $ nix-shell swh.nix
    ```
    ...
    • Mar 23 2019, 11:28 PM
    • 257 Lines
  • $ ./swh-weekly-report.py
    Tasks (subscribed):
    - T1534 | PostgreSQL replication issues between prado and somerset
    - T1276 | swh-journal: Add tests
    ...
    • Mar 20 2019, 11:07 PM
    • 18 Lines
  • <Multi_key> <d> <r> : "https://forge.softwareheritage.org/diffusion" # repositories
    <Multi_key> <d> <t> : "https://forge.softwareheritage.org/tasks"
    <Multi_key> <d> <p> : "https://forge.softwareheritage.org/paste"
    <Multi_key> <d> <g> : "https://wiki.softwareheritage.org/wiki/Git_style_guide"
    <Multi_key> <d> <d> : "https://docs.softwareheritage.org/devel/getting-started.html#getting-started"
    ...
    • Mar 20 2019, 9:55 AM
    • 7 Lines
  • (swh) archit@work-pc:~/swh-environment/swh-lister$ python
    Python 3.6.7 (default, Oct 22 2018, 11:32:17)
    [GCC 8.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import logging
    ...
    • Mar 19 2019, 10:22 PM
    • 100 Lines
  • [<FrameSummary file /usr/lib/python3.5/runpy.py, line 193 in _run_module_as_main>, <FrameSummary file /usr/lib/python3.5/runpy.py, line 85 in _run_code>, <FrameSummary file /usr/lib/python3/dist-packages/celery/__main__.py, line 20 in <module>>, <FrameSummary file /usr/lib/python3/dist-packages/celery/__main__.py, line 16 in main>, <FrameSummary file /usr/lib/python3/dist-packages/celery/bin/celery.py, line 322 in main>, <FrameSummary file /usr/lib/python3/dist-packages/celery/bin/celery.py, line 496 in execute_from_commandline>, <FrameSummary file /usr/lib/python3/dist-packages/celery/bin/base.py, line 275 in execute_from_commandline>, <FrameSummary file /usr/lib/python3/dist-packages/celery/bin/celery.py, line 488 in handle_argv>, <FrameSummary file /usr/lib/python3/dist-packages/celery/bin/celery.py, line 420 in execute>, <FrameSummary file /usr/lib/python3/dist-packages/celery/bin/worker.py, line 223 in run_from_argv>, <FrameSummary file /usr/lib/python3/dist-packages/celery/bin/base.py, line 238 in...
    • Mar 18 2019, 5:17 PM
    • 1 Line
  • Package Version Location
    --------------------------- ------------- -------------------------------------------------
    aiohttp 4.0.0a0
    alabaster 0.7.12
    amqp 2.4.2
    ...
    • Mar 18 2019, 11:19 AM
    • 150 Lines
  • From: ardumont@softwareheritage.org
    To: sysadmin@fsf.org
    Cc: swh-devel@inria.fr
    Subject: [swh] GNU listing adaptation please?
    Fcc: sent
    ...
    • Mar 12 2019, 5:52 PM
    • 38 Lines
  • Using the Software Heritage Graph Dataset
    =========================================
    This README contains instructions on how to use the different formats the
    *Software Heritage graph dataset* is distributed as.
    ...
    • Mar 11 2019, 4:06 PM
    • 129 Lines
  • *** swh-deploy: starting test run on moma.internal.softwareheritage.org...
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Loading facts
    ...
    • Mar 4 2019, 3:28 PM
    • 59 Lines
  • swh-web_1 | starting the swh-web server
    swh-web_1 | [2019-03-04 12:25:26 +0000] [1] [INFO] Starting gunicorn 19.9.0
    swh-web_1 | [2019-03-04 12:25:26 +0000] [1] [INFO] Listening at: http://0.0.0.0:5004 (1)
    swh-web_1 | [2019-03-04 12:25:26 +0000] [1] [INFO] Using worker: sync
    swh-web_1 | [2019-03-04 12:25:26 +0000] [13] [INFO] Booting worker with pid: 13
    ...
    • Mar 4 2019, 1:29 PM
    • 53 Lines
  • swh-web_1 | TypeError: 'NoneType' object is not iterable
    swh-web_1 | [2019-03-03 14:26:03 +0000] [18] [ERROR] Error handling request /static/img/icons/swh-logo-deposit-192x192.png
    swh-web_1 | Traceback (most recent call last):
    swh-web_1 | File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    swh-web_1 | self.handle_request(listener, req, client, addr)
    ...
    • Mar 3 2019, 4:31 PM
    • 223 Lines
  • update revision_metadata
    set translated_metadata = origin_intrinsic_metadata.metadata
    from origin_intrinsic_metadata
    where revision_metadata.id=origin_intrinsic_metadata.from_revision and revision_metadata.translated_metadata='{"@context": "https://doi.org/10.5063/schema/codemeta-2.0"}' and origin_intrinsic_metadata.metadata != '{"@context": "https://doi.org/10.5063/schema/codemeta-2.0"}';
    • Mar 2 2019, 9:50 AM
    • 4 Lines
  • DELETE FROM revision_metadata
    WHERE translated_metadata = '{"@context": "https://doi.org/10.5063/schema/codemeta-2.0"}'::jsonb ;
    DELETE FROM origin_intrinsic_metadata
    WHERE metadata = '{"@context": "https://doi.org/10.5063/schema/codemeta-2.0"}'::jsonb ;
    • Mar 1 2019, 2:32 PM
    • 5 Lines
  • *** swh-deploy: starting test run on moma.internal.softwareheritage.org...
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Loading facts
    ...
    • Feb 28 2019, 6:51 PM
    • 59 Lines
  • Old :
    CREATE TABLE public.directory (
    id public.sha1_git NOT NULL,
    dir_entries bigint[],
    ...
    • Feb 19 2019, 11:48 AM
    • 33 Lines
  • Feb 14 17:46:13 worker01 python3[2344]: [2019-02-14 17:46:13,609: ERROR/ForkPoolWorker-1] Task swh.deposit.loader.tasks.ChecksDepositTsk[e201d7a8-6f19-4248-a08c-d10874c2e6a3] raised unexpected: AttributeError("'DepositChecker' object has no attribute 'log'",)
    Feb 14 17:46:13 worker01 python3[2344]: Traceback (most recent call last):
    Feb 14 17:46:13 worker01 python3[2344]: File "/usr/lib/python3/dist-packages/swh/deposit/loader/checker.py", line 21, in check
    Feb 14 17:46:13 worker01 python3[2344]: self.client.check(deposit_check_url)
    Feb 14 17:46:13 worker01 python3[2344]: File "/usr/lib/python3/dist-packages/swh/deposit/client/__init__.py", line 208, in check
    ...
    • Feb 14 2019, 7:01 PM
    • 42 Lines
  • Feb 14 14:24:05 worker01 python3[4534]: [2019-02-14 14:24:05,077: ERROR/ForkPoolWorker-1] Task swh.lister.gitlab.tasks.IncrementalGitLabLister[f7f2d92d-bfa9-4994-a637-e3859a73f432] raised unexpected: TypeError('incremental_gitlab_lister() takes 0 positional arguments but 1 was given',)
    Feb 14 14:24:05 worker01 python3[4534]: Traceback (most recent call last):
    Feb 14 14:24:05 worker01 python3[4534]: File "/usr/lib/python3/dist-packages/celery/app/trace.py", line 382, in trace_task
    Feb 14 14:24:05 worker01 python3[4534]: R = retval = fun(*args, **kwargs)
    Feb 14 14:24:05 worker01 python3[4534]: File "/usr/lib/python3/dist-packages/swh/scheduler/task.py", line 45, in __call__
    ...
    • Feb 14 2019, 3:39 PM
    • 18 Lines
  • rng | frq | bar
    -------------+----------+--------------------------------
    [4,73) | 22473062 | ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
    [72,142) | 4225673 | ■■■■■■
    [141,211) | 3351946 | ■■■■
    ...
    • Feb 12 2019, 4:41 PM
    • 22 Lines
  • goal: Make the stretch-swh's build for the scheduler ok
    Relevant commit:
    ```
    cd swh-scheduler
    ...
    • Feb 11 2019, 4:24 PM
    • 38 Lines
  • ============================= test session starts ==============================
    platform linux -- Python 3.5.3, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
    rootdir: /<<PKGBUILDDIR>>, inifile:
    plugins: postgresql-1.3.4, hypothesis-3.6.1, celery-4.2.1
    collected 42 items
    ...
    • Feb 11 2019, 11:15 AM
    • 68 Lines
  • Started by timer
    Running in Durability level: MAX_SURVIVABILITY
    Loading library swh@master
    Attempting to resolve master from remote references...
    > git --version # timeout=10
    ...
    • Feb 7 2019, 3:38 PM
    • 138 Lines
  • gemspec | codemeta | pkginfo | npm | maven | total | total_nonempty
    ---------+----------+---------+---------+--------+----------+----------------
    143688 | 139 | 3525 | 1313459 | 447697 | 17730303 | 2033490
    • Feb 4 2019, 10:55 AM
    • 3 Lines
  • #!/bin/bash
    # wrapper to run GitHub Licensee license detection tool form a git clone of its
    # repo, setting up the appropriate Ruby load path
    ...
    • Feb 2 2019, 1:35 PM
    • 8 Lines
  • Jan 31 13:04:36 storage0 python3 [2434334]: 2019-01-31 13:04:36 [2434334] [ERROR] canceling statement due to statement timeout
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request ()
    File "/usr/lib/python3/dist-packages/flask/app.py", line 1598, in dispatch_request
    ...
    • Jan 31 2019, 2:11 PM
    • 31 Lines
  • *** swh-deploy: deploying on moma.internal.softwareheritage.org...
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Loading facts
    ...
    • Jan 25 2019, 2:49 PM
    • 51 Lines
  • swh-scheduler-api_1 | [INFO] werkzeug -- 172.20.0.15 - - [24/Jan/2019 10:48:57] "POST /create_tasks HTTP/1.1" 200 -
    swh-scheduler-api_1 | [INFO] werkzeug -- 172.20.0.15 - - [24/Jan/2019 10:48:57] "POST /create_tasks HTTP/1.1" 200 -
    swh-scheduler-api_1 | [INFO] werkzeug -- 172.20.0.15 - - [24/Jan/2019 10:48:57] "POST /create_tasks HTTP/1.1" 200 -
    swh-scheduler-api_1 | [ERROR] root -- relation "tmp_task" already exists
    ...
    • Jan 24 2019, 11:56 AM
    • 47 Lines
  • $ pip3 show pyld | grep Version
    Version: 1.0.3
    $ python3
    Python 3.5.3 (default, Sep 27 2018, 17:25:39)
    [GCC 6.3.0 20170516] on linux
    ...
    • Jan 16 2019, 5:57 PM
    • 29 Lines
  • >>> import pprint
    >>> import swh.indexer.storage.api.client
    >>> s = swh.indexer.storage.api.client.RemoteStorage(url='http://uffizi.internal.softwareheritage.org:5007/')
    >>> pprint.pprint(s.origin_intrinsic_metadata_search_fulltext(['James']))
    [{'from_revision': b'\xd4bM\xa6\x9eH\x06\x15\x0c\x1ap\xbc\x84~\x11\x17'
    ...
    • Jan 14 2019, 1:40 PM
    • 85 Lines
    • Python
  • import sys
    import time
    #import kafka
    ...
    • Jan 10 2019, 1:38 PM
    • 32 Lines
  • # Copyright (C) 2018 The Software Heritage developers
    # See the AUTHORS file at the top-level directory of this distribution
    # License: GNU General Public License version 3, or any later version
    # See top-level LICENSE file for more information
    ...
    • Jan 10 2019, 11:53 AM
    • 50 Lines
    • Python
  • -- DONE
    CREATE TABLE ctas_dataset_dir_to_rev
    WITH (format = 'TEXTFILE', external_location =
    's3://softwareheritage/edges_dataset/dir_to_rev/', field_delimiter = ' ')
    AS SELECT to_hex(directory.id) as source, to_hex(target) as dest
    ...
    • Jan 7 2019, 7:15 PM
    • 72 Lines
    • SQL
  • swh-environment $ pip install $( ./bin/pip-swh-packages --with-testing )
    swh-storage[schemata,listener][testing] should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+
    • Dec 20 2018, 3:31 PM
    • 2 Lines
  • ```
    $ pifpaf run postgresql -- pytest
    WARNING [pifpaf.drivers] `psutil.Popen(pid=23360, status='terminated')` is already gone, sending SIGKILL to its process group
    ERROR [pifpaf] sequence item 0: expected str instance, bytes found
    ```
    ...
    • Dec 20 2018, 3:24 PM
    • 45 Lines
  • swh-scheduler-api_1 | ERROR:root:fe_sendauth: no password supplied
    swh-scheduler-api_1 | Traceback (most recent call last):
    swh-scheduler-api_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    swh-scheduler-api_1 | rv = self.dispatch_request()
    swh-scheduler-api_1 | File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
    ...
    • Dec 17 2018, 3:22 PM
    • 19 Lines
  • ✘ dev@desktop5  ~/swh-environment/swh-docs   master  git pull
    Already up-to-date.
    dev@desktop5  ~/swh-environment/swh-docs   master  tox -r -e sphinx-dev
    GLOB sdist-make: /home/dev/swh-environment/swh-docs/setup.py
    sphinx-dev recreate: /home/dev/swh-environment/swh-docs/.tox/sphinx-dev
    ...
    • Nov 29 2018, 6:29 PM
    • 46 Lines
  • When i have something like:
    ```
    if a:
    r = do_something(a)
    ...
    • Nov 22 2018, 11:46 AM
    • 21 Lines
  • Hypothesis issue with old version: https://github.com/HypothesisWorks/hypothesis/issues/290
    build output:
    ```
    ============================= test session starts ==============================
    ...
    • Nov 19 2018, 11:38 AM
    • 98 Lines
  • pytest swh/indexer/tests/storage/test_storage.py -x
    ========================================================================================================= test session starts =========================================================================================================
    platform linux -- Python 3.5.3, pytest-3.9.3, py-1.7.0, pluggy-0.8.0
    hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/dev/swh-environment/swh-indexer/.hypothesis/examples')
    rootdir: /home/dev/swh-environment/swh-indexer, inifile: pytest.ini
    ...
    • Nov 15 2018, 10:52 AM
    • 99 Lines
  • diff <(grep 'def ' swh/storage/in_memory.py | grep -v 'def _' | sed -e 's/(.*//' | sort) <(grep 'def ' swh/storage/storage.py | grep -v 'def _' | sed -e 's/(.*//' | sort)
    0a1
    > def add_to_objstorage
    3a5
    > def content_get
    ...
    • Nov 15 2018, 10:07 AM
    • 21 Lines
    • Diff
  • | context | file_name | counted | percentage | percentage | percentage on 3,424,000,000 files |
    |-------------------------------|-------------------|----------------------|------------|------------|-----------------------------------|
    | CodeMeta | CODE | 320 | 0.00% | | 8.85E-06 |
    | haskell | .cabal | 676053 | 1.27% | 0.01% | 0.01870298068 |
    | java- Maven | pom.xml | 15509125 | 29.03% | 0.43% | 0.4290593566 |
    ...
    • Nov 12 2018, 5:11 PM
    • 33 Lines
  • def _naive_sig(param_names):
    return inspect.Signature([
    inspect.Parameter(name, inspect.Parameter.POSITIONAL_OR_KEYWORD)
    for name in param_names])
    ...
    • Nov 9 2018, 1:07 PM
    • 82 Lines
    • Python
  • def content_mimetype_missing(self, mimetypes, db=None, cur=None):
    """Generates mimetypes missing from storage.
    Args:
    mimetypes (iterable): iterable of dict with keys:
    ...
    • Nov 8 2018, 12:03 PM
    • 13 Lines
    • Python
  • commit edebe6a4a42bae99a1819898b14bb0951cfe6b8b
    Author: Valentin Lorentz <vlorentz@softwareheritage.org>
    Date: Mon Nov 5 14:54:06 2018 +0100
    Remove testrepo.zip.
    ...
    • Nov 5 2018, 2:52 PM
    • 64 Lines
    • Diff
  • GLOB sdist-make: /home/morane/Documents/code/swh-environment/swh-indexer/setup.py
    flake8 recreate: /home/morane/Documents/code/swh-environment/swh-indexer/.tox/flake8
    flake8 installdeps: flake8
    flake8 installed: flake8==3.6.0,mccabe==0.6.1,pkg-resources==0.0.0,pycodestyle==2.4.0,pyflakes==2.0.0
    flake8 runtests: PYTHONHASHSEED='3997505493'
    ...
    • Oct 30 2018, 4:31 PM
    • 3,664 Lines
  • dev@desktop5  ~/swh-environment/swh-indexer   master  git pull
    remote: Counting objects: 14, done.
    remote: Compressing objects: 100% (14/14), done.
    remote: Total 14 (delta 10), reused 0 (delta 0)
    Unpacking objects: 100% (14/14), done.
    ...
    • Oct 29 2018, 10:22 AM
    • 14 Lines
  • If it's urgent to redeploy indexers, here is what i foresee in the
    current state of affairs,
    - After review and acceptance, merge the diffs:
    ...
    • Oct 27 2018, 11:16 AM
    • 58 Lines
  • celery.worker.strategy: INFO: Received task: swh.indexer.tests.test_origin_metadata.test_revision_metadata_task[785b1155-89f2-4aec-ac50-c2d3eb34b4d3]
    celery.app.trace: ERROR: Task swh.indexer.tests.test_origin_metadata.test_revision_metadata_task[785b1155-89f2-4aec-ac50-c2d3eb34b4d3] raised unexpected: EncodeError(TypeError("b'8dbb6aeb036e7fd80664eb8bfd1507881af1ba9f' is not JSON serializable",),)
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
    ...
    • Oct 25 2018, 3:08 PM
    • 81 Lines
  • ======================================================================
    FAIL: test_pipeline (swh.indexer.tests.test_origin_metadata.TestOriginMetadata)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    ...
    • Oct 25 2018, 2:47 PM
    • 26 Lines