Page MenuHomeSoftware Heritage
Paste Active Pastes
  • softwareheritage=> copy (select distinct date_offset, encode(date_offset_bytes, 'escape') from revision) to stdout;
    69 +0109
    765 +1245
    -137 -0217
    360 +0600
    ...
    • Fri, Jan 21, 9:21 AM
    • 446 Lines
  • class Statsd:
    # ...
    @contextmanager
    def status_gauge(self, metric_name: str, statuses: Collection[str], common_tags: Optional[Dict[str, str]] = None):
    current_status = Optional[str] = None
    ...
    • Thu, Jan 20, 1:46 PM
    • 19 Lines
    • Python
  • Jan 19 16:11:15 worker0 python3[3375647]: [2022-01-19 16:11:15,120: INFO/MainProcess] Received task: swh.loader.svn.tasks.DumpMountAndLoadSvnRepository[8028cd34-c06f-4dda-b109-a1b315dafabb]
    Jan 19 16:11:18 worker0 python3[3375658]: [2022-01-19 16:11:18,753: INFO/ForkPoolWorker-1] Load origin 'https://plugins.svn.wordpress.org/wp-activity-log-for-woocommerce' with type 'svn'
    Jan 19 21:12:10 worker0 python3[3375658]: [2022-01-19 21:12:10,701: INFO/ForkPoolWorker-1] Processing revisions [1-168825] for {'swh-origin': 'https://plugins.svn.wordpress.org/wp-activity-log-for-woocommerce', 'remote_url': 'file:///tmp/swh.loader.svn.dxrrwh6j-3375658/swh.loader.svn.fw2r9j6a-3375658/tmpiw_ugkd8', 'local_url': b'/tmp/swh.loader.svn.ft2fxai2-3375658/tmpiw_ugkd8', 'uuid': b'b8457f37-d9ea-0310-8a92-e5e31aec5664'}
    Jan 19 21:18:07 worker0 python3[3375658]: [2022-01-19 21:18:07,339: INFO/ForkPoolWorker-1] Task swh.loader.svn.tasks.DumpMountAndLoadSvnRepository[8028cd34-c06f-4dda-b109-a1b315dafabb] succeeded in 18412.19621137576s...
    • Thu, Jan 20, 9:59 AM
    • 4 Lines
  • git,https://scm.gforge.inria.fr/anonscm/git/simty/simty.git
    git,https://github.com/unitystation/unitystation
    svn,http://svn.code.sf.net/p/sauerbraten/code
    git,https://gitlab.com/inkscape/inkscape
    git,https://github.com/keybase/client
    ...
    • Mon, Jan 17, 1:54 PM
    • 25 Lines
  • 12:37 $ git diff
    diff --git a/swh/web/common/origin_save.py b/swh/web/common/origin_save.py
    index 3749d134..6696427b 100644
    --- a/swh/web/common/origin_save.py
    +++ b/swh/web/common/origin_save.py
    ...
    • Thu, Jan 13, 12:38 PM
    • 20 Lines
  • insert into new_task(id, type, arguments, next_run, current_interval, status, policy,
    retries_left, priority)
    select id, type, arguments, next_run, current_interval, status, policy,
    retries_left, priority
    from task
    ...
    • Wed, Jan 12, 1:47 PM
    • 33 Lines
  • -- SWH DB schema upgrade
    -- from_version: 32
    -- to_version: 33
    -- description: Archive old new_task/new_task-run tasks
    ...
    • Mon, Jan 10, 3:52 PM
    • 223 Lines
  • >>> s = '<entry xmlns:testns="https://example.org"> <testns:foo> <testns:bar url="http://example.org/1" /> <testns:bar url="http://example.org/2" /></testns:foo> </entry>'
    >>> pprint(xmltodict.parse(s, process_namespaces=True, dict_constructor=dict))
    {'entry': {'https://example.org:foo': {'https://example.org:bar': [{'@url': 'http://example.org/1',
    '@xmlns': OrderedDict([('testns',
    'https://example.org')])},
    ...
    • Mon, Jan 10, 2:25 PM
    • 6 Lines
    • Python
  • ```
    16:28:34 softwareheritage-scheduler@belvedere:5432=> select type, policy, count(*) from task where type like 'load-%' and status !='disabled' and priority is null group by type, policy;
    +--------------------+-----------+---------+
    | type | policy | count |
    +--------------------+-----------+---------+
    ...
    • Fri, Jan 7, 5:13 PM
    • 21 Lines
  • 16:45:09 swh-web@belvedere:5432=> select * from save_origin_request where origin_url='https://github.com/globalbioticinteractions/usnm-ixodes' order by id desc limit 3;
    +--------+-------------------------------+------------+---------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+--------+
    | id | request_date | visit_type | origin_url | status | loading_task_id | visit_date | loading_task_status | visit_status | user_ids | note |
    +--------+-------------------------------+------------+---------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+--------+
    | 105121 | 2022-01-07 15:14:52.39695+00 | git | https://github.com/globalbioticinteractions/usnm-ixodes | accepted | 404639723 | (null) | succeeded | (null) | (null) | (null) |
    ...
    • Fri, Jan 7, 4:46 PM
    • 8 Lines
  • 11:03:56 Build Architecture: amd64
    11:03:56 Build Type: full
    11:03:56 Build-Space: 109680
    11:03:56 Build-Time: 288
    11:03:56 Distribution: unstable-swh
    ...
    • Tue, Jan 4, 12:14 PM
    • 26 Lines
  • (.venv) jenkins@30f908325208:~/swh-environment/swh-search$ python3 -c "from frozendict import frozendict; frozendict({})"
    Segmentation fault (core dumped)
    (.venv) jenkins@30f908325208:~/swh-environment/swh-search$ ls
    AUTHORS CONTRIBUTORS MANIFEST.in README.md docs mypy.ini package.json pytest.ini requirements-test.txt setup.cfg swh tox.ini
    CODE_OF_CONDUCT.md LICENSE Makefile core es_config node_modules pyproject.toml requirements-swh.txt requirements.txt setup.py swh.search.egg-info yarn.lock
    ...
    • Dec 21 2021, 4:52 PM
    • 62 Lines
  • pytest -s
    ============================================================================================================================= test session starts =============================================================================================================================
    platform linux -- Python 3.7.3, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
    rootdir: /home/test/swh-environment/swh-search, configfile: pytest.ini
    plugins: django-test-migrations-1.2.0, postgresql-2.6.1, swh.core-1.0.0, xdist-2.5.0, forked-1.4.0, redis-2.3.0, flask-1.2.0, asyncio-0.16.0, dash-2.0.0, django-4.5.2, requests-mock-1.9.3, mock-3.6.1, hypothesis-6.31.6, swh.journal-0.9.2.dev2+g4e5e009
    ...
    • Dec 21 2021, 1:30 PM
    • 95 Lines
  • ./bin/update
    Already up to date.
    perl: warning: Setting locale failed.
    perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    ...
    • Dec 21 2021, 12:43 PM
    • 161 Lines
  • vagrant plugin install vagrant-snapshot
    Installing the 'vagrant-snapshot' plugin. This can take a few minutes...
    Building native extensions. This could take a while...
    Vagrant failed to properly resolve required dependencies. These
    errors can commonly be caused by misconfigured plugin installations
    ...
    • Dec 14 2021, 10:01 AM
    • 20 Lines
  • #!/bin/bash
    id=$1
    echo Getting the logs...
    ...
    • Dec 14 2021, 9:46 AM
    • 21 Lines
    • Bash Scripting
  • 15:59:24 softwareheritage-scheduler@belvedere:5432=> select * from listed_origins where visit_type='cvs' limit 10;
    +--------------------------------------+----------------------------------------------+------------+------------------------+---------+-------------------------------+-------------------------------+------------------------+
    | lister_id | url | visit_type | extra_loader_arguments | enabled | first_seen | last_seen | last_update |
    +--------------------------------------+----------------------------------------------+------------+------------------------+---------+-------------------------------+-------------------------------+------------------------+
    | b678cfc3-2780-4186-9186-d78a14bd4958 | https://cvs.code.sf.net/p/a-a-p/code | cvs | {} | t | 2021-06-01 20:24:22.899139+00 | 2021-10-15 04:34:05.365731+00 | 2018-08-12 00:00:00+00 |
    ...
    • Dec 9 2021, 3:59 PM
    • 16 Lines
  • *** swh-deploy: starting test run on moma.internal.softwareheritage.org...
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Dec 8 2021, 3:03 PM
    • 27 Lines
  • diff --git a/swh/loader/svn/ra.py b/swh/loader/svn/ra.py
    index 8a51c49..d88496f 100644
    --- a/swh/loader/svn/ra.py
    +++ b/swh/loader/svn/ra.py
    @@ -11,6 +11,7 @@ from __future__ import annotations
    ...
    • Dec 8 2021, 1:42 PM
    • 245 Lines
  • ```
    Dec 08 08:31:43 thyssen package-helper[6666]: /etc/unbound/unbound.conf.d/listen-docker.conf:1: error: syntax error
    Dec 08 08:31:43 thyssen package-helper[6666]: read /etc/unbound/unbound.conf failed: 1 errors in configuration file
    Dec 08 08:31:43 thyssen package-helper[6669]: /etc/unbound/unbound.conf.d/listen-docker.conf:1: error: syntax error
    Dec 08 08:31:43 thyssen package-helper[6669]: read /etc/unbound/unbound.conf failed: 1 errors in configuration file
    ...
    • Dec 8 2021, 9:36 AM
    • 10 Lines
  • Somehow for now, the pergamon manifest is not complete/incorrect (and it's not clear where exactly we fall short).
    At boot time, we have the puppet master running while it should not. Puppet master must be served through apache (ssl configured and what not).
    So we need to make the provisioning step pass twice (for some unknown reason as of yet as well).
    Then connect to the node, stop and deactivate the puppet master service.
    Then restart the apache service so it serves properly puppet (as master).
    ...
    • Dec 7 2021, 10:27 AM
    • 25 Lines
  • vagrant vm:
    ```
    root@thyssen:~# puppet agent --test
    Info: Using configured environment 'production'
    ...
    • Dec 7 2021, 10:19 AM
    • 76 Lines
  • production:
    ```
    List of databases
    +------------------------------------+--------------+----------+---------+---------+-----------------------+
    | Name | Owner | Encoding | Collate | Ctype | Access privileges |
    ...
    • Dec 6 2021, 5:57 PM
    • 65 Lines
  • 15:04:47 swh-scheduler@db1:5432=> select * from task where type = 'list-debian-distribution';
    +-[ RECORD 1 ]-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | id | 1182486 |
    | type | list-debian-distribution |
    | arguments | {"args": [], "kwargs": {"suites": ["stretch", "buster", "bullseye"], "components": ["main", "contrib", "non-free"], "mirror_url": "http://deb.debian.org/debian/", "distribution": "Debian"}} |
    ...
    • Dec 6 2021, 3:05 PM
    • 24 Lines
  • root@pergamon:~# clush -b -w @swh-workers systemctl status swh-worker@loader_nixguix | grep succeeded
    Dec 05 18:28:52 worker15 python3[2936713]: [2021-12-05 18:28:52,509: INFO/ForkPoolWorker-1] Task swh.loader.package.nixguix.tasks.LoadNixguix[d38e7d14-4894-4851-9ec3-7fa533d74049] succeeded in 28061.734233491123s: {'status': 'eventful', 'snapshot_id': '3568e78b54169b3995efa0acfa7c88a45d0056aa'}
    Dec 05 10:40:20 worker16 python3[2974229]: [2021-12-05 10:40:20,535: INFO/ForkPoolWorker-1] Task swh.loader.package.nixguix.tasks.LoadNixguix[51f86173-0b8e-496f-8d05-fb396247d197] succeeded in 9895.689101913944s: {'status': 'eventful', 'snapshot_id': '0415952acb92d5a8ad03c1f9d60a81459cb78aa5'}
    • Dec 6 2021, 9:33 AM
    • 3 Lines
  • (swh) ✔ ~/swh/swh-environment/docker [master|✚ 1⚑ 41]
    16:01 $ doco exec swh-scheduler /bin/bash
    swh@ab68bb37fea4:/$ psql service=swh-scheduler
    psql (12.9 (Debian 12.9-1.pgdg110+1), server 12.8 (Debian 12.8-1.pgdg100+1))
    Type "help" for help.
    ...
    • Dec 3 2021, 4:02 PM
    • 22 Lines
  • 14:09:12 swh-scheduler@db1:5432=> begin;
    BEGIN
    Time: 5.818 ms
    14:09:14 *swh-scheduler@db1:5432=> update listed_origins set extra_loader_arguments = extra_loader_arguments - 'date' where visit_type='deb';
    UPDATE 35105
    ...
    • Dec 3 2021, 2:09 PM
    • 15 Lines
  • 13:38:27 softwareheritage-scheduler@belvedere:5432=> select * from task where type = 'list-gitlab-incremental';
    +-----------+-------------------------+--------------------------------------------------------------------------------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------+
    | id | type | arguments | next_run | current_interval | status | policy | retries_left | priority |
    +-----------+-------------------------+--------------------------------------------------------------------------------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------+
    | 399914374 | list-gitlab-incremental | {"args": [], "kwargs": {"url": "https://foss.heptapod.net/api/v4/", "name": "heptapod"}} | 2021-12-04 08:56:02.284533+00 | 1 day | next_run_not_scheduled | recurring | 3 | (null) |
    ...
    • Dec 3 2021, 1:38 PM
    • 16 Lines
  • root@riverside:~# for volume in data redis kafka zookeeper symbolicator clickhouse; do echo docker volume create --name=sentry-$volume; done
    docker volume create --name=sentry-data
    docker volume create --name=sentry-redis
    docker volume create --name=sentry-kafka
    docker volume create --name=sentry-zookeeper
    ...
    • Dec 3 2021, 10:11 AM
    • 26 Lines
  • from functools import wraps
    def handle_raise_on_commit(f):
    @wraps(f)
    ...
    • Dec 2 2021, 5:02 PM
    • 16 Lines
    • Python
  • swh-loader_1 | [2021-12-02 13:43:44,472: ERROR/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[fb2923f8-a344-4a4d-9340-0897f6bab6fc] raised unexpected: TypeError("__init__() missing 1 required positional argument: 'date'")
    swh-loader_1 | Traceback (most recent call last):
    swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 450, in trace_task
    swh-loader_1 | R = retval = fun(*args, **kwargs)
    swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/task.py", line 55, in __call__
    ...
    • Dec 2 2021, 2:46 PM
    • 33 Lines
  • class DirectoryData:
    date: datetime
    """
    Date of the directory in the isochrone frontier
    """
    ...
    • Dec 1 2021, 5:13 PM
    • 11 Lines
    • Python
  • @dataclass(eq=True, frozen=True)
    class DirectoryData:
    """Object representing the data associated to a directory in the provenance model,
    where `date` is the date of the directory in the isochrone frontier, and `flat` is a
    flag acknowledging that a flat model for the elements outside the frontier has
    ...
    • Dec 1 2021, 5:12 PM
    • 10 Lines
    • Python
  • P1225 tryout
    $ ipython
    In [1]: 1626111425534 / 1e3
    Out[1]: 1626111425.534
    ...
    • Nov 26 2021, 10:40 AM
    • 22 Lines
  • worker11: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 553
    worker11: Error: /Stage[main]/Profile::Systemd_journal::Journalbeat/User[journalbeat]/ensure: change from 'present' to 'absent' failed: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 553
    worker10: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 522
    worker10: Error: /Stage[main]/Profile::Systemd_journal::Journalbeat/User[journalbeat]/ensure: change from 'present' to 'absent' failed: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 522
    worker12: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 516
    ...
    • Nov 26 2021, 10:20 AM
    • 31 Lines
  • root@bardo:~# puppet agent --test
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Nov 24 2021, 4:22 PM
    • 149 Lines
  • root@rp1:/etc# diff -U3 /etc/apt/apt.conf.d/50unattended-upgrades.ucf-old /etc/apt/apt.conf.d/50unattended-upgrades
    --- /etc/apt/apt.conf.d/50unattended-upgrades.ucf-old 2021-11-23 18:02:23.908766790 +0000
    +++ /etc/apt/apt.conf.d/50unattended-upgrades 2021-11-24 10:34:33.004766790 +0000
    @@ -1,68 +1,164 @@
    -// This file is managed by Puppet. DO NOT EDIT.
    ...
    • Nov 24 2021, 11:38 AM
    • 205 Lines
  • $ vagrant up admin-bardo
    Bringing machine 'admin-bardo' up with 'libvirt' provider...
    ==> admin-bardo: Creating image (snapshot of base box volume).
    ==> admin-bardo: Creating domain with the following settings...
    ==> admin-bardo: -- Name: puppet-environment_admin-bardo
    ...
    • Nov 23 2021, 4:07 PM
    • 691 Lines
  • vagrant up admin-bardo
    Bringing machine 'admin-bardo' up with 'libvirt' provider...
    ==> admin-bardo: Creating image (snapshot of base box volume).
    ==> admin-bardo: Creating domain with the following settings...
    ==> admin-bardo: -- Name: puppet-environment_admin-bardo
    ...
    • Nov 22 2021, 1:34 PM
    • 117 Lines
  • (swh) ✔ ~/swh/swh-environment/swh-loader-svn [master|✚ 2⚑ 10]
    16:43 $ git diff
    diff --git a/swh/loader/svn/loader.py b/swh/loader/svn/loader.py
    index eac96a3..698babe 100644
    --- a/swh/loader/svn/loader.py
    ...
    • Nov 18 2021, 4:54 PM
    • 60 Lines
  • diff --git a/swh/scheduler/celery_backend/recurrent_visits.py b/swh/scheduler/celery_backend/recurrent_visits.py
    index 8f43d8e..eacf8de 100644
    --- a/swh/scheduler/celery_backend/recurrent_visits.py
    +++ b/swh/scheduler/celery_backend/recurrent_visits.py
    @@ -23,6 +23,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Tuple
    ...
    • Nov 16 2021, 2:16 PM
    • 93 Lines
  • diff --git a/swh/journal/client.py b/swh/journal/client.py
    index ff6f0ef..344f629 100644
    --- a/swh/journal/client.py
    +++ b/swh/journal/client.py
    @@ -297,13 +297,13 @@ class JournalClient:
    ...
    • Nov 16 2021, 12:40 PM
    • 19 Lines
  • 16:01:14 swh@db1:5432=> select * from origin_visit_status ovs inner join origin o on ovs.origin=o.id where date > '2021-11-10 15:00' and type='npm' limit 10;
    +---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+---------+-----------------------------------------------------------+
    | origin | visit | date | status | metadata | snapshot | type | id | url |
    +---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+---------+-----------------------------------------------------------+
    | 1796362 | 1 | 2021-11-10 15:00:22.221484+00 | full | (null) | \x034b93485d1fc0a0efc9dd3447b1f7a92e278041 | npm | 1796362 | https://www.npmjs.com/package/angular-phonecat-nexen |
    ...
    • Nov 10 2021, 4:04 PM
    • 18 Lines
  • root@worker18:/home/ardumont# tree -L 3 /tmp/tmp0tm0182g/
    /tmp/tmp0tm0182g/
    ├── swh.loader.svn.2zdkkl_5-1740204
    │   └── tmpi0q7t29s
    │   ├── conf
    ...
    • Nov 10 2021, 9:42 AM
    • 20 Lines
  • swhworker@worker17:~$ /usr/bin/time -v swh loader run svn https://profs.scienze.univr.it/posenato/svn/sw/CSTNU
    INFO:swh.loader.svn.SvnLoader:Load origin 'https://profs.scienze.univr.it/posenato/svn/sw/CSTNU' with type 'svn'
    INFO:swh.loader.svn.SvnLoader:Processing revisions [297-619] for {'swh-origin': 'https://profs.scienze.univr.it/posenato/svn/sw/CSTNU', 'remote_url': 'file:///tmp/tmpsd2aj26l/swh.loader.svn.xkj39oxf-930164/tmpphfs60dp', 'local_url': b'/tmp/swh.loader.svn.kqus7k3a-930164/tmpphfs60dp', 'uuid': b'782453a1-1937-45d1-8845-2a6fcc2839b7'}
    {'status': 'eventful'}
    Command being timed: "swh loader run svn https://profs.scienze.univr.it/posenato/svn/sw/CSTNU"
    ...
    • Nov 8 2021, 6:57 PM
    • 27 Lines
  • swh-loader_1 | [2021-10-29 21:21:17,320: INFO/ForkPoolWorker-1] Processing revisions [1-1089] for {'swh-origin': 'http://svn.code.sf.net/p/blinkensisters/code', 'remote_url': 'file:///tmp/tmpi_mjbfaq/swh.loader.svn.zyez62lm-83/tmpcsww9pus', 'local_url': b'/tmp/swh.loader.svn.tx68am7w-83/tmpcsww9pus', 'uuid': b'943a9807-df12-0410-b0c0-cd0da6ec65e4'}
    swh-loader_1 | [2021-10-29 21:21:17,325: DEBUG/ForkPoolWorker-1] rev: 1, swhrev: d0fde2b7bc24f3b6d6316a079e310905f2b6179e, dir: 7a2faa45330bbfdb9d7ffca331d57073fb8ba61f
    swh-loader_1 | [2021-10-29 21:21:17,443: DEBUG/ForkPoolWorker-1] rev: 2, swhrev: 56eccaa1166c813a33b90b2b90db1658b1fd3b6e, dir: d4b04aa0c1fc8da2e2092cdb9956a0db9d18e86c
    swh-loader_1 | [2021-10-29 21:21:17,449: DEBUG/ForkPoolWorker-1] rev: 3, swhrev: 695e144d7820bd28d6b796e5930b7a3998e4b464, dir: 6744db600724f5f95eef06a7926c9e33a1f3d620
    swh-loader_1 | [2021-10-29 21:21:17,702: DEBUG/ForkPoolWorker-1] rev: 4, swhrev: 39d2270022f5ffec7fff9f0e11700776ff827075, dir: 150465caf99e23ed5e505f874e1d2249f6036dc8
    ...
    • Nov 2 2021, 1:48 PM
    • 253 Lines
  • swh-loader_1 | [2021-10-28 15:14:21,272: DEBUG/ForkPoolWorker-1] Mounting dump file with "svnadmin load".
    swh-loader_1 | [2021-10-28 15:14:26,735: INFO/ForkPoolWorker-1] Processing revisions [1-253] for {'swh-origin': 'http://svn.code.sf.net/p/proxytunnel/code/', 'remote_url': 'file:///tmp/tmp8oo46bm8/swh.loader.svn.0znrosz2-84/tmp1wjhoail', 'local_url': b'/tmp/swh.loader.svn.wpdwou1s-84/tmp1wjhoail', 'uuid': b'bc163920-b10d-0410-b2c5-a5491ca2ceef'}
    swh-loader_1 | [2021-10-28 15:14:26,739: DEBUG/ForkPoolWorker-1] rev: 1, swhrev: 93a685b99d596bf5bbd20cbd14752ff55fb6da7c, dir: 75ed58f260bfa4102d0e09657803511f5f0ab372
    swh-loader_1 | [2021-10-28 15:14:26,739: DEBUG/ForkPoolWorker-1] Checking hash computations on revision 1...
    swh-loader_1 | [2021-10-28 15:14:26,754: DEBUG/ForkPoolWorker-1] rev: 2, swhrev: eb093a0018af3b7e114023347b68217123efa326, dir: 99b8ce06292b877fd89143183217962e38943ca8
    ...
    • Oct 28 2021, 5:16 PM
    • 376 Lines
  • swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': ['swh.loader.svn.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.git.tasks', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', 'swh.loader.package.deposit.tasks', 'swh.loader.package.nixguix.tasks', 'swh.loader.package.npm.tasks', 'swh.loader.package.opam.tasks', 'swh.loader.package.pypi.tasks', 'swh.lister.bitbucket.tasks', 'swh.lister.cgit.tasks', 'swh.lister.cran.tasks', 'swh.lister.debian.tasks', 'swh.lister.gitea.tasks', 'swh.lister.github.tasks', 'swh.lister.gitlab.tasks', 'swh.lister.gnu.tasks', 'swh.lister.launchpad.tasks', 'swh.lister.npm.tasks', 'swh.lister.opam.tasks', 'swh.lister.packagist.tasks', 'swh.lister.phabricator.tasks', 'swh.lister.pypi.tasks', 'swh.lister.sourceforge.tasks', 'swh.lister.tuleap.tasks', 'swh.deposit.loader.tasks']}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    ...
    • Oct 27 2021, 3:42 PM
    • 108 Lines
  • /usr/bin/time -v swh-doco exec swh-loader swh loader run git https://github.com/scikit-learn/scikit-learn
    + cd /home/tony/work/inria/repo/swh/swh-environment/docker
    + docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader swh loader run git https://github.com/scikit-learn/scikit-learn
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/scikit-learn/scikit-learn' with type 'git'
    Enumerating objects: 206538, done.
    ...
    • Oct 26 2021, 4:16 PM
    • 47 Lines
  • /usr/bin/time -v swh-doco exec swh-loader swh loader run git https://git.code.sf.net/u/bsomervi/hamlib.git
    + cd /home/tony/work/inria/repo/swh/swh-environment/docker
    + docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader swh loader run git https://git.code.sf.net/u/bsomervi/hamlib.git
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://git.code.sf.net/u/bsomervi/hamlib.git' with type 'git'
    Enumerating objects: 52811, done.
    ...
    • Oct 26 2021, 10:32 AM
    • 33 Lines
  • #+title: About dropping ingestion of PR-like branches in the loader git
    * synthesis
    Code got adapted so the loader git actually retrieves the full packfile (D6550). Then
    ...
    • Oct 25 2021, 9:46 PM
    • 523 Lines
  • (swh) ✔ ~/swh/swh-environment/docker [master|✚ 2⚑ 35]
    14:21 $ doco exec swh-scheduler-runner swh scheduler origin grab-next tar 10
    lister_id,url,visit_type,extra_loader_arguments,last_update,enabled,first_seen,last_seen
    efd50b96-ecb1-47db-bfdf-56f2247769c3,https://cran.r-project.org/package=aaSEA,tar,"{'artifacts': [{'url': 'https://cran.r-project.org/src/contrib/aaSEA_1.1.0.tar.gz', 'version': '1.1.0'}]}",2019-11-09 15:35:46+00:00,True,2021-10-22 12:14:20.515523+00:00,2021-10-22 12:14:20.515523+00:00
    efd50b96-ecb1-47db-bfdf-56f2247769c3,https://cran.r-project.org/package=AATtools,tar,"{'artifacts': [{'url': 'https://cran.r-project.org/src/contrib/AATtools_0.0.1.tar.gz', 'version': '0.0.1'}]}",2020-06-09 16:49:30+00:00,True,2021-10-22 12:14:20.515523+00:00,2021-10-22 12:14:20.515523+00:00
    ...
    • Oct 22 2021, 2:59 PM
    • 60 Lines
  • swh-scheduler-orchestrator_1 | wait-for-it: waiting for swh-scheduler:5008 without a timeout
    swh-scheduler-orchestrator_1 | wait-for-it: swh-scheduler:5008 is available after 0 seconds
    swh-scheduler-orchestrator_1 | Starting the swh-scheduler start-orchestrator
    swh-scheduler-orchestrator_1 | wait-for-it: waiting for amqp:5672 without a timeout
    swh-scheduler-orchestrator_1 | wait-for-it: amqp:5672 is available after 0 seconds
    ...
    • Oct 22 2021, 12:27 PM
    • 85 Lines
  • Looks like override no longer works for me:
    ```
    $ docker-compose --version
    docker-compose version 1.29.2, build unknown
    $ docker --version
    ...
    • Oct 21 2021, 4:16 PM
    • 75 Lines
  • @given(
    new_snapshot(min_size=4, max_size=4)
    )
    @pytest.mark.parametrize("browse_context", ["log"])
    def test_snapshot_log_with_id(client, browse_context, new_snapshot):
    ...
    • Oct 21 2021, 11:44 AM
    • 12 Lines
    • Python
  • swh_graph-replayer.11.gxas1odxle08@shsrvc2 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
    swh_graph-replayer.11.gxas1odxle08@shsrvc2 | ERROR:swh.journal.client.rdkafka:FAIL [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: 1 request(s) timed out: disconnect (after 598635ms in state UP, 1 identical error(s) suppressed)
    swh_graph-replayer.8.movkrxdxo4js@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    swh_graph-replayer.8.movkrxdxo4js@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    swh_graph-replayer.3.obpkqy8uyh3n@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    ...
    • Oct 19 2021, 11:43 AM
    • 14 Lines
  • swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | ### CONFIG FILE ###
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | storage:
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | cls: remote
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | args:
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | url: http://storage:5002/
    ...
    • Oct 18 2021, 2:47 PM
    • 55 Lines
  • {"topics":
    [
    {"topic": "__consumer_offsets"},
    {"topic": "swh.journal.indexed.content_ctags"},
    ...
    • Oct 14 2021, 4:08 PM
    • 28 Lines
    • JSON
  • $ make -C docs/ clean html
    make: Entering directory '/home/tony/work/inria/repo/swh/swh-environment/swh-docs/docs'
    sphinx-build -M clean "." "_build" --color
    Removing everything under '_build'...
    bin/ln-sphinx-subprojects --remove
    ...
    • Oct 13 2021, 5:21 PM
    • 178 Lines
  • started with timing option, will keep track of elapsed time
    loading transposed graph...
    transposed graph loaded (duration: PT1M38.291779679S).
    loading revision timestamps...
    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    ...
    • Oct 12 2021, 4:03 PM
    • 9 Lines
  • # to run in venv ve with "fil-profile run script.py"
    import os
    import subprocess
    ...
    • Oct 5 2021, 5:53 PM
    • 12 Lines
  • root@worker0:~# cat /etc/softwareheritage/loader_npm.yml
    ---
    storage:
    cls: pipeline
    steps:
    ...
    • Oct 5 2021, 12:30 PM
    • 22 Lines
  • ddouard  (e) swh   master … 12  ~/s/s/docker  cat celeryconfig.py
    CELERY_ACCEPT_CONTENT = [
    'application/x-msgpack'
    ]
    ddouard  (e) swh   master … 12  ~/s/s/docker  echo $CELERY_BROKER_URL
    ...
    • Oct 4 2021, 3:31 PM
    • 14 Lines
  • 61e140a70e718dc0010df38aedbbb1a2f35, refs/pull/9760/head: 839b0a7981b9dbc5110c5ca579e535f7e40ec707, refs/pull/9773/head: c5ee0d8a679a0f249487f5133b78ad22f6eade26, refs/pull/9778/head: f117f67f519eee8a6613734d8f3f30693c50483a, refs/pull/9786/head: fd49257d17f90ecf95258da825a828bac00ae384, refs/pull/9816/head: d5f9c0c5ded8494c040fa0c3cb99117000e77765, refs/pull/9829/head: 5440bb57aa5b7a44a327c006b2a6af93621831a0, refs/pull/9838/head: 149445797b27cb782e51be6c2b7ddb8c6260dfc4, refs/pull/9841/head: dfd245dbbc58d9eda67dd77f925539b8b09b66f8, refs/pull/9842/head: 9975f88532fb0ae9fd5b822f5456deb61e28c784, refs/pull/9868/head: c51ad1418e815cad44efc335d80a39cfce0a7980, refs/pull/987/head: d1f0ce1d57c22f9729d613bffcee0d85c1d606fe, refs/pull/9888/head: 4b3c742ba717292f45fd1ec2eabd232b6f80eb76, refs/pull/9909/head: ed861c41f78f536d31b80613b75bc78b5957d8ff, refs/pull/9966/head: 1183a88a4980161543d60b9cfd9b3af95ad17a0b, refs/pull/9972/head: 0338a59fc909fc31233254c5f9ea871ea02b372e, refs/pull/9990/head: d487aeb574d032de026c8a554c4106f41a2c68a7, refs/pull/9998/head: bfeaaf9bd4072dc47664835a21fcdfbd1756f959, refs/remotes/mozilla/try: 956f8edb78be56a014d3082e5e4db65f035590cf, refs/remotes/sfackler/dynamic_lib: d8a37160416cdd2cc01817688c77a580bcbb67fe, refs/reviewable/pr30095/r1: db61fc79afb0980df08575fc9bb513f7f4a8b7a2, refs/reviewable/pr30095/r2: 727fb6342be37074ee17af1057ebabbaafe79144, refs/reviewable/pr30095/r3: dc3b857c3145a819bf44f1261bb12f82f2fcd1f3, refs/reviewable/pr30095/r4: 3068d80a7cd478a0887f5cc424b7ef47f2bffa1e, refs/reviewable/pr30426/r1: 1daa2253287f5db2497c720c57ff6a824e0fb40f, refs/reviewable/pr30426/r2: 75a21310acb645e8fdb6aed50118e5a4a68802d1, refs/reviewable/pr30426/r3: ad2315df841160f4ecee576e5a9aa52055ebc8a6, refs/reviewable/pr30426/r4: 7718f24df1abc6f667d3730e47fbe787a41d93ff, refs/reviewable/pr30426/r5: 7cc26712b711cbc115e6f8147c4bdc969fe73f65, refs/reviewable/pr30426/r6: b651488aad44fb85fc6dd53f7c53e315ca816c20, refs/reviewable/pr30900/r1: d5c9577e3743f8c327a430f14bbeee9a306eba84, refs/pull/89494/head: e34fd546111c3cad82c91a9f466284d55a53e5fb, refs/pull/89495/head: 1c2ad79f9d789b0d449a1bc4dfe7d7ddc2ed4109, refs/pull/89499/head: 0e4a8f9fb9123cb3406509e618ac6184e2c3f936, refs/pull/89500/head: a28a78f247a0f02521e97eabf98e790085a4a753, refs/pull/89501/head: a1f7872c3ff98a91273fc5fbb5a900258e422a3c, refs/pull/89502/head: 199b33f0d77c3fae1c2c982029df7168899f8aba, refs/pull/89504/head: fdd8a0dde53e09b2f0ec869088fcb1d882031bd6, refs/pull/89505/head: 5b6604825e20ecd962c7021131de6a20cc73a32e, refs/pull/89506/head: a034af20f96fdf175fc7821e47458361a8102b75, refs/pull/89507/head: e41bb97c253bf99a33ee077578d876d3d6b94148, refs/pull/89508/head: bce86219830e386d1247364b6018ede2e85d3036, refs/pull/89509/head: 11140ff1a0dd3a395dd3ca1488bf580559f782f8, refs/pull/89510/head: 33178d2093f735606076b2741c8eadea87aecd05, refs/pull/89511/head: 55a3c8636da6266e22d0c077cd4a58d40a0613ed, refs/pull/89512/head: 5c5dde8f0a1e7a3f735dce63d6b81680d24c56d9
    Enumerating objects: 25767, done.
    Counting objects: 100% (12873/12873), done.
    Compressing objects: 100% (5543/5543), done.
    Total 25767 (delta 9966), reused 7330 (delta 7330), pack-reused 12894
    ...
    • Oct 4 2021, 10:41 AM
    • 7 Lines
  • kind: ScaledObject
    metadata:
    name: loaders-operators
    spec:
    scaleTargetRef:
    ...
    • Oct 4 2021, 9:19 AM
    • 39 Lines
    • YAML
  • Hello,
    The current loader git consumes a lot of memory depending on the size of
    the repository. It's fetching the full packfile of unknown
    references/refs (filtered by last snapshot's references), then parses
    ...
    • Oct 2 2021, 10:56 AM
    • 154 Lines
  • (Pdb++) [n for n in refs.keys() if not utils.ignore_branch_name(n)]
    [b'HEAD', b'refs/heads/master', b'refs/pull/10/head', b'refs/pull/100/head', b'refs/pull/102/head', b'refs/pull/103/head', b'refs/pull/105/head', b'refs/pull/106/head', b'refs/pull/107/head', b'refs/pull/108/head', b'refs/pull/109/head', b'refs/pull/11/head', b'refs/pull/110/head', b'refs/pull/111/head', b'refs/pull/112/head', b'refs/pull/113/head', b'refs/pull/114/head', b'refs/pull/115/head', b'refs/pull/118/head', b'refs/pull/119/head', b'refs/pull/12/head', b'refs/pull/120/head', b'refs/pull/121/head', b'refs/pull/122/head', b'refs/pull/123/head', b'refs/pull/124/head', b'refs/pull/125/head', b'refs/pull/126/head', b'refs/pull/127/head', b'refs/pull/128/head', b'refs/pull/129/head', b'refs/pull/13/head', b'refs/pull/130/head', b'refs/pull/131/head', b'refs/pull/133/head', b'refs/pull/134/head', b'refs/pull/135/head', b'refs/pull/136/head', b'refs/pull/137/head', b'refs/pull/138/head', b'refs/pull/139/head', b'refs/pull/14/head'...
    • Oct 1 2021, 4:51 PM
    • 3 Lines
  • storage:
    cls: local
    db: postgresql:///?service=swh-storage
    objstorage:
    cls: remote
    ...
    • Oct 1 2021, 12:02 PM
    • 14 Lines
    • YAML
  • * cozy/cozy-stack
    ** docker-dev with override
    ...
    • Sep 30 2021, 6:46 PM
    • 230 Lines
  • #!/bin/bash
    # SPDX-FileCopyrightText: © 2021 Stefano Zacchiroli <zack@upsilon.cc>
    # SPDX-License-Identifier: GPL-3.0-or-later
    ...
    • Sep 30 2021, 5:00 PM
    • 18 Lines
  • Setting up postgresql-14 (14.0-1.pgdg+1) ...
    Creating new PostgreSQL cluster 14/main ...
    /usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions
    The files belonging to this database system will be owned by user "postgres".
    This user must also own the server process.
    ...
    • Sep 30 2021, 4:25 PM
    • 22 Lines
  • (ve) swhworker@worker0:~$ time swh loader -C /etc/softwareheritage/loader_git.yml run git https://github.com/torvalds/l
    inux
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/torvalds/linux' with type 'git'
    Enumerating objects: 8345251, done.
    Total 8345251 (delta 0), reused 0 (delta 0), pack-reused 8345251
    ...
    • Sep 30 2021, 10:06 AM
    • 23 Lines
  • # in swh/web/tests/conftest/py
    @pytest.fixture(scope='session')
    def django_db_modify_db_settings(django_db_modify_db_settings_parallel_suffix, postgresql_proc):
    from django.conf import settings as django_settings
    django_settings.DATABASES['default'].update({
    ...
    • Sep 29 2021, 4:19 PM
    • 12 Lines
  • Sep 29 12:51:52 worker18 python3[1046]: [2021-09-29 12:51:52,181: INFO/ForkPoolWorker-7] Processing revisions [1-0] for {'swh-origin': 'https://svn.code.sf.net/p/penguinsdbtools2018/svn', 'remote_url': 'file:///tmp/tmpjtb68puq/swh.loader.s
    vn.n58ub6q5-1046/tmpf9j3ln2g', 'local_url': b'/tmp/swh.loader.svn.ntglfflw-1046/tmpf9j3ln2g', 'uuid': b'd8f5688d-efb6-4dc1-b967-41be2f097118'}
    Sep 29 12:51:52 worker18 python3[1046]: [2021-09-29 12:51:52,182: ERROR/ForkPoolWorker-7] ('No such revision 1', 160006)
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 457, in fetch_data
    ...
    • Sep 29 2021, 3:05 PM
    • 22 Lines
  • Sep 28 15:23:50 worker0 python3[1078211]: [2021-09-28 15:23:50,779: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:51 worker0 python3[1078211]: [2021-09-28 15:23:51,222: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:52 worker0 python3[1078211]: [2021-09-28 15:23:52,122: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:52 worker0 python3[1078211]: [2021-09-28 15:23:52,126: ERROR/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[9fd77287-dc5b-4f63-9f15-516438b8e42e] raised unexpected: RetryError('<Future at 0x7f0027780d68 state=
    finished raised RemoteException>')
    ...
    • Sep 28 2021, 5:25 PM
    • 37 Lines
  •  ddouard  (e) swh   master ✚ 2 … 11  ~/s/s/swh-storage  mypy swh
    swh/storage/replay.py:133: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[SkippedContent]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:134: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[Content]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:137: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[SkippedContent]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:152: error: Argument 1 to "metadata_authority_add" of "StorageInterface" has incompatible type "Set[Any]"; expected "List[MetadataAuthority]"
    ...
    • Sep 28 2021, 4:54 PM
    • 7 Lines
  • diff --git a/swh/web/api/views/origin_save.py b/swh/web/api/views/origin_save.py
    index a2d3366f..6cba7e4e 100644
    --- a/swh/web/api/views/origin_save.py
    +++ b/swh/web/api/views/origin_save.py
    @@ -11,6 +11,7 @@ from swh.web.common.origin_save import (
    ...
    • Sep 28 2021, 4:48 PM
    • 262 Lines
  • DEBUG:swh.storage.replay:Inserting 200 raw_extrinsic_metadata objects
    Traceback (most recent call last):
    File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.15.0', 'console_scripts', 'swh')()
    File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
    ...
    • Sep 28 2021, 4:46 PM
    • 51 Lines
  • storage:
    cls: remote
    # url: http://localhost:5002/
    # url: http://storage0.internal.staging.swh.network:5002
    url: http://moma.internal.softwareheritage.org:5002/
    ...
    • Sep 28 2021, 2:07 PM
    • 96 Lines
    • YAML
  • egrep -v '^".*",".*",".*"$' license-blobs.csv
    "swh:1:cnt:03e1933241b8c3878d81c0184d7f2fd3d8cd6185","037d40bc6bcb42dfd740be545dbdf2df3405442f","LICENSE
    "
    "swh:1:cnt:65a5c662900ee946583147129720563fd4ba286d","40e9258799f752fe25d7518155c615c1c497b7ac","LICENSE.md
    "
    ...
    • Sep 27 2021, 4:50 PM
    • 11 Lines
  • #+title: Patching Dulwich to decrease memory footprint
    #+author: ardumont
    In the following analysis, we will executes multiple ingestions with and without the
    patched [1] dulwich version.
    ...
    • Sep 27 2021, 10:01 AM
    • 567 Lines
  • Case with some edge cases:
    ```
    $ opam show --color never --normalise --root $PWD -f all-versions opam-state
    2.0~alpha5 2.0.0~beta 2.0.0~beta3 2.0.0~beta3.1 2.0.0~beta5 2.0.0~rc 2.0.0~rc2 2.0.0~rc3 2.0.0 2.0.1 2.0.2 2.0.3 2.0.4 2.0.5 2.0.6 2.0.7 2.0.8 2.0.9 2.1.0~beta2 2.1.0~beta4 2.1.0~rc 2.1.0~rc2 2.1.0
    ```
    ...
    • Sep 24 2021, 4:14 PM
    • 78 Lines
  • visit_type=hg; sleep=300; while true; do for policy in never_visited_oldest_update_first already_visited_order_by_lag ; do echo "$(date) scheduling $visit_type origins with policy ${policy}"; SWH_C
    ONFIG_FILENAME=/etc/softwareheritage/scheduler/listener-runner.yml swh scheduler -C /etc/softwareheritage/scheduler/listener-runner.yml origin send-to-celery --policy $policy $visit_type; echo "$(date) sleep
    $sleep" ; sleep $sleep; done; done
    • Sep 23 2021, 11:11 AM
    • 3 Lines
  •  ddouard  (e) swh   master … 2  ~/s/s/swh-web  cat swh/web/django_db_backend/base.py
    from django.db.backends.postgresql.base import * # NOQA
    from django.db.backends.postgresql.base import DatabaseWrapper as _DatabaseWrapper
    # dirty hack to allow using a postgresql:// libpq connection URI as db name...
    ...
    • Sep 22 2021, 6:15 PM
    • 12 Lines
  • - First run of the foss heptapod origins scheduled
    - They failed due to a dangling configuration key from the old mercurial loader (fixed)
    -> all origins have failed
    -> they have now their entry in the origin-visit-stats table
    -> scheduler metrics are updated
    ...
    • Sep 22 2021, 11:37 AM
    • 413 Lines
  • $ pwd
    /home/tony/scratch/opam/test-opam-root
    $ opam init --reinit --bare --no-setup --root $PWD opam.ocaml.org https://opam.ocaml.org
    $ opam repository add --root $PWD satysfi-external https://github.com/gfngfn/satysfi-external-repo.git
    $ opam repository add --root $PWD opam-windows-repository https://github.com/vouillon/opam-windows-repository.git
    ...
    • Sep 21 2021, 1:54 PM
    • 56 Lines
  • tony  yavin4  ~  %  opam init --reinit --bare --no-setup --root ~/opamtest instance-foo https://opam.ocaml.org
    [NOTE] Will configure from built-in defaults.
    Checking for available remotes: rsync and local, git, mercurial, darcs. Perfect!
    <><> Fetching repository information ><><><><><><><><><><><><><><><><><><><><><>
    ...
    • Sep 20 2021, 12:30 PM
    • 8 Lines
  • ```
    swhworker@worker17:~$ time SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_oneshot.yml swh loader run mercurial_from_disk https://foss.heptapod.net/heptapod/heptapod
    INFO:swh.loader.mercurial.LoaderFromDisk:Load origin 'https://foss.heptapod.net/heptapod/heptapod' with type 'hg'
    applying clone bundle from https://cellar-c2.services.clever-cloud.com/heptapod-foss-clonebundles/heptapod/heptapod-2020-07-22-zstd-v2.hg
    adding changesets
    ...
    • Sep 17 2021, 11:13 AM
    • 60 Lines
  • diff --git a/swh/lister/gitlab/lister.py b/swh/lister/gitlab/lister.py
    index 5937256..074906d 100644
    --- a/swh/lister/gitlab/lister.py
    +++ b/swh/lister/gitlab/lister.py
    @@ -203,10 +203,11 @@ class GitLabLister(Lister[GitLabListerState, PageResult]):
    ...
    • Sep 16 2021, 5:47 PM
    • 29 Lines
  • - worker0.staging is running within a venv with D6240 (more extid filtering) and D6268 (build snapshot)
    - worker17 is running with the current latest mercurial packaged (no optim)
    But the filtering is still happening in the mercurial loader (without D6275)
    # tl; dr
    ...
    • Sep 16 2021, 4:04 PM
    • 136 Lines
  • vagrant up test
    Bringing machine 'test' up with 'libvirt' provider...
    ==> test: Creating image (snapshot of base box volume).
    ==> test: Creating domain with the following settings...
    ==> test: -- Name: puppet-environment_test
    ...
    • Sep 15 2021, 5:26 PM
    • 62 Lines
  • cqlsh:swh> DESCRIBE extid
    CREATE TABLE swh.extid (
    extid_type ascii,
    extid blob,
    ...
    • Sep 15 2021, 2:23 PM
    • 28 Lines
  • seq 2 8 | parallel -t ssh root@parasilo-{} nodetool flush
    seq 2 8 | parallel -t ssh root@parasilo-{} systemctl stop cassandra
    seq 2 8 | parallel -t ssh root@parasilo-{} sync
    seq 2 8 | xargs -t -n1 -i{} ssh root@parasilo-{} 'echo 3 > /proc/sys/vm/drop_caches'
    seq 2 8 | parallel -t ssh root@parasilo-{} systemctl start cassandra
    • Sep 15 2021, 12:04 PM
    • 5 Lines
    • Bash Scripting
  • from swh.storage import get_storage
    from swh.model.hashutil import hash_to_bytes
    import sys
    import time
    ...
    • Sep 15 2021, 11:18 AM
    • 31 Lines
    • Python
  • 10:36 guest@softwareheritage => select type, count(*) from origin_visit_status where status='full' and snapshot is null group by type;
    type │ count
    ──────┼───────
    git │ 1
    hg │ 62959
    ...
    • Sep 15 2021, 11:05 AM
    • 9 Lines
  • (swh) ✘-2 ~/swh/swh-environment/swh-loader-mercurial [master|✔]
    15:05 $ tox -r
    GLOB sdist-make: /home/anlambert/swh/swh-environment/swh-loader-mercurial/setup.py
    black recreate: /home/anlambert/swh/swh-environment/swh-loader-mercurial/.tox/black
    black installdeps: black==19.10b0
    ...
    • Sep 14 2021, 3:08 PM
    • 306 Lines
  • ```
    10:51:05 softwareheritage@belvedere:5432=> select * from origin o inner join origin_visit_status ovs on o.id=ovs.origin where url = 'https://github.com/CocoaPods/Specs' order by date desc limit 10;
    +---------+------------------------------------+---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+
    | id | url | origin | visit | date | status | metadata | snapshot | type |
    ...
    • Sep 14 2021, 10:53 AM
    • 41 Lines
  • [2] is the log used to extract the following list
    ```
    $ gzip -dc [2] | awk '{print $13" "$15 }' | sed -e 's/\\nTraceback//' | sort | uniq
    ```
    ...
    • Sep 13 2021, 6:10 PM
    • 258 Lines
  • See [1] for the full logs filtered on the command:
    ```
    $ gzip -dc [1] | awk '{print $13" "$15 }' | sed -e 's/\\nTraceback//' | sort | uniq
    ```
    ...
    • Sep 13 2021, 5:59 PM
    • 357 Lines
  • 00959a167bd98452c98ce73382f4b42179d53d32
    00a867beb2ad8e203f242e9843d2e88de0856cda
    028e9890a9287b35851c48ca351641743542d030
    030a51a49b3239769928872be9ac6d435ab14a61
    036594a6bbec926c21fa073e2404a5f760d35a43
    ...
    • Sep 10 2021, 6:08 PM
    • 166 Lines