Page MenuHomeSoftware Heritage
Create Paste
Paste Active Pastes

Active Pastes
Use Results
Edit Query

  • P1261 all timezone offsets in the DB
    softwareheritage=> copy (select distinct date_offset, encode(date_offset_bytes, 'escape') from revision) to stdout;
    69 +0109
    765 +1245
    -137 -0217
    360 +0600
    ...
    • Jan 21 2022, 9:21 AM
    • 446 Lines
    Author: vlorentz
  • P1260 (An Untitled Masterwork)
    class Statsd:
    # ...
    @contextmanager
    def status_gauge(self, metric_name: str, statuses: Collection[str], common_tags: Optional[Dict[str, str]] = None):
    current_status = Optional[str] = None
    ...
    • Jan 20 2022, 1:46 PM
    • 19 Lines
    • Python
    Author: douardda
  • P1259 staging: monster svn repository success
    Jan 19 16:11:15 worker0 python3[3375647]: [2022-01-19 16:11:15,120: INFO/MainProcess] Received task: swh.loader.svn.tasks.DumpMountAndLoadSvnRepository[8028cd34-c06f-4dda-b109-a1b315dafabb]
    Jan 19 16:11:18 worker0 python3[3375658]: [2022-01-19 16:11:18,753: INFO/ForkPoolWorker-1] Load origin 'https://plugins.svn.wordpress.org/wp-activity-log-for-woocommerce' with type 'svn'
    Jan 19 21:12:10 worker0 python3[3375658]: [2022-01-19 21:12:10,701: INFO/ForkPoolWorker-1] Processing revisions [1-168825] for {'swh-origin': 'https://plugins.svn.wordpress.org/wp-activity-log-for-woocommerce', 'remote_url': 'file:///tmp/swh.loader.svn.dxrrwh6j-3375658/swh.loader.svn.fw2r9j6a-3375658/tmpiw_ugkd8', 'local_url': b'/tmp/swh.loader.svn.ft2fxai2-3375658/tmpiw_ugkd8', 'uuid': b'b8457f37-d9ea-0310-8a92-e5e31aec5664'}
    Jan 19 21:18:07 worker0 python3[3375658]: [2022-01-19 21:18:07,339: INFO/ForkPoolWorker-1] Task swh.loader.svn.tasks.DumpMountAndLoadSvnRepository[8028cd34-c06f-4dda-b109-a1b315dafabb] succeeded in 18412.19621137576s...
    • Jan 20 2022, 9:59 AM
    • 4 Lines
    Author: ardumont
  • P1258 (An Untitled Masterwork)
    git,https://scm.gforge.inria.fr/anonscm/git/simty/simty.git
    git,https://github.com/unitystation/unitystation
    svn,http://svn.code.sf.net/p/sauerbraten/code
    git,https://gitlab.com/inkscape/inkscape
    git,https://github.com/keybase/client
    ...
    • Jan 17 2022, 1:54 PM
    • 25 Lines
    Author: ardumont
  • P1255 (An Untitled Masterwork)
    12:37 $ git diff
    diff --git a/swh/web/common/origin_save.py b/swh/web/common/origin_save.py
    index 3749d134..6696427b 100644
    --- a/swh/web/common/origin_save.py
    +++ b/swh/web/common/origin_save.py
    ...
    • Jan 13 2022, 12:38 PM
    • 20 Lines
    Author: anlambert
  • P1254 cancelled migration query running for too long plus task_add piling up...
    insert into new_task(id, type, arguments, next_run, current_interval, status, policy,
    retries_left, priority)
    select id, type, arguments, next_run, current_interval, status, policy,
    retries_left, priority
    from task
    ...
    • Jan 12 2022, 1:47 PM
    • 33 Lines
    Author: ardumont
  • P1253 Clean up scheduler db script -- ran on dump extracted from swh-scheduler db (staging) and mounted back on swh-scheduler-dev db (docker)
    -- SWH DB schema upgrade
    -- from_version: 32
    -- to_version: 33
    -- description: Archive old new_task/new_task-run tasks
    ...
    • Jan 10 2022, 3:52 PM
    • 223 Lines
    Author: ardumont
  • P1252 xmltodict xmlns
    >>> s = '<entry xmlns:testns="https://example.org"> <testns:foo> <testns:bar url="http://example.org/1" /> <testns:bar url="http://example.org/2" /></testns:foo> </entry>'
    >>> pprint(xmltodict.parse(s, process_namespaces=True, dict_constructor=dict))
    {'entry': {'https://example.org:foo': {'https://example.org:bar': [{'@url': 'http://example.org/1',
    '@xmlns': OrderedDict([('testns',
    'https://example.org')])},
    ...
    • Jan 10 2022, 2:25 PM
    • 6 Lines
    • Python
    Author: vlorentz
  • P1251 besides expected load-nixguix, why is there still recurring tasks created?
    ```
    16:28:34 softwareheritage-scheduler@belvedere:5432=> select type, policy, count(*) from task where type like 'load-%' and status !='disabled' and priority is null group by type, policy;
    +--------------------+-----------+---------+
    | type | policy | count |
    +--------------------+-----------+---------+
    ...
    • Jan 7 2022, 5:13 PM
    • 21 Lines
    Author: ardumont
  • P1250 (An Untitled Masterwork)
    16:45:09 swh-web@belvedere:5432=> select * from save_origin_request where origin_url='https://github.com/globalbioticinteractions/usnm-ixodes' order by id desc limit 3;
    +--------+-------------------------------+------------+---------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+--------+
    | id | request_date | visit_type | origin_url | status | loading_task_id | visit_date | loading_task_status | visit_status | user_ids | note |
    +--------+-------------------------------+------------+---------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+--------+
    | 105121 | 2022-01-07 15:14:52.39695+00 | git | https://github.com/globalbioticinteractions/usnm-ixodes | accepted | 404639723 | (null) | succeeded | (null) | (null) | (null) |
    ...
    • Jan 7 2022, 4:46 PM
    • 8 Lines
    Author: ardumont
  • P1248 debian package signing issue
    11:03:56 Build Architecture: amd64
    11:03:56 Build Type: full
    11:03:56 Build-Space: 109680
    11:03:56 Build-Time: 288
    11:03:56 Distribution: unstable-swh
    ...
    • Jan 4 2022, 12:14 PM
    • 26 Lines
    Author: anlambert
  • P1247 frozendict segfault
    (.venv) jenkins@30f908325208:~/swh-environment/swh-search$ python3 -c "from frozendict import frozendict; frozendict({})"
    Segmentation fault (core dumped)
    (.venv) jenkins@30f908325208:~/swh-environment/swh-search$ ls
    AUTHORS CONTRIBUTORS MANIFEST.in README.md docs mypy.ini package.json pytest.ini requirements-test.txt setup.cfg swh tox.ini
    CODE_OF_CONDUCT.md LICENSE Makefile core es_config node_modules pyproject.toml requirements-swh.txt requirements.txt setup.py swh.search.egg-info yarn.lock
    ...
    • Dec 21 2021, 4:52 PM
    • 62 Lines
    Author: vlorentz
  • P1246 swh-search in chroot
    pytest -s
    ============================================================================================================================= test session starts =============================================================================================================================
    platform linux -- Python 3.7.3, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
    rootdir: /home/test/swh-environment/swh-search, configfile: pytest.ini
    plugins: django-test-migrations-1.2.0, postgresql-2.6.1, swh.core-1.0.0, xdist-2.5.0, forked-1.4.0, redis-2.3.0, flask-1.2.0, asyncio-0.16.0, dash-2.0.0, django-4.5.2, requests-mock-1.9.3, mock-3.6.1, hypothesis-6.31.6, swh.journal-0.9.2.dev2+g4e5e009
    ...
    • Dec 21 2021, 1:30 PM
    • 95 Lines
    Author: vlorentz
  • P1245 mr update in chroot
    ./bin/update
    Already up to date.
    perl: warning: Setting locale failed.
    perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    ...
    • Dec 21 2021, 12:43 PM
    • 161 Lines
    Author: vlorentz
  • P1244 sadness, no snapshot plugin for vagrant (with libvirt provider)
    vagrant plugin install vagrant-snapshot
    Installing the 'vagrant-snapshot' plugin. This can take a few minutes...
    Building native extensions. This could take a while...
    Vagrant failed to properly resolve required dependencies. These
    errors can commonly be caused by misconfigured plugin installations
    ...
    • Dec 14 2021, 10:01 AM
    • 20 Lines
    Author: ardumont
  • P1243 check docker-dev build request durations
    #!/bin/bash
    id=$1
    echo Getting the logs...
    ...
    • Dec 14 2021, 9:46 AM
    • 21 Lines
    • Bash Scripting
    Author: vsellier
  • P1242 sampled actual listed cvs origins so far
    15:59:24 softwareheritage-scheduler@belvedere:5432=> select * from listed_origins where visit_type='cvs' limit 10;
    +--------------------------------------+----------------------------------------------+------------+------------------------+---------+-------------------------------+-------------------------------+------------------------+
    | lister_id | url | visit_type | extra_loader_arguments | enabled | first_seen | last_seen | last_update |
    +--------------------------------------+----------------------------------------------+------------+------------------------+---------+-------------------------------+-------------------------------+------------------------+
    | b678cfc3-2780-4186-9186-d78a14bd4958 | https://cvs.code.sf.net/p/a-a-p/code | cvs | {} | t | 2021-06-01 20:24:22.899139+00 | 2021-10-15 04:34:05.365731+00 | 2018-08-12 00:00:00+00 |
    ...
    • Dec 9 2021, 3:59 PM
    • 16 Lines
    Author: ardumont
  • P1241 (An Untitled Masterwork)
    *** swh-deploy: starting test run on moma.internal.softwareheritage.org...
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Dec 8 2021, 3:03 PM
    • 27 Lines
    Author: anlambert
  • P1240 subversion loader svn:exernals support (quick and dirty POC)
    diff --git a/swh/loader/svn/ra.py b/swh/loader/svn/ra.py
    index 8a51c49..d88496f 100644
    --- a/swh/loader/svn/ra.py
    +++ b/swh/loader/svn/ra.py
    @@ -11,6 +11,7 @@ from __future__ import annotations
    ...
    • Dec 8 2021, 1:42 PM
    • 245 Lines
    Author: anlambert
  • P1239 (bullseye) unbound 1.13 not too happy about listen-docker.conf instruction ([from buster] unbound 1.9)
    ```
    Dec 08 08:31:43 thyssen package-helper[6666]: /etc/unbound/unbound.conf.d/listen-docker.conf:1: error: syntax error
    Dec 08 08:31:43 thyssen package-helper[6666]: read /etc/unbound/unbound.conf failed: 1 errors in configuration file
    Dec 08 08:31:43 thyssen package-helper[6669]: /etc/unbound/unbound.conf.d/listen-docker.conf:1: error: syntax error
    Dec 08 08:31:43 thyssen package-helper[6669]: read /etc/unbound/unbound.conf failed: 1 errors in configuration file
    ...
    • Dec 8 2021, 9:36 AM
    • 10 Lines
    Author: ardumont
  • P1238 "vagrant up --no-destroy-on-error pergamon" workaround for now
    Somehow for now, the pergamon manifest is not complete/incorrect (and it's not clear where exactly we fall short).
    At boot time, we have the puppet master running while it should not. Puppet master must be served through apache (ssl configured and what not).
    So we need to make the provisioning step pass twice (for some unknown reason as of yet as well).
    Then connect to the node, stop and deactivate the puppet master service.
    Then restart the apache service so it serves properly puppet (as master).
    ...
    • Dec 7 2021, 10:27 AM
    • 25 Lines
    Author: ardumont
  • P1237 puppet agent --test failing at creation time without java installed in puppet manifest
    vagrant vm:
    ```
    root@thyssen:~# puppet agent --test
    Info: Using configured environment 'production'
    ...
    • Dec 7 2021, 10:19 AM
    • 76 Lines
    Author: ardumont
  • P1236 collation configuration discrepancy between production and staging dbs
    production:
    ```
    List of databases
    +------------------------------------+--------------+----------+---------+---------+-----------------------+
    | Name | Owner | Encoding | Collate | Ctype | Access privileges |
    ...
    • Dec 6 2021, 5:57 PM
    • 65 Lines
    Author: ardumont
  • P1235 staging list debian distribution
    15:04:47 swh-scheduler@db1:5432=> select * from task where type = 'list-debian-distribution';
    +-[ RECORD 1 ]-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | id | 1182486 |
    | type | list-debian-distribution |
    | arguments | {"args": [], "kwargs": {"suites": ["stretch", "buster", "bullseye"], "components": ["main", "contrib", "non-free"], "mirror_url": "http://deb.debian.org/debian/", "distribution": "Debian"}} |
    ...
    • Dec 6 2021, 3:05 PM
    • 24 Lines
    Author: ardumont
  • P1234 Load nixguix finished!
    root@pergamon:~# clush -b -w @swh-workers systemctl status swh-worker@loader_nixguix | grep succeeded
    Dec 05 18:28:52 worker15 python3[2936713]: [2021-12-05 18:28:52,509: INFO/ForkPoolWorker-1] Task swh.loader.package.nixguix.tasks.LoadNixguix[d38e7d14-4894-4851-9ec3-7fa533d74049] succeeded in 28061.734233491123s: {'status': 'eventful', 'snapshot_id': '3568e78b54169b3995efa0acfa7c88a45d0056aa'}
    Dec 05 10:40:20 worker16 python3[2974229]: [2021-12-05 10:40:20,535: INFO/ForkPoolWorker-1] Task swh.loader.package.nixguix.tasks.LoadNixguix[51f86173-0b8e-496f-8d05-fb396247d197] succeeded in 9895.689101913944s: {'status': 'eventful', 'snapshot_id': '0415952acb92d5a8ad03c1f9d60a81459cb78aa5'}
    • Dec 6 2021, 9:33 AM
    • 3 Lines
    Author: ardumont
  • P1233 debian lister with last_update
    (swh) ✔ ~/swh/swh-environment/docker [master|✚ 1⚑ 41]
    16:01 $ doco exec swh-scheduler /bin/bash
    swh@ab68bb37fea4:/$ psql service=swh-scheduler
    psql (12.9 (Debian 12.9-1.pgdg110+1), server 12.8 (Debian 12.8-1.pgdg100+1))
    Type "help" for help.
    ...
    • Dec 3 2021, 4:02 PM
    • 22 Lines
    Author: anlambert
  • P1232 drop date field in debian lister extra arguments
    14:09:12 swh-scheduler@db1:5432=> begin;
    BEGIN
    Time: 5.818 ms
    14:09:14 *swh-scheduler@db1:5432=> update listed_origins set extra_loader_arguments = extra_loader_arguments - 'date' where visit_type='deb';
    UPDATE 35105
    ...
    • Dec 3 2021, 2:09 PM
    • 15 Lines
    Author: ardumont
  • P1231 lister example
    13:38:27 softwareheritage-scheduler@belvedere:5432=> select * from task where type = 'list-gitlab-incremental';
    +-----------+-------------------------+--------------------------------------------------------------------------------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------+
    | id | type | arguments | next_run | current_interval | status | policy | retries_left | priority |
    +-----------+-------------------------+--------------------------------------------------------------------------------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------+
    | 399914374 | list-gitlab-incremental | {"args": [], "kwargs": {"url": "https://foss.heptapod.net/api/v4/", "name": "heptapod"}} | 2021-12-04 08:56:02.284533+00 | 1 day | next_run_not_scheduled | recurring | 3 | (null) |
    ...
    • Dec 3 2021, 1:38 PM
    • 16 Lines
    Author: ardumont
  • P1230 vagrant ssh riverside: dist-upgrade then puppet agent ok
    root@riverside:~# for volume in data redis kafka zookeeper symbolicator clickhouse; do echo docker volume create --name=sentry-$volume; done
    docker volume create --name=sentry-data
    docker volume create --name=sentry-redis
    docker volume create --name=sentry-kafka
    docker volume create --name=sentry-zookeeper
    ...
    • Dec 3 2021, 10:11 AM
    • 26 Lines
    Author: ardumont
  • P1229 Command-Line Input
    from functools import wraps
    def handle_raise_on_commit(f):
    @wraps(f)
    ...
    • Dec 2 2021, 5:02 PM
    • 16 Lines
    • Python
    Author: olasd
  • P1228 (An Untitled Masterwork)
    swh-loader_1 | [2021-12-02 13:43:44,472: ERROR/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[fb2923f8-a344-4a4d-9340-0897f6bab6fc] raised unexpected: TypeError("__init__() missing 1 required positional argument: 'date'")
    swh-loader_1 | Traceback (most recent call last):
    swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 450, in trace_task
    swh-loader_1 | R = retval = fun(*args, **kwargs)
    swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/task.py", line 55, in __call__
    ...
    • Dec 2 2021, 2:46 PM
    • 33 Lines
    Author: anlambert
  • P1227 (An Untitled Masterwork)
    class DirectoryData:
    date: datetime
    """
    Date of the directory in the isochrone frontier
    """
    ...
    • Dec 1 2021, 5:13 PM
    • 11 Lines
    • Python
    Author: vlorentz
  • P1226 (An Untitled Masterwork)
    @dataclass(eq=True, frozen=True)
    class DirectoryData:
    """Object representing the data associated to a directory in the provenance model,
    where `date` is the date of the directory in the isochrone frontier, and `flat` is a
    flag acknowledging that a flat model for the elements outside the frontier has
    ...
    • Dec 1 2021, 5:12 PM
    • 10 Lines
    • Python
    Author: vlorentz
  • P1225 tryout
    $ ipython
    In [1]: 1626111425534 / 1e3
    Out[1]: 1626111425.534
    ...
    • Nov 26 2021, 10:40 AM
    • 22 Lines
    Author: ardumont
  • P1224 puppet agent --test on workers complain about journalbeat
    worker11: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 553
    worker11: Error: /Stage[main]/Profile::Systemd_journal::Journalbeat/User[journalbeat]/ensure: change from 'present' to 'absent' failed: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 553
    worker10: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 522
    worker10: Error: /Stage[main]/Profile::Systemd_journal::Journalbeat/User[journalbeat]/ensure: change from 'present' to 'absent' failed: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 522
    worker12: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 516
    ...
    • Nov 26 2021, 10:20 AM
    • 31 Lines
    Author: ardumont
  • P1223 breaking deps makes puppet agent --test refuse to comply
    root@bardo:~# puppet agent --test
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Nov 24 2021, 4:22 PM
    • 149 Lines
    Author: ardumont
  • P1222 admin-rp1 diff on unattended upgrade configuration file
    root@rp1:/etc# diff -U3 /etc/apt/apt.conf.d/50unattended-upgrades.ucf-old /etc/apt/apt.conf.d/50unattended-upgrades
    --- /etc/apt/apt.conf.d/50unattended-upgrades.ucf-old 2021-11-23 18:02:23.908766790 +0000
    +++ /etc/apt/apt.conf.d/50unattended-upgrades 2021-11-24 10:34:33.004766790 +0000
    @@ -1,68 +1,164 @@
    -// This file is managed by Puppet. DO NOT EDIT.
    ...
    • Nov 24 2021, 11:38 AM
    • 205 Lines
    Author: ardumont
  • P1221 some more vagrant up sadness
    $ vagrant up admin-bardo
    Bringing machine 'admin-bardo' up with 'libvirt' provider...
    ==> admin-bardo: Creating image (snapshot of base box volume).
    ==> admin-bardo: Creating domain with the following settings...
    ==> admin-bardo: -- Name: puppet-environment_admin-bardo
    ...
    • Nov 23 2021, 4:07 PM
    • 691 Lines
    Author: ardumont
  • P1220 vagrant up sadness
    vagrant up admin-bardo
    Bringing machine 'admin-bardo' up with 'libvirt' provider...
    ==> admin-bardo: Creating image (snapshot of base box volume).
    ==> admin-bardo: Creating domain with the following settings...
    ==> admin-bardo: -- Name: puppet-environment_admin-bardo
    ...
    • Nov 22 2021, 1:34 PM
    • 117 Lines
    Author: ardumont
  • P1219 SvnLoaderFromRemoteDump optimization
    (swh) ✔ ~/swh/swh-environment/swh-loader-svn [master|✚ 2⚑ 10]
    16:43 $ git diff
    diff --git a/swh/loader/svn/loader.py b/swh/loader/svn/loader.py
    index eac96a3..698babe 100644
    --- a/swh/loader/svn/loader.py
    ...
    • Nov 18 2021, 4:54 PM
    • 60 Lines
    Author: anlambert
  • P1218 Command-Line Input
    diff --git a/swh/scheduler/celery_backend/recurrent_visits.py b/swh/scheduler/celery_backend/recurrent_visits.py
    index 8f43d8e..eacf8de 100644
    --- a/swh/scheduler/celery_backend/recurrent_visits.py
    +++ b/swh/scheduler/celery_backend/recurrent_visits.py
    @@ -23,6 +23,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Tuple
    ...
    • Nov 16 2021, 2:16 PM
    • 93 Lines
    Author: olasd
  • P1217 Command-Line Input
    diff --git a/swh/journal/client.py b/swh/journal/client.py
    index ff6f0ef..344f629 100644
    --- a/swh/journal/client.py
    +++ b/swh/journal/client.py
    @@ -297,13 +297,13 @@ class JournalClient:
    ...
    • Nov 16 2021, 12:40 PM
    • 19 Lines
    Author: olasd
  • P1216 (An Untitled Masterwork)
    16:01:14 swh@db1:5432=> select * from origin_visit_status ovs inner join origin o on ovs.origin=o.id where date > '2021-11-10 15:00' and type='npm' limit 10;
    +---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+---------+-----------------------------------------------------------+
    | origin | visit | date | status | metadata | snapshot | type | id | url |
    +---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+---------+-----------------------------------------------------------+
    | 1796362 | 1 | 2021-11-10 15:00:22.221484+00 | full | (null) | \x034b93485d1fc0a0efc9dd3447b1f7a92e278041 | npm | 1796362 | https://www.npmjs.com/package/angular-phonecat-nexen |
    ...
    • Nov 10 2021, 4:04 PM
    • 18 Lines
    Author: ardumont
  • P1215 temporary/working directories mess with the loader svn somewhere
    root@worker18:/home/ardumont# tree -L 3 /tmp/tmp0tm0182g/
    /tmp/tmp0tm0182g/
    ├── swh.loader.svn.2zdkkl_5-1740204
    │   └── tmpi0q7t29s
    │   ├── conf
    ...
    • Nov 10 2021, 9:42 AM
    • 20 Lines
    Author: ardumont
  • P1214 (An Untitled Masterwork)
    swhworker@worker17:~$ /usr/bin/time -v swh loader run svn https://profs.scienze.univr.it/posenato/svn/sw/CSTNU
    INFO:swh.loader.svn.SvnLoader:Load origin 'https://profs.scienze.univr.it/posenato/svn/sw/CSTNU' with type 'svn'
    INFO:swh.loader.svn.SvnLoader:Processing revisions [297-619] for {'swh-origin': 'https://profs.scienze.univr.it/posenato/svn/sw/CSTNU', 'remote_url': 'file:///tmp/tmpsd2aj26l/swh.loader.svn.xkj39oxf-930164/tmpphfs60dp', 'local_url': b'/tmp/swh.loader.svn.kqus7k3a-930164/tmpphfs60dp', 'uuid': b'782453a1-1937-45d1-8845-2a6fcc2839b7'}
    {'status': 'eventful'}
    Command being timed: "swh loader run svn https://profs.scienze.univr.it/posenato/svn/sw/CSTNU"
    ...
    • Nov 8 2021, 6:57 PM
    • 27 Lines
    Author: ardumont
  • P1213 svn loader link handling bug
    swh-loader_1 | [2021-10-29 21:21:17,320: INFO/ForkPoolWorker-1] Processing revisions [1-1089] for {'swh-origin': 'http://svn.code.sf.net/p/blinkensisters/code', 'remote_url': 'file:///tmp/tmpi_mjbfaq/swh.loader.svn.zyez62lm-83/tmpcsww9pus', 'local_url': b'/tmp/swh.loader.svn.tx68am7w-83/tmpcsww9pus', 'uuid': b'943a9807-df12-0410-b0c0-cd0da6ec65e4'}
    swh-loader_1 | [2021-10-29 21:21:17,325: DEBUG/ForkPoolWorker-1] rev: 1, swhrev: d0fde2b7bc24f3b6d6316a079e310905f2b6179e, dir: 7a2faa45330bbfdb9d7ffca331d57073fb8ba61f
    swh-loader_1 | [2021-10-29 21:21:17,443: DEBUG/ForkPoolWorker-1] rev: 2, swhrev: 56eccaa1166c813a33b90b2b90db1658b1fd3b6e, dir: d4b04aa0c1fc8da2e2092cdb9956a0db9d18e86c
    swh-loader_1 | [2021-10-29 21:21:17,449: DEBUG/ForkPoolWorker-1] rev: 3, swhrev: 695e144d7820bd28d6b796e5930b7a3998e4b464, dir: 6744db600724f5f95eef06a7926c9e33a1f3d620
    swh-loader_1 | [2021-10-29 21:21:17,702: DEBUG/ForkPoolWorker-1] rev: 4, swhrev: 39d2270022f5ffec7fff9f0e11700776ff827075, dir: 150465caf99e23ed5e505f874e1d2249f6036dc8
    ...
    • Nov 2 2021, 1:48 PM
    • 253 Lines
    Author: anlambert
  • P1212 (An Untitled Masterwork)
    swh-loader_1 | [2021-10-28 15:14:21,272: DEBUG/ForkPoolWorker-1] Mounting dump file with "svnadmin load".
    swh-loader_1 | [2021-10-28 15:14:26,735: INFO/ForkPoolWorker-1] Processing revisions [1-253] for {'swh-origin': 'http://svn.code.sf.net/p/proxytunnel/code/', 'remote_url': 'file:///tmp/tmp8oo46bm8/swh.loader.svn.0znrosz2-84/tmp1wjhoail', 'local_url': b'/tmp/swh.loader.svn.wpdwou1s-84/tmp1wjhoail', 'uuid': b'bc163920-b10d-0410-b2c5-a5491ca2ceef'}
    swh-loader_1 | [2021-10-28 15:14:26,739: DEBUG/ForkPoolWorker-1] rev: 1, swhrev: 93a685b99d596bf5bbd20cbd14752ff55fb6da7c, dir: 75ed58f260bfa4102d0e09657803511f5f0ab372
    swh-loader_1 | [2021-10-28 15:14:26,739: DEBUG/ForkPoolWorker-1] Checking hash computations on revision 1...
    swh-loader_1 | [2021-10-28 15:14:26,754: DEBUG/ForkPoolWorker-1] rev: 2, swhrev: eb093a0018af3b7e114023347b68217123efa326, dir: 99b8ce06292b877fd89143183217962e38943ca8
    ...
    • Oct 28 2021, 5:16 PM
    • 376 Lines
    Author: anlambert
  • P1211 swh-doco logs -f swh-scheduler-schedule-recurrent
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': ['swh.loader.svn.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.git.tasks', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', 'swh.loader.package.deposit.tasks', 'swh.loader.package.nixguix.tasks', 'swh.loader.package.npm.tasks', 'swh.loader.package.opam.tasks', 'swh.loader.package.pypi.tasks', 'swh.lister.bitbucket.tasks', 'swh.lister.cgit.tasks', 'swh.lister.cran.tasks', 'swh.lister.debian.tasks', 'swh.lister.gitea.tasks', 'swh.lister.github.tasks', 'swh.lister.gitlab.tasks', 'swh.lister.gnu.tasks', 'swh.lister.launchpad.tasks', 'swh.lister.npm.tasks', 'swh.lister.opam.tasks', 'swh.lister.packagist.tasks', 'swh.lister.phabricator.tasks', 'swh.lister.pypi.tasks', 'swh.lister.sourceforge.tasks', 'swh.lister.tuleap.tasks', 'swh.deposit.loader.tasks']}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    ...
    • Oct 27 2021, 3:42 PM
    • 108 Lines
    Author: ardumont
  • P1210 /usr/bin/time -v swh-doco exec swh-loader swh loader run git https://github.com/scikit-learn/scikit-learn
    /usr/bin/time -v swh-doco exec swh-loader swh loader run git https://github.com/scikit-learn/scikit-learn
    + cd /home/tony/work/inria/repo/swh/swh-environment/docker
    + docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader swh loader run git https://github.com/scikit-learn/scikit-learn
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/scikit-learn/scikit-learn' with type 'git'
    Enumerating objects: 206538, done.
    ...
    • Oct 26 2021, 4:16 PM
    • 47 Lines
    Author: ardumont
  • P1209 ingestion ok
    /usr/bin/time -v swh-doco exec swh-loader swh loader run git https://git.code.sf.net/u/bsomervi/hamlib.git
    + cd /home/tony/work/inria/repo/swh/swh-environment/docker
    + docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader swh loader run git https://git.code.sf.net/u/bsomervi/hamlib.git
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://git.code.sf.net/u/bsomervi/hamlib.git' with type 'git'
    Enumerating objects: 52811, done.
    ...
    • Oct 26 2021, 10:32 AM
    • 33 Lines
    Author: ardumont
  • P1208 sampled origins: Patch (drop PR branches) or no patch (current version), failure to ingest with huge packfile
    #+title: About dropping ingestion of PR-like branches in the loader git
    * synthesis
    Code got adapted so the loader git actually retrieves the full packfile (D6550). Then
    ...
    • Oct 25 2021, 9:46 PM
    • 523 Lines
    Author: ardumont
  • P1207 (An Untitled Masterwork)
    (swh) ✔ ~/swh/swh-environment/docker [master|✚ 2⚑ 35]
    14:21 $ doco exec swh-scheduler-runner swh scheduler origin grab-next tar 10
    lister_id,url,visit_type,extra_loader_arguments,last_update,enabled,first_seen,last_seen
    efd50b96-ecb1-47db-bfdf-56f2247769c3,https://cran.r-project.org/package=aaSEA,tar,"{'artifacts': [{'url': 'https://cran.r-project.org/src/contrib/aaSEA_1.1.0.tar.gz', 'version': '1.1.0'}]}",2019-11-09 15:35:46+00:00,True,2021-10-22 12:14:20.515523+00:00,2021-10-22 12:14:20.515523+00:00
    efd50b96-ecb1-47db-bfdf-56f2247769c3,https://cran.r-project.org/package=AATtools,tar,"{'artifacts': [{'url': 'https://cran.r-project.org/src/contrib/AATtools_0.0.1.tar.gz', 'version': '0.0.1'}]}",2020-06-09 16:49:30+00:00,True,2021-10-22 12:14:20.515523+00:00,2021-10-22 12:14:20.515523+00:00
    ...
    • Oct 22 2021, 2:59 PM
    • 60 Lines
    Author: anlambert
  • P1206 swh-scheduler-orchestrator is running
    swh-scheduler-orchestrator_1 | wait-for-it: waiting for swh-scheduler:5008 without a timeout
    swh-scheduler-orchestrator_1 | wait-for-it: swh-scheduler:5008 is available after 0 seconds
    swh-scheduler-orchestrator_1 | Starting the swh-scheduler start-orchestrator
    swh-scheduler-orchestrator_1 | wait-for-it: waiting for amqp:5672 without a timeout
    swh-scheduler-orchestrator_1 | wait-for-it: amqp:5672 is available after 0 seconds
    ...
    • Oct 22 2021, 12:27 PM
    • 85 Lines
    Author: ardumont
  • P1205 docker-compose no longer allows override on containers
    Looks like override no longer works for me:
    ```
    $ docker-compose --version
    docker-compose version 1.29.2, build unknown
    $ docker --version
    ...
    • Oct 21 2021, 4:16 PM
    • 75 Lines
    Author: ardumont
  • P1204 snapshot test
    @given(
    new_snapshot(min_size=4, max_size=4)
    )
    @pytest.mark.parametrize("browse_context", ["log"])
    def test_snapshot_log_with_id(client, browse_context, new_snapshot):
    ...
    • Oct 21 2021, 11:44 AM
    • 12 Lines
    • Python
    Author: jayeshv
  • P1203 (An Untitled Masterwork)
    swh_graph-replayer.11.gxas1odxle08@shsrvc2 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
    swh_graph-replayer.11.gxas1odxle08@shsrvc2 | ERROR:swh.journal.client.rdkafka:FAIL [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: 1 request(s) timed out: disconnect (after 598635ms in state UP, 1 identical error(s) suppressed)
    swh_graph-replayer.8.movkrxdxo4js@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    swh_graph-replayer.8.movkrxdxo4js@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    swh_graph-replayer.3.obpkqy8uyh3n@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    ...
    • Oct 19 2021, 11:43 AM
    • 14 Lines
    Author: douardda
  • P1202 (An Untitled Masterwork)
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | ### CONFIG FILE ###
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | storage:
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | cls: remote
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | args:
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | url: http://storage:5002/
    ...
    • Oct 18 2021, 2:47 PM
    • 55 Lines
    Author: douardda
  • P1201 staging: topics to move to storage1
    {"topics":
    [
    {"topic": "__consumer_offsets"},
    {"topic": "swh.journal.indexed.content_ctags"},
    ...
    • Oct 14 2021, 4:08 PM
    • 28 Lines
    • JSON
    Author: vsellier
  • P1200 `make -C docs clean html` failure to build the devel docs
    $ make -C docs/ clean html
    make: Entering directory '/home/tony/work/inria/repo/swh/swh-environment/swh-docs/docs'
    sphinx-build -M clean "." "_build" --color
    Removing everything under '_build'...
    bin/ln-sphinx-subprojects --remove
    ...
    • Oct 13 2021, 5:21 PM
    • 178 Lines
    Author: ardumont
  • P1199 Command-Line Input
    started with timing option, will keep track of elapsed time
    loading transposed graph...
    transposed graph loaded (duration: PT1M38.291779679S).
    loading revision timestamps...
    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    ...
    • Oct 12 2021, 4:03 PM
    • 9 Lines
    Author: zack
  • P1197 fil-profile run script.py
    # to run in venv ve with "fil-profile run script.py"
    import os
    import subprocess
    ...
    • Oct 5 2021, 5:53 PM
    • 12 Lines
    Author: ardumont
  • P1196 (An Untitled Masterwork)
    root@worker0:~# cat /etc/softwareheritage/loader_npm.yml
    ---
    storage:
    cls: pipeline
    steps:
    ...
    • Oct 5 2021, 12:30 PM
    • 22 Lines
    Author: ardumont
  • P1195 (An Untitled Masterwork)
    ddouard  (e) swh   master … 12  ~/s/s/docker  cat celeryconfig.py
    CELERY_ACCEPT_CONTENT = [
    'application/x-msgpack'
    ]
    ddouard  (e) swh   master … 12  ~/s/s/docker  echo $CELERY_BROKER_URL
    ...
    • Oct 4 2021, 3:31 PM
    • 14 Lines
    Author: douardda
  • P1194 unknown reference in git loader with partial snapshot run warning becomes tedious...
    61e140a70e718dc0010df38aedbbb1a2f35, refs/pull/9760/head: 839b0a7981b9dbc5110c5ca579e535f7e40ec707, refs/pull/9773/head: c5ee0d8a679a0f249487f5133b78ad22f6eade26, refs/pull/9778/head: f117f67f519eee8a6613734d8f3f30693c50483a, refs/pull/9786/head: fd49257d17f90ecf95258da825a828bac00ae384, refs/pull/9816/head: d5f9c0c5ded8494c040fa0c3cb99117000e77765, refs/pull/9829/head: 5440bb57aa5b7a44a327c006b2a6af93621831a0, refs/pull/9838/head: 149445797b27cb782e51be6c2b7ddb8c6260dfc4, refs/pull/9841/head: dfd245dbbc58d9eda67dd77f925539b8b09b66f8, refs/pull/9842/head: 9975f88532fb0ae9fd5b822f5456deb61e28c784, refs/pull/9868/head: c51ad1418e815cad44efc335d80a39cfce0a7980, refs/pull/987/head: d1f0ce1d57c22f9729d613bffcee0d85c1d606fe, refs/pull/9888/head: 4b3c742ba717292f45fd1ec2eabd232b6f80eb76, refs/pull/9909/head: ed861c41f78f536d31b80613b75bc78b5957d8ff, refs/pull/9966/head: 1183a88a4980161543d60b9cfd9b3af95ad17a0b, refs/pull/9972/head: 0338a59fc909fc31233254c5f9ea871ea02b372e, refs/pull/9990/head: d487aeb574d032de026c8a554c4106f41a2c68a7, refs/pull/9998/head: bfeaaf9bd4072dc47664835a21fcdfbd1756f959, refs/remotes/mozilla/try: 956f8edb78be56a014d3082e5e4db65f035590cf, refs/remotes/sfackler/dynamic_lib: d8a37160416cdd2cc01817688c77a580bcbb67fe, refs/reviewable/pr30095/r1: db61fc79afb0980df08575fc9bb513f7f4a8b7a2, refs/reviewable/pr30095/r2: 727fb6342be37074ee17af1057ebabbaafe79144, refs/reviewable/pr30095/r3: dc3b857c3145a819bf44f1261bb12f82f2fcd1f3, refs/reviewable/pr30095/r4: 3068d80a7cd478a0887f5cc424b7ef47f2bffa1e, refs/reviewable/pr30426/r1: 1daa2253287f5db2497c720c57ff6a824e0fb40f, refs/reviewable/pr30426/r2: 75a21310acb645e8fdb6aed50118e5a4a68802d1, refs/reviewable/pr30426/r3: ad2315df841160f4ecee576e5a9aa52055ebc8a6, refs/reviewable/pr30426/r4: 7718f24df1abc6f667d3730e47fbe787a41d93ff, refs/reviewable/pr30426/r5: 7cc26712b711cbc115e6f8147c4bdc969fe73f65, refs/reviewable/pr30426/r6: b651488aad44fb85fc6dd53f7c53e315ca816c20, refs/reviewable/pr30900/r1: d5c9577e3743f8c327a430f14bbeee9a306eba84, refs/pull/89494/head: e34fd546111c3cad82c91a9f466284d55a53e5fb, refs/pull/89495/head: 1c2ad79f9d789b0d449a1bc4dfe7d7ddc2ed4109, refs/pull/89499/head: 0e4a8f9fb9123cb3406509e618ac6184e2c3f936, refs/pull/89500/head: a28a78f247a0f02521e97eabf98e790085a4a753, refs/pull/89501/head: a1f7872c3ff98a91273fc5fbb5a900258e422a3c, refs/pull/89502/head: 199b33f0d77c3fae1c2c982029df7168899f8aba, refs/pull/89504/head: fdd8a0dde53e09b2f0ec869088fcb1d882031bd6, refs/pull/89505/head: 5b6604825e20ecd962c7021131de6a20cc73a32e, refs/pull/89506/head: a034af20f96fdf175fc7821e47458361a8102b75, refs/pull/89507/head: e41bb97c253bf99a33ee077578d876d3d6b94148, refs/pull/89508/head: bce86219830e386d1247364b6018ede2e85d3036, refs/pull/89509/head: 11140ff1a0dd3a395dd3ca1488bf580559f782f8, refs/pull/89510/head: 33178d2093f735606076b2741c8eadea87aecd05, refs/pull/89511/head: 55a3c8636da6266e22d0c077cd4a58d40a0613ed, refs/pull/89512/head: 5c5dde8f0a1e7a3f735dce63d6b81680d24c56d9
    Enumerating objects: 25767, done.
    Counting objects: 100% (12873/12873), done.
    Compressing objects: 100% (5543/5543), done.
    Total 25767 (delta 9966), reused 7330 (delta 7330), pack-reused 12894
    ...
    • Oct 4 2021, 10:41 AM
    • 7 Lines
    Author: ardumont
  • P1193 keda configuration for docker environment
    kind: ScaledObject
    metadata:
    name: loaders-operators
    spec:
    scaleTargetRef:
    ...
    • Oct 4 2021, 9:19 AM
    • 39 Lines
    • YAML
    Author: vsellier
  • P1192 [draft-ml-devel] About reducing the loader-git memory consumption and overall work
    Hello,
    The current loader git consumes a lot of memory depending on the size of
    the repository. It's fetching the full packfile of unknown
    references/refs (filtered by last snapshot's references), then parses
    ...
    • Oct 2 2021, 10:56 AM
    • 154 Lines
    Author: ardumont
  • P1191 (An Untitled Masterwork)
    (Pdb++) [n for n in refs.keys() if not utils.ignore_branch_name(n)]
    [b'HEAD', b'refs/heads/master', b'refs/pull/10/head', b'refs/pull/100/head', b'refs/pull/102/head', b'refs/pull/103/head', b'refs/pull/105/head', b'refs/pull/106/head', b'refs/pull/107/head', b'refs/pull/108/head', b'refs/pull/109/head', b'refs/pull/11/head', b'refs/pull/110/head', b'refs/pull/111/head', b'refs/pull/112/head', b'refs/pull/113/head', b'refs/pull/114/head', b'refs/pull/115/head', b'refs/pull/118/head', b'refs/pull/119/head', b'refs/pull/12/head', b'refs/pull/120/head', b'refs/pull/121/head', b'refs/pull/122/head', b'refs/pull/123/head', b'refs/pull/124/head', b'refs/pull/125/head', b'refs/pull/126/head', b'refs/pull/127/head', b'refs/pull/128/head', b'refs/pull/129/head', b'refs/pull/13/head', b'refs/pull/130/head', b'refs/pull/131/head', b'refs/pull/133/head', b'refs/pull/134/head', b'refs/pull/135/head', b'refs/pull/136/head', b'refs/pull/137/head', b'refs/pull/138/head', b'refs/pull/139/head', b'refs/pull/14/head'...
    • Oct 1 2021, 4:51 PM
    • 3 Lines
    Author: ardumont
  • P1190 (An Untitled Masterwork)
    storage:
    cls: local
    db: postgresql:///?service=swh-storage
    objstorage:
    cls: remote
    ...
    • Oct 1 2021, 12:02 PM
    • 14 Lines
    • YAML
    Author: vlorentz
  • P1189 draft: ryout runs on loader git improvments
    * cozy/cozy-stack
    ** docker-dev with override
    ...
    • Sep 30 2021, 6:46 PM
    • 230 Lines
    Author: ardumont
  • P1187 import-ics
    #!/bin/bash
    # SPDX-FileCopyrightText: © 2021 Stefano Zacchiroli <zack@upsilon.cc>
    # SPDX-License-Identifier: GPL-3.0-or-later
    ...
    • Sep 30 2021, 5:00 PM
    • 18 Lines
    Author: zack
  • P1186 Command-Line Input
    Setting up postgresql-14 (14.0-1.pgdg+1) ...
    Creating new PostgreSQL cluster 14/main ...
    /usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions
    The files belonging to this database system will be owned by user "postgres".
    This user must also own the server process.
    ...
    • Sep 30 2021, 4:25 PM
    • 22 Lines
    Author: zack
  • P1185 computers say no to D6377
    (ve) swhworker@worker0:~$ time swh loader -C /etc/softwareheritage/loader_git.yml run git https://github.com/torvalds/l
    inux
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/torvalds/linux' with type 'git'
    Enumerating objects: 8345251, done.
    Total 8345251 (delta 0), reused 0 (delta 0), pack-reused 8345251
    ...
    • Sep 30 2021, 10:06 AM
    • 23 Lines
    Author: ardumont
  • P1184 (An Untitled Masterwork)
    # in swh/web/tests/conftest/py
    @pytest.fixture(scope='session')
    def django_db_modify_db_settings(django_db_modify_db_settings_parallel_suffix, postgresql_proc):
    from django.conf import settings as django_settings
    django_settings.DATABASES['default'].update({
    ...
    • Sep 29 2021, 4:19 PM
    • 12 Lines
    Author: douardda
  • P1183 other recurring svn loader issue
    Sep 29 12:51:52 worker18 python3[1046]: [2021-09-29 12:51:52,181: INFO/ForkPoolWorker-7] Processing revisions [1-0] for {'swh-origin': 'https://svn.code.sf.net/p/penguinsdbtools2018/svn', 'remote_url': 'file:///tmp/tmpjtb68puq/swh.loader.s
    vn.n58ub6q5-1046/tmpf9j3ln2g', 'local_url': b'/tmp/swh.loader.svn.ntglfflw-1046/tmpf9j3ln2g', 'uuid': b'd8f5688d-efb6-4dc1-b967-41be2f097118'}
    Sep 29 12:51:52 worker18 python3[1046]: [2021-09-29 12:51:52,182: ERROR/ForkPoolWorker-7] ('No such revision 1', 160006)
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 457, in fetch_data
    ...
    • Sep 29 2021, 3:05 PM
    • 22 Lines
    Author: ardumont
  • P1182 (An Untitled Masterwork)
    Sep 28 15:23:50 worker0 python3[1078211]: [2021-09-28 15:23:50,779: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:51 worker0 python3[1078211]: [2021-09-28 15:23:51,222: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:52 worker0 python3[1078211]: [2021-09-28 15:23:52,122: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:52 worker0 python3[1078211]: [2021-09-28 15:23:52,126: ERROR/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[9fd77287-dc5b-4f63-9f15-516438b8e42e] raised unexpected: RetryError('<Future at 0x7f0027780d68 state=
    finished raised RemoteException>')
    ...
    • Sep 28 2021, 5:25 PM
    • 37 Lines
    Author: ardumont
  • P1181 (An Untitled Masterwork)
     ddouard  (e) swh   master ✚ 2 … 11  ~/s/s/swh-storage  mypy swh
    swh/storage/replay.py:133: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[SkippedContent]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:134: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[Content]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:137: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[SkippedContent]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:152: error: Argument 1 to "metadata_authority_add" of "StorageInterface" has incompatible type "Set[Any]"; expected "List[MetadataAuthority]"
    ...
    • Sep 28 2021, 4:54 PM
    • 7 Lines
    Author: douardda
  • P1180 (An Untitled Masterwork)
    diff --git a/swh/web/api/views/origin_save.py b/swh/web/api/views/origin_save.py
    index a2d3366f..6cba7e4e 100644
    --- a/swh/web/api/views/origin_save.py
    +++ b/swh/web/api/views/origin_save.py
    @@ -11,6 +11,7 @@ from swh.web.common.origin_save import (
    ...
    • Sep 28 2021, 4:48 PM
    • 262 Lines
    Author: anlambert
  • P1179 (An Untitled Masterwork)
    DEBUG:swh.storage.replay:Inserting 200 raw_extrinsic_metadata objects
    Traceback (most recent call last):
    File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.15.0', 'console_scripts', 'swh')()
    File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
    ...
    • Sep 28 2021, 4:46 PM
    • 51 Lines
    Author: douardda
  • P1178 swh-web config
    storage:
    cls: remote
    # url: http://localhost:5002/
    # url: http://storage0.internal.staging.swh.network:5002
    url: http://moma.internal.softwareheritage.org:5002/
    ...
    • Sep 28 2021, 2:07 PM
    • 96 Lines
    • YAML
    Author: anlambert
  • P1177 Command-Line Input
    egrep -v '^".*",".*",".*"$' license-blobs.csv
    "swh:1:cnt:03e1933241b8c3878d81c0184d7f2fd3d8cd6185","037d40bc6bcb42dfd740be545dbdf2df3405442f","LICENSE
    "
    "swh:1:cnt:65a5c662900ee946583147129720563fd4ba286d","40e9258799f752fe25d7518155c615c1c497b7ac","LICENSE.md
    "
    ...
    • Sep 27 2021, 4:50 PM
    • 11 Lines
    Author: zack
  • P1176 Patching dulwich to decrease memory footprint
    #+title: Patching Dulwich to decrease memory footprint
    #+author: ardumont
    In the following analysis, we will executes multiple ingestions with and without the
    patched [1] dulwich version.
    ...
    • Sep 27 2021, 10:01 AM
    • 567 Lines
    Author: ardumont
  • P1175 version ordering in opam is a bit specific
    Case with some edge cases:
    ```
    $ opam show --color never --normalise --root $PWD -f all-versions opam-state
    2.0~alpha5 2.0.0~beta 2.0.0~beta3 2.0.0~beta3.1 2.0.0~beta5 2.0.0~rc 2.0.0~rc2 2.0.0~rc3 2.0.0 2.0.1 2.0.2 2.0.3 2.0.4 2.0.5 2.0.6 2.0.7 2.0.8 2.0.9 2.1.0~beta2 2.1.0~beta4 2.1.0~rc 2.1.0~rc2 2.1.0
    ```
    ...
    • Sep 24 2021, 4:14 PM
    • 78 Lines
    Author: ardumont
  • P1174 (An Untitled Masterwork)
    visit_type=hg; sleep=300; while true; do for policy in never_visited_oldest_update_first already_visited_order_by_lag ; do echo "$(date) scheduling $visit_type origins with policy ${policy}"; SWH_C
    ONFIG_FILENAME=/etc/softwareheritage/scheduler/listener-runner.yml swh scheduler -C /etc/softwareheritage/scheduler/listener-runner.yml origin send-to-celery --policy $policy $visit_type; echo "$(date) sleep
    $sleep" ; sleep $sleep; done; done
    • Sep 23 2021, 11:11 AM
    • 3 Lines
    Author: ardumont
  • P1173 (An Untitled Masterwork)
     ddouard  (e) swh   master … 2  ~/s/s/swh-web  cat swh/web/django_db_backend/base.py
    from django.db.backends.postgresql.base import * # NOQA
    from django.db.backends.postgresql.base import DatabaseWrapper as _DatabaseWrapper
    # dirty hack to allow using a postgresql:// libpq connection URI as db name...
    ...
    • Sep 22 2021, 6:15 PM
    • 12 Lines
    Author: douardda
  • P1172 a bit of a scheduling predicament!?
    - First run of the foss heptapod origins scheduled
    - They failed due to a dangling configuration key from the old mercurial loader (fixed)
    -> all origins have failed
    -> they have now their entry in the origin-visit-stats table
    -> scheduler metrics are updated
    ...
    • Sep 22 2021, 11:37 AM
    • 413 Lines
    Author: ardumont
  • P1171 opam can deal with multi-instances indeed
    $ pwd
    /home/tony/scratch/opam/test-opam-root
    $ opam init --reinit --bare --no-setup --root $PWD opam.ocaml.org https://opam.ocaml.org
    $ opam repository add --root $PWD satysfi-external https://github.com/gfngfn/satysfi-external-repo.git
    $ opam repository add --root $PWD opam-windows-repository https://github.com/vouillon/opam-windows-repository.git
    ...
    • Sep 21 2021, 1:54 PM
    • 56 Lines
    Author: ardumont
  • P1170 opam root taking around 214Mib
    tony  yavin4  ~  %  opam init --reinit --bare --no-setup --root ~/opamtest instance-foo https://opam.ocaml.org
    [NOTE] Will configure from built-in defaults.
    Checking for available remotes: rsync and local, git, mercurial, darcs. Perfect!
    <><> Fetching repository information ><><><><><><><><><><><><><><><><><><><><><>
    ...
    • Sep 20 2021, 12:30 PM
    • 8 Lines
    Author: ardumont
  • P1169 Large repository rounds 1st is long, 2nd is failing faster (because the error is blocking)
    ```
    swhworker@worker17:~$ time SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_oneshot.yml swh loader run mercurial_from_disk https://foss.heptapod.net/heptapod/heptapod
    INFO:swh.loader.mercurial.LoaderFromDisk:Load origin 'https://foss.heptapod.net/heptapod/heptapod' with type 'hg'
    applying clone bundle from https://cellar-c2.services.clever-cloud.com/heptapod-foss-clonebundles/heptapod/heptapod-2020-07-22-zstd-v2.hg
    adding changesets
    ...
    • Sep 17 2021, 11:13 AM
    • 60 Lines
    Author: ardumont
  • P1168 (An Untitled Masterwork)
    diff --git a/swh/lister/gitlab/lister.py b/swh/lister/gitlab/lister.py
    index 5937256..074906d 100644
    --- a/swh/lister/gitlab/lister.py
    +++ b/swh/lister/gitlab/lister.py
    @@ -203,10 +203,11 @@ class GitLabLister(Lister[GitLabListerState, PageResult]):
    ...
    • Sep 16 2021, 5:47 PM
    • 29 Lines
    Author: anlambert
  • P1167 small perf comparison on mercurial loader diffs
    - worker0.staging is running within a venv with D6240 (more extid filtering) and D6268 (build snapshot)
    - worker17 is running with the current latest mercurial packaged (no optim)
    But the filtering is still happening in the mercurial loader (without D6275)
    # tl; dr
    ...
    • Sep 16 2021, 4:04 PM
    • 136 Lines
    Author: ardumont
  • P1166 (An Untitled Masterwork)
    vagrant up test
    Bringing machine 'test' up with 'libvirt' provider...
    ==> test: Creating image (snapshot of base box volume).
    ==> test: Creating domain with the following settings...
    ==> test: -- Name: puppet-environment_test
    ...
    • Sep 15 2021, 5:26 PM
    • 62 Lines
    Author: ardumont
  • P1165 cassandra extid allow filtering
    cqlsh:swh> DESCRIBE extid
    CREATE TABLE swh.extid (
    extid_type ascii,
    extid blob,
    ...
    • Sep 15 2021, 2:23 PM
    • 28 Lines
    Author: vlorentz
  • P1164 flush cassandra buffers
    seq 2 8 | parallel -t ssh root@parasilo-{} nodetool flush
    seq 2 8 | parallel -t ssh root@parasilo-{} systemctl stop cassandra
    seq 2 8 | parallel -t ssh root@parasilo-{} sync
    seq 2 8 | xargs -t -n1 -i{} ssh root@parasilo-{} 'echo 3 > /proc/sys/vm/drop_caches'
    seq 2 8 | parallel -t ssh root@parasilo-{} systemctl start cassandra
    • Sep 15 2021, 12:04 PM
    • 5 Lines
    • Bash Scripting
    Author: vsellier
  • P1163 directory_ls.py
    from swh.storage import get_storage
    from swh.model.hashutil import hash_to_bytes
    import sys
    import time
    ...
    • Sep 15 2021, 11:18 AM
    • 31 Lines
    • Python
    Author: vsellier
  • P1162 Full visits with null snapshots
    10:36 guest@softwareheritage => select type, count(*) from origin_visit_status where status='full' and snapshot is null group by type;
    type │ count
    ──────┼───────
    git │ 1
    hg │ 62959
    ...
    • Sep 15 2021, 11:05 AM
    • 9 Lines
    Author: olasd
  • P1161 (An Untitled Masterwork)
    (swh) ✘-2 ~/swh/swh-environment/swh-loader-mercurial [master|✔]
    15:05 $ tox -r
    GLOB sdist-make: /home/anlambert/swh/swh-environment/swh-loader-mercurial/setup.py
    black recreate: /home/anlambert/swh/swh-environment/swh-loader-mercurial/.tox/black
    black installdeps: black==19.10b0
    ...
    • Sep 14 2021, 3:08 PM
    • 306 Lines
    Author: anlambert
  • P1160 CocoaPods/Specs runs
    ```
    10:51:05 softwareheritage@belvedere:5432=> select * from origin o inner join origin_visit_status ovs on o.id=ovs.origin where url = 'https://github.com/CocoaPods/Specs' order by date desc limit 10;
    +---------+------------------------------------+---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+
    | id | url | origin | visit | date | status | metadata | snapshot | type |
    ...
    • Sep 14 2021, 10:53 AM
    • 41 Lines
    Author: ardumont
  • P1159 load-failing-on-unpacking
    [2] is the log used to extract the following list
    ```
    $ gzip -dc [2] | awk '{print $13" "$15 }' | sed -e 's/\\nTraceback//' | sort | uniq
    ```
    ...
    • Sep 13 2021, 6:10 PM
    • 258 Lines
    Author: ardumont
  • P1158 20210913-load-failing-on-404.txt
    See [1] for the full logs filtered on the command:
    ```
    $ gzip -dc [1] | awk '{print $13" "$15 }' | sed -e 's/\\nTraceback//' | sort | uniq
    ```
    ...
    • Sep 13 2021, 5:59 PM
    • 357 Lines
    Author: ardumont
  • P1157 Revisions with non-empty metadata field and missing raw extrinsic metadata
    00959a167bd98452c98ce73382f4b42179d53d32
    00a867beb2ad8e203f242e9843d2e88de0856cda
    028e9890a9287b35851c48ca351641743542d030
    030a51a49b3239769928872be9ac6d435ab14a61
    036594a6bbec926c21fa073e2404a5f760d35a43
    ...
    • Sep 10 2021, 6:08 PM
    • 166 Lines
    Author: olasd
First
Prev
Next
About · Developer information · Legal