Page MenuHomeSoftware Heritage
Paste Active Pastes
  • 10:04:07 swh-scheduler@db1:5432=> select first_seen, url from listed_origins where visit_type='bzr' and url like 'http://%' order by first_seen desc;
    select first_seen, url from listed_origins where visit_type='bzr' and url like 'http://%' order by first_seen desc;
    +-------------------------------+--------------------------------------------------------------------+
    | first_seen | url |
    +-------------------------------+--------------------------------------------------------------------+
    ...
    • Feb 15 2022, 10:05 AM
    • 83 Lines
  • diff ok.org ko.org
    9c9
    < charset-normalizer==2.0.11
    ---
    > charset-normalizer==2.0.12
    ...
    • Feb 14 2022, 3:52 PM
    • 25 Lines
  • dpkg: error processing archive /var/cache/apt/archives/python3-swh.loader.bzr_1.1.0-1~swh2~bpo10+1_all.deb (--unpack):
    trying to overwrite '/usr/lib/python3/dist-packages/swh/loader/__init__.py', which is also in package python3-swh.loader.core 2.5.0-1~swh1~bpo10+1
    Errors were encountered while processing:
    /var/cache/apt/archives/python3-swh.loader.bzr_1.1.0-1~swh2~bpo10+1_all.deb
    • Feb 11 2022, 5:43 PM
    • 4 Lines
  • sbuild --nolog --batch --no-clean-source --no-run-lintian --arch-all --source --build-dep-resolver=aptitude --extra-repository='deb http://incoming.debian.org/debian-buildd/ buildd-unstable main' --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'
    dpkg-source: info: using source format '3.0 (quilt)'
    dpkg-source: info: building breezy using existing ./breezy_3.2.1+bzr7585.orig.tar.gz
    dpkg-source: warning: upstream signing key but no upstream tarball signature
    dpkg-source: info: using patch list from debian/patches/series
    ...
    • Feb 11 2022, 5:14 PM
    • 49,805 Lines
  • Feb 11 13:46:52 kelvingrove keycloak[1436401]: 13:46:52,362 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-109) Uncaught server error: org.keycloak.authentication.AuthenticationFlowException: authenticator: direct-grant-validate-password
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.DefaultAuthenticationFlow.processSingleFlowExecutionModel(DefaultAuthenticationFlow.java:423)
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:247)
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:976)
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.protocol.oidc.endpoints.TokenEndpoint.resourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:592)
    ...
    • Feb 11 2022, 2:51 PM
    • 77 Lines
  • $ swh auth --client-id swh-web generate-token vlorentz
    Password:
    Traceback (most recent call last):
    File "/home/dev/.local/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core', 'console_scripts', 'swh')())
    ...
    • Feb 11 2022, 2:43 PM
    • 30 Lines
  • def origin_visit_find_latest_status_by_date(
    origin_url: str, visit_date: datetime.datetime
    ) -> Optional[OriginVisitInfo]:
    """Retrieve origin visit status whose date is most recent than the provided visit_date.
    ...
    • Feb 10 2022, 9:22 AM
    • 30 Lines
    • Python
  • 0: swh:1:dir:d24c739c57c68a5cfa7a0042bfbd796f875d1e0c
    successors:
    swh:1:dir:bd1898f4ea14c895bb34219c5495f1c9840eab2a asm [perms: 16384]
    swh:1:dir:294d460f1960619445897b62d8aa0bddb6d81f40 uapi [perms: 16384]
    ...
    • Feb 9 2022, 4:05 PM
    • 51 Lines
  • package org.softwareheritage.graph.utils;
    import it.unimi.dsi.big.webgraph.NodeIterator;
    import org.softwareheritage.graph.SwhUnidirectionalGraph;
    import org.softwareheritage.graph.labels.DirEntry;
    ...
    • Feb 9 2022, 3:58 PM
    • 77 Lines
    • Java
  • swh-lister_1 | [2022-02-08 10:29:25,384: INFO/MainProcess] Task swh.lister.maven.tasks.FullMavenLister[95fa1dcb-6fea-41b7-82e3-e07162d45f2d] received
    swh-lister_1 | [2022-02-08 10:29:45,424: INFO/ForkPoolWorker-1] Downloading text index from http://swh-lister-maven-nginx/export.fld.
    swh-lister_1 | [2022-02-08 10:30:05,455: ERROR/ForkPoolWorker-1] Task swh.lister.maven.tasks.FullMavenLister[95fa1dcb-6fea-41b7-82e3-e07162d45f2d] raised unexpected: ValueError("can not serialize 'datetime.datetime' object where tzinfo=None")
    swh-lister_1 | Traceback (most recent call last):
    swh-lister_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 450, in trace_task
    ...
    • Feb 8 2022, 11:31 AM
    • 37 Lines
  • public class DumpProperties {
    public static void main(String[] args) throws IOException {
    String graphPath = args[0];
    SwhBidirectionalGraph graph = SwhBidirectionalGraph.load(graphPath);
    ...
    • Feb 7 2022, 5:56 PM
    • 50 Lines
  • 17:49:20 swh-scheduler@db1:5432=> select now(), count(*) from listed_origins where visit_type='bzr';
    select now(), count(*) from listed_origins where visit_type='bzr';
    +-------------------------------+-------+
    | now | count |
    +-------------------------------+-------+
    ...
    • Feb 7 2022, 5:50 PM
    • 309 Lines
  • ```
    @remote_api_endpoint("origin/visit_status/find_by_date")
    def origin_visit_status_find_by_date(
    self, origin: str, visit_date: datetime.datetime
    ) -> Optional[OriginVisit]:
    ...
    • Feb 4 2022, 3:55 PM
    • 18 Lines
  • diff --git a/swh/web/common/origin_visits.py b/swh/web/common/origin_visits.py
    index 6dc2201a..023c7aa8 100644
    --- a/swh/web/common/origin_visits.py
    +++ b/swh/web/common/origin_visits.py
    @@ -42,43 +42,45 @@ def get_origin_visits(origin_info: OriginInfo) -> List[OriginVisitInfo]:
    ...
    • Feb 3 2022, 6:07 PM
    • 96 Lines
  • diff --git a/swh/web/common/archive.py b/swh/web/common/archive.py
    index e9e877f6..dca07e30 100644
    --- a/swh/web/common/archive.py
    +++ b/swh/web/common/archive.py
    @@ -703,7 +703,7 @@ def lookup_revision_log(rev_sha1_git, limit):
    ...
    • Feb 3 2022, 3:56 PM
    • 13 Lines
  • time curl https://webapp1.internal.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/
    [{"origin_url":"https://foss.heptapod.net/mercurial/hgview","visit_type":"hg","save_request_date":"2022-02-02T10:15:01.448199+00:00","save_request_status":"accepted","save_task_status":"failed","visit_status":"failed","visit_date":"2022-02-02T10:15:17.559967+00:00","loading_task_id":405935267,"note":null},{"origin_url":"https://foss.heptapod.net/mercurial/hgview","visit_type":"hg","save_request_date":"2022-02-02T09:48:23.326128+00:00","save_request_status":"accepted","save_task_status":"succeeded","visit_status":"full","visit_date":"2022-02-02T09:48:36.296855+00:00","loading_task_id":405934604,"note":null},{"origin_url":"https://foss.heptapod.net/mercurial/hgview","visit_type":"hg","save_request_date":"2022-02-02T09:44:36.829049+00:00","save_request_status":"accepted","save_task_status":"succeeded","visit_status":"full","visit_date":"2022-02-02T09:44:46.020548+00:00","loading_task_id"...
    • Feb 2 2022, 11:36 AM
    • 2 Lines
  • $ varnishncsa -F '%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" %D' | grep -E " [0-9]{7,}$"
    ...
    128.93.166.14 - - [02/Feb/2022:09:35:11 +0000] "GET http://archive.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/ HTTP/1.1" 502 341 "-" "python-requests/2.21.0" 60058264
    128.93.166.14 - - [02/Feb/2022:09:41:16 +0000] "POST http://archive.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/ HTTP/1.1" 502 341 "-" "python-requests/2.21.0" 60057275
    128.93.166.14 - - [02/Feb/2022:09:44:36 +0000] "POST http://archive.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/ HTTP/1.1" 502 341 "-" "python-requests/2.21.0" 60060106
    ...
    • Feb 2 2022, 11:04 AM
    • 6 Lines
  • root@pergamon:~# sudo -u nobody /bin/bash
    nobody@pergamon:/root$ chronic /usr/local/bin/weekly-planning-bot
    + DEST=ardumont@softwareheritage.org
    + AUTHOR=swh-team@inria.fr
    + TEMPLATE_ID=6YKT5osoST-amJQ0MusH2Q
    ...
    • Jan 31 2022, 11:40 AM
    • 103 Lines
  • root@dali:~# cat backup.sh
    #!/bin/bash
    i=0
    dataset=data/postgresql
    ...
    • Jan 27 2022, 10:25 AM
    • 20 Lines
    • Bash Scripting
  • $ sudo virsh list
    Id Name State
    ----------------------------------------------
    2 puppet-environment_pergamon running
    4 puppet-environment_riverside running
    ...
    • Jan 27 2022, 10:15 AM
    • 15 Lines
  • use anyhow::Result;
    use tempfile::TempDir;
    use std::{
    env, io,
    ...
    • Jan 26 2022, 9:46 PM
    • 377 Lines
    • rust
  • write(4, "0032have 16617535684faf9de30620de83667214297a36b8\n0032have ee8f3955c0ef4c7a90a5f707c37c4cb3711ff626\n"..., 1604) = 1604
    read(3, "0008", 4) = 4
    read(3, "NAK\n", 4) = 4
    brk(0x5558db682000) = 0x5558db682000
    write(4, "0032have 1e1490a38504419e349caa1b7d55d5c141a9bccb\n0032have 90e94b160c7f647ddffda707f5e3c0c66c170df8\n"..., 1604) = 1604
    ...
    • Jan 26 2022, 2:42 PM
    • 81 Lines
  • softwareheritage=> copy (select distinct date_offset, encode(date_offset_bytes, 'escape') from revision) to stdout;
    69 +0109
    765 +1245
    -137 -0217
    360 +0600
    ...
    • Jan 21 2022, 9:21 AM
    • 446 Lines
  • class Statsd:
    # ...
    @contextmanager
    def status_gauge(self, metric_name: str, statuses: Collection[str], common_tags: Optional[Dict[str, str]] = None):
    current_status = Optional[str] = None
    ...
    • Jan 20 2022, 1:46 PM
    • 19 Lines
    • Python
  • Jan 19 16:11:15 worker0 python3[3375647]: [2022-01-19 16:11:15,120: INFO/MainProcess] Received task: swh.loader.svn.tasks.DumpMountAndLoadSvnRepository[8028cd34-c06f-4dda-b109-a1b315dafabb]
    Jan 19 16:11:18 worker0 python3[3375658]: [2022-01-19 16:11:18,753: INFO/ForkPoolWorker-1] Load origin 'https://plugins.svn.wordpress.org/wp-activity-log-for-woocommerce' with type 'svn'
    Jan 19 21:12:10 worker0 python3[3375658]: [2022-01-19 21:12:10,701: INFO/ForkPoolWorker-1] Processing revisions [1-168825] for {'swh-origin': 'https://plugins.svn.wordpress.org/wp-activity-log-for-woocommerce', 'remote_url': 'file:///tmp/swh.loader.svn.dxrrwh6j-3375658/swh.loader.svn.fw2r9j6a-3375658/tmpiw_ugkd8', 'local_url': b'/tmp/swh.loader.svn.ft2fxai2-3375658/tmpiw_ugkd8', 'uuid': b'b8457f37-d9ea-0310-8a92-e5e31aec5664'}
    Jan 19 21:18:07 worker0 python3[3375658]: [2022-01-19 21:18:07,339: INFO/ForkPoolWorker-1] Task swh.loader.svn.tasks.DumpMountAndLoadSvnRepository[8028cd34-c06f-4dda-b109-a1b315dafabb] succeeded in 18412.19621137576s...
    • Jan 20 2022, 9:59 AM
    • 4 Lines
  • git,https://scm.gforge.inria.fr/anonscm/git/simty/simty.git
    git,https://github.com/unitystation/unitystation
    svn,http://svn.code.sf.net/p/sauerbraten/code
    git,https://gitlab.com/inkscape/inkscape
    git,https://github.com/keybase/client
    ...
    • Jan 17 2022, 1:54 PM
    • 25 Lines
  • 12:37 $ git diff
    diff --git a/swh/web/common/origin_save.py b/swh/web/common/origin_save.py
    index 3749d134..6696427b 100644
    --- a/swh/web/common/origin_save.py
    +++ b/swh/web/common/origin_save.py
    ...
    • Jan 13 2022, 12:38 PM
    • 20 Lines
  • insert into new_task(id, type, arguments, next_run, current_interval, status, policy,
    retries_left, priority)
    select id, type, arguments, next_run, current_interval, status, policy,
    retries_left, priority
    from task
    ...
    • Jan 12 2022, 1:47 PM
    • 33 Lines
  • -- SWH DB schema upgrade
    -- from_version: 32
    -- to_version: 33
    -- description: Archive old new_task/new_task-run tasks
    ...
    • Jan 10 2022, 3:52 PM
    • 223 Lines
  • >>> s = '<entry xmlns:testns="https://example.org"> <testns:foo> <testns:bar url="http://example.org/1" /> <testns:bar url="http://example.org/2" /></testns:foo> </entry>'
    >>> pprint(xmltodict.parse(s, process_namespaces=True, dict_constructor=dict))
    {'entry': {'https://example.org:foo': {'https://example.org:bar': [{'@url': 'http://example.org/1',
    '@xmlns': OrderedDict([('testns',
    'https://example.org')])},
    ...
    • Jan 10 2022, 2:25 PM
    • 6 Lines
    • Python
  • ```
    16:28:34 softwareheritage-scheduler@belvedere:5432=> select type, policy, count(*) from task where type like 'load-%' and status !='disabled' and priority is null group by type, policy;
    +--------------------+-----------+---------+
    | type | policy | count |
    +--------------------+-----------+---------+
    ...
    • Jan 7 2022, 5:13 PM
    • 21 Lines
  • 16:45:09 swh-web@belvedere:5432=> select * from save_origin_request where origin_url='https://github.com/globalbioticinteractions/usnm-ixodes' order by id desc limit 3;
    +--------+-------------------------------+------------+---------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+--------+
    | id | request_date | visit_type | origin_url | status | loading_task_id | visit_date | loading_task_status | visit_status | user_ids | note |
    +--------+-------------------------------+------------+---------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+--------+
    | 105121 | 2022-01-07 15:14:52.39695+00 | git | https://github.com/globalbioticinteractions/usnm-ixodes | accepted | 404639723 | (null) | succeeded | (null) | (null) | (null) |
    ...
    • Jan 7 2022, 4:46 PM
    • 8 Lines
  • 11:03:56 Build Architecture: amd64
    11:03:56 Build Type: full
    11:03:56 Build-Space: 109680
    11:03:56 Build-Time: 288
    11:03:56 Distribution: unstable-swh
    ...
    • Jan 4 2022, 12:14 PM
    • 26 Lines
  • (.venv) jenkins@30f908325208:~/swh-environment/swh-search$ python3 -c "from frozendict import frozendict; frozendict({})"
    Segmentation fault (core dumped)
    (.venv) jenkins@30f908325208:~/swh-environment/swh-search$ ls
    AUTHORS CONTRIBUTORS MANIFEST.in README.md docs mypy.ini package.json pytest.ini requirements-test.txt setup.cfg swh tox.ini
    CODE_OF_CONDUCT.md LICENSE Makefile core es_config node_modules pyproject.toml requirements-swh.txt requirements.txt setup.py swh.search.egg-info yarn.lock
    ...
    • Dec 21 2021, 4:52 PM
    • 62 Lines
  • pytest -s
    ============================================================================================================================= test session starts =============================================================================================================================
    platform linux -- Python 3.7.3, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
    rootdir: /home/test/swh-environment/swh-search, configfile: pytest.ini
    plugins: django-test-migrations-1.2.0, postgresql-2.6.1, swh.core-1.0.0, xdist-2.5.0, forked-1.4.0, redis-2.3.0, flask-1.2.0, asyncio-0.16.0, dash-2.0.0, django-4.5.2, requests-mock-1.9.3, mock-3.6.1, hypothesis-6.31.6, swh.journal-0.9.2.dev2+g4e5e009
    ...
    • Dec 21 2021, 1:30 PM
    • 95 Lines
  • ./bin/update
    Already up to date.
    perl: warning: Setting locale failed.
    perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    ...
    • Dec 21 2021, 12:43 PM
    • 161 Lines
  • vagrant plugin install vagrant-snapshot
    Installing the 'vagrant-snapshot' plugin. This can take a few minutes...
    Building native extensions. This could take a while...
    Vagrant failed to properly resolve required dependencies. These
    errors can commonly be caused by misconfigured plugin installations
    ...
    • Dec 14 2021, 10:01 AM
    • 20 Lines
  • #!/bin/bash
    id=$1
    echo Getting the logs...
    ...
    • Dec 14 2021, 9:46 AM
    • 21 Lines
    • Bash Scripting
  • 15:59:24 softwareheritage-scheduler@belvedere:5432=> select * from listed_origins where visit_type='cvs' limit 10;
    +--------------------------------------+----------------------------------------------+------------+------------------------+---------+-------------------------------+-------------------------------+------------------------+
    | lister_id | url | visit_type | extra_loader_arguments | enabled | first_seen | last_seen | last_update |
    +--------------------------------------+----------------------------------------------+------------+------------------------+---------+-------------------------------+-------------------------------+------------------------+
    | b678cfc3-2780-4186-9186-d78a14bd4958 | https://cvs.code.sf.net/p/a-a-p/code | cvs | {} | t | 2021-06-01 20:24:22.899139+00 | 2021-10-15 04:34:05.365731+00 | 2018-08-12 00:00:00+00 |
    ...
    • Dec 9 2021, 3:59 PM
    • 16 Lines
  • *** swh-deploy: starting test run on moma.internal.softwareheritage.org...
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Dec 8 2021, 3:03 PM
    • 27 Lines
  • diff --git a/swh/loader/svn/ra.py b/swh/loader/svn/ra.py
    index 8a51c49..d88496f 100644
    --- a/swh/loader/svn/ra.py
    +++ b/swh/loader/svn/ra.py
    @@ -11,6 +11,7 @@ from __future__ import annotations
    ...
    • Dec 8 2021, 1:42 PM
    • 245 Lines
  • ```
    Dec 08 08:31:43 thyssen package-helper[6666]: /etc/unbound/unbound.conf.d/listen-docker.conf:1: error: syntax error
    Dec 08 08:31:43 thyssen package-helper[6666]: read /etc/unbound/unbound.conf failed: 1 errors in configuration file
    Dec 08 08:31:43 thyssen package-helper[6669]: /etc/unbound/unbound.conf.d/listen-docker.conf:1: error: syntax error
    Dec 08 08:31:43 thyssen package-helper[6669]: read /etc/unbound/unbound.conf failed: 1 errors in configuration file
    ...
    • Dec 8 2021, 9:36 AM
    • 10 Lines
  • Somehow for now, the pergamon manifest is not complete/incorrect (and it's not clear where exactly we fall short).
    At boot time, we have the puppet master running while it should not. Puppet master must be served through apache (ssl configured and what not).
    So we need to make the provisioning step pass twice (for some unknown reason as of yet as well).
    Then connect to the node, stop and deactivate the puppet master service.
    Then restart the apache service so it serves properly puppet (as master).
    ...
    • Dec 7 2021, 10:27 AM
    • 25 Lines
  • vagrant vm:
    ```
    root@thyssen:~# puppet agent --test
    Info: Using configured environment 'production'
    ...
    • Dec 7 2021, 10:19 AM
    • 76 Lines
  • production:
    ```
    List of databases
    +------------------------------------+--------------+----------+---------+---------+-----------------------+
    | Name | Owner | Encoding | Collate | Ctype | Access privileges |
    ...
    • Dec 6 2021, 5:57 PM
    • 65 Lines
  • 15:04:47 swh-scheduler@db1:5432=> select * from task where type = 'list-debian-distribution';
    +-[ RECORD 1 ]-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | id | 1182486 |
    | type | list-debian-distribution |
    | arguments | {"args": [], "kwargs": {"suites": ["stretch", "buster", "bullseye"], "components": ["main", "contrib", "non-free"], "mirror_url": "http://deb.debian.org/debian/", "distribution": "Debian"}} |
    ...
    • Dec 6 2021, 3:05 PM
    • 24 Lines
  • root@pergamon:~# clush -b -w @swh-workers systemctl status swh-worker@loader_nixguix | grep succeeded
    Dec 05 18:28:52 worker15 python3[2936713]: [2021-12-05 18:28:52,509: INFO/ForkPoolWorker-1] Task swh.loader.package.nixguix.tasks.LoadNixguix[d38e7d14-4894-4851-9ec3-7fa533d74049] succeeded in 28061.734233491123s: {'status': 'eventful', 'snapshot_id': '3568e78b54169b3995efa0acfa7c88a45d0056aa'}
    Dec 05 10:40:20 worker16 python3[2974229]: [2021-12-05 10:40:20,535: INFO/ForkPoolWorker-1] Task swh.loader.package.nixguix.tasks.LoadNixguix[51f86173-0b8e-496f-8d05-fb396247d197] succeeded in 9895.689101913944s: {'status': 'eventful', 'snapshot_id': '0415952acb92d5a8ad03c1f9d60a81459cb78aa5'}
    • Dec 6 2021, 9:33 AM
    • 3 Lines
  • (swh) ✔ ~/swh/swh-environment/docker [master|✚ 1⚑ 41]
    16:01 $ doco exec swh-scheduler /bin/bash
    swh@ab68bb37fea4:/$ psql service=swh-scheduler
    psql (12.9 (Debian 12.9-1.pgdg110+1), server 12.8 (Debian 12.8-1.pgdg100+1))
    Type "help" for help.
    ...
    • Dec 3 2021, 4:02 PM
    • 22 Lines
  • 14:09:12 swh-scheduler@db1:5432=> begin;
    BEGIN
    Time: 5.818 ms
    14:09:14 *swh-scheduler@db1:5432=> update listed_origins set extra_loader_arguments = extra_loader_arguments - 'date' where visit_type='deb';
    UPDATE 35105
    ...
    • Dec 3 2021, 2:09 PM
    • 15 Lines
  • 13:38:27 softwareheritage-scheduler@belvedere:5432=> select * from task where type = 'list-gitlab-incremental';
    +-----------+-------------------------+--------------------------------------------------------------------------------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------+
    | id | type | arguments | next_run | current_interval | status | policy | retries_left | priority |
    +-----------+-------------------------+--------------------------------------------------------------------------------------------------+-------------------------------+------------------+------------------------+-----------+--------------+----------+
    | 399914374 | list-gitlab-incremental | {"args": [], "kwargs": {"url": "https://foss.heptapod.net/api/v4/", "name": "heptapod"}} | 2021-12-04 08:56:02.284533+00 | 1 day | next_run_not_scheduled | recurring | 3 | (null) |
    ...
    • Dec 3 2021, 1:38 PM
    • 16 Lines
  • root@riverside:~# for volume in data redis kafka zookeeper symbolicator clickhouse; do echo docker volume create --name=sentry-$volume; done
    docker volume create --name=sentry-data
    docker volume create --name=sentry-redis
    docker volume create --name=sentry-kafka
    docker volume create --name=sentry-zookeeper
    ...
    • Dec 3 2021, 10:11 AM
    • 26 Lines
  • from functools import wraps
    def handle_raise_on_commit(f):
    @wraps(f)
    ...
    • Dec 2 2021, 5:02 PM
    • 16 Lines
    • Python
  • swh-loader_1 | [2021-12-02 13:43:44,472: ERROR/ForkPoolWorker-1] Task swh.loader.package.debian.tasks.LoadDebian[fb2923f8-a344-4a4d-9340-0897f6bab6fc] raised unexpected: TypeError("__init__() missing 1 required positional argument: 'date'")
    swh-loader_1 | Traceback (most recent call last):
    swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 450, in trace_task
    swh-loader_1 | R = retval = fun(*args, **kwargs)
    swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/task.py", line 55, in __call__
    ...
    • Dec 2 2021, 2:46 PM
    • 33 Lines
  • class DirectoryData:
    date: datetime
    """
    Date of the directory in the isochrone frontier
    """
    ...
    • Dec 1 2021, 5:13 PM
    • 11 Lines
    • Python
  • @dataclass(eq=True, frozen=True)
    class DirectoryData:
    """Object representing the data associated to a directory in the provenance model,
    where `date` is the date of the directory in the isochrone frontier, and `flat` is a
    flag acknowledging that a flat model for the elements outside the frontier has
    ...
    • Dec 1 2021, 5:12 PM
    • 10 Lines
    • Python
  • P1225 tryout
    $ ipython
    In [1]: 1626111425534 / 1e3
    Out[1]: 1626111425.534
    ...
    • Nov 26 2021, 10:40 AM
    • 22 Lines
  • worker11: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 553
    worker11: Error: /Stage[main]/Profile::Systemd_journal::Journalbeat/User[journalbeat]/ensure: change from 'present' to 'absent' failed: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 553
    worker10: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 522
    worker10: Error: /Stage[main]/Profile::Systemd_journal::Journalbeat/User[journalbeat]/ensure: change from 'present' to 'absent' failed: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 522
    worker12: Error: Could not delete user journalbeat: Execution of '/usr/sbin/userdel journalbeat' returned 8: userdel: user journalbeat is currently used by process 516
    ...
    • Nov 26 2021, 10:20 AM
    • 31 Lines
  • root@bardo:~# puppet agent --test
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Nov 24 2021, 4:22 PM
    • 149 Lines
  • root@rp1:/etc# diff -U3 /etc/apt/apt.conf.d/50unattended-upgrades.ucf-old /etc/apt/apt.conf.d/50unattended-upgrades
    --- /etc/apt/apt.conf.d/50unattended-upgrades.ucf-old 2021-11-23 18:02:23.908766790 +0000
    +++ /etc/apt/apt.conf.d/50unattended-upgrades 2021-11-24 10:34:33.004766790 +0000
    @@ -1,68 +1,164 @@
    -// This file is managed by Puppet. DO NOT EDIT.
    ...
    • Nov 24 2021, 11:38 AM
    • 205 Lines
  • $ vagrant up admin-bardo
    Bringing machine 'admin-bardo' up with 'libvirt' provider...
    ==> admin-bardo: Creating image (snapshot of base box volume).
    ==> admin-bardo: Creating domain with the following settings...
    ==> admin-bardo: -- Name: puppet-environment_admin-bardo
    ...
    • Nov 23 2021, 4:07 PM
    • 691 Lines
  • vagrant up admin-bardo
    Bringing machine 'admin-bardo' up with 'libvirt' provider...
    ==> admin-bardo: Creating image (snapshot of base box volume).
    ==> admin-bardo: Creating domain with the following settings...
    ==> admin-bardo: -- Name: puppet-environment_admin-bardo
    ...
    • Nov 22 2021, 1:34 PM
    • 117 Lines
  • (swh) ✔ ~/swh/swh-environment/swh-loader-svn [master|✚ 2⚑ 10]
    16:43 $ git diff
    diff --git a/swh/loader/svn/loader.py b/swh/loader/svn/loader.py
    index eac96a3..698babe 100644
    --- a/swh/loader/svn/loader.py
    ...
    • Nov 18 2021, 4:54 PM
    • 60 Lines
  • diff --git a/swh/scheduler/celery_backend/recurrent_visits.py b/swh/scheduler/celery_backend/recurrent_visits.py
    index 8f43d8e..eacf8de 100644
    --- a/swh/scheduler/celery_backend/recurrent_visits.py
    +++ b/swh/scheduler/celery_backend/recurrent_visits.py
    @@ -23,6 +23,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Tuple
    ...
    • Nov 16 2021, 2:16 PM
    • 93 Lines
  • diff --git a/swh/journal/client.py b/swh/journal/client.py
    index ff6f0ef..344f629 100644
    --- a/swh/journal/client.py
    +++ b/swh/journal/client.py
    @@ -297,13 +297,13 @@ class JournalClient:
    ...
    • Nov 16 2021, 12:40 PM
    • 19 Lines
  • 16:01:14 swh@db1:5432=> select * from origin_visit_status ovs inner join origin o on ovs.origin=o.id where date > '2021-11-10 15:00' and type='npm' limit 10;
    +---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+---------+-----------------------------------------------------------+
    | origin | visit | date | status | metadata | snapshot | type | id | url |
    +---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+---------+-----------------------------------------------------------+
    | 1796362 | 1 | 2021-11-10 15:00:22.221484+00 | full | (null) | \x034b93485d1fc0a0efc9dd3447b1f7a92e278041 | npm | 1796362 | https://www.npmjs.com/package/angular-phonecat-nexen |
    ...
    • Nov 10 2021, 4:04 PM
    • 18 Lines
  • root@worker18:/home/ardumont# tree -L 3 /tmp/tmp0tm0182g/
    /tmp/tmp0tm0182g/
    ├── swh.loader.svn.2zdkkl_5-1740204
    │   └── tmpi0q7t29s
    │   ├── conf
    ...
    • Nov 10 2021, 9:42 AM
    • 20 Lines
  • swhworker@worker17:~$ /usr/bin/time -v swh loader run svn https://profs.scienze.univr.it/posenato/svn/sw/CSTNU
    INFO:swh.loader.svn.SvnLoader:Load origin 'https://profs.scienze.univr.it/posenato/svn/sw/CSTNU' with type 'svn'
    INFO:swh.loader.svn.SvnLoader:Processing revisions [297-619] for {'swh-origin': 'https://profs.scienze.univr.it/posenato/svn/sw/CSTNU', 'remote_url': 'file:///tmp/tmpsd2aj26l/swh.loader.svn.xkj39oxf-930164/tmpphfs60dp', 'local_url': b'/tmp/swh.loader.svn.kqus7k3a-930164/tmpphfs60dp', 'uuid': b'782453a1-1937-45d1-8845-2a6fcc2839b7'}
    {'status': 'eventful'}
    Command being timed: "swh loader run svn https://profs.scienze.univr.it/posenato/svn/sw/CSTNU"
    ...
    • Nov 8 2021, 6:57 PM
    • 27 Lines
  • swh-loader_1 | [2021-10-29 21:21:17,320: INFO/ForkPoolWorker-1] Processing revisions [1-1089] for {'swh-origin': 'http://svn.code.sf.net/p/blinkensisters/code', 'remote_url': 'file:///tmp/tmpi_mjbfaq/swh.loader.svn.zyez62lm-83/tmpcsww9pus', 'local_url': b'/tmp/swh.loader.svn.tx68am7w-83/tmpcsww9pus', 'uuid': b'943a9807-df12-0410-b0c0-cd0da6ec65e4'}
    swh-loader_1 | [2021-10-29 21:21:17,325: DEBUG/ForkPoolWorker-1] rev: 1, swhrev: d0fde2b7bc24f3b6d6316a079e310905f2b6179e, dir: 7a2faa45330bbfdb9d7ffca331d57073fb8ba61f
    swh-loader_1 | [2021-10-29 21:21:17,443: DEBUG/ForkPoolWorker-1] rev: 2, swhrev: 56eccaa1166c813a33b90b2b90db1658b1fd3b6e, dir: d4b04aa0c1fc8da2e2092cdb9956a0db9d18e86c
    swh-loader_1 | [2021-10-29 21:21:17,449: DEBUG/ForkPoolWorker-1] rev: 3, swhrev: 695e144d7820bd28d6b796e5930b7a3998e4b464, dir: 6744db600724f5f95eef06a7926c9e33a1f3d620
    swh-loader_1 | [2021-10-29 21:21:17,702: DEBUG/ForkPoolWorker-1] rev: 4, swhrev: 39d2270022f5ffec7fff9f0e11700776ff827075, dir: 150465caf99e23ed5e505f874e1d2249f6036dc8
    ...
    • Nov 2 2021, 1:48 PM
    • 253 Lines
  • swh-loader_1 | [2021-10-28 15:14:21,272: DEBUG/ForkPoolWorker-1] Mounting dump file with "svnadmin load".
    swh-loader_1 | [2021-10-28 15:14:26,735: INFO/ForkPoolWorker-1] Processing revisions [1-253] for {'swh-origin': 'http://svn.code.sf.net/p/proxytunnel/code/', 'remote_url': 'file:///tmp/tmp8oo46bm8/swh.loader.svn.0znrosz2-84/tmp1wjhoail', 'local_url': b'/tmp/swh.loader.svn.wpdwou1s-84/tmp1wjhoail', 'uuid': b'bc163920-b10d-0410-b2c5-a5491ca2ceef'}
    swh-loader_1 | [2021-10-28 15:14:26,739: DEBUG/ForkPoolWorker-1] rev: 1, swhrev: 93a685b99d596bf5bbd20cbd14752ff55fb6da7c, dir: 75ed58f260bfa4102d0e09657803511f5f0ab372
    swh-loader_1 | [2021-10-28 15:14:26,739: DEBUG/ForkPoolWorker-1] Checking hash computations on revision 1...
    swh-loader_1 | [2021-10-28 15:14:26,754: DEBUG/ForkPoolWorker-1] rev: 2, swhrev: eb093a0018af3b7e114023347b68217123efa326, dir: 99b8ce06292b877fd89143183217962e38943ca8
    ...
    • Oct 28 2021, 5:16 PM
    • 376 Lines
  • swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': ['swh.loader.svn.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.git.tasks', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', 'swh.loader.package.deposit.tasks', 'swh.loader.package.nixguix.tasks', 'swh.loader.package.npm.tasks', 'swh.loader.package.opam.tasks', 'swh.loader.package.pypi.tasks', 'swh.lister.bitbucket.tasks', 'swh.lister.cgit.tasks', 'swh.lister.cran.tasks', 'swh.lister.debian.tasks', 'swh.lister.gitea.tasks', 'swh.lister.github.tasks', 'swh.lister.gitlab.tasks', 'swh.lister.gnu.tasks', 'swh.lister.launchpad.tasks', 'swh.lister.npm.tasks', 'swh.lister.opam.tasks', 'swh.lister.packagist.tasks', 'swh.lister.phabricator.tasks', 'swh.lister.pypi.tasks', 'swh.lister.sourceforge.tasks', 'swh.lister.tuleap.tasks', 'swh.deposit.loader.tasks']}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    ...
    • Oct 27 2021, 3:42 PM
    • 108 Lines
  • /usr/bin/time -v swh-doco exec swh-loader swh loader run git https://github.com/scikit-learn/scikit-learn
    + cd /home/tony/work/inria/repo/swh/swh-environment/docker
    + docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader swh loader run git https://github.com/scikit-learn/scikit-learn
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/scikit-learn/scikit-learn' with type 'git'
    Enumerating objects: 206538, done.
    ...
    • Oct 26 2021, 4:16 PM
    • 47 Lines
  • /usr/bin/time -v swh-doco exec swh-loader swh loader run git https://git.code.sf.net/u/bsomervi/hamlib.git
    + cd /home/tony/work/inria/repo/swh/swh-environment/docker
    + docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader swh loader run git https://git.code.sf.net/u/bsomervi/hamlib.git
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://git.code.sf.net/u/bsomervi/hamlib.git' with type 'git'
    Enumerating objects: 52811, done.
    ...
    • Oct 26 2021, 10:32 AM
    • 33 Lines
  • #+title: About dropping ingestion of PR-like branches in the loader git
    * synthesis
    Code got adapted so the loader git actually retrieves the full packfile (D6550). Then
    ...
    • Oct 25 2021, 9:46 PM
    • 523 Lines
  • (swh) ✔ ~/swh/swh-environment/docker [master|✚ 2⚑ 35]
    14:21 $ doco exec swh-scheduler-runner swh scheduler origin grab-next tar 10
    lister_id,url,visit_type,extra_loader_arguments,last_update,enabled,first_seen,last_seen
    efd50b96-ecb1-47db-bfdf-56f2247769c3,https://cran.r-project.org/package=aaSEA,tar,"{'artifacts': [{'url': 'https://cran.r-project.org/src/contrib/aaSEA_1.1.0.tar.gz', 'version': '1.1.0'}]}",2019-11-09 15:35:46+00:00,True,2021-10-22 12:14:20.515523+00:00,2021-10-22 12:14:20.515523+00:00
    efd50b96-ecb1-47db-bfdf-56f2247769c3,https://cran.r-project.org/package=AATtools,tar,"{'artifacts': [{'url': 'https://cran.r-project.org/src/contrib/AATtools_0.0.1.tar.gz', 'version': '0.0.1'}]}",2020-06-09 16:49:30+00:00,True,2021-10-22 12:14:20.515523+00:00,2021-10-22 12:14:20.515523+00:00
    ...
    • Oct 22 2021, 2:59 PM
    • 60 Lines
  • swh-scheduler-orchestrator_1 | wait-for-it: waiting for swh-scheduler:5008 without a timeout
    swh-scheduler-orchestrator_1 | wait-for-it: swh-scheduler:5008 is available after 0 seconds
    swh-scheduler-orchestrator_1 | Starting the swh-scheduler start-orchestrator
    swh-scheduler-orchestrator_1 | wait-for-it: waiting for amqp:5672 without a timeout
    swh-scheduler-orchestrator_1 | wait-for-it: amqp:5672 is available after 0 seconds
    ...
    • Oct 22 2021, 12:27 PM
    • 85 Lines
  • Looks like override no longer works for me:
    ```
    $ docker-compose --version
    docker-compose version 1.29.2, build unknown
    $ docker --version
    ...
    • Oct 21 2021, 4:16 PM
    • 75 Lines
  • @given(
    new_snapshot(min_size=4, max_size=4)
    )
    @pytest.mark.parametrize("browse_context", ["log"])
    def test_snapshot_log_with_id(client, browse_context, new_snapshot):
    ...
    • Oct 21 2021, 11:44 AM
    • 12 Lines
    • Python
  • swh_graph-replayer.11.gxas1odxle08@shsrvc2 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
    swh_graph-replayer.11.gxas1odxle08@shsrvc2 | ERROR:swh.journal.client.rdkafka:FAIL [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: 1 request(s) timed out: disconnect (after 598635ms in state UP, 1 identical error(s) suppressed)
    swh_graph-replayer.8.movkrxdxo4js@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    swh_graph-replayer.8.movkrxdxo4js@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    swh_graph-replayer.3.obpkqy8uyh3n@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    ...
    • Oct 19 2021, 11:43 AM
    • 14 Lines
  • swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | ### CONFIG FILE ###
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | storage:
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | cls: remote
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | args:
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | url: http://storage:5002/
    ...
    • Oct 18 2021, 2:47 PM
    • 55 Lines
  • {"topics":
    [
    {"topic": "__consumer_offsets"},
    {"topic": "swh.journal.indexed.content_ctags"},
    ...
    • Oct 14 2021, 4:08 PM
    • 28 Lines
    • JSON
  • $ make -C docs/ clean html
    make: Entering directory '/home/tony/work/inria/repo/swh/swh-environment/swh-docs/docs'
    sphinx-build -M clean "." "_build" --color
    Removing everything under '_build'...
    bin/ln-sphinx-subprojects --remove
    ...
    • Oct 13 2021, 5:21 PM
    • 178 Lines
  • started with timing option, will keep track of elapsed time
    loading transposed graph...
    transposed graph loaded (duration: PT1M38.291779679S).
    loading revision timestamps...
    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    ...
    • Oct 12 2021, 4:03 PM
    • 9 Lines
  • # to run in venv ve with "fil-profile run script.py"
    import os
    import subprocess
    ...
    • Oct 5 2021, 5:53 PM
    • 12 Lines
  • root@worker0:~# cat /etc/softwareheritage/loader_npm.yml
    ---
    storage:
    cls: pipeline
    steps:
    ...
    • Oct 5 2021, 12:30 PM
    • 22 Lines
  • ddouard  (e) swh   master … 12  ~/s/s/docker  cat celeryconfig.py
    CELERY_ACCEPT_CONTENT = [
    'application/x-msgpack'
    ]
    ddouard  (e) swh   master … 12  ~/s/s/docker  echo $CELERY_BROKER_URL
    ...
    • Oct 4 2021, 3:31 PM
    • 14 Lines
  • 61e140a70e718dc0010df38aedbbb1a2f35, refs/pull/9760/head: 839b0a7981b9dbc5110c5ca579e535f7e40ec707, refs/pull/9773/head: c5ee0d8a679a0f249487f5133b78ad22f6eade26, refs/pull/9778/head: f117f67f519eee8a6613734d8f3f30693c50483a, refs/pull/9786/head: fd49257d17f90ecf95258da825a828bac00ae384, refs/pull/9816/head: d5f9c0c5ded8494c040fa0c3cb99117000e77765, refs/pull/9829/head: 5440bb57aa5b7a44a327c006b2a6af93621831a0, refs/pull/9838/head: 149445797b27cb782e51be6c2b7ddb8c6260dfc4, refs/pull/9841/head: dfd245dbbc58d9eda67dd77f925539b8b09b66f8, refs/pull/9842/head: 9975f88532fb0ae9fd5b822f5456deb61e28c784, refs/pull/9868/head: c51ad1418e815cad44efc335d80a39cfce0a7980, refs/pull/987/head: d1f0ce1d57c22f9729d613bffcee0d85c1d606fe, refs/pull/9888/head: 4b3c742ba717292f45fd1ec2eabd232b6f80eb76, refs/pull/9909/head: ed861c41f78f536d31b80613b75bc78b5957d8ff, refs/pull/9966/head: 1183a88a4980161543d60b9cfd9b3af95ad17a0b, refs/pull/9972/head: 0338a59fc909fc31233254c5f9ea871ea02b372e, refs/pull/9990/head: d487aeb574d032de026c8a554c4106f41a2c68a7, refs/pull/9998/head: bfeaaf9bd4072dc47664835a21fcdfbd1756f959, refs/remotes/mozilla/try: 956f8edb78be56a014d3082e5e4db65f035590cf, refs/remotes/sfackler/dynamic_lib: d8a37160416cdd2cc01817688c77a580bcbb67fe, refs/reviewable/pr30095/r1: db61fc79afb0980df08575fc9bb513f7f4a8b7a2, refs/reviewable/pr30095/r2: 727fb6342be37074ee17af1057ebabbaafe79144, refs/reviewable/pr30095/r3: dc3b857c3145a819bf44f1261bb12f82f2fcd1f3, refs/reviewable/pr30095/r4: 3068d80a7cd478a0887f5cc424b7ef47f2bffa1e, refs/reviewable/pr30426/r1: 1daa2253287f5db2497c720c57ff6a824e0fb40f, refs/reviewable/pr30426/r2: 75a21310acb645e8fdb6aed50118e5a4a68802d1, refs/reviewable/pr30426/r3: ad2315df841160f4ecee576e5a9aa52055ebc8a6, refs/reviewable/pr30426/r4: 7718f24df1abc6f667d3730e47fbe787a41d93ff, refs/reviewable/pr30426/r5: 7cc26712b711cbc115e6f8147c4bdc969fe73f65, refs/reviewable/pr30426/r6: b651488aad44fb85fc6dd53f7c53e315ca816c20, refs/reviewable/pr30900/r1: d5c9577e3743f8c327a430f14bbeee9a306eba84, refs/pull/89494/head: e34fd546111c3cad82c91a9f466284d55a53e5fb, refs/pull/89495/head: 1c2ad79f9d789b0d449a1bc4dfe7d7ddc2ed4109, refs/pull/89499/head: 0e4a8f9fb9123cb3406509e618ac6184e2c3f936, refs/pull/89500/head: a28a78f247a0f02521e97eabf98e790085a4a753, refs/pull/89501/head: a1f7872c3ff98a91273fc5fbb5a900258e422a3c, refs/pull/89502/head: 199b33f0d77c3fae1c2c982029df7168899f8aba, refs/pull/89504/head: fdd8a0dde53e09b2f0ec869088fcb1d882031bd6, refs/pull/89505/head: 5b6604825e20ecd962c7021131de6a20cc73a32e, refs/pull/89506/head: a034af20f96fdf175fc7821e47458361a8102b75, refs/pull/89507/head: e41bb97c253bf99a33ee077578d876d3d6b94148, refs/pull/89508/head: bce86219830e386d1247364b6018ede2e85d3036, refs/pull/89509/head: 11140ff1a0dd3a395dd3ca1488bf580559f782f8, refs/pull/89510/head: 33178d2093f735606076b2741c8eadea87aecd05, refs/pull/89511/head: 55a3c8636da6266e22d0c077cd4a58d40a0613ed, refs/pull/89512/head: 5c5dde8f0a1e7a3f735dce63d6b81680d24c56d9
    Enumerating objects: 25767, done.
    Counting objects: 100% (12873/12873), done.
    Compressing objects: 100% (5543/5543), done.
    Total 25767 (delta 9966), reused 7330 (delta 7330), pack-reused 12894
    ...
    • Oct 4 2021, 10:41 AM
    • 7 Lines
  • kind: ScaledObject
    metadata:
    name: loaders-operators
    spec:
    scaleTargetRef:
    ...
    • Oct 4 2021, 9:19 AM
    • 39 Lines
    • YAML
  • Hello,
    The current loader git consumes a lot of memory depending on the size of
    the repository. It's fetching the full packfile of unknown
    references/refs (filtered by last snapshot's references), then parses
    ...
    • Oct 2 2021, 10:56 AM
    • 154 Lines
  • (Pdb++) [n for n in refs.keys() if not utils.ignore_branch_name(n)]
    [b'HEAD', b'refs/heads/master', b'refs/pull/10/head', b'refs/pull/100/head', b'refs/pull/102/head', b'refs/pull/103/head', b'refs/pull/105/head', b'refs/pull/106/head', b'refs/pull/107/head', b'refs/pull/108/head', b'refs/pull/109/head', b'refs/pull/11/head', b'refs/pull/110/head', b'refs/pull/111/head', b'refs/pull/112/head', b'refs/pull/113/head', b'refs/pull/114/head', b'refs/pull/115/head', b'refs/pull/118/head', b'refs/pull/119/head', b'refs/pull/12/head', b'refs/pull/120/head', b'refs/pull/121/head', b'refs/pull/122/head', b'refs/pull/123/head', b'refs/pull/124/head', b'refs/pull/125/head', b'refs/pull/126/head', b'refs/pull/127/head', b'refs/pull/128/head', b'refs/pull/129/head', b'refs/pull/13/head', b'refs/pull/130/head', b'refs/pull/131/head', b'refs/pull/133/head', b'refs/pull/134/head', b'refs/pull/135/head', b'refs/pull/136/head', b'refs/pull/137/head', b'refs/pull/138/head', b'refs/pull/139/head', b'refs/pull/14/head'...
    • Oct 1 2021, 4:51 PM
    • 3 Lines
  • storage:
    cls: local
    db: postgresql:///?service=swh-storage
    objstorage:
    cls: remote
    ...
    • Oct 1 2021, 12:02 PM
    • 14 Lines
    • YAML
  • * cozy/cozy-stack
    ** docker-dev with override
    ...
    • Sep 30 2021, 6:46 PM
    • 230 Lines
  • #!/bin/bash
    # SPDX-FileCopyrightText: © 2021 Stefano Zacchiroli <zack@upsilon.cc>
    # SPDX-License-Identifier: GPL-3.0-or-later
    ...
    • Sep 30 2021, 5:00 PM
    • 18 Lines
  • Setting up postgresql-14 (14.0-1.pgdg+1) ...
    Creating new PostgreSQL cluster 14/main ...
    /usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions
    The files belonging to this database system will be owned by user "postgres".
    This user must also own the server process.
    ...
    • Sep 30 2021, 4:25 PM
    • 22 Lines
  • (ve) swhworker@worker0:~$ time swh loader -C /etc/softwareheritage/loader_git.yml run git https://github.com/torvalds/l
    inux
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/torvalds/linux' with type 'git'
    Enumerating objects: 8345251, done.
    Total 8345251 (delta 0), reused 0 (delta 0), pack-reused 8345251
    ...
    • Sep 30 2021, 10:06 AM
    • 23 Lines
  • # in swh/web/tests/conftest/py
    @pytest.fixture(scope='session')
    def django_db_modify_db_settings(django_db_modify_db_settings_parallel_suffix, postgresql_proc):
    from django.conf import settings as django_settings
    django_settings.DATABASES['default'].update({
    ...
    • Sep 29 2021, 4:19 PM
    • 12 Lines
  • Sep 29 12:51:52 worker18 python3[1046]: [2021-09-29 12:51:52,181: INFO/ForkPoolWorker-7] Processing revisions [1-0] for {'swh-origin': 'https://svn.code.sf.net/p/penguinsdbtools2018/svn', 'remote_url': 'file:///tmp/tmpjtb68puq/swh.loader.s
    vn.n58ub6q5-1046/tmpf9j3ln2g', 'local_url': b'/tmp/swh.loader.svn.ntglfflw-1046/tmpf9j3ln2g', 'uuid': b'd8f5688d-efb6-4dc1-b967-41be2f097118'}
    Sep 29 12:51:52 worker18 python3[1046]: [2021-09-29 12:51:52,182: ERROR/ForkPoolWorker-7] ('No such revision 1', 160006)
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 457, in fetch_data
    ...
    • Sep 29 2021, 3:05 PM
    • 22 Lines
  • Sep 28 15:23:50 worker0 python3[1078211]: [2021-09-28 15:23:50,779: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:51 worker0 python3[1078211]: [2021-09-28 15:23:51,222: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:52 worker0 python3[1078211]: [2021-09-28 15:23:52,122: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:52 worker0 python3[1078211]: [2021-09-28 15:23:52,126: ERROR/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[9fd77287-dc5b-4f63-9f15-516438b8e42e] raised unexpected: RetryError('<Future at 0x7f0027780d68 state=
    finished raised RemoteException>')
    ...
    • Sep 28 2021, 5:25 PM
    • 37 Lines
  •  ddouard  (e) swh   master ✚ 2 … 11  ~/s/s/swh-storage  mypy swh
    swh/storage/replay.py:133: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[SkippedContent]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:134: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[Content]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:137: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[SkippedContent]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:152: error: Argument 1 to "metadata_authority_add" of "StorageInterface" has incompatible type "Set[Any]"; expected "List[MetadataAuthority]"
    ...
    • Sep 28 2021, 4:54 PM
    • 7 Lines
  • diff --git a/swh/web/api/views/origin_save.py b/swh/web/api/views/origin_save.py
    index a2d3366f..6cba7e4e 100644
    --- a/swh/web/api/views/origin_save.py
    +++ b/swh/web/api/views/origin_save.py
    @@ -11,6 +11,7 @@ from swh.web.common.origin_save import (
    ...
    • Sep 28 2021, 4:48 PM
    • 262 Lines
  • DEBUG:swh.storage.replay:Inserting 200 raw_extrinsic_metadata objects
    Traceback (most recent call last):
    File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.15.0', 'console_scripts', 'swh')()
    File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
    ...
    • Sep 28 2021, 4:46 PM
    • 51 Lines