Page MenuHomeSoftware Heritage
Create Paste
Paste Active Pastes

Active Pastes
Use Results
Edit Query

  • P1222 admin-rp1 diff on unattended upgrade configuration file
    root@rp1:/etc# diff -U3 /etc/apt/apt.conf.d/50unattended-upgrades.ucf-old /etc/apt/apt.conf.d/50unattended-upgrades
    --- /etc/apt/apt.conf.d/50unattended-upgrades.ucf-old 2021-11-23 18:02:23.908766790 +0000
    +++ /etc/apt/apt.conf.d/50unattended-upgrades 2021-11-24 10:34:33.004766790 +0000
    @@ -1,68 +1,164 @@
    -// This file is managed by Puppet. DO NOT EDIT.
    ...
    • Nov 24 2021, 11:38 AM
    • 205 Lines
    Author: ardumont
  • P1221 some more vagrant up sadness
    $ vagrant up admin-bardo
    Bringing machine 'admin-bardo' up with 'libvirt' provider...
    ==> admin-bardo: Creating image (snapshot of base box volume).
    ==> admin-bardo: Creating domain with the following settings...
    ==> admin-bardo: -- Name: puppet-environment_admin-bardo
    ...
    • Nov 23 2021, 4:07 PM
    • 691 Lines
    Author: ardumont
  • P1220 vagrant up sadness
    vagrant up admin-bardo
    Bringing machine 'admin-bardo' up with 'libvirt' provider...
    ==> admin-bardo: Creating image (snapshot of base box volume).
    ==> admin-bardo: Creating domain with the following settings...
    ==> admin-bardo: -- Name: puppet-environment_admin-bardo
    ...
    • Nov 22 2021, 1:34 PM
    • 117 Lines
    Author: ardumont
  • P1219 SvnLoaderFromRemoteDump optimization
    (swh) ✔ ~/swh/swh-environment/swh-loader-svn [master|✚ 2⚑ 10]
    16:43 $ git diff
    diff --git a/swh/loader/svn/loader.py b/swh/loader/svn/loader.py
    index eac96a3..698babe 100644
    --- a/swh/loader/svn/loader.py
    ...
    • Nov 18 2021, 4:54 PM
    • 60 Lines
    Author: anlambert
  • P1218 Command-Line Input
    diff --git a/swh/scheduler/celery_backend/recurrent_visits.py b/swh/scheduler/celery_backend/recurrent_visits.py
    index 8f43d8e..eacf8de 100644
    --- a/swh/scheduler/celery_backend/recurrent_visits.py
    +++ b/swh/scheduler/celery_backend/recurrent_visits.py
    @@ -23,6 +23,7 @@ from typing import TYPE_CHECKING, Any, Dict, List, Tuple
    ...
    • Nov 16 2021, 2:16 PM
    • 93 Lines
    Author: olasd
  • P1217 Command-Line Input
    diff --git a/swh/journal/client.py b/swh/journal/client.py
    index ff6f0ef..344f629 100644
    --- a/swh/journal/client.py
    +++ b/swh/journal/client.py
    @@ -297,13 +297,13 @@ class JournalClient:
    ...
    • Nov 16 2021, 12:40 PM
    • 19 Lines
    Author: olasd
  • P1216 (An Untitled Masterwork)
    16:01:14 swh@db1:5432=> select * from origin_visit_status ovs inner join origin o on ovs.origin=o.id where date > '2021-11-10 15:00' and type='npm' limit 10;
    +---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+---------+-----------------------------------------------------------+
    | origin | visit | date | status | metadata | snapshot | type | id | url |
    +---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+---------+-----------------------------------------------------------+
    | 1796362 | 1 | 2021-11-10 15:00:22.221484+00 | full | (null) | \x034b93485d1fc0a0efc9dd3447b1f7a92e278041 | npm | 1796362 | https://www.npmjs.com/package/angular-phonecat-nexen |
    ...
    • Nov 10 2021, 4:04 PM
    • 18 Lines
    Author: ardumont
  • P1215 temporary/working directories mess with the loader svn somewhere
    root@worker18:/home/ardumont# tree -L 3 /tmp/tmp0tm0182g/
    /tmp/tmp0tm0182g/
    ├── swh.loader.svn.2zdkkl_5-1740204
    │   └── tmpi0q7t29s
    │   ├── conf
    ...
    • Nov 10 2021, 9:42 AM
    • 20 Lines
    Author: ardumont
  • P1214 (An Untitled Masterwork)
    swhworker@worker17:~$ /usr/bin/time -v swh loader run svn https://profs.scienze.univr.it/posenato/svn/sw/CSTNU
    INFO:swh.loader.svn.SvnLoader:Load origin 'https://profs.scienze.univr.it/posenato/svn/sw/CSTNU' with type 'svn'
    INFO:swh.loader.svn.SvnLoader:Processing revisions [297-619] for {'swh-origin': 'https://profs.scienze.univr.it/posenato/svn/sw/CSTNU', 'remote_url': 'file:///tmp/tmpsd2aj26l/swh.loader.svn.xkj39oxf-930164/tmpphfs60dp', 'local_url': b'/tmp/swh.loader.svn.kqus7k3a-930164/tmpphfs60dp', 'uuid': b'782453a1-1937-45d1-8845-2a6fcc2839b7'}
    {'status': 'eventful'}
    Command being timed: "swh loader run svn https://profs.scienze.univr.it/posenato/svn/sw/CSTNU"
    ...
    • Nov 8 2021, 6:57 PM
    • 27 Lines
    Author: ardumont
  • P1213 svn loader link handling bug
    swh-loader_1 | [2021-10-29 21:21:17,320: INFO/ForkPoolWorker-1] Processing revisions [1-1089] for {'swh-origin': 'http://svn.code.sf.net/p/blinkensisters/code', 'remote_url': 'file:///tmp/tmpi_mjbfaq/swh.loader.svn.zyez62lm-83/tmpcsww9pus', 'local_url': b'/tmp/swh.loader.svn.tx68am7w-83/tmpcsww9pus', 'uuid': b'943a9807-df12-0410-b0c0-cd0da6ec65e4'}
    swh-loader_1 | [2021-10-29 21:21:17,325: DEBUG/ForkPoolWorker-1] rev: 1, swhrev: d0fde2b7bc24f3b6d6316a079e310905f2b6179e, dir: 7a2faa45330bbfdb9d7ffca331d57073fb8ba61f
    swh-loader_1 | [2021-10-29 21:21:17,443: DEBUG/ForkPoolWorker-1] rev: 2, swhrev: 56eccaa1166c813a33b90b2b90db1658b1fd3b6e, dir: d4b04aa0c1fc8da2e2092cdb9956a0db9d18e86c
    swh-loader_1 | [2021-10-29 21:21:17,449: DEBUG/ForkPoolWorker-1] rev: 3, swhrev: 695e144d7820bd28d6b796e5930b7a3998e4b464, dir: 6744db600724f5f95eef06a7926c9e33a1f3d620
    swh-loader_1 | [2021-10-29 21:21:17,702: DEBUG/ForkPoolWorker-1] rev: 4, swhrev: 39d2270022f5ffec7fff9f0e11700776ff827075, dir: 150465caf99e23ed5e505f874e1d2249f6036dc8
    ...
    • Nov 2 2021, 1:48 PM
    • 253 Lines
    Author: anlambert
  • P1212 (An Untitled Masterwork)
    swh-loader_1 | [2021-10-28 15:14:21,272: DEBUG/ForkPoolWorker-1] Mounting dump file with "svnadmin load".
    swh-loader_1 | [2021-10-28 15:14:26,735: INFO/ForkPoolWorker-1] Processing revisions [1-253] for {'swh-origin': 'http://svn.code.sf.net/p/proxytunnel/code/', 'remote_url': 'file:///tmp/tmp8oo46bm8/swh.loader.svn.0znrosz2-84/tmp1wjhoail', 'local_url': b'/tmp/swh.loader.svn.wpdwou1s-84/tmp1wjhoail', 'uuid': b'bc163920-b10d-0410-b2c5-a5491ca2ceef'}
    swh-loader_1 | [2021-10-28 15:14:26,739: DEBUG/ForkPoolWorker-1] rev: 1, swhrev: 93a685b99d596bf5bbd20cbd14752ff55fb6da7c, dir: 75ed58f260bfa4102d0e09657803511f5f0ab372
    swh-loader_1 | [2021-10-28 15:14:26,739: DEBUG/ForkPoolWorker-1] Checking hash computations on revision 1...
    swh-loader_1 | [2021-10-28 15:14:26,754: DEBUG/ForkPoolWorker-1] rev: 2, swhrev: eb093a0018af3b7e114023347b68217123efa326, dir: 99b8ce06292b877fd89143183217962e38943ca8
    ...
    • Oct 28 2021, 5:16 PM
    • 376 Lines
    Author: anlambert
  • P1211 swh-doco logs -f swh-scheduler-schedule-recurrent
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': ['swh.loader.svn.tasks', 'swh.loader.mercurial.tasks', 'swh.loader.git.tasks', 'swh.loader.package.archive.tasks', 'swh.loader.package.cran.tasks', 'swh.loader.package.debian.tasks', 'swh.loader.package.deposit.tasks', 'swh.loader.package.nixguix.tasks', 'swh.loader.package.npm.tasks', 'swh.loader.package.opam.tasks', 'swh.loader.package.pypi.tasks', 'swh.lister.bitbucket.tasks', 'swh.lister.cgit.tasks', 'swh.lister.cran.tasks', 'swh.lister.debian.tasks', 'swh.lister.gitea.tasks', 'swh.lister.github.tasks', 'swh.lister.gitlab.tasks', 'swh.lister.gnu.tasks', 'swh.lister.launchpad.tasks', 'swh.lister.npm.tasks', 'swh.lister.opam.tasks', 'swh.lister.packagist.tasks', 'swh.lister.phabricator.tasks', 'swh.lister.pypi.tasks', 'swh.lister.sourceforge.tasks', 'swh.lister.tuleap.tasks', 'swh.deposit.loader.tasks']}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    swh-scheduler-schedule-recurrent_1 | DEBUG:swh.scheduler.celery_backend.config:Creating a Celery app with {'task_soft_time_limit': 0, 'task_broker': 'amqp://guest:guest@amqp/%2f', 'task_queues': [<unbound Queue celery -> <unbound Exchange celery(direct)> -> celery>], 'broker_transport_options': {'max_retries': 1}, 'task_modules': []}
    ...
    • Oct 27 2021, 3:42 PM
    • 108 Lines
    Author: ardumont
  • P1210 /usr/bin/time -v swh-doco exec swh-loader swh loader run git https://github.com/scikit-learn/scikit-learn
    /usr/bin/time -v swh-doco exec swh-loader swh loader run git https://github.com/scikit-learn/scikit-learn
    + cd /home/tony/work/inria/repo/swh/swh-environment/docker
    + docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader swh loader run git https://github.com/scikit-learn/scikit-learn
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/scikit-learn/scikit-learn' with type 'git'
    Enumerating objects: 206538, done.
    ...
    • Oct 26 2021, 4:16 PM
    • 47 Lines
    Author: ardumont
  • P1209 ingestion ok
    /usr/bin/time -v swh-doco exec swh-loader swh loader run git https://git.code.sf.net/u/bsomervi/hamlib.git
    + cd /home/tony/work/inria/repo/swh/swh-environment/docker
    + docker-compose -f docker-compose.yml -f docker-compose.override.yml exec swh-loader swh loader run git https://git.code.sf.net/u/bsomervi/hamlib.git
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://git.code.sf.net/u/bsomervi/hamlib.git' with type 'git'
    Enumerating objects: 52811, done.
    ...
    • Oct 26 2021, 10:32 AM
    • 33 Lines
    Author: ardumont
  • P1208 sampled origins: Patch (drop PR branches) or no patch (current version), failure to ingest with huge packfile
    #+title: About dropping ingestion of PR-like branches in the loader git
    * synthesis
    Code got adapted so the loader git actually retrieves the full packfile (D6550). Then
    ...
    • Oct 25 2021, 9:46 PM
    • 523 Lines
    Author: ardumont
  • P1207 (An Untitled Masterwork)
    (swh) ✔ ~/swh/swh-environment/docker [master|✚ 2⚑ 35]
    14:21 $ doco exec swh-scheduler-runner swh scheduler origin grab-next tar 10
    lister_id,url,visit_type,extra_loader_arguments,last_update,enabled,first_seen,last_seen
    efd50b96-ecb1-47db-bfdf-56f2247769c3,https://cran.r-project.org/package=aaSEA,tar,"{'artifacts': [{'url': 'https://cran.r-project.org/src/contrib/aaSEA_1.1.0.tar.gz', 'version': '1.1.0'}]}",2019-11-09 15:35:46+00:00,True,2021-10-22 12:14:20.515523+00:00,2021-10-22 12:14:20.515523+00:00
    efd50b96-ecb1-47db-bfdf-56f2247769c3,https://cran.r-project.org/package=AATtools,tar,"{'artifacts': [{'url': 'https://cran.r-project.org/src/contrib/AATtools_0.0.1.tar.gz', 'version': '0.0.1'}]}",2020-06-09 16:49:30+00:00,True,2021-10-22 12:14:20.515523+00:00,2021-10-22 12:14:20.515523+00:00
    ...
    • Oct 22 2021, 2:59 PM
    • 60 Lines
    Author: anlambert
  • P1206 swh-scheduler-orchestrator is running
    swh-scheduler-orchestrator_1 | wait-for-it: waiting for swh-scheduler:5008 without a timeout
    swh-scheduler-orchestrator_1 | wait-for-it: swh-scheduler:5008 is available after 0 seconds
    swh-scheduler-orchestrator_1 | Starting the swh-scheduler start-orchestrator
    swh-scheduler-orchestrator_1 | wait-for-it: waiting for amqp:5672 without a timeout
    swh-scheduler-orchestrator_1 | wait-for-it: amqp:5672 is available after 0 seconds
    ...
    • Oct 22 2021, 12:27 PM
    • 85 Lines
    Author: ardumont
  • P1205 docker-compose no longer allows override on containers
    Looks like override no longer works for me:
    ```
    $ docker-compose --version
    docker-compose version 1.29.2, build unknown
    $ docker --version
    ...
    • Oct 21 2021, 4:16 PM
    • 75 Lines
    Author: ardumont
  • P1204 snapshot test
    @given(
    new_snapshot(min_size=4, max_size=4)
    )
    @pytest.mark.parametrize("browse_context", ["log"])
    def test_snapshot_log_with_id(client, browse_context, new_snapshot):
    ...
    • Oct 21 2021, 11:44 AM
    • 12 Lines
    • Python
    Author: jayeshv
  • P1203 (An Untitled Masterwork)
    swh_graph-replayer.11.gxas1odxle08@shsrvc2 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
    swh_graph-replayer.11.gxas1odxle08@shsrvc2 | ERROR:swh.journal.client.rdkafka:FAIL [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: 1 request(s) timed out: disconnect (after 598635ms in state UP, 1 identical error(s) suppressed)
    swh_graph-replayer.8.movkrxdxo4js@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    swh_graph-replayer.8.movkrxdxo4js@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    swh_graph-replayer.3.obpkqy8uyh3n@shsrvc3 | WARNING:swh.journal.client.rdkafka:REQTMOUT [rdkafka#consumer-1] [thrd:sasl_ssl://broker1.journal.staging.swh.network:9093/bootstrap]: sasl_ssl://broker1.journal.staging.swh.network:9093/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 0 partially-sent requests
    ...
    • Oct 19 2021, 11:43 AM
    • 14 Lines
    Author: douardda
  • P1202 (An Untitled Masterwork)
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | ### CONFIG FILE ###
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | storage:
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | cls: remote
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | args:
    swh_graph-replayer.1.wjhrw9wnh5vw@shsrvc2 | url: http://storage:5002/
    ...
    • Oct 18 2021, 2:47 PM
    • 55 Lines
    Author: douardda
  • P1201 staging: topics to move to storage1
    {"topics":
    [
    {"topic": "__consumer_offsets"},
    {"topic": "swh.journal.indexed.content_ctags"},
    ...
    • Oct 14 2021, 4:08 PM
    • 28 Lines
    • JSON
    Author: vsellier
  • P1200 `make -C docs clean html` failure to build the devel docs
    $ make -C docs/ clean html
    make: Entering directory '/home/tony/work/inria/repo/swh/swh-environment/swh-docs/docs'
    sphinx-build -M clean "." "_build" --color
    Removing everything under '_build'...
    bin/ln-sphinx-subprojects --remove
    ...
    • Oct 13 2021, 5:21 PM
    • 178 Lines
    Author: ardumont
  • P1199 Command-Line Input
    started with timing option, will keep track of elapsed time
    loading transposed graph...
    transposed graph loaded (duration: PT1M38.291779679S).
    loading revision timestamps...
    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    ...
    • Oct 12 2021, 4:03 PM
    • 9 Lines
    Author: zack
  • P1197 fil-profile run script.py
    # to run in venv ve with "fil-profile run script.py"
    import os
    import subprocess
    ...
    • Oct 5 2021, 5:53 PM
    • 12 Lines
    Author: ardumont
  • P1196 (An Untitled Masterwork)
    root@worker0:~# cat /etc/softwareheritage/loader_npm.yml
    ---
    storage:
    cls: pipeline
    steps:
    ...
    • Oct 5 2021, 12:30 PM
    • 22 Lines
    Author: ardumont
  • P1195 (An Untitled Masterwork)
    ddouard  (e) swh   master … 12  ~/s/s/docker  cat celeryconfig.py
    CELERY_ACCEPT_CONTENT = [
    'application/x-msgpack'
    ]
    ddouard  (e) swh   master … 12  ~/s/s/docker  echo $CELERY_BROKER_URL
    ...
    • Oct 4 2021, 3:31 PM
    • 14 Lines
    Author: douardda
  • P1194 unknown reference in git loader with partial snapshot run warning becomes tedious...
    61e140a70e718dc0010df38aedbbb1a2f35, refs/pull/9760/head: 839b0a7981b9dbc5110c5ca579e535f7e40ec707, refs/pull/9773/head: c5ee0d8a679a0f249487f5133b78ad22f6eade26, refs/pull/9778/head: f117f67f519eee8a6613734d8f3f30693c50483a, refs/pull/9786/head: fd49257d17f90ecf95258da825a828bac00ae384, refs/pull/9816/head: d5f9c0c5ded8494c040fa0c3cb99117000e77765, refs/pull/9829/head: 5440bb57aa5b7a44a327c006b2a6af93621831a0, refs/pull/9838/head: 149445797b27cb782e51be6c2b7ddb8c6260dfc4, refs/pull/9841/head: dfd245dbbc58d9eda67dd77f925539b8b09b66f8, refs/pull/9842/head: 9975f88532fb0ae9fd5b822f5456deb61e28c784, refs/pull/9868/head: c51ad1418e815cad44efc335d80a39cfce0a7980, refs/pull/987/head: d1f0ce1d57c22f9729d613bffcee0d85c1d606fe, refs/pull/9888/head: 4b3c742ba717292f45fd1ec2eabd232b6f80eb76, refs/pull/9909/head: ed861c41f78f536d31b80613b75bc78b5957d8ff, refs/pull/9966/head: 1183a88a4980161543d60b9cfd9b3af95ad17a0b, refs/pull/9972/head: 0338a59fc909fc31233254c5f9ea871ea02b372e, refs/pull/9990/head: d487aeb574d032de026c8a554c4106f41a2c68a7, refs/pull/9998/head: bfeaaf9bd4072dc47664835a21fcdfbd1756f959, refs/remotes/mozilla/try: 956f8edb78be56a014d3082e5e4db65f035590cf, refs/remotes/sfackler/dynamic_lib: d8a37160416cdd2cc01817688c77a580bcbb67fe, refs/reviewable/pr30095/r1: db61fc79afb0980df08575fc9bb513f7f4a8b7a2, refs/reviewable/pr30095/r2: 727fb6342be37074ee17af1057ebabbaafe79144, refs/reviewable/pr30095/r3: dc3b857c3145a819bf44f1261bb12f82f2fcd1f3, refs/reviewable/pr30095/r4: 3068d80a7cd478a0887f5cc424b7ef47f2bffa1e, refs/reviewable/pr30426/r1: 1daa2253287f5db2497c720c57ff6a824e0fb40f, refs/reviewable/pr30426/r2: 75a21310acb645e8fdb6aed50118e5a4a68802d1, refs/reviewable/pr30426/r3: ad2315df841160f4ecee576e5a9aa52055ebc8a6, refs/reviewable/pr30426/r4: 7718f24df1abc6f667d3730e47fbe787a41d93ff, refs/reviewable/pr30426/r5: 7cc26712b711cbc115e6f8147c4bdc969fe73f65, refs/reviewable/pr30426/r6: b651488aad44fb85fc6dd53f7c53e315ca816c20, refs/reviewable/pr30900/r1: d5c9577e3743f8c327a430f14bbeee9a306eba84, refs/pull/89494/head: e34fd546111c3cad82c91a9f466284d55a53e5fb, refs/pull/89495/head: 1c2ad79f9d789b0d449a1bc4dfe7d7ddc2ed4109, refs/pull/89499/head: 0e4a8f9fb9123cb3406509e618ac6184e2c3f936, refs/pull/89500/head: a28a78f247a0f02521e97eabf98e790085a4a753, refs/pull/89501/head: a1f7872c3ff98a91273fc5fbb5a900258e422a3c, refs/pull/89502/head: 199b33f0d77c3fae1c2c982029df7168899f8aba, refs/pull/89504/head: fdd8a0dde53e09b2f0ec869088fcb1d882031bd6, refs/pull/89505/head: 5b6604825e20ecd962c7021131de6a20cc73a32e, refs/pull/89506/head: a034af20f96fdf175fc7821e47458361a8102b75, refs/pull/89507/head: e41bb97c253bf99a33ee077578d876d3d6b94148, refs/pull/89508/head: bce86219830e386d1247364b6018ede2e85d3036, refs/pull/89509/head: 11140ff1a0dd3a395dd3ca1488bf580559f782f8, refs/pull/89510/head: 33178d2093f735606076b2741c8eadea87aecd05, refs/pull/89511/head: 55a3c8636da6266e22d0c077cd4a58d40a0613ed, refs/pull/89512/head: 5c5dde8f0a1e7a3f735dce63d6b81680d24c56d9
    Enumerating objects: 25767, done.
    Counting objects: 100% (12873/12873), done.
    Compressing objects: 100% (5543/5543), done.
    Total 25767 (delta 9966), reused 7330 (delta 7330), pack-reused 12894
    ...
    • Oct 4 2021, 10:41 AM
    • 7 Lines
    Author: ardumont
  • P1193 keda configuration for docker environment
    kind: ScaledObject
    metadata:
    name: loaders-operators
    spec:
    scaleTargetRef:
    ...
    • Oct 4 2021, 9:19 AM
    • 39 Lines
    • YAML
    Author: vsellier
  • P1192 [draft-ml-devel] About reducing the loader-git memory consumption and overall work
    Hello,
    The current loader git consumes a lot of memory depending on the size of
    the repository. It's fetching the full packfile of unknown
    references/refs (filtered by last snapshot's references), then parses
    ...
    • Oct 2 2021, 10:56 AM
    • 154 Lines
    Author: ardumont
  • P1191 (An Untitled Masterwork)
    (Pdb++) [n for n in refs.keys() if not utils.ignore_branch_name(n)]
    [b'HEAD', b'refs/heads/master', b'refs/pull/10/head', b'refs/pull/100/head', b'refs/pull/102/head', b'refs/pull/103/head', b'refs/pull/105/head', b'refs/pull/106/head', b'refs/pull/107/head', b'refs/pull/108/head', b'refs/pull/109/head', b'refs/pull/11/head', b'refs/pull/110/head', b'refs/pull/111/head', b'refs/pull/112/head', b'refs/pull/113/head', b'refs/pull/114/head', b'refs/pull/115/head', b'refs/pull/118/head', b'refs/pull/119/head', b'refs/pull/12/head', b'refs/pull/120/head', b'refs/pull/121/head', b'refs/pull/122/head', b'refs/pull/123/head', b'refs/pull/124/head', b'refs/pull/125/head', b'refs/pull/126/head', b'refs/pull/127/head', b'refs/pull/128/head', b'refs/pull/129/head', b'refs/pull/13/head', b'refs/pull/130/head', b'refs/pull/131/head', b'refs/pull/133/head', b'refs/pull/134/head', b'refs/pull/135/head', b'refs/pull/136/head', b'refs/pull/137/head', b'refs/pull/138/head', b'refs/pull/139/head', b'refs/pull/14/head'...
    • Oct 1 2021, 4:51 PM
    • 3 Lines
    Author: ardumont
  • P1190 (An Untitled Masterwork)
    storage:
    cls: local
    db: postgresql:///?service=swh-storage
    objstorage:
    cls: remote
    ...
    • Oct 1 2021, 12:02 PM
    • 14 Lines
    • YAML
    Author: vlorentz
  • P1189 draft: ryout runs on loader git improvments
    * cozy/cozy-stack
    ** docker-dev with override
    ...
    • Sep 30 2021, 6:46 PM
    • 230 Lines
    Author: ardumont
  • P1187 import-ics
    #!/bin/bash
    # SPDX-FileCopyrightText: © 2021 Stefano Zacchiroli <zack@upsilon.cc>
    # SPDX-License-Identifier: GPL-3.0-or-later
    ...
    • Sep 30 2021, 5:00 PM
    • 18 Lines
    Author: zack
  • P1186 Command-Line Input
    Setting up postgresql-14 (14.0-1.pgdg+1) ...
    Creating new PostgreSQL cluster 14/main ...
    /usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions
    The files belonging to this database system will be owned by user "postgres".
    This user must also own the server process.
    ...
    • Sep 30 2021, 4:25 PM
    • 22 Lines
    Author: zack
  • P1185 computers say no to D6377
    (ve) swhworker@worker0:~$ time swh loader -C /etc/softwareheritage/loader_git.yml run git https://github.com/torvalds/l
    inux
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/torvalds/linux' with type 'git'
    Enumerating objects: 8345251, done.
    Total 8345251 (delta 0), reused 0 (delta 0), pack-reused 8345251
    ...
    • Sep 30 2021, 10:06 AM
    • 23 Lines
    Author: ardumont
  • P1184 (An Untitled Masterwork)
    # in swh/web/tests/conftest/py
    @pytest.fixture(scope='session')
    def django_db_modify_db_settings(django_db_modify_db_settings_parallel_suffix, postgresql_proc):
    from django.conf import settings as django_settings
    django_settings.DATABASES['default'].update({
    ...
    • Sep 29 2021, 4:19 PM
    • 12 Lines
    Author: douardda
  • P1183 other recurring svn loader issue
    Sep 29 12:51:52 worker18 python3[1046]: [2021-09-29 12:51:52,181: INFO/ForkPoolWorker-7] Processing revisions [1-0] for {'swh-origin': 'https://svn.code.sf.net/p/penguinsdbtools2018/svn', 'remote_url': 'file:///tmp/tmpjtb68puq/swh.loader.s
    vn.n58ub6q5-1046/tmpf9j3ln2g', 'local_url': b'/tmp/swh.loader.svn.ntglfflw-1046/tmpf9j3ln2g', 'uuid': b'd8f5688d-efb6-4dc1-b967-41be2f097118'}
    Sep 29 12:51:52 worker18 python3[1046]: [2021-09-29 12:51:52,182: ERROR/ForkPoolWorker-7] ('No such revision 1', 160006)
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 457, in fetch_data
    ...
    • Sep 29 2021, 3:05 PM
    • 22 Lines
    Author: ardumont
  • P1182 (An Untitled Masterwork)
    Sep 28 15:23:50 worker0 python3[1078211]: [2021-09-28 15:23:50,779: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:51 worker0 python3[1078211]: [2021-09-28 15:23:51,222: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:52 worker0 python3[1078211]: [2021-09-28 15:23:52,122: WARNING/ForkPoolWorker-1] Retrying RPC call
    Sep 28 15:23:52 worker0 python3[1078211]: [2021-09-28 15:23:52,126: ERROR/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[9fd77287-dc5b-4f63-9f15-516438b8e42e] raised unexpected: RetryError('<Future at 0x7f0027780d68 state=
    finished raised RemoteException>')
    ...
    • Sep 28 2021, 5:25 PM
    • 37 Lines
    Author: ardumont
  • P1181 (An Untitled Masterwork)
     ddouard  (e) swh   master ✚ 2 … 11  ~/s/s/swh-storage  mypy swh
    swh/storage/replay.py:133: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[SkippedContent]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:134: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[Content]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:137: error: Argument 1 to "collision_aware_content_add" has incompatible type "Callable[[List[SkippedContent]], Dict[str, int]]"; expected "Callable[[Iterable[Any]], None]"
    swh/storage/replay.py:152: error: Argument 1 to "metadata_authority_add" of "StorageInterface" has incompatible type "Set[Any]"; expected "List[MetadataAuthority]"
    ...
    • Sep 28 2021, 4:54 PM
    • 7 Lines
    Author: douardda
  • P1180 (An Untitled Masterwork)
    diff --git a/swh/web/api/views/origin_save.py b/swh/web/api/views/origin_save.py
    index a2d3366f..6cba7e4e 100644
    --- a/swh/web/api/views/origin_save.py
    +++ b/swh/web/api/views/origin_save.py
    @@ -11,6 +11,7 @@ from swh.web.common.origin_save import (
    ...
    • Sep 28 2021, 4:48 PM
    • 262 Lines
    Author: anlambert
  • P1179 (An Untitled Masterwork)
    DEBUG:swh.storage.replay:Inserting 200 raw_extrinsic_metadata objects
    Traceback (most recent call last):
    File "/usr/bin/swh", line 11, in <module>
    load_entry_point('swh.core==0.15.0', 'console_scripts', 'swh')()
    File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
    ...
    • Sep 28 2021, 4:46 PM
    • 51 Lines
    Author: douardda
  • P1178 swh-web config
    storage:
    cls: remote
    # url: http://localhost:5002/
    # url: http://storage0.internal.staging.swh.network:5002
    url: http://moma.internal.softwareheritage.org:5002/
    ...
    • Sep 28 2021, 2:07 PM
    • 96 Lines
    • YAML
    Author: anlambert
  • P1177 Command-Line Input
    egrep -v '^".*",".*",".*"$' license-blobs.csv
    "swh:1:cnt:03e1933241b8c3878d81c0184d7f2fd3d8cd6185","037d40bc6bcb42dfd740be545dbdf2df3405442f","LICENSE
    "
    "swh:1:cnt:65a5c662900ee946583147129720563fd4ba286d","40e9258799f752fe25d7518155c615c1c497b7ac","LICENSE.md
    "
    ...
    • Sep 27 2021, 4:50 PM
    • 11 Lines
    Author: zack
  • P1176 Patching dulwich to decrease memory footprint
    #+title: Patching Dulwich to decrease memory footprint
    #+author: ardumont
    In the following analysis, we will executes multiple ingestions with and without the
    patched [1] dulwich version.
    ...
    • Sep 27 2021, 10:01 AM
    • 567 Lines
    Author: ardumont
  • P1175 version ordering in opam is a bit specific
    Case with some edge cases:
    ```
    $ opam show --color never --normalise --root $PWD -f all-versions opam-state
    2.0~alpha5 2.0.0~beta 2.0.0~beta3 2.0.0~beta3.1 2.0.0~beta5 2.0.0~rc 2.0.0~rc2 2.0.0~rc3 2.0.0 2.0.1 2.0.2 2.0.3 2.0.4 2.0.5 2.0.6 2.0.7 2.0.8 2.0.9 2.1.0~beta2 2.1.0~beta4 2.1.0~rc 2.1.0~rc2 2.1.0
    ```
    ...
    • Sep 24 2021, 4:14 PM
    • 78 Lines
    Author: ardumont
  • P1174 (An Untitled Masterwork)
    visit_type=hg; sleep=300; while true; do for policy in never_visited_oldest_update_first already_visited_order_by_lag ; do echo "$(date) scheduling $visit_type origins with policy ${policy}"; SWH_C
    ONFIG_FILENAME=/etc/softwareheritage/scheduler/listener-runner.yml swh scheduler -C /etc/softwareheritage/scheduler/listener-runner.yml origin send-to-celery --policy $policy $visit_type; echo "$(date) sleep
    $sleep" ; sleep $sleep; done; done
    • Sep 23 2021, 11:11 AM
    • 3 Lines
    Author: ardumont
  • P1173 (An Untitled Masterwork)
     ddouard  (e) swh   master … 2  ~/s/s/swh-web  cat swh/web/django_db_backend/base.py
    from django.db.backends.postgresql.base import * # NOQA
    from django.db.backends.postgresql.base import DatabaseWrapper as _DatabaseWrapper
    # dirty hack to allow using a postgresql:// libpq connection URI as db name...
    ...
    • Sep 22 2021, 6:15 PM
    • 12 Lines
    Author: douardda
  • P1172 a bit of a scheduling predicament!?
    - First run of the foss heptapod origins scheduled
    - They failed due to a dangling configuration key from the old mercurial loader (fixed)
    -> all origins have failed
    -> they have now their entry in the origin-visit-stats table
    -> scheduler metrics are updated
    ...
    • Sep 22 2021, 11:37 AM
    • 413 Lines
    Author: ardumont
  • P1171 opam can deal with multi-instances indeed
    $ pwd
    /home/tony/scratch/opam/test-opam-root
    $ opam init --reinit --bare --no-setup --root $PWD opam.ocaml.org https://opam.ocaml.org
    $ opam repository add --root $PWD satysfi-external https://github.com/gfngfn/satysfi-external-repo.git
    $ opam repository add --root $PWD opam-windows-repository https://github.com/vouillon/opam-windows-repository.git
    ...
    • Sep 21 2021, 1:54 PM
    • 56 Lines
    Author: ardumont
  • P1170 opam root taking around 214Mib
    tony  yavin4  ~  %  opam init --reinit --bare --no-setup --root ~/opamtest instance-foo https://opam.ocaml.org
    [NOTE] Will configure from built-in defaults.
    Checking for available remotes: rsync and local, git, mercurial, darcs. Perfect!
    <><> Fetching repository information ><><><><><><><><><><><><><><><><><><><><><>
    ...
    • Sep 20 2021, 12:30 PM
    • 8 Lines
    Author: ardumont
  • P1169 Large repository rounds 1st is long, 2nd is failing faster (because the error is blocking)
    ```
    swhworker@worker17:~$ time SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_oneshot.yml swh loader run mercurial_from_disk https://foss.heptapod.net/heptapod/heptapod
    INFO:swh.loader.mercurial.LoaderFromDisk:Load origin 'https://foss.heptapod.net/heptapod/heptapod' with type 'hg'
    applying clone bundle from https://cellar-c2.services.clever-cloud.com/heptapod-foss-clonebundles/heptapod/heptapod-2020-07-22-zstd-v2.hg
    adding changesets
    ...
    • Sep 17 2021, 11:13 AM
    • 60 Lines
    Author: ardumont
  • P1168 (An Untitled Masterwork)
    diff --git a/swh/lister/gitlab/lister.py b/swh/lister/gitlab/lister.py
    index 5937256..074906d 100644
    --- a/swh/lister/gitlab/lister.py
    +++ b/swh/lister/gitlab/lister.py
    @@ -203,10 +203,11 @@ class GitLabLister(Lister[GitLabListerState, PageResult]):
    ...
    • Sep 16 2021, 5:47 PM
    • 29 Lines
    Author: anlambert
  • P1167 small perf comparison on mercurial loader diffs
    - worker0.staging is running within a venv with D6240 (more extid filtering) and D6268 (build snapshot)
    - worker17 is running with the current latest mercurial packaged (no optim)
    But the filtering is still happening in the mercurial loader (without D6275)
    # tl; dr
    ...
    • Sep 16 2021, 4:04 PM
    • 136 Lines
    Author: ardumont
  • P1166 (An Untitled Masterwork)
    vagrant up test
    Bringing machine 'test' up with 'libvirt' provider...
    ==> test: Creating image (snapshot of base box volume).
    ==> test: Creating domain with the following settings...
    ==> test: -- Name: puppet-environment_test
    ...
    • Sep 15 2021, 5:26 PM
    • 62 Lines
    Author: ardumont
  • P1165 cassandra extid allow filtering
    cqlsh:swh> DESCRIBE extid
    CREATE TABLE swh.extid (
    extid_type ascii,
    extid blob,
    ...
    • Sep 15 2021, 2:23 PM
    • 28 Lines
    Author: vlorentz
  • P1164 flush cassandra buffers
    seq 2 8 | parallel -t ssh root@parasilo-{} nodetool flush
    seq 2 8 | parallel -t ssh root@parasilo-{} systemctl stop cassandra
    seq 2 8 | parallel -t ssh root@parasilo-{} sync
    seq 2 8 | xargs -t -n1 -i{} ssh root@parasilo-{} 'echo 3 > /proc/sys/vm/drop_caches'
    seq 2 8 | parallel -t ssh root@parasilo-{} systemctl start cassandra
    • Sep 15 2021, 12:04 PM
    • 5 Lines
    • Bash Scripting
    Author: vsellier
  • P1163 directory_ls.py
    from swh.storage import get_storage
    from swh.model.hashutil import hash_to_bytes
    import sys
    import time
    ...
    • Sep 15 2021, 11:18 AM
    • 31 Lines
    • Python
    Author: vsellier
  • P1162 Full visits with null snapshots
    10:36 guest@softwareheritage => select type, count(*) from origin_visit_status where status='full' and snapshot is null group by type;
    type │ count
    ──────┼───────
    git │ 1
    hg │ 62959
    ...
    • Sep 15 2021, 11:05 AM
    • 9 Lines
    Author: olasd
  • P1161 (An Untitled Masterwork)
    (swh) ✘-2 ~/swh/swh-environment/swh-loader-mercurial [master|✔]
    15:05 $ tox -r
    GLOB sdist-make: /home/anlambert/swh/swh-environment/swh-loader-mercurial/setup.py
    black recreate: /home/anlambert/swh/swh-environment/swh-loader-mercurial/.tox/black
    black installdeps: black==19.10b0
    ...
    • Sep 14 2021, 3:08 PM
    • 306 Lines
    Author: anlambert
  • P1160 CocoaPods/Specs runs
    ```
    10:51:05 softwareheritage@belvedere:5432=> select * from origin o inner join origin_visit_status ovs on o.id=ovs.origin where url = 'https://github.com/CocoaPods/Specs' order by date desc limit 10;
    +---------+------------------------------------+---------+-------+-------------------------------+---------+----------+--------------------------------------------+------+
    | id | url | origin | visit | date | status | metadata | snapshot | type |
    ...
    • Sep 14 2021, 10:53 AM
    • 41 Lines
    Author: ardumont
  • P1159 load-failing-on-unpacking
    [2] is the log used to extract the following list
    ```
    $ gzip -dc [2] | awk '{print $13" "$15 }' | sed -e 's/\\nTraceback//' | sort | uniq
    ```
    ...
    • Sep 13 2021, 6:10 PM
    • 258 Lines
    Author: ardumont
  • P1158 20210913-load-failing-on-404.txt
    See [1] for the full logs filtered on the command:
    ```
    $ gzip -dc [1] | awk '{print $13" "$15 }' | sed -e 's/\\nTraceback//' | sort | uniq
    ```
    ...
    • Sep 13 2021, 5:59 PM
    • 357 Lines
    Author: ardumont
  • P1157 Revisions with non-empty metadata field and missing raw extrinsic metadata
    00959a167bd98452c98ce73382f4b42179d53d32
    00a867beb2ad8e203f242e9843d2e88de0856cda
    028e9890a9287b35851c48ca351641743542d030
    030a51a49b3239769928872be9ac6d435ab14a61
    036594a6bbec926c21fa073e2404a5f760d35a43
    ...
    • Sep 10 2021, 6:08 PM
    • 166 Lines
    Author: olasd
  • P1155 Log buffer stats
    diff --git a/swh/storage/proxies/filter.py b/swh/storage/proxies/filter.py
    index 14bff3cc..ff73e850 100644
    --- a/swh/storage/proxies/filter.py
    +++ b/swh/storage/proxies/filter.py
    @@ -16,6 +16,10 @@ from swh.model.model import (
    ...
    • Sep 7 2021, 10:10 AM
    • 137 Lines
    Author: vsellier
  • P1154 (An Untitled Masterwork)
    [2021-09-03 13:38:39,438: ERROR/ForkPoolWorker-21] Bundle cooking failed.
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/vault/cookers/base.py", line 130, in cook
    self.prepare_bundle()
    File "/usr/lib/python3/dist-packages/swh/vault/cookers/git_bare.py", line 145, in prepare_bundle
    ...
    • Sep 3 2021, 5:41 PM
    • 61 Lines
    Author: anlambert
  • P1153 corrupted revision
    In [8]: from swh.storage import get_storage
    In [9]: s = get_storage('remote', url='http://moma.internal.softwareheritage.org:5002/')
    In [10]: r, = s.revision_get([bytes.fromhex('40b469ef825bec99054bec0fcaec224abe2e9cef')])
    ...
    • Sep 3 2021, 5:38 PM
    • 11 Lines
    • Python
    Author: olasd
  • P1152 (An Untitled Masterwork)
    [2021-09-03 14:41:41,875: ERROR/ForkPoolWorker-17] Bundle cooking failed.
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/vault/cookers/base.py", line 130, in cook
    self.prepare_bundle()
    File "/usr/lib/python3/dist-packages/swh/vault/cookers/git_bare.py", line 145, in prepare_bundle
    ...
    • Sep 3 2021, 5:29 PM
    • 35 Lines
    Author: anlambert
  • P1151 swh-vault cooking error
    Sep 03 12:33:12 worker12 python3[2525920]: [2021-09-03 12:33:12,243: INFO/MainProcess] Received task: swh.vault.cooking_tasks.SWHCookingTask[49a1a587-113d-4a47-b2f0-8a3345f7a344]
    Sep 03 12:33:12 worker12 python3[2614585]: Initialized empty Git repository in /tmp/swh-vault-gitbare-hfnujjye/clone.git/
    Sep 03 12:33:12 worker12 python3[2525940]: [2021-09-03 12:33:12,975: WARNING/ForkPoolWorker-16] Retrying RPC call
    Sep 03 12:33:13 worker12 python3[2525940]: [2021-09-03 12:33:13,191: WARNING/ForkPoolWorker-16] Retrying RPC call
    ...
    • Sep 3 2021, 2:41 PM
    • 48 Lines
    Author: ardumont
  • P1150 Command-Line Input
    2021-04-30 10:40:44,401 __main__ ERROR Could not parse revision metadata 69b3c7915fe7301980851946f8f8d32912359443
    Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 1161, in main
    handle_row(row, storage, deposit_cur, dry_run)
    File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 1076, in handle_row
    ...
    • Sep 3 2021, 2:10 PM
    • 4,067 Lines
    Author: olasd
  • P1149 scheduler metrics output tryout
    12:31:24 softwareheritage-scheduler@belvedere:5432=> select l.name, l.instance_name, sm.visit_type,
    softwareheritage-scheduler-> extract(epoch from sm.last_update) as last_update,
    softwareheritage-scheduler-> sm.origins_known, sm.origins_enabled, sm.origins_never_visited,
    softwareheritage-scheduler-> sm.origins_with_pending_changes
    softwareheritage-scheduler-> from scheduler_metrics sm
    ...
    • Sep 3 2021, 9:35 AM
    • 62 Lines
    Author: ardumont
  • P1148 vault debian package build with graph client tests ok
    swh-debian-build-unstable
    ++ git branch
    ++ grep '*'
    ++ cut '-d ' -f2
    + CURRENT_BRANCH=debian/unstable-swh
    ...
    • Sep 1 2021, 4:38 PM
    • 2,025 Lines
    Author: ardumont
  • P1147 debian package successful build
    unstable:
    ```
    $ swh-debian-build-unstable
    ++ git branch
    ++ grep '*'
    ...
    • Sep 1 2021, 3:18 PM
    • 1,339 Lines
    Author: ardumont
  • P1144 (An Untitled Masterwork)
    diff --git a/setup.py b/setup.py
    index d72b153..a7100a1 100755
    --- a/setup.py
    +++ b/setup.py
    @@ -74,8 +74,10 @@ def finalize_options(self):
    ...
    • Aug 31 2021, 2:38 PM
    • 53 Lines
    • Diff
    Author: vlorentz
  • P1143 Command-Line Input
    .
    ├── AUTHORS
    ├── CODE_OF_CONDUCT.md
    ├── CONTRIBUTORS
    ├── debian
    ...
    • Aug 31 2021, 2:31 PM
    • 98 Lines
    Author: olasd
  • P1142 Command-Line Input
    sbuild (Debian sbuild) 0.81.2 (31 January 2021) on mirzakhani.olasd.eu
    +==============================================================================+
    | swh-search 0.11.3-2~swh1 (amd64) Tue, 31 Aug 2021 12:28:45 +0000 |
    +==============================================================================+
    ...
    • Aug 31 2021, 2:30 PM
    • 1,593 Lines
    Author: olasd
  • P1141 GraphQL mvp rough schema
    origin(url) {
    url
    snapshots
    }
    ...
    • Aug 31 2021, 10:55 AM
    • 39 Lines
    • JSON
    Author: jayeshv
  • P1140 some basic pdb bootstrap example
    ... $ cd swh-loader-core
    $ ...  master $$ arc patch D6158
    $ ...  arcpatch-D6158 $ # edit swh/loader/package/loader.py and add some breakpoing
    $ ...  arcpatch-D6158 $ grep -B25 -A2 pdb swh/loader/package/loader.py
    ...  arcpatch-D6158 $ pytest --log-level=debug --ff -x -vv -s swh/loader/package/jar/tests/test_jar.py -k test_jar_visit_with_release_artifact_no_prior_visit
    ...
    • Aug 31 2021, 9:52 AM
    • 30 Lines
    Author: ardumont
  • P1139 github graphql org repos
    {
    viewer {
    login
    }
    organization(login: "rails") {
    ...
    • Aug 30 2021, 5:18 PM
    • 12 Lines
    Author: vlorentz
  • P1138 graphql repositories
    {
    search(query: "org:rails", type: REPOSITORY, first: 100) {
    repositoryCount
    edges {
    node {
    ...
    • Aug 30 2021, 5:17 PM
    • 16 Lines
    • JSON
    Author: jayeshv
  • P1137 maven jar loader tests failure
    (swh) boris@debian:~/swh-environment/swh-loader-core/swh/loader/package/jar$ pytest -v .
    ================================================ test session starts =================================================
    platform linux -- Python 3.7.3, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/boris/.virtualenvs/swh/bin/python3
    cachedir: .pytest_cache
    hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/boris/swh-environment/swh-loader-core/swh/loader/package/jar/.hypothesis/examples')
    ...
    • Aug 30 2021, 10:24 AM
    • 369 Lines
    • Python
    Author: borisbaldassari
  • P1136 old graphql draft
    commit 4f000e884c3010c2bb04bbb516aaf0ba234ccf4f
    Author: Valentin Lorentz <vlorentz@softwareheritage.org>
    Date: Fri Jun 28 14:57:57 2019 +0200
    [WIP] graphql API
    ...
    • Aug 30 2021, 10:05 AM
    • 529 Lines
    • Diff
    Author: vlorentz
  • P1135 es_open_unfreeze_from_journalctl.sh and es_open_unfreeze.sh:
    Manual installation so far.
    es_open_unfreeze_from_journalctl.sh:
    ```
    root@logstash0:~# cat /usr/local/bin/es_open_unfreeze_from_journalctl.sh
    ...
    • Aug 27 2021, 2:37 PM
    • 77 Lines
    Author: ardumont
  • P1134 current load-git tasks status
    11:54:44 softwareheritage-scheduler@belvedere:5432=> select status, count(*) from task where type='load-git' and priority is null group by status;
    +------------------------+-----------+
    | status | count |
    +------------------------+-----------+
    | next_run_not_scheduled | 17 |
    ...
    • Aug 26 2021, 12:31 PM
    • 19 Lines
    Author: ardumont
  • P1133 old yarnpkg support
    diff --git a/setup.py b/setup.py
    index 3929a46..bf3108d 100755
    --- a/setup.py
    +++ b/setup.py
    @@ -132,8 +132,8 @@ def generate_parser(dest_path):
    ...
    • Aug 25 2021, 1:57 PM
    • 14 Lines
    • Diff
    Author: vlorentz
  • P1131 (An Untitled Masterwork)
    function updateVaultItemList(vaultUrl, vaultItems) {
    window.localStorage.setItem('swh-vault-cooking-tasks', JSON.stringify(vaultItems));
    return cy.visit(vaultUrl);
    }
    ...
    • Aug 24 2021, 3:42 PM
    • 11 Lines
    • Javascript
    Author: anlambert
  • P1130 vault.spec.js: Remove vaultItems from global variables
    commit 9a4422102ff79124d637b80aab0fdecfff0169e0 (HEAD -> prepare-vault)
    Author: Valentin Lorentz <vlorentz@softwareheritage.org>
    Date: Tue Aug 24 10:19:22 2021 +0200
    vault.spec.js: Remove vaultItems from global variables
    ...
    • Aug 24 2021, 10:52 AM
    • 64 Lines
    • Diff
    Author: vlorentz
  • P1129 (An Untitled Masterwork)
    root@belvedere:~# puppet agent --enable; puppet agent --test --noop; puppet agent --disable
    Info: Using configured environment 'production'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Aug 23 2021, 5:38 PM
    • 44 Lines
    Author: ardumont
  • P1128 gbp buildpackage for buster-swh of monitoring-plugins-systemd
    $ swh-debian-build-stable
    ++ git branch
    ++ grep '*'
    ++ cut '-d ' -f2
    + CURRENT_BRANCH=debian/buster-swh
    ...
    • Aug 23 2021, 3:09 PM
    • 955 Lines
    Author: ardumont
  • P1127 error
    Storing in /Users/vanishaagrawal/Library/Application Support/ActivityTracker-nodejs/activity
    (node:15864) UnhandledPromiseRejectionWarning: Error: spawn /snapshot/Cross-Platform-Activity-Tracker/node_modules/.pnpm/active-win@7.6.1/node_modules/active-win/main ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:269:19)
    at onErrorNT (internal/child_process.js:467:16)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)
    ...
    • Aug 21 2021, 2:51 PM
    • 33 Lines
    Author: vanisha10
  • P1126 search.yml
    search:
    cls: elasticsearch
    hosts:
    - http://127.0.0.1:9200/
    • Aug 20 2021, 10:59 AM
    • 4 Lines
    Author: KShivendu
  • P1125 swh-{indexer,scheduler,search}-journal-client containers exit (unhealthy)
    Attaching to docker_swh-indexer-journal-client_1
    swh-indexer-journal-client_1 | Using pip from /srv/softwareheritage/venv/bin/pip
    swh-indexer-journal-client_1 | Installed Python packages:
    swh-indexer-journal-client_1 | Package Version
    swh-indexer-journal-client_1 | --------------------- -----------
    ...
    • Aug 20 2021, 10:49 AM
    • 339 Lines
    Author: KShivendu
  • P1124 restart a backfill where it has stopped previously
    #!/bin/zsh
    #set -x
    setopt shwordsplit
    cat >/tmp/script.py <<EOF
    ...
    • Aug 18 2021, 2:37 PM
    • 57 Lines
    • Bash Scripting
    Author: vsellier
  • P1123 create index-fossology-license-for-partition
    insert into task_type(
    type,
    description,
    backend_name,
    default_interval, min_interval, max_interval, backoff_factor,
    ...
    • Aug 18 2021, 11:22 AM
    • 12 Lines
    • SQL
    Author: vlorentz
  • P1122 generate backfill command for a given range (for sha1)
    import sys
    begin=0
    end=0x1000000
    #partitions=10
    ...
    • Aug 17 2021, 12:34 PM
    • 35 Lines
    • Python
    Author: vsellier
  • P1121 bakfill script for sha1 based ranges
    #!/bin/zsh
    # From
    setopt shwordsplit
    ...
    • Aug 17 2021, 12:33 PM
    • 28 Lines
    • Bash Scripting
    Author: vsellier
  • P1120 add swh-search static dir to django
    diff --git a/swh/web/settings/common.py b/swh/web/settings/common.py
    index ac6e4878..d4bb93e3 100644
    --- a/swh/web/settings/common.py
    +++ b/swh/web/settings/common.py
    @@ -12,6 +12,7 @@ import os
    ...
    • Aug 16 2021, 12:37 PM
    • 27 Lines
    Author: anlambert
  • P1119 staging - deposit logs
    Aug 09 15:43:53 deposit python3[886]: 2021-08-09 15:43:53 [886] swh.deposit.auth:WARNING Error during cache token retrieval: Signature has expired.
    Aug 09 15:43:53 deposit python3[886]: 2021-08-09 15:43:53 [886] gunicorn.access:INFO 127.0.0.1 - hal-preprod [09/Aug/2021:15:43:53 +0000] "POST /1/hal-preprod/ HTTP/1.1" 201 1358 "-" "-"
    Aug 09 15:43:54 deposit python3[896]: 2021-08-09 15:43:54 [896] gunicorn.access:INFO 127.0.0.1 - hal-preprod [09/Aug/2021:15:43:54 +0000] "POST /1/hal-preprod/483/media/ HTTP/1.1" 201 1384 "-" "-"
    Aug 09 15:48:54 deposit python3[896]: 2021-08-09 15:48:54 [896] swh.deposit.auth:WARNING Error during cache token retrieval: Signature has expired.
    Aug 09 15:53:55 deposit python3[881]: 2021-08-09 15:53:55 [881] swh.deposit.auth:WARNING Error during cache token retrieval: Signature has expired.
    ...
    • Aug 10 2021, 5:05 PM
    • 29 Lines
    Author: ardumont
  • P1118 Cassandra missing group by replica
    diff --git a/swh/storage/cassandra/cql.py b/swh/storage/cassandra/cql.py
    index 89662459..d4941c70 100644
    --- a/swh/storage/cassandra/cql.py
    +++ b/swh/storage/cassandra/cql.py
    @@ -25,6 +25,7 @@
    ...
    • Aug 6 2021, 6:52 PM
    • 85 Lines
    • Diff
    Author: vlorentz
  • P1117 full list of opam origins targeting an empty snapshot + worker logs on failing origin with 404 (2nd comment)
    ```
    16:09 <labor[m]> ardumont: could you give me the full list of opam packages + versions ending up with a 404 ?
    16:32 <+ardumont> my best approximation for this would be origins whose status ended up in an empty snapshot
    ```
    ...
    • Aug 6 2021, 4:36 PM
    • 1,023 Lines
    Author: ardumont
First
Prev
Next
About · Developer information · Legal