Page MenuHomeSoftware Heritage
Create Paste
Paste Active Pastes

Active Pastes
Use Results
Edit Query

  • P1374 prod: loader frequency
    09:52:37 softwareheritage-scheduler@belvedere:5432=> select * from task_type where type like 'load-%' or type like 'list-%';
    +------------------------------+--------------------------------------------------------------+-----------------------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
    | type | description | backend_name | default_interval | min_interval | max_interval | backoff_factor | max_queue_length | num_retries | retry_delay |
    +------------------------------+--------------------------------------------------------------+-----------------------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
    | load-hg | Update an origin of type mercurial | swh.loader.mercurial.tasks.LoadMercurial | 64 days | 1 day | 64 days | 2 | 5000 | 3 | (null) |
    ...
    • Jun 3 2022, 9:55 AM
    • 55 Lines
    Author: ardumont
  • P1373 staging scheduling frequency
    09:49:06 swh-scheduler@db1:5432=> select * from task_type where type like 'load-%';
    +-----------------------+--------------------------------------------------------------+----------------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
    | type | description | backend_name | default_interval | min_interval | max_interval | backoff_factor | max_queue_length | num_retries | retry_delay |
    +-----------------------+--------------------------------------------------------------+----------------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
    | load-svn-from-archive | Loading svn repositories from svn dump | swh.loader.svn.tasks.MountAndLoadSvnRepository | 1 day | 1 day | 1 day | 1 | 1000 | (null) | (null) |
    ...
    • Jun 3 2022, 9:52 AM
    • 26 Lines
    Author: ardumont
  • P1372 (An Untitled Masterwork)
    $ docker-compose exec swh-idx-storage swh db --config-file /indexer_storage.yml upgrade indexer_storage
    ERROR:swh.core.db.db_utils:Failed to connect to `postgresql:///?service=swh-indexers`
    Traceback (most recent call last):
    File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/db/db_utils.py", line 241, in swh_db_module
    db = connect_to_conninfo(db_or_conninfo)
    ...
    • Jun 2 2022, 3:06 PM
    • 44 Lines
    Author: vlorentz
  • P1371 still github origins listed from maven with exotic urls
    psql service=admin-staging-swh-scheduler
    Pager usage is off.
    Null display is "(null)".
    Timing is on.
    Border style is 2.
    ...
    • Jun 2 2022, 2:06 PM
    • 776 Lines
    Author: ardumont
  • P1370 indexer cli says no
    ```
    swhworker@worker0:~$ dpkg -l python3-swh.indexer | grep ii
    ii python3-swh.indexer 1.1.0-1~swh1~bpo10+1 all Software Heritage Content Indexerswhworker@worker0:~$ echo $SWH_CONFIG_FILENAME
    /etc/softwareheritage/indexer/journal_client.yml
    swhworker@worker0:~$ /usr/bin/swh indexer --config-file $SWH_CONFIG_FILENAME journal-client indexer '*'
    ...
    • Jun 1 2022, 4:43 PM
    • 19 Lines
    Author: ardumont
  • P1369 old maven listing with github origins non-canonicalized
    10:26:05 swh-scheduler@db1:5432=> with maven_lister_ids as (
    swh-scheduler(> select id from listers where name='maven'
    swh-scheduler(> ) select now(), visit_type, url
    swh-scheduler-> from
    swh-scheduler-> listed_origins lo1
    ...
    • Jun 1 2022, 10:46 AM
    • 28,051 Lines
    Author: ardumont
  • P1368 Blog I: Software Heritage
    ---
    layout: post
    title: "Software Heritage"
    description: "About Software Heritage, what we stand for, and where we are headed"
    date:
    ...
    • May 26 2022, 11:58 AM
    • 52 Lines
    Author: VickyMerzOwn
  • P1367 diff deps between last green and current green builds
    diff /tmp/tox-swh-web.now.txt /tmp/tox-swh-web-last-green.txt
    0a1,3
    > aiohttp==3.8.1
    > aiohttp-utils==3.1.1
    > aiosignal==1.2.0
    ...
    • May 25 2022, 5:33 PM
    • 123 Lines
    Author: ardumont
  • P1366 (An Untitled Masterwork)
    [19/May/2022 08:29:58] [DEBUG] Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/rest_framework/views.py", line 483, in dispatch
    self.initial(request, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/rest_framework/views.py", line 402, in initial
    self.check_throttles(request)
    ...
    • May 19 2022, 11:18 AM
    • 12 Lines
    Author: douardda
  • P1365 (An Untitled Masterwork)
    anlambert@carnavalet:/tmp$ python3
    Python 3.9.2 (default, Feb 28 2021, 17:03:44)
    [GCC 10.2.1 20210110] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import dulwich
    ...
    • May 17 2022, 1:43 PM
    • 56 Lines
    Author: anlambert
  • P1364 swh search kibana profiling
    [
    // Default query template (Don't run)
    {
    "query": {
    "bool": {
    ...
    • May 16 2022, 8:52 PM
    • 183 Lines
    • JSON
    Author: KShivendu
  • P1363 (An Untitled Masterwork)
    swh-scheduler=> select url from listed_origins where lister_id = 'ab79de00-a2d3-44ac-90a8-f53c35c5d893' and visit_type = 'git';
    -------------------------------------------------------------------------------------------------------------------------------------
    ${basedir}
    ${base.url}.git
    ${custom.gitprivate.url}simplerest.git
    ...
    • May 12 2022, 6:05 PM
    • 72 Lines
    Author: anlambert
  • P1362 (An Untitled Masterwork)
    11:53:48 swh-scheduler@db1:5432=> select * from task where type='list-maven-full';
    +-[ RECORD 1 ]-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | id | 31171944 |
    | type | list-maven-full |
    | arguments | {"args": [], "kwargs": {"url": "https://repo1.maven.org/maven2/", "index_url": "https://maven-exporter.internal.staging.swh.network/export-maven-central.fld"}} |
    ...
    • May 12 2022, 11:54 AM
    • 23 Lines
    Author: ardumont
  • P1361 (An Untitled Masterwork)
    % join <( cat 2021-03-23/compressed/graph.edges.stats.txt | sort ) <( cat 2022-04-25/compressed/graph.edges.stats.txt | sort ) | while read t a b; do echo $t $(( 1. * b / a )); done
    dir:cnt 1.6619934106831602
    dir:dir 1.5117835202334333
    dir:rev 2.5607389960164033
    ori:snp 1.1175533201104195
    ...
    • May 8 2022, 12:32 PM
    • 22 Lines
    Author: seirl
  • P1360 (An Untitled Masterwork)
    seirl@vab ../softwareheritage/ssd/graph/2022-04-25 % cat compressed/graph.nodes.count.txt
    25340003875
    seirl@vab ../softwareheritage/ssd/graph/2022-04-25 % cat compressed/graph.nodes.stats.txt
    cnt 12213798094
    dir 10173900390
    ...
    • May 8 2022, 11:15 AM
    • 26 Lines
    Author: seirl
  • P1359 (An Untitled Masterwork)
    ~/dati/src/rust/coreutils/src/uu/sort   main  rg fork
    ✘  ~/dati/src/rust/coreutils/src/uu/sort   main  rg spawn
    src/check.rs
    41: thread::spawn({
    ...
    • May 6 2022, 4:50 PM
    • 12 Lines
    Author: zack
  • P1358 stupid directories
    select directory_id, from_utf8(name), type, target, perms from directory_entry where from_utf8(name) in ('.', '..', '') or strpos(from_utf8(name), '/') > 0
    "directory_id","","type","target","perms"
    "5002df6cc84b3245e9554f1b4ef286876cbb0856",".","dir","3dc4aa1834136cbff6f4451c677f8c18f28c56f2","16384"
    ...
    • May 4 2022, 5:01 PM
    • 545 Lines
    Author: seirl
  • P1357 (An Untitled Masterwork)
    seirl@vab ../softwareheritage/ssd/graph/2021-03-23 % du -shc --apparent-size compressed/* | sort -h :(
    9 compressed/graph.persons.count.txt
    11 compressed/graph.labels.count.txt
    12 compressed/graph.nodes.count.txt
    13 compressed/graph.edges.count.txt
    ...
    • Apr 29 2022, 3:11 PM
    • 49 Lines
    Author: seirl
  • P1356 SQL fork repos
    softwareheritage=> select count(*) from (select url as child, (select count(*) from origin_visit where origin=origin.id) as child_visits, encode(metadata, 'escape')::json->'parent'->'full_name' as parent, (select count(*) from origin_visit, origin as o where origin_visit.origin=o.id and o.url= concat('https://github.com/'::text, (encode(metadata, 'escape')::json->'parent'->>'full_name')::text)) as parent_visits from raw_extrinsic_metadata, origin where digest(url, 'sha1')=decode(substring(target, 11), 'hex') and target>'swh:1:ori:' and target < 'swh:1:orz' and format='application/vnd.github.v3+json' and encode(metadata, 'escape') not like '%\\3%' and encode(metadata, 'escape') not like '%\\\\%' and encode(metadata, 'escape')::json->'parent'->'full_name' is not null) as t where t.child_visits=1 and t.parent_visits>0;
    ^[[A^[[A
    count
    -------
    10642
    ...
    • Apr 29 2022, 1:48 PM
    • 12 Lines
    Author: vlorentz
  • P1355 (An Untitled Masterwork)
    diff --git a/swh/loader/package/maven/loader.py b/swh/loader/package/maven/loader.py
    index d664e67..e2d9d05 100644
    --- a/swh/loader/package/maven/loader.py
    +++ b/swh/loader/package/maven/loader.py
    @@ -1,8 +1,10 @@
    ...
    • Apr 28 2022, 5:52 PM
    • 39 Lines
    Author: anlambert
  • P1354 (An Untitled Masterwork)
    (swh) $ pip freeze | grep swh.core
    -e git+ssh://git@forge.softwareheritage.org/diffusion/DCORE/swh-core.git@d374b6002955bf3591c84335b6bcc3521ff78061#egg=swh.core
    (swh) $ pip show swh.core | grep Version:
    Version: 1.0.1.dev3+gd374b60
    • Apr 28 2022, 2:09 PM
    • 4 Lines
    Author: zack
  • P1353 fork speedup!!!
    swhworker@worker3:~$ url=https://github.com/progval/Limnoria ; /usr/bin/time swh loader run git $url lister_name=github lister_instance_name=github
    WARNING:swh.core.cli:Could not load subcommand graph: DistributionNotFound(Requirement.parse('py4j'), None)
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/progval/Limnoria' with type 'git'
    INFO:swh.loader.git.loader:Listed 819 refs for repo https://github.com/progval/Limnoria
    {'status': 'uneventful'} for origin 'https://github.com/progval/Limnoria'
    ...
    • Apr 28 2022, 1:34 PM
    • 28 Lines
    Author: ardumont
  • P1352 4.612300505552681% whose ingestion is roughly done in staging
    listed origins:
    ```
    10:24:59 swh@db1:5432=> select now(), count(*) from origin where url like 'https://repo1.maven.org/maven2/%'
    swh-> ;
    +------------------------------+--------+
    ...
    • Apr 28 2022, 10:31 AM
    • 31 Lines
    Author: ardumont
  • P1351 (An Untitled Masterwork)
    $ cat example.py
    import contextlib
    @contextlib.contextmanager
    def f():
    ...
    • Apr 26 2022, 11:54 AM
    • 21 Lines
    Author: vlorentz
  • P1350 test-inter-node-network.sh
    #!/usr/bin/env bash
    # set -x
    export KUBECONFIG=staging-workers.yaml
    ...
    • Apr 26 2022, 11:28 AM
    • 26 Lines
    Author: ardumont
  • P1349 (An Untitled Masterwork)
    ERROR:swh.dataset.journalprocessor:Exporter ORCExporter: error while exporting the object: {'origin': 'https://www.npmjs.com/package/@reactionaries/hookd', 'visit': 170, 'date': datetime.datetime(2020, 9, 15, 18, 11, 37, 416264
    , tzinfo=datetime.timezone.utc), 'status': 'full', 'snapshot': b'\x15\xa3i\x0c\x05\x86PT\x19\\_~k7\x1cY$V\xdb\xe5', 'metadata': None}
    Traceback (most recent call last):
    File "/home/seirl/swh-environment/swh-dataset/swh/dataset/journalprocessor.py", line 467, in process_message
    [message.key()],
    ...
    • Apr 25 2022, 2:23 PM
    • 31 Lines
    Author: seirl
  • P1348 git annex sync --content (libvirt debian 11.3 with zfs image)
    swh  tony  yavin4  ~  work  …  isos  libvirt  debian   master  1+  %  sha512sum -c swh-debian-zfs-11.3-amd64-20220422-1444.qcow2.sha512
    swh-debian-zfs-11.3-amd64-20220422-1444.qcow2: OK
    swh  tony  yavin4  ~  work  …  isos  libvirt  debian   master  2+  %  ls -lah
    total 841M
    drwxr-xr-x 2 tony tony 16 Apr 22 17:00 .
    ...
    • Apr 22 2022, 5:06 PM
    • 132 Lines
    Author: ardumont
  • P1347 git annex sync --content (libvirt debian 11.3 image)
    swh  tony  yavin4  ~  work  …  isos  libvirt  debian   master  %  ls -lah
    total 576M
    drwxr-xr-x 2 tony tony 8 Apr 22 16:08 .
    drwxr-xr-x 3 tony tony 3 Mar 12 19:25 ..
    lrwxrwxrwx 1 tony tony 203 May 18 2021 swh-debian-10.6-amd64-20201012-1352.qcow2 -> ../../../.git/annex/objects/6M/K7/SHA256E-s540454212--96a6280841959cf9be85ee5d07afa7cf53d31ced56369cb66c536bd1b61f2d7c/SHA256E-s540454212--96a6280841959cf9be85ee5d07afa7cf53d31ced56369cb66c536bd1b61f2d7c
    ...
    • Apr 22 2022, 4:13 PM
    • 442 Lines
    Author: ardumont
  • P1346 package build debian_bullseye.qemu.json
    packer build debian_bullseye.qemu.json
    qemu: output will be in this color.
    ==> qemu: Retrieving ISO
    ==> qemu: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.3.0-amd64-netinst.iso
    ...
    • Apr 22 2022, 3:54 PM
    • 686 Lines
    Author: ardumont
  • P1345 (An Untitled Masterwork)
    14:06 <+moranegg> olasd: (or any other git annex expert) I have done a git sync to add one pdf and seems lot of stuff has happened during this sync, Is there a way to check that this is normal and that I didn't break anything?
    14:08 <+olasd> there's a bunch of pdfs that are in the git repo instead of the annex (events/2021-11-30-swh5years/community-meetup/) but that commit was in january
    14:11 <+moranegg> interesting, I had in January other issues
    14:12 <+olasd> I managed a git annex get of the pdf you've added just fine
    14:15 <+moranegg> great, I hope I haven't deleted anything with the sync
    ...
    • Apr 22 2022, 2:41 PM
    • 16 Lines
    Author: ardumont
  • P1344 Error related to D7491
    if has_add_forge_now_permission(request.user):
    for item in page.object_list:
    history = AddForgeNowRequestHistory.objects.filter(request=item ).filter(actor_role = "MODERATOR").order_by(
    "id"
    )
    ...
    • Apr 21 2022, 9:39 PM
    • 18 Lines
    Author: anirudhlakhotia
  • P1343 packer build debian_bullseye_zfs.json
    packer build debian_bullseye_zfs.json
    proxmox: output will be in this color.
    ==> proxmox: Retrieving ISO
    ==> proxmox: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.3.0-amd64-netinst.iso
    ...
    • Apr 21 2022, 4:06 PM
    • 1,671 Lines
    Author: ardumont
  • P1342 packer build debian_bullseye.json
    packer build debian_bullseye.json
    proxmox: output will be in this color.
    ==> proxmox: Retrieving ISO
    ==> proxmox: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.3.0-amd64-netinst.iso
    ...
    • Apr 21 2022, 3:30 PM
    • 1,189 Lines
    Author: ardumont
  • P1341 zpool create that pool
    root@elastic-worker0:~# puppet agent --test
    Info: Using configured environment 'staging'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Apr 20 2022, 11:54 AM
    • 47 Lines
    Author: ardumont
  • P1340 (An Untitled Masterwork)
    softwareheritage-scheduler=> select url, last_scheduled, last_visit from origin_visit_stats where url = 'http://svn.code.sf.net/p/csreadgen/code/';
    url | last_scheduled | last_visit
    ------------------------------------------+-------------------------------+-------------------------------
    http://svn.code.sf.net/p/csreadgen/code/ | 2022-04-13 09:42:21.829297+00 | 2022-04-13 09:42:26.456121+00
    (1 row)
    • Apr 13 2022, 5:27 PM
    • 5 Lines
    Author: anlambert
  • P1339 (An Untitled Masterwork)
    Traceback (most recent call last):
    File "/home/seirl/swh-environment/.venv/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core', 'console_scripts', 'swh')())
    File "/home/seirl/swh-environment/swh-core/swh/core/cli/__init__.py", line 185, in main
    return swh(auto_envvar_prefix="SWH")
    ...
    • Apr 12 2022, 5:33 PM
    • 49 Lines
    Author: seirl
  • P1338 (An Untitled Masterwork)
    journal:
    brokers:
    - kafka1.internal.softwareheritage.org:9094
    - kafka2.internal.softwareheritage.org:9094
    - kafka3.internal.softwareheritage.org:9094
    ...
    • Apr 12 2022, 5:27 PM
    • 11 Lines
    Author: seirl
  • P1337 (An Untitled Masterwork)
    softwareheritage=> SELECT count(*) FROM origin_visit ov INNER JOIN origin o ON o.id = ov.origin WHERE o.url = 'https://subversion.renater.fr/anonscm/svn/panda';
    count
    --------
    132706
    (1 row)
    • Apr 12 2022, 12:02 PM
    • 5 Lines
    Author: anlambert
  • P1336 (An Untitled Masterwork)
    class SupportsObjectType(Protocol):
    object_type: str
    _L: List[SupportsObjectType] = [
    Person,
    ...
    • Apr 11 2022, 10:29 AM
    • 29 Lines
    • Python
    Author: vlorentz
  • P1335 (An Untitled Masterwork)
    Traceback (most recent call last):
    File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
    File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
    ...
    • Apr 8 2022, 5:37 PM
    • 26 Lines
    Author: anlambert
  • P1334 (An Untitled Masterwork)
    13:42 $ swh search -C ~/.config/swh/search.yml rpc-serve
    * Serving Flask app 'swh.search.api.server' (lazy loading)
    * Environment: production
    WARNING: This is a development server. Do not use it in a production deployment.
    Use a production WSGI server instead.
    ...
    • Apr 7 2022, 1:45 PM
    • 35 Lines
    Author: anlambert
  • P1333 query 3
    query test {
    origin(url: "https://github.com/python/cpython") {
    url
    id
    visits(first: 2, after: "Mg==") {
    ...
    • Apr 6 2022, 6:01 PM
    • 50 Lines
    Author: jayeshv
  • P1332 query 2
    query rels {
    release(SWHId: "d28343dc3ad53a411ae3685e7d6a7866c8c22d6b") {
    id
    date
    name
    ...
    • Apr 6 2022, 5:58 PM
    • 15 Lines
    Author: jayeshv
  • P1331 query1
    query snapshots {
    snapshot(SWHId: "6f1e3067acba64891af07680367a38cc1b9cb57a") {
    branches(first: 10) {
    nodes {
    name
    ...
    • Apr 6 2022, 5:57 PM
    • 31 Lines
    Author: jayeshv
  • P1330 setup-db.sh
    #!/usr/bin/env bash
    set -xe
    DBPORT=5432
    ...
    • Apr 6 2022, 4:36 PM
    • 15 Lines
    Author: ardumont
  • P1329 come on vagrant, why, why!?
    $ sudo pidof dnsmasq
    240165 240164 240049 240048
    $ ls /var/lib/libvirt
    boot dnsmasq images qemu sanlock
    $ ls /var/lib/libvirt/dnsmasq
    ...
    • Apr 4 2022, 5:08 PM
    • 25 Lines
    Author: ardumont
  • P1328 docker-compose.override.yml
    version: '2'
    services:
    ...
    ...
    • Apr 4 2022, 1:30 PM
    • 8 Lines
    Author: ardumont
  • P1327 (An Untitled Masterwork)
    diff --git java/src/main/java/org/softwareheritage/graph/compress/LabelMapBuilder.java java/src/main/java/org/softwareheritage/graph/compress/LabelMapBuilder.java
    index 600870e..0825266 100644
    --- java/src/main/java/org/softwareheritage/graph/compress/LabelMapBuilder.java
    +++ java/src/main/java/org/softwareheritage/graph/compress/LabelMapBuilder.java
    @@ -143,7 +143,7 @@ public class LabelMapBuilder {
    ...
    • Apr 1 2022, 4:45 PM
    • 104 Lines
    • Diff
    Author: seirl
  • P1326 graphql-sample1
    query testquery {
    origin(url: "https://github.com/python/cpython") {
    url
    id
    visits(first: 2, after: "Mg==") {
    ...
    • Apr 1 2022, 3:36 PM
    • 45 Lines
    Author: jayeshv
  • P1325 docker-compose restart doesn't restart all containers
    (swh) parallels@ubuntu-linux-20-04-desktop:~/Desktop/swh/swh-environment/docker$ sudo docker-compose restart
    Restarting docker_swh-web-cron_1 ...
    Restarting docker_swh-vault-worker_1 ...
    Restarting docker_swh-indexer-journal-client_1 ...
    Restarting docker_swh-lister_1 ...
    ...
    • Apr 1 2022, 10:02 AM
    • 304 Lines
    Author: anirudhlakhotia
  • P1324 [POC] Use new endpoint of deposit private API for datatables in swh-web
    diff --git a/swh/web/admin/deposit.py b/swh/web/admin/deposit.py
    index ddbfe304..1d653f36 100644
    --- a/swh/web/admin/deposit.py
    +++ b/swh/web/admin/deposit.py
    @@ -3,21 +3,18 @@
    ...
    • Mar 30 2022, 6:04 PM
    • 132 Lines
    Author: anlambert
  • P1323 [POC] Deposit private API endpoint for datatables to list deposits
    diff --git a/swh/deposit/api/private/deposit_list.py b/swh/deposit/api/private/deposit_list.py
    index 4bb40e31..e8c07e31 100644
    --- a/swh/deposit/api/private/deposit_list.py
    +++ b/swh/deposit/api/private/deposit_list.py
    @@ -3,10 +3,20 @@
    ...
    • Mar 30 2022, 6:03 PM
    • 162 Lines
    Author: anlambert
  • P1322 (An Untitled Masterwork)
    # swh-core
    diff --git a/swh/core/db/common.py b/swh/core/db/common.py
    index 65e6d75..6f89058 100644
    --- a/swh/core/db/common.py
    +++ b/swh/core/db/common.py
    ...
    • Mar 30 2022, 1:45 PM
    • 76 Lines
    Author: douardda
  • P1321 somehow, no add-forge-now migration run in cypress tests...?
    make test-frontend-ui-cmd
    rm -f swh-web-test.sqlite3
    django-admin migrate --settings=swh.web.settings.tests
    Operations to perform:
    Apply all migrations: admin, auth, contenttypes, sessions, swh_web_add_forge_now, swh_web_auth, swh_web_common
    ...
    • Mar 29 2022, 5:50 PM
    • 49 Lines
    Author: ardumont
  • P1320 confluent-kafka installation-related error on running bin/install
    ERROR: Command errored out with exit status 1:
    command: /home/parallels/.virtualenvs/swh/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0_w3ghla/confluent-kafka/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0_w3ghla/confluent-kafka/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-tx1t5lrk
    cwd: /tmp/pip-install-0_w3ghla/confluent-kafka/
    Complete output (116 lines):
    running bdist_wheel
    ...
    • Mar 28 2022, 10:14 PM
    • 122 Lines
    Author: anirudhlakhotia
  • P1319 (An Untitled Masterwork)
    codemeta:identifier does not allow propertyvalues (source: https://github.com/codemeta/codemeta/blob/master/codemeta.jsonld#L36 ; `"@type": "@id"` means the value must be a URI, but propertyvalues are not URIs), so we need to replace:
    <codemeta:identifier>
    <codemeta:type>schema:PropertyValue</codemeta:type>
    <schema:propertyID>HAL-ID</schema:propertyID>
    ...
    • Mar 28 2022, 4:26 PM
    • 15 Lines
    Author: vlorentz
  • P1318 KShivendu question and replies from us
    11:14 <KShivendu> That's great! Now try fixing some easy hacks and write a draft for your proposal and share it with us :)
    12:59 <KShivendu> Hey vsellier can you please tell me about how we have deployed the elasticsearch cluster. for example: what kind of a machine(s) is it, and where is it hosted?
    13:00 <KShivendu> also, did we face any challenge in this because of the scale? if yes, please briefly describe them :)
    13:25 <+ardumont> KShivendu: where services are exposed is described in https://docs.softwareheritage.org/sysadm/network-architecture/service-urls.html
    13:25 <+ardumont> ah he is not here
    ...
    • Mar 28 2022, 1:53 PM
    • 11 Lines
    Author: ardumont
  • P1317 (An Untitled Masterwork)
    describe('Browse requests list tests', function() {
    beforeEach(function() {
    this.addForgeNowUrl = this.Urls.forge_add();
    this.listAddForgeRequestsUrl = this.Urls.add_forge_request_list_datatables();
    });
    ...
    • Mar 25 2022, 3:02 PM
    • 48 Lines
    Author: ardumont
  • P1316 (An Untitled Masterwork)
    softwareheritage=> SELECT DISTINCT ON (ov.visit) ov.visit, ov.type, ov.date, ovs.date as status_date, ovs.status
    FROM origin_visit ov
    INNER JOIN origin o ON o.id = ov.origin
    INNER JOIN origin_visit_status ovs USING (origin, visit)
    WHERE o.url = 'https://github.com/python/cpython'
    ...
    • Mar 25 2022, 2:27 PM
    • 78 Lines
    Author: anlambert
  • P1315 pyorc_no_zoneinfo.patch
    diff --git a/CHANGELOG.rst b/CHANGELOG.rst
    index e417bcd..198150f 100644
    --- a/CHANGELOG.rst
    +++ b/CHANGELOG.rst
    @@ -57,7 +57,7 @@ Added
    ...
    • Mar 22 2022, 4:43 PM
    • 148 Lines
    • Diff
    Author: vlorentz
  • P1314 failed attempt at error coverage
    new file cypress/fixtures/add-forge-now-create-request-fail.json
    @@ -0,0 +1,8 @@
    +{
    + "status": 400,
    + "ok": false,
    ...
    • Mar 18 2022, 3:13 PM
    • 43 Lines
    Author: ardumont
  • P1313 (An Untitled Masterwork)
    make test-frontend-ui-cmd
    rm -f swh-web-test.sqlite3
    django-admin migrate --settings=swh.web.settings.tests -v0 2>/dev/null
    cat swh/web/tests/create_test_admin.py | django-admin shell --settings=swh.web.settings.tests
    cat swh/web/tests/create_test_users.py | django-admin shell --settings=swh.web.settings.tests
    ...
    • Mar 17 2022, 11:08 AM
    • 62 Lines
    Author: ardumont
  • P1311 trying to create a basic user with the proper add-forge-now permissions and it's just making excuses not to create it...
    swh/web/tests/create_test_users.py:
    ```
    from swh.web.auth.utils import SWH_AMBASSADOR_PERMISSION, SWH_MODERATOR_PERMISSION
    ...
    • Mar 15 2022, 11:11 AM
    • 106 Lines
    Author: ardumont
  • P1310 query plan origin_visit_get_latest
    softwareheritage=> explain select ov.* from origin_visit ov INNER JOIN origin o ON o.id = ov.origin WHERE o.url = 'svn://scm.gforge.inria.fr/svnroot/fpg-ipc/' ORDER BY ov.visit DESC LIMIT 1;
    QUERY PLAN
    ----------------------------------------------------------------------------------------------------------------
    Limit (cost=633.57..633.57 rows=1 width=28)
    -> Sort (cost=633.57..633.59 rows=7 width=28)
    ...
    • Mar 11 2022, 11:53 AM
    • 29 Lines
    Author: vlorentz
  • P1306 ~/.config/git/config
    [alias]
    checkout-from-staging = "!f() { git fetch -nf https://forge.softwareheritage.org/source/staging.git phabricator/diff/$1:diff/$1 && git checkout diff/$1; }; f"
    • Mar 8 2022, 2:33 PM
    • 2 Lines
    Author: ardumont
  • P1305 GraphQL query example
    query test {
    origin(url: "https://github.com/python/cpython") {
    id
    url
    visits(first: 2) {
    ...
    • Mar 8 2022, 9:16 AM
    • 46 Lines
    Author: jayeshv
  • P1304 deposit metadata split
    softwareheritage=> \copy (select encode(metadata, 'escape') from raw_extrinsic_metadata where target='swh:1:dir:5d6f2287a9e2e08aaad3f31f833c82d50bcf3c88') to stdout
    <
    ?
    x
    m
    ...
    • Mar 2 2022, 2:48 PM
    • 71 Lines
    Author: vlorentz
  • P1303 objstorage async in sync in async
    diff --git a/swh/objstorage/backends/azure.py b/swh/objstorage/backends/azure.py
    index be7aeaa..80d8fba 100644
    --- a/swh/objstorage/backends/azure.py
    +++ b/swh/objstorage/backends/azure.py
    @@ -78,12 +78,23 @@ def get_container_url(
    ...
    • Feb 28 2022, 5:06 PM
    • 28 Lines
    • Diff
    Author: vlorentz
  • P1302 xmlschema uri
    ✘ dev@desktop5  ~/swh-environment/swh-deposit   validate-metadata ●  pip3 show xmlschema
    Name: xmlschema
    Version: 1.9.2
    Summary: An XML Schema validator and decoder
    Home-page: https://github.com/sissaschool/xmlschema
    ...
    • Feb 24 2022, 6:35 PM
    • 20 Lines
    Author: vlorentz
  • P1301 (An Untitled Masterwork)
    ### Strucure of answer with short answer
    <details>
    <summary> use this template for dropdown </summary>
    <br>
    ...
    • Feb 23 2022, 5:01 PM
    • 7 Lines
    Author: anlambert
  • P1300 (An Untitled Masterwork)
    class SwhGraphTinkerpopProvider extends WebGraphTinkerpopProvider {
    @Override
    public ImmutableGraph load() {
    var graph = SwhBidirectionalImmutableGraph.loadLabelled(...);
    graph.loadCommitAuthors(LoadMethod.MAPPED);
    ...
    • Feb 23 2022, 4:32 PM
    • 19 Lines
    • Java
    Author: seirl
  • P1299 (An Untitled Masterwork)
    Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install python3-swh.web' returned 100: Reading package lists...
    Building dependency tree...
    Reading state information...
    Some packages could not be installed. This may mean that you have
    requested an impossible situation or if you are using the unstable
    ...
    • Feb 23 2022, 3:29 PM
    • 24 Lines
    Author: anlambert
  • P1298 python <string>
    $ echo "world" > "<string>"
    $ python3 -c "hello"
    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    world
    ...
    • Feb 23 2022, 11:59 AM
    • 6 Lines
    Author: vlorentz
  • P1297 Sample GraphQL query
    query myquery {
    origin(url: "https://github.com/python/cpython") {
    url
    id
    visits(first: 2) {
    ...
    • Feb 23 2022, 10:51 AM
    • 27 Lines
    Author: jayeshv
  • P1294 cassandra vms and resources destruction in 2 times
    swh  tony  yavin4  ~  work  …  sysadm-provisioning  azure  terraform   master  2⬆  8⚑  %  terraform apply
    azurerm_resource_group.euwest-vault: Refreshing state... [id=/subscriptions/49b7f681-8efc-4689-8524-870fc0c1db09/resourceGroups/euwest-vault]
    azurerm_resource_group.euwest-cassandra: Refreshing state... [id=/subscriptions/49b7f681-8efc-4689-8524-870fc0c1db09/resourceGroups/euwest-cassandra]
    azurerm_storage_account.vault-storage: Refreshing state... [id=/subscriptions/49b7f681-8efc-4689-8524-870fc0c1db09/resourceGroups/euwest-vault/providers/Microsoft.Storage/storageAccounts/swhvaultstorage]
    azurerm_network_interface.storage-interface["storage01"]: Refreshing state... [id=/subscriptions/49b7f681-8efc-4689-8524-870fc0c1db09/resourceGroups/euwest-servers/providers/Microsoft.Network/networkInterfaces/storage01-interface]
    ...
    • Feb 18 2022, 4:33 PM
    • 1,094 Lines
    Author: ardumont
  • P1293 bzr launchpad fun
    ipython
    Python 3.9.2 (default, Feb 28 2021, 17:03:44)
    Type 'copyright', 'credits' or 'license' for more information
    IPython 7.27.0 -- An enhanced Interactive Python. Type '?' for help.
    ...
    • Feb 16 2022, 4:14 PM
    • 235 Lines
    Author: ardumont
  • P1291 (An Untitled Masterwork)
    [2022-02-15 13:42:59,440: INFO/MainProcess] Task swh.loader.cvs.tasks.LoadCvsRepository[9ea64ca6-744b-46b0-a5dd-754ad1a7ae99] received
    swh-loader_1 | [2022-02-15 13:43:02,110: INFO/ForkPoolWorker-1] Load origin 'pserver://anonymous:anonymous@a.cvs.sourceforge.net/cvsroot/mx4j/mx4j' with type 'cvs'
    swh-loader_1 | Request: BEGIN AUTH REQUEST
    swh-loader_1 | /cvsroot/mx4j
    swh-loader_1 | anonymous
    ...
    • Feb 15 2022, 2:46 PM
    • 22 Lines
    Author: anlambert
  • P1290 last 5 deposit in staging
    $ psql service=staging-swh-deposit -c '\x' -c 'select now(), * from deposit order by id desc limit 5;'
    ```
    +-------------------------------+-----+-------------------------------+-------------------------------+------------------------------------------+----------------------------------------------------+--------+-----------+---------------+-----------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+--------------+---------------------------------------------------------------------------+
    | now | id | reception_date | complete_date | external_id | swhid | status | client_id | collection_id | parent_id | status_detail | swhid_context | check_task_id | load_task_id | origin_url |
    ...
    • Feb 15 2022, 10:33 AM
    • 16 Lines
    Author: ardumont
  • P1289 new sourceforge bzr origins
    10:04:07 swh-scheduler@db1:5432=> select first_seen, url from listed_origins where visit_type='bzr' and url like 'http://%' order by first_seen desc;
    select first_seen, url from listed_origins where visit_type='bzr' and url like 'http://%' order by first_seen desc;
    +-------------------------------+--------------------------------------------------------------------+
    | first_seen | url |
    +-------------------------------+--------------------------------------------------------------------+
    ...
    • Feb 15 2022, 10:05 AM
    • 83 Lines
    Author: ardumont
  • P1288 deps build diff
    diff ok.org ko.org
    9c9
    < charset-normalizer==2.0.11
    ---
    > charset-normalizer==2.0.12
    ...
    • Feb 14 2022, 3:52 PM
    • 25 Lines
    Author: ardumont
  • P1287 oh come on...
    dpkg: error processing archive /var/cache/apt/archives/python3-swh.loader.bzr_1.1.0-1~swh2~bpo10+1_all.deb (--unpack):
    trying to overwrite '/usr/lib/python3/dist-packages/swh/loader/__init__.py', which is also in package python3-swh.loader.core 2.5.0-1~swh1~bpo10+1
    Errors were encountered while processing:
    /var/cache/apt/archives/python3-swh.loader.bzr_1.1.0-1~swh2~bpo10+1_all.deb
    • Feb 11 2022, 5:43 PM
    • 4 Lines
    Author: ardumont
  • P1286 sbuild python3-breezy
    sbuild --nolog --batch --no-clean-source --no-run-lintian --arch-all --source --build-dep-resolver=aptitude --extra-repository='deb http://incoming.debian.org/debian-buildd/ buildd-unstable main' --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'
    dpkg-source: info: using source format '3.0 (quilt)'
    dpkg-source: info: building breezy using existing ./breezy_3.2.1+bzr7585.orig.tar.gz
    dpkg-source: warning: upstream signing key but no upstream tarball signature
    dpkg-source: info: using patch list from debian/patches/series
    ...
    • Feb 11 2022, 5:14 PM
    • 49,805 Lines
    Author: ardumont
  • P1285 keycloak error
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: 13:46:52,362 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-109) Uncaught server error: org.keycloak.authentication.AuthenticationFlowException: authenticator: direct-grant-validate-password
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.DefaultAuthenticationFlow.processSingleFlowExecutionModel(DefaultAuthenticationFlow.java:423)
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:247)
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:976)
    Feb 11 13:46:52 kelvingrove keycloak[1436401]: at org.keycloak.keycloak-services@10.0.2//org.keycloak.protocol.oidc.endpoints.TokenEndpoint.resourceOwnerPasswordCredentialsGrant(TokenEndpoint.java:592)
    ...
    • Feb 11 2022, 2:51 PM
    • 77 Lines
    Author: vsellier
  • P1284 swh auth error
    $ swh auth --client-id swh-web generate-token vlorentz
    Password:
    Traceback (most recent call last):
    File "/home/dev/.local/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core', 'console_scripts', 'swh')())
    ...
    • Feb 11 2022, 2:43 PM
    • 30 Lines
    Author: vlorentz
  • P1283 origin_visit_find_latest_status_by_date
    def origin_visit_find_latest_status_by_date(
    origin_url: str, visit_date: datetime.datetime
    ) -> Optional[OriginVisitInfo]:
    """Retrieve origin visit status whose date is most recent than the provided visit_date.
    ...
    • Feb 10 2022, 9:22 AM
    • 30 Lines
    • Python
    Author: ardumont
  • P1282 SwhGraph dumped properties
    0: swh:1:dir:d24c739c57c68a5cfa7a0042bfbd796f875d1e0c
    successors:
    swh:1:dir:bd1898f4ea14c895bb34219c5495f1c9840eab2a asm [perms: 16384]
    swh:1:dir:294d460f1960619445897b62d8aa0bddb6d81f40 uapi [perms: 16384]
    ...
    • Feb 9 2022, 4:05 PM
    • 51 Lines
    Author: seirl
  • P1281 DumpProperties.java
    package org.softwareheritage.graph.utils;
    import it.unimi.dsi.big.webgraph.NodeIterator;
    import org.softwareheritage.graph.SwhUnidirectionalGraph;
    import org.softwareheritage.graph.labels.DirEntry;
    ...
    • Feb 9 2022, 3:58 PM
    • 77 Lines
    • Java
    Author: seirl
  • P1280 docker run says no to naive datetime ¯\_(ツ)_/¯
    swh-lister_1 | [2022-02-08 10:29:25,384: INFO/MainProcess] Task swh.lister.maven.tasks.FullMavenLister[95fa1dcb-6fea-41b7-82e3-e07162d45f2d] received
    swh-lister_1 | [2022-02-08 10:29:45,424: INFO/ForkPoolWorker-1] Downloading text index from http://swh-lister-maven-nginx/export.fld.
    swh-lister_1 | [2022-02-08 10:30:05,455: ERROR/ForkPoolWorker-1] Task swh.lister.maven.tasks.FullMavenLister[95fa1dcb-6fea-41b7-82e3-e07162d45f2d] raised unexpected: ValueError("can not serialize 'datetime.datetime' object where tzinfo=None")
    swh-lister_1 | Traceback (most recent call last):
    swh-lister_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 450, in trace_task
    ...
    • Feb 8 2022, 11:31 AM
    • 37 Lines
    Author: ardumont
  • P1279 DumpProperties.java
    public class DumpProperties {
    public static void main(String[] args) throws IOException {
    String graphPath = args[0];
    SwhBidirectionalGraph graph = SwhBidirectionalGraph.load(graphPath);
    ...
    • Feb 7 2022, 5:56 PM
    • 50 Lines
    Author: seirl
  • P1278 bzr sourceforge origins
    17:49:20 swh-scheduler@db1:5432=> select now(), count(*) from listed_origins where visit_type='bzr';
    select now(), count(*) from listed_origins where visit_type='bzr';
    +-------------------------------+-------+
    | now | count |
    +-------------------------------+-------+
    ...
    • Feb 7 2022, 5:50 PM
    • 309 Lines
    Author: ardumont
  • P1276 swh.storage.interface new endpoint proposal
    ```
    @remote_api_endpoint("origin/visit_status/find_by_date")
    def origin_visit_status_find_by_date(
    self, origin: str, visit_date: datetime.datetime
    ) -> Optional[OriginVisit]:
    ...
    • Feb 4 2022, 3:55 PM
    • 18 Lines
    Author: ardumont
  • P1275 (An Untitled Masterwork)
    diff --git a/swh/web/common/origin_visits.py b/swh/web/common/origin_visits.py
    index 6dc2201a..023c7aa8 100644
    --- a/swh/web/common/origin_visits.py
    +++ b/swh/web/common/origin_visits.py
    @@ -42,43 +42,45 @@ def get_origin_visits(origin_info: OriginInfo) -> List[OriginVisitInfo]:
    ...
    • Feb 3 2022, 6:07 PM
    • 96 Lines
    Author: anlambert
  • P1274 (An Untitled Masterwork)
    diff --git a/swh/web/common/archive.py b/swh/web/common/archive.py
    index e9e877f6..dca07e30 100644
    --- a/swh/web/common/archive.py
    +++ b/swh/web/common/archive.py
    @@ -703,7 +703,7 @@ def lookup_revision_log(rev_sha1_git, limit):
    ...
    • Feb 3 2022, 3:56 PM
    • 13 Lines
    Author: anlambert
  • P1272 subsequent cache queries are fine
    time curl https://webapp1.internal.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/
    [{"origin_url":"https://foss.heptapod.net/mercurial/hgview","visit_type":"hg","save_request_date":"2022-02-02T10:15:01.448199+00:00","save_request_status":"accepted","save_task_status":"failed","visit_status":"failed","visit_date":"2022-02-02T10:15:17.559967+00:00","loading_task_id":405935267,"note":null},{"origin_url":"https://foss.heptapod.net/mercurial/hgview","visit_type":"hg","save_request_date":"2022-02-02T09:48:23.326128+00:00","save_request_status":"accepted","save_task_status":"succeeded","visit_status":"full","visit_date":"2022-02-02T09:48:36.296855+00:00","loading_task_id":405934604,"note":null},{"origin_url":"https://foss.heptapod.net/mercurial/hgview","visit_type":"hg","save_request_date":"2022-02-02T09:44:36.829049+00:00","save_request_status":"accepted","save_task_status":"succeeded","visit_status":"full","visit_date":"2022-02-02T09:44:46.020548+00:00","loading_task_id"...
    • Feb 2 2022, 11:36 AM
    • 2 Lines
    Author: ardumont
  • P1271 scn timeouts
    $ varnishncsa -F '%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i" %D' | grep -E " [0-9]{7,}$"
    ...
    128.93.166.14 - - [02/Feb/2022:09:35:11 +0000] "GET http://archive.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/ HTTP/1.1" 502 341 "-" "python-requests/2.21.0" 60058264
    128.93.166.14 - - [02/Feb/2022:09:41:16 +0000] "POST http://archive.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/ HTTP/1.1" 502 341 "-" "python-requests/2.21.0" 60057275
    128.93.166.14 - - [02/Feb/2022:09:44:36 +0000] "POST http://archive.softwareheritage.org/api/1/origin/save/hg/url/https://foss.heptapod.net/mercurial/hgview/ HTTP/1.1" 502 341 "-" "python-requests/2.21.0" 60060106
    ...
    • Feb 2 2022, 11:04 AM
    • 6 Lines
    Author: vsellier
  • P1270 "http call as user nobody" says no
    root@pergamon:~# sudo -u nobody /bin/bash
    nobody@pergamon:/root$ chronic /usr/local/bin/weekly-planning-bot
    + DEST=ardumont@softwareheritage.org
    + AUTHOR=swh-team@inria.fr
    + TEMPLATE_ID=6YKT5osoST-amJQ0MusH2Q
    ...
    • Jan 31 2022, 11:40 AM
    • 103 Lines
    Author: ardumont
  • P1269 quick and dirty postgresql backup script based on zfs snapshot
    root@dali:~# cat backup.sh
    #!/bin/bash
    i=0
    dataset=data/postgresql
    ...
    • Jan 27 2022, 10:25 AM
    • 20 Lines
    • Bash Scripting
    Author: vsellier
  • P1268 snap' dat' vm
    $ sudo virsh list
    Id Name State
    ----------------------------------------------
    2 puppet-environment_pergamon running
    4 puppet-environment_riverside running
    ...
    • Jan 27 2022, 10:15 AM
    • 15 Lines
    Author: ardumont
  • P1267 Command-Line Input
    use anyhow::Result;
    use tempfile::TempDir;
    use std::{
    env, io,
    ...
    • Jan 26 2022, 9:46 PM
    • 377 Lines
    • rust
    Author: olasd
  • P1265 git smart strace
    write(4, "0032have 16617535684faf9de30620de83667214297a36b8\n0032have ee8f3955c0ef4c7a90a5f707c37c4cb3711ff626\n"..., 1604) = 1604
    read(3, "0008", 4) = 4
    read(3, "NAK\n", 4) = 4
    brk(0x5558db682000) = 0x5558db682000
    write(4, "0032have 1e1490a38504419e349caa1b7d55d5c141a9bccb\n0032have 90e94b160c7f647ddffda707f5e3c0c66c170df8\n"..., 1604) = 1604
    ...
    • Jan 26 2022, 2:42 PM
    • 81 Lines
    Author: vlorentz
First
Prev
Next
About · Developer information · Legal