Page MenuHomeSoftware Heritage
  • dev-dart@desktop5:~/pub-dev/app$ git diff
    diff --git a/app/lib/frontend/handlers/custom_api.dart b/app/lib/frontend/handlers/custom_api.dart
    index 8342cd1d..4d5bf622 100644
    --- a/app/lib/frontend/handlers/custom_api.dart
    +++ b/app/lib/frontend/handlers/custom_api.dart
    ...
    • Aug 31 2022, 1:02 PM
    • 41 Lines
    • Diff
  • $ zstdcat ../../../2021-03-23/license-blobs.csv.zst | grep 7ca4503e91c6be4be3afc8a64972ef38f9c7d034
    swh:1:cnt:a9d56615c3fc82b7a5b2ac8a9ecbe70007010f7a,7ca4503e91c6be4be3afc8a64972ef38f9c7d034,LICENSE
    swh:1:cnt:a9d56615c3fc82b7a5b2ac8a9ecbe70007010f7a,7ca4503e91c6be4be3afc8a64972ef38f9c7d034,LICENSE.md
    $ zstdcat ../../../2022-04-25/license-blobs.csv.zst | grep 7ca4503e91c6be4be3afc8a64972ef38f9c7d034
    swh:1:cnt:a9d56615c3fc82b7a5b2ac8a9ecbe70007010f7a,7ca4503e91c6be4be3afc8a64972ef38f9c7d034,LICENSE
    ...
    • Aug 31 2022, 11:37 AM
    • 7 Lines
  • ~/.../sysadm-provisioning/proxmox/terraform/production master ❯ terraform plan -destroy -target rancher2_cluster.production_cassandra
    rancher2_cluster.production_cassandra: Refreshing state... [id=c-pcndd]
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
    - destroy
    ...
    • Aug 30 2022, 3:40 PM
    • 516 Lines
  • (swh) ✔ ~/swh/swh-environment/swh-storage [master ↓·1|✚ 1⚑ 17]
    14:06 $ git diff -U13
    diff --git a/swh/storage/sql/40-funcs.sql b/swh/storage/sql/40-funcs.sql
    index 626f0b3d..ce3a298d 100644
    --- a/swh/storage/sql/40-funcs.sql
    ...
    • Aug 30 2022, 2:06 PM
    • 43 Lines
  • import time
    import grpc
    from swh.graph.rpc import swhgraph_pb2, swhgraph_pb2_grpc
    with grpc.insecure_channel("granet.internal.softwareheritage.org:50091") as c:
    ...
    • Aug 29 2022, 6:06 PM
    • 19 Lines
  • terraform destroy -target=rancher2_cluster.cluster-graphql3
    rancher2_cluster.cluster-graphql3: Refreshing state... [id=c-9mqhw]
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
    - destroy
    ...
    • Aug 26 2022, 12:35 PM
    • 657 Lines
  • ```
    $ terraform destroy -target=rancher2_cluster.deployment_internship
    rancher2_cluster.deployment_internship: Refreshing state... [id=c-fvnrx]
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
    ...
    • Aug 23 2022, 5:04 PM
    • 560 Lines
  • (swh) ✔ ~/swh/swh-environment/swh-scheduler/sql [master ↑·1|⚑ 3]
    15:23 $ ls ../swh/scheduler/sql/*.sql
    ../swh/scheduler/sql/10-superuser-init.sql ../swh/scheduler/sql/30-schema.sql ../swh/scheduler/sql/40-func.sql ../swh/scheduler/sql/50-data.sql ../swh/scheduler/sql/60-indexes.sql
    • Aug 22 2022, 3:25 PM
    • 3 Lines
  • from typing import Dict, List, Optional, Union
    from rdflib import URIRef, XSD, Literal, Graph, BNode
    from swh.indexer.codemeta import CROSSWALK_TABLE, add_list
    ...
    • Aug 19 2022, 4:32 PM
    • 61 Lines
    • Python
  • query RepoFiles {
    repository(owner: "SoftwareHeritage", name: "swh-web") {
    object(expression: "HEAD:") {
    # Top-level.
    ... on Tree {
    ...
    • Aug 18 2022, 1:44 PM
    • 44 Lines
  • query RepoFiles {
    repository(owner: "SoftwareHeritage", name: "swh-web") {
    object(expression: "master:") {
    ... on Tree {
    entries {
    ...
    • Aug 18 2022, 1:17 PM
    • 18 Lines
  • (swh) ben@ben-deb:~/swh-environment/swh-web(plop *+)$ mypy .
    swh/web/common/typing.py:10: error: Cannot find implementation or library stub for module named "swh.core.api.classes"
    swh/web/common/typing.py:11: error: Cannot find implementation or library stub for module named "swh.model.swhids"
    swh/web/config.py:9: error: Cannot find implementation or library stub for module named "swh.core"
    swh/web/config.py:10: error: Cannot find implementation or library stub for module named "swh.counters"
    ...
    • Aug 18 2022, 11:20 AM
    • 16 Lines
  • 14:56 $ make test
    python3 -m pytest --hypothesis-profile=swh-web-fast ./swh/web/tests
    Traceback (most recent call last):
    File "/home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/pytest_django/plugin.py", line 179, in _handle_import_error
    yield
    ...
    • Aug 17 2022, 2:58 PM
    • 79 Lines
  • ben@ben-deb:~/swh-environment(master)$ workon swh
    WARNING:swh.core.cli:Could not load subcommand identify: ModuleNotFoundError("No module named 'swh.model.swhids'")
    WARNING:swh.core.cli:Could not load subcommand lister: ImportError("cannot import name 'remote_api_endpoint' from 'swh.core.api' (unknown location)")
    WARNING:swh.core.cli:Could not load subcommand scheduler: ImportError("cannot import name 'remote_api_endpoint' from 'swh.core.api' (unknown location)")
    WARNING:swh.core.cli:Could not load subcommand deposit: ModuleNotFoundError("No module named 'swh.model.swhids'")
    ...
    • Aug 17 2022, 2:16 PM
    • 7 Lines
  • (swh) ✔ ~/swh/swh-environment/swh-web [add-forge-now-url-validation L|⚑ 150]
    13:58 $ ls ~/.virtualenvs/swh/lib/python3.9/site-packages/swh*
    /home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/swh.auth.egg-link /home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/swh.indexer.egg-link /home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/swh.objstorage.egg-link
    /home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/swh.core.egg-link /home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/swh.journal.egg-link /home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/swh.objstorage.replayer.egg-link
    /home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/swh.counters.egg-link /home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/swh.lister.egg-link /home/anlambert/.virtualenvs/swh/lib/python3.9/site-packages/swh.perfecthash.egg-link
    ...
    • Aug 17 2022, 2:02 PM
    • 11 Lines
  • (swh) ben@ben-deb:~/swh-environment(master)$ pip freeze | grep swh
    -e git+https://forge.softwareheritage.org/source/swh-auth.git@6c85751c2be061a890605e0a6256548f24bcc20f#egg=swh.auth
    -e git+https://forge.softwareheritage.org/source/swh-core.git@a0e61ab86973f5df3054bd7b343daf59a03415d0#egg=swh.core
    -e git+https://forge.softwareheritage.org/source/swh-counters.git@9d5a710268a1346a93b7c54ba3529b3afcde57c5#egg=swh.counters
    -e git+https://forge.softwareheritage.org/source/swh-dataset.git@68f9bd2028b6be0674a29c4660dfd67edd3c4acf#egg=swh.dataset
    ...
    • Aug 17 2022, 1:54 PM
    • 30 Lines
  • (swh) ben@ben-deb:~/swh-environment/swh-web(rename-denied-status *)$ pytest swh/web/tests/add_forge_now/test_models.py::test_allowed_next_statuses
    Traceback (most recent call last):
    File "/home/ben/.virtualenvs/swh/bin/pytest", line 8, in <module>
    sys.exit(console_main())
    File "/home/ben/.virtualenvs/swh/lib/python3.9/site-packages/_pytest/config/__init__.py", line 187, in console_main
    ...
    • Aug 17 2022, 1:00 PM
    • 47 Lines
  • (swh) ben@ben-deb:~/swh-environment/swh-objstorage(master *)$ pip install -e .
    Obtaining file:///home/ben/swh-environment/swh-objstorage
    Installing build dependencies ... done
    Checking if build backend supports build_editable ... done
    Getting requirements to build editable ... done
    ...
    • Aug 17 2022, 12:58 PM
    • 101 Lines
  • P1422 test ko
    (swh) ben@ben-deb:~/swh-environment/swh-web(rename-denied-status *)$ pytest swh/web/tests/add_forge_now/test_models.py::test_allowed_next_statuses
    ========================================================== test session starts ==========================================================
    platform linux -- Python 3.9.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
    django: settings: swh.web.settings.tests (from ini)
    rootdir: /home/ben/swh-environment/swh-web, configfile: pytest.ini
    ...
    • Aug 17 2022, 12:00 PM
    • 129 Lines
  • 15:36 $ curl "https://gitlab.com/api/v4/projects?id_after=105227&imported=false&last_activity_after=2022-07-01T00%3A00%3A00%2B00%3A00&membership=false&order_by=id&owned=false&page=1&pagination=keyset&per_page=50&repository_checksum_failed=false&simple=false&sort=asc&starred=false&statistics=false&wiki_checksum_failed=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false" | jq '.[] | "\(.id) \(.last_activity_at)"'
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 44816 0 44816 0 0 61475 0 --:--:-- --:--:-- --:--:-- 61475
    "105308 2022-08-04T04:38:44.891Z"
    ...
    • Aug 11 2022, 3:39 PM
    • 110 Lines
  • 15:17 $ curl -i "https://gitlab.com/api/v4/projects?pagination=keyset&per_page=50&order_by=id&sort=asc&last_activity_after=2022-07-01T00:00:00Z"
    HTTP/2 200
    date: Thu, 11 Aug 2022 13:18:00 GMT
    content-type: application/json
    cache-control: max-age=0, private, must-revalidate
    ...
    • Aug 11 2022, 3:21 PM
    • 30 Lines
  • The test:
    ```
    def test_directory_metadata_indexer_single_root_dir(self):
    metadata_indexer = DirectoryMetadataIndexer(config=DIRECTORY_METADATA_CONFIG)
    fill_obj_storage(metadata_indexer.objstorage)
    ...
    • Aug 6 2022, 7:20 AM
    • 98 Lines
  • 2022-08-05T13:54:55.399239842Z swhtest_mirror0_vault.1.3pezvycvogx9@libra | ERROR:root:git_bare swh:1:rev:81d10c7544d07429ab97dc296415ba3ae0371855 was not found.
    2022-08-05T13:54:55.399352171Z swhtest_mirror0_vault.1.3pezvycvogx9@libra | Traceback (most recent call last):
    2022-08-05T13:54:55.399381061Z swhtest_mirror0_vault.1.3pezvycvogx9@libra | File "/usr/lib/python3/dist-packages/flask/app.py", line 1813, in full_dispatch_request
    2022-08-05T13:54:55.399401207Z swhtest_mirror0_vault.1.3pezvycvogx9@libra | rv = self.dispatch_request()
    ...
    • Aug 5 2022, 4:23 PM
    • 20 Lines
  • Configuration Summary
    Description Part Number
    System Localization ABF - France - French Localization
    OS Localization ABF - France - French Localization
    ...
    • Aug 5 2022, 12:11 PM
    • 32 Lines
  • package org.softwareheritage.graph.utils;
    import it.unimi.dsi.big.webgraph.LazyLongIterator;
    import java.util.Random;
    ...
    • Aug 4 2022, 11:44 PM
    • 132 Lines
    • Java
  • apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
    name: argocd-server
    ...
    • Jul 28 2022, 3:58 PM
    • 28 Lines
  • packer build debian_bullseye_zfs.json
    proxmox: output will be in this color.
    ==> proxmox: Retrieving ISO
    ==> proxmox: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.4.0-amd64-netinst.iso
    ...
    • Jul 27 2022, 2:49 PM
    • 1,654 Lines
  • packer build debian_bullseye.json
    proxmox: output will be in this color.
    ==> proxmox: Retrieving ISO
    ==> proxmox: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.4.0-amd64-netinst.iso
    ...
    • Jul 27 2022, 2:32 PM
    • 1,171 Lines
  • kubectl get all -A
    NAMESPACE NAME READY STATUS RESTARTS AGE
    cattle-fleet-system pod/fleet-agent-96f6f455c-qhtwd 1/1 Running 68 3d14h
    cattle-system pod/cattle-cluster-agent-7c5484cbb7-rwsmh 1/1 Running 64 (149m ago) 3d15h
    cattle-system pod/cattle-node-agent-4wvwv 1/1 Running 0 3d15h
    ...
    • Jul 25 2022, 11:19 AM
    • 54 Lines
  • swh/indexer/tests/test_cli.py:791:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    /home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/testing.py:408: in invoke
    return_value = cli.main(args=args or (), prog_name=prog_name, **extra)
    /home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py:1055: in main
    ...
    • Jul 21 2022, 6:50 PM
    • 38 Lines
  • swh/indexer/tests/utils.py:712:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    swh/indexer/tests/utils.py:702: in assert_results_ok
    self.assertEqual(expected_results, actual_results)
    E AssertionError: Lists differ: [Cont[48 chars]ol={'configuration': {'type': 'library', 'debi[839 chars]='')] != [Cont[48 chars]ol={'id': -7322148152319774267, 'name': 'file'[846 chars]='')]
    ...
    • Jul 21 2022, 4:26 PM
    • 17 Lines
  • Collecting grpcio-tools (from swh.graph)
    Downloading https://files.pythonhosted.org/packages/44/79/3a5e1290fce95fcd61d0f380e542e372e91f6ef159ae71e746530116c90d/grpcio-tools-1.47.0.tar.gz (2.2MB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    ...
    • Jul 20 2022, 11:49 AM
    • 18 Lines
  • 10:30:57 softwareheritage-scheduler@belvedere:5432=> select now(), status, count(*) from task where type = 'index-origin-metadata' group by status;
    +-------------------------------+------------------------+---------+
    | now | status | count |
    +-------------------------------+------------------------+---------+
    | 2022-07-19 08:35:10.207831+00 | next_run_not_scheduled | 9804975 |
    ...
    • Jul 19 2022, 10:36 AM
    • 12 Lines
  • swhworker@worker17:~$ url=https://github.com/liferay/liferay-portal; /usr/bin/time -v swh loader run git $url lister_name=github lister_instance_name=github pack_size_bytes=34359738368 | tee liferay-portal-pack-size-limit-32g.txt
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/liferay/liferay-portal' with type 'git'
    Enumerating objects: 14950278, done.
    Counting objects: 100% (647/647), done.
    Compressing objects: 100% (468/468), done.
    ...
    • Jul 13 2022, 2:05 PM
    • 31 Lines
  • ```
    |----------+---------+-------+-------------------+----------------+-----------------+-----------------+----------------+-----------------|
    | Instance | vCPU(s) | RAM | Temporary storage | Pay as you go | 1 year reserved | 3 year reserved | Spot | Add to estimate |
    |----------+---------+-------+-------------------+----------------+-----------------+-----------------+----------------+-----------------|
    | DS2 v2 | 2 | 7 GiB | 14 GiB | €94.4220/month | €57.1461/month | €39.1504/month | €21.5095/month | |
    ...
    • Jul 11 2022, 4:49 PM
    • 27 Lines
  • 16:38:14 softwareheritage-scheduler@belvedere:5432=> select now(), status, count(*) from task where type = 'index-fossology-license-partition' and status != 'completed' group by status;
    +-----+--------+-------+
    | now | status | count |
    +-----+--------+-------+
    +-----+--------+-------+
    ...
    • Jul 11 2022, 4:39 PM
    • 17 Lines
  • objstorage:
    cls: azure-prefixed
    accounts:
    '0':
    account_name: <account-0>
    ...
    • Jul 7 2022, 1:46 PM
    • 67 Lines
  • softwareheritage-indexer=> select * from indexer_configuration;
    id | tool_name | tool_version | tool_configuration
    -----------+-------------------------+-----------------------+----------------------------------------------------------------------------------------------
    1 | nomos | 3.1.0rc2-31-ga2cbb8c | {"command_line": "nomossa <filepath>"}
    5 | universal-ctags | ~git7859817b | {"command_line": "ctags --fields=+lnz --sort=no --links=no --output-format=json <filepath>"}
    ...
    • Jul 6 2022, 4:00 PM
    • 24 Lines
  • ```
    14:10:10 swh-deposit@db1:5432=> select * from deposit_client dc inner join auth_user u on user_ptr_id=u.id where u.username='hal';
    +-[ RECORD 1 ]-+--------------------------------------------------------------------------------+
    | user_ptr_id | 2 |
    | collections | {1} |
    ...
    • Jul 6 2022, 2:11 PM
    • 22 Lines
  • -- number of origin metadata, per domain
    softwareheritage=> select (select url from metadata_authority where metadata_authority.id=authority_id), count(*) from raw_extrinsic_metadata where target > 'swh:1:ori:' and target < 'swh:1:orz' group by authority_id;
    url | count
    ------------------------------------------------+---------
    ...
    • Jul 5 2022, 11:47 AM
    • 29 Lines
  • diff --git a/swh/indexer/metadata_dictionary/cff.py b/swh/indexer/metadata_dictionary/cff.py
    index 48be831..b278e5c 100644
    --- a/swh/indexer/metadata_dictionary/cff.py
    +++ b/swh/indexer/metadata_dictionary/cff.py
    @@ -38,19 +38,20 @@ def normalize_authors(self, d: List[dict]) -> Dict[str, list]:
    ...
    • Jul 5 2022, 10:30 AM
    • 31 Lines
    • Diff
  • diff --git a/swh/indexer/metadata_dictionary/cff.py b/swh/indexer/metadata_dictionary/cff.py
    index 48be831..731cab1 100644
    --- a/swh/indexer/metadata_dictionary/cff.py
    +++ b/swh/indexer/metadata_dictionary/cff.py
    @@ -20,6 +20,24 @@ class CffMapping(DictMapping, SingleFileIntrinsicMapping):
    ...
    • Jul 4 2022, 8:28 PM
    • 56 Lines
    • Diff
  • From 1e8c3d694e2775afd01d80031765f919603eef10 Mon Sep 17 00:00:00 2001
    From: Antoine Pietri <antoine.pietri1@gmail.com>
    Date: Thu, 30 Jun 2022 15:47:06 +0200
    Subject: [PATCH] docs: fix relative include of swhgraph.proto
    ...
    • Jun 30 2022, 3:48 PM
    • 33 Lines
    • Diff
  • grep -v ".zip" guix-sources.json | grep -v ".jar\|.7z\|.gem\|.gz\|.tgz\|.tar.\|.tbz\|.tgz\|output\|integrity\|urls\|inferred\|url\|{\|}\|\]\|\[\|git_ref\|"git"\|hg\|no-origin\|false\|svn_revision\|name\|"svn"\|crates"
    "https://common-lisp.net/project/asdf/archives/asdf-3.3.5.lisp"
    "https://linux-libre.fsfla.org/pub/linux-libre/releases/5.17.14-gnu/deblob-check"
    "https://linux-libre.fsfla.org/pub/linux-libre/releases/5.17.14-gnu/deblob-5.17"
    "http://mccarrolllab.com/download/1276/"
    ...
    • Jun 30 2022, 3:19 PM
    • 139 Lines
  • swh_graph-replayer.11.vlqkq2mupe8t@shsrvc2 | ERROR:swh.storage.replay:Object has id a03d79c816c7ffebe3e8d90d7d395482075b086f, but it should be c8c1462ff14300cc081475b0d97da679eab5a8ff: Release(name=b'debian/0.26.1-0ubuntu2.2', message=b'tagging package python-glance-store version debian/0.26.1-0ubuntu2.2\n', target=hash_to_bytes('e4c28d0c01ea1274e8e9b28f113572c36211d538'), target_type=ObjectType.REVISION, synthetic=False, author=Person(fullname=b'Corey Bryant <corey.bryant@canonical.com>', name=b'Corey Bryant', email=b'corey.bryant@canonical.com'), date=TimestampWithTimezone(timestamp=Timestamp(seconds=1560178555, microseconds=0), offset_bytes=b'-0400'), metadata=None, id=hash_to_bytes('a03d79c816c7ffebe3e8d90d7d395482075b086f'), raw_manifest=None)
    • Jun 30 2022, 1:57 PM
    • 1 Line
  • privileged: true
    object_types:
    - content
    - skipped_content
    - directory
    ...
    • Jun 29 2022, 5:32 PM
    • 20 Lines
  • {
    "license": [
    "LGPL-2.1-only",
    "GPL-3.0-or-later"
    ]
    ...
    • Jun 28 2022, 4:26 PM
    • 12 Lines
    • JSON
  • terraform apply
    module.worker18.proxmox_vm_qemu.node: Refreshing state... [id=uffizi/qemu/141]
    module.kelvingrove.proxmox_vm_qemu.node: Refreshing state... [id=branly/qemu/123]
    module.provenance-client01.proxmox_vm_qemu.node: Refreshing state... [id=uffizi/qemu/126]
    module.counters1.proxmox_vm_qemu.node: Refreshing state... [id=branly/qemu/139]
    ...
    • Jun 27 2022, 4:37 PM
    • 3,788 Lines
  • $ terraform apply
    module.webapp1.proxmox_vm_qemu.node: Refreshing state... [id=hypervisor3/qemu/125]
    module.counters1.proxmox_vm_qemu.node: Refreshing state... [id=branly/qemu/139]
    module.search1.proxmox_vm_qemu.node: Refreshing state... [id=branly/qemu/136]
    module.worker17.proxmox_vm_qemu.node: Refreshing state... [id=uffizi/qemu/143]
    ...
    • Jun 27 2022, 3:38 PM
    • 139 Lines
  • root@getty:~# /usr/local/sbin/create_kafka_users_rocquencourt.sh snyk-prod-01
    Creating user snyk-prod-01, with unprivileged access to consumer group prefix snyk-prod-01-
    Password for user snyk-prod-01:
    Setting user credentials
    Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
    ...
    • Jun 27 2022, 2:14 PM
    • 15 Lines
  • scheduler:
    cls: remote
    url: http://swh-scheduler:5008/
    vault:
    ...
    • Jun 21 2022, 2:31 PM
    • 9 Lines
  • anlambert@carnavalet:/tmp$ cvs -z3 -d:pserver:anonymous@a.cvs.sourceforge.net:/cvsroot/javanms co -P jnms-area-service
    cvs checkout: Updating jnms-area-service
    U jnms-area-service/.classpath
    U jnms-area-service/.packaging
    U jnms-area-service/.project
    ...
    • Jun 20 2022, 3:18 PM
    • 234 Lines
  • (defun create-user (user)
    (when-unmapped-with-update (:user (forgerie-core:user-username user))
    (let*
    ((avatar (forgerie-core:user-avatar user))
    (avatar
    ...
    • Jun 18 2022, 4:08 PM
    • 75 Lines
  • def normalize_description(self, description):
    r"""Try to re-decode ``description`` as UTF-16, as this is a somewhat common
    mistake that causes issues in the database because of null bytes in JSON.
    >>> NpmMapping().normalize_description("foo bar")
    ...
    • Jun 17 2022, 6:05 PM
    • 71 Lines
    • Python
  • Metadata on Social Code Repositories
    ====================================
    MeSoCoRe is a vocabulary which complements ontologies like schema.org/CodeMeta
    and DOAP in describing software projects. While the latter are meant to describe
    ...
    • Jun 16 2022, 6:00 PM
    • 77 Lines
  • 17:08 $ swh counters -C ~/.config/swh/counters.yml journal-client
    {'brokers': ['kafka1.internal.softwareheritage.org:9094', 'kafka2.internal.softwareheritage.org:9094', 'kafka3.internal.softwareheritage.org:9094', 'kafka4.internal.softwareheritage.org:9094'], 'prefix': 'swh.journal.objects', 'group_id': 'anlambert.origin_counts.dev4', 'object_types': ['origin'], 'batch_size': 1000, 'sasl.mechanism': 'SCRAM-SHA-512', 'security.protocol': 'SASL_SSL', 'sasl.username': 'mirror-test-ro', 'sasl.password': '**********', 'stop_after_objects': None}
    Traceback (most recent call last):
    File "/home/anlambert/.virtualenvs/swh/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core', 'console_scripts', 'swh')())
    ...
    • Jun 16 2022, 5:11 PM
    • 26 Lines
  • (swh) anlambert@carnavalet:/tmp$ bzr branch lp:~jelmer/bzr-git/dulwich
    Branched 236 revisions.
    (swh) anlambert@carnavalet:/tmp$ cd dulwich/
    (swh) anlambert@carnavalet:/tmp/dulwich$ brz tags
    bzr-svn-0.1 ?
    ...
    • Jun 14 2022, 3:11 PM
    • 26 Lines
  • from laptop ok (yavin4):
    ```
    kubectl exec -ti -n gitlab-system deployment/gitlab-toolbox -- /srv/gitlab/bin/rails console
    Defaulted container "toolbox" out of: toolbox, certificates (init), configure (init)
    ...
    • Jun 13 2022, 3:42 PM
    • 51 Lines
  • morane@hplaptopft0:~/Documents/work/swh/presentations/slides$ git push
    Username for 'https://forge.softwareheritage.org': moranegg
    Password for 'https://moranegg@forge.softwareheritage.org':
    Enumerating objects: 9, done.
    Counting objects: 100% (9/9), done.
    ...
    • Jun 8 2022, 11:12 PM
    • 13 Lines
  • journal_client:
    cls: kafka
    brokers:
    - broker1.journal.staging.swh.network:9093
    group_id: {group_id}
    ...
    • Jun 7 2022, 1:29 PM
    • 30 Lines
  • {
    "tar.gz": 124760,
    "tar.xz": 23499,
    "tar.bz2": 10652,
    "git": 6847,
    ...
    • Jun 6 2022, 8:34 PM
    • 56 Lines
    • JSON
  • 16:04:03 swh-scheduler@db1:5432=> with maven_lister_ids as (
    select id from listers where name='maven'
    ) select now(), visit_type, url
    from
    listed_origins lo1
    ...
    • Jun 3 2022, 4:12 PM
    • 1,910 Lines
  • swh/web/tests/api/views/test_metadata.py:81:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    swh/web/tests/conftest.py:1106: in inner
    item.ihook.pytest_runtest_call(item=item)
    ../../../.virtualenvs/swh/lib/python3.9/site-packages/pluggy/_hooks.py:265: in __call__
    ...
    • Jun 3 2022, 11:25 AM
    • 72 Lines
  • 11:00 $ pytest -sv --hypothesis-verbosity=verbose swh/web/tests/api/views/test_metadata.py::test_api_raw_extrinsic_metadata_scroll
    ================================================================================================================================== test session starts ==================================================================================================================================
    platform linux -- Python 3.9.2, pytest-7.1.2, pluggy-1.0.0 -- /home/anlambert/.virtualenvs/swh/bin/python
    cachedir: .pytest_cache
    hypothesis profile 'swh-web-fast-with-verbose-verbosity' -> deadline=None, max_examples=5, verbosity=Verbosity.verbose, database=DirectoryBasedExampleDatabase('/home/anlambert/swh/swh-environment/swh-web/.hypothesis/examples'), suppress_health_check=[HealthCheck.too_slow, HealthCheck.filter_too_much, HealthCheck.function_scoped_fixture]
    ...
    • Jun 3 2022, 11:02 AM
    • 102 Lines
  • 09:52:37 softwareheritage-scheduler@belvedere:5432=> select * from task_type where type like 'load-%' or type like 'list-%';
    +------------------------------+--------------------------------------------------------------+-----------------------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
    | type | description | backend_name | default_interval | min_interval | max_interval | backoff_factor | max_queue_length | num_retries | retry_delay |
    +------------------------------+--------------------------------------------------------------+-----------------------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
    | load-hg | Update an origin of type mercurial | swh.loader.mercurial.tasks.LoadMercurial | 64 days | 1 day | 64 days | 2 | 5000 | 3 | (null) |
    ...
    • Jun 3 2022, 9:55 AM
    • 55 Lines
  • 09:49:06 swh-scheduler@db1:5432=> select * from task_type where type like 'load-%';
    +-----------------------+--------------------------------------------------------------+----------------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
    | type | description | backend_name | default_interval | min_interval | max_interval | backoff_factor | max_queue_length | num_retries | retry_delay |
    +-----------------------+--------------------------------------------------------------+----------------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
    | load-svn-from-archive | Loading svn repositories from svn dump | swh.loader.svn.tasks.MountAndLoadSvnRepository | 1 day | 1 day | 1 day | 1 | 1000 | (null) | (null) |
    ...
    • Jun 3 2022, 9:52 AM
    • 26 Lines
  • $ docker-compose exec swh-idx-storage swh db --config-file /indexer_storage.yml upgrade indexer_storage
    ERROR:swh.core.db.db_utils:Failed to connect to `postgresql:///?service=swh-indexers`
    Traceback (most recent call last):
    File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/db/db_utils.py", line 241, in swh_db_module
    db = connect_to_conninfo(db_or_conninfo)
    ...
    • Jun 2 2022, 3:06 PM
    • 44 Lines
  • psql service=admin-staging-swh-scheduler
    Pager usage is off.
    Null display is "(null)".
    Timing is on.
    Border style is 2.
    ...
    • Jun 2 2022, 2:06 PM
    • 776 Lines
  • ```
    swhworker@worker0:~$ dpkg -l python3-swh.indexer | grep ii
    ii python3-swh.indexer 1.1.0-1~swh1~bpo10+1 all Software Heritage Content Indexerswhworker@worker0:~$ echo $SWH_CONFIG_FILENAME
    /etc/softwareheritage/indexer/journal_client.yml
    swhworker@worker0:~$ /usr/bin/swh indexer --config-file $SWH_CONFIG_FILENAME journal-client indexer '*'
    ...
    • Jun 1 2022, 4:43 PM
    • 19 Lines
  • 10:26:05 swh-scheduler@db1:5432=> with maven_lister_ids as (
    swh-scheduler(> select id from listers where name='maven'
    swh-scheduler(> ) select now(), visit_type, url
    swh-scheduler-> from
    swh-scheduler-> listed_origins lo1
    ...
    • Jun 1 2022, 10:46 AM
    • 28,051 Lines
  • ---
    layout: post
    title: "Software Heritage"
    description: "About Software Heritage, what we stand for, and where we are headed"
    date:
    ...
    • May 26 2022, 11:58 AM
    • 52 Lines
  • diff /tmp/tox-swh-web.now.txt /tmp/tox-swh-web-last-green.txt
    0a1,3
    > aiohttp==3.8.1
    > aiohttp-utils==3.1.1
    > aiosignal==1.2.0
    ...
    • May 25 2022, 5:33 PM
    • 123 Lines
  • [19/May/2022 08:29:58] [DEBUG] Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/rest_framework/views.py", line 483, in dispatch
    self.initial(request, *args, **kwargs)
    File "/usr/lib/python3/dist-packages/rest_framework/views.py", line 402, in initial
    self.check_throttles(request)
    ...
    • May 19 2022, 11:18 AM
    • 12 Lines
  • anlambert@carnavalet:/tmp$ python3
    Python 3.9.2 (default, Feb 28 2021, 17:03:44)
    [GCC 10.2.1 20210110] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import dulwich
    ...
    • May 17 2022, 1:43 PM
    • 56 Lines
  • [
    // Default query template (Don't run)
    {
    "query": {
    "bool": {
    ...
    • May 16 2022, 8:52 PM
    • 183 Lines
    • JSON
  • swh-scheduler=> select url from listed_origins where lister_id = 'ab79de00-a2d3-44ac-90a8-f53c35c5d893' and visit_type = 'git';
    -------------------------------------------------------------------------------------------------------------------------------------
    ${basedir}
    ${base.url}.git
    ${custom.gitprivate.url}simplerest.git
    ...
    • May 12 2022, 6:05 PM
    • 72 Lines
  • 11:53:48 swh-scheduler@db1:5432=> select * from task where type='list-maven-full';
    +-[ RECORD 1 ]-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | id | 31171944 |
    | type | list-maven-full |
    | arguments | {"args": [], "kwargs": {"url": "https://repo1.maven.org/maven2/", "index_url": "https://maven-exporter.internal.staging.swh.network/export-maven-central.fld"}} |
    ...
    • May 12 2022, 11:54 AM
    • 23 Lines
  • % join <( cat 2021-03-23/compressed/graph.edges.stats.txt | sort ) <( cat 2022-04-25/compressed/graph.edges.stats.txt | sort ) | while read t a b; do echo $t $(( 1. * b / a )); done
    dir:cnt 1.6619934106831602
    dir:dir 1.5117835202334333
    dir:rev 2.5607389960164033
    ori:snp 1.1175533201104195
    ...
    • May 8 2022, 12:32 PM
    • 22 Lines
  • seirl@vab ../softwareheritage/ssd/graph/2022-04-25 % cat compressed/graph.nodes.count.txt
    25340003875
    seirl@vab ../softwareheritage/ssd/graph/2022-04-25 % cat compressed/graph.nodes.stats.txt
    cnt 12213798094
    dir 10173900390
    ...
    • May 8 2022, 11:15 AM
    • 26 Lines
  • ~/dati/src/rust/coreutils/src/uu/sort   main  rg fork
    ✘  ~/dati/src/rust/coreutils/src/uu/sort   main  rg spawn
    src/check.rs
    41: thread::spawn({
    ...
    • May 6 2022, 4:50 PM
    • 12 Lines
  • select directory_id, from_utf8(name), type, target, perms from directory_entry where from_utf8(name) in ('.', '..', '') or strpos(from_utf8(name), '/') > 0
    "directory_id","","type","target","perms"
    "5002df6cc84b3245e9554f1b4ef286876cbb0856",".","dir","3dc4aa1834136cbff6f4451c677f8c18f28c56f2","16384"
    ...
    • May 4 2022, 5:01 PM
    • 545 Lines
  • seirl@vab ../softwareheritage/ssd/graph/2021-03-23 % du -shc --apparent-size compressed/* | sort -h :(
    9 compressed/graph.persons.count.txt
    11 compressed/graph.labels.count.txt
    12 compressed/graph.nodes.count.txt
    13 compressed/graph.edges.count.txt
    ...
    • Apr 29 2022, 3:11 PM
    • 49 Lines
  • softwareheritage=> select count(*) from (select url as child, (select count(*) from origin_visit where origin=origin.id) as child_visits, encode(metadata, 'escape')::json->'parent'->'full_name' as parent, (select count(*) from origin_visit, origin as o where origin_visit.origin=o.id and o.url= concat('https://github.com/'::text, (encode(metadata, 'escape')::json->'parent'->>'full_name')::text)) as parent_visits from raw_extrinsic_metadata, origin where digest(url, 'sha1')=decode(substring(target, 11), 'hex') and target>'swh:1:ori:' and target < 'swh:1:orz' and format='application/vnd.github.v3+json' and encode(metadata, 'escape') not like '%\\3%' and encode(metadata, 'escape') not like '%\\\\%' and encode(metadata, 'escape')::json->'parent'->'full_name' is not null) as t where t.child_visits=1 and t.parent_visits>0;
    ^[[A^[[A
    count
    -------
    10642
    ...
    • Apr 29 2022, 1:48 PM
    • 12 Lines
  • diff --git a/swh/loader/package/maven/loader.py b/swh/loader/package/maven/loader.py
    index d664e67..e2d9d05 100644
    --- a/swh/loader/package/maven/loader.py
    +++ b/swh/loader/package/maven/loader.py
    @@ -1,8 +1,10 @@
    ...
    • Apr 28 2022, 5:52 PM
    • 39 Lines
  • (swh) $ pip freeze | grep swh.core
    -e git+ssh://git@forge.softwareheritage.org/diffusion/DCORE/swh-core.git@d374b6002955bf3591c84335b6bcc3521ff78061#egg=swh.core
    (swh) $ pip show swh.core | grep Version:
    Version: 1.0.1.dev3+gd374b60
    • Apr 28 2022, 2:09 PM
    • 4 Lines
  • swhworker@worker3:~$ url=https://github.com/progval/Limnoria ; /usr/bin/time swh loader run git $url lister_name=github lister_instance_name=github
    WARNING:swh.core.cli:Could not load subcommand graph: DistributionNotFound(Requirement.parse('py4j'), None)
    INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/progval/Limnoria' with type 'git'
    INFO:swh.loader.git.loader:Listed 819 refs for repo https://github.com/progval/Limnoria
    {'status': 'uneventful'} for origin 'https://github.com/progval/Limnoria'
    ...
    • Apr 28 2022, 1:34 PM
    • 28 Lines
  • listed origins:
    ```
    10:24:59 swh@db1:5432=> select now(), count(*) from origin where url like 'https://repo1.maven.org/maven2/%'
    swh-> ;
    +------------------------------+--------+
    ...
    • Apr 28 2022, 10:31 AM
    • 31 Lines
  • $ cat example.py
    import contextlib
    @contextlib.contextmanager
    def f():
    ...
    • Apr 26 2022, 11:54 AM
    • 21 Lines
  • #!/usr/bin/env bash
    # set -x
    export KUBECONFIG=staging-workers.yaml
    ...
    • Apr 26 2022, 11:28 AM
    • 26 Lines
  • ERROR:swh.dataset.journalprocessor:Exporter ORCExporter: error while exporting the object: {'origin': 'https://www.npmjs.com/package/@reactionaries/hookd', 'visit': 170, 'date': datetime.datetime(2020, 9, 15, 18, 11, 37, 416264
    , tzinfo=datetime.timezone.utc), 'status': 'full', 'snapshot': b'\x15\xa3i\x0c\x05\x86PT\x19\\_~k7\x1cY$V\xdb\xe5', 'metadata': None}
    Traceback (most recent call last):
    File "/home/seirl/swh-environment/swh-dataset/swh/dataset/journalprocessor.py", line 467, in process_message
    [message.key()],
    ...
    • Apr 25 2022, 2:23 PM
    • 31 Lines
  • swh  tony  yavin4  ~  work  …  isos  libvirt  debian   master  1+  %  sha512sum -c swh-debian-zfs-11.3-amd64-20220422-1444.qcow2.sha512
    swh-debian-zfs-11.3-amd64-20220422-1444.qcow2: OK
    swh  tony  yavin4  ~  work  …  isos  libvirt  debian   master  2+  %  ls -lah
    total 841M
    drwxr-xr-x 2 tony tony 16 Apr 22 17:00 .
    ...
    • Apr 22 2022, 5:06 PM
    • 132 Lines
  • swh  tony  yavin4  ~  work  …  isos  libvirt  debian   master  %  ls -lah
    total 576M
    drwxr-xr-x 2 tony tony 8 Apr 22 16:08 .
    drwxr-xr-x 3 tony tony 3 Mar 12 19:25 ..
    lrwxrwxrwx 1 tony tony 203 May 18 2021 swh-debian-10.6-amd64-20201012-1352.qcow2 -> ../../../.git/annex/objects/6M/K7/SHA256E-s540454212--96a6280841959cf9be85ee5d07afa7cf53d31ced56369cb66c536bd1b61f2d7c/SHA256E-s540454212--96a6280841959cf9be85ee5d07afa7cf53d31ced56369cb66c536bd1b61f2d7c
    ...
    • Apr 22 2022, 4:13 PM
    • 442 Lines
  • packer build debian_bullseye.qemu.json
    qemu: output will be in this color.
    ==> qemu: Retrieving ISO
    ==> qemu: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.3.0-amd64-netinst.iso
    ...
    • Apr 22 2022, 3:54 PM
    • 686 Lines
  • 14:06 <+moranegg> olasd: (or any other git annex expert) I have done a git sync to add one pdf and seems lot of stuff has happened during this sync, Is there a way to check that this is normal and that I didn't break anything?
    14:08 <+olasd> there's a bunch of pdfs that are in the git repo instead of the annex (events/2021-11-30-swh5years/community-meetup/) but that commit was in january
    14:11 <+moranegg> interesting, I had in January other issues
    14:12 <+olasd> I managed a git annex get of the pdf you've added just fine
    14:15 <+moranegg> great, I hope I haven't deleted anything with the sync
    ...
    • Apr 22 2022, 2:41 PM
    • 16 Lines
  • if has_add_forge_now_permission(request.user):
    for item in page.object_list:
    history = AddForgeNowRequestHistory.objects.filter(request=item ).filter(actor_role = "MODERATOR").order_by(
    "id"
    )
    ...
    • Apr 21 2022, 9:39 PM
    • 18 Lines
  • packer build debian_bullseye_zfs.json
    proxmox: output will be in this color.
    ==> proxmox: Retrieving ISO
    ==> proxmox: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.3.0-amd64-netinst.iso
    ...
    • Apr 21 2022, 4:06 PM
    • 1,671 Lines
  • packer build debian_bullseye.json
    proxmox: output will be in this color.
    ==> proxmox: Retrieving ISO
    ==> proxmox: Trying https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.3.0-amd64-netinst.iso
    ...
    • Apr 21 2022, 3:30 PM
    • 1,189 Lines
  • root@elastic-worker0:~# puppet agent --test
    Info: Using configured environment 'staging'
    Info: Retrieving pluginfacts
    Info: Retrieving plugin
    Info: Retrieving locales
    ...
    • Apr 20 2022, 11:54 AM
    • 47 Lines
  • softwareheritage-scheduler=> select url, last_scheduled, last_visit from origin_visit_stats where url = 'http://svn.code.sf.net/p/csreadgen/code/';
    url | last_scheduled | last_visit
    ------------------------------------------+-------------------------------+-------------------------------
    http://svn.code.sf.net/p/csreadgen/code/ | 2022-04-13 09:42:21.829297+00 | 2022-04-13 09:42:26.456121+00
    (1 row)
    • Apr 13 2022, 5:27 PM
    • 5 Lines
  • Traceback (most recent call last):
    File "/home/seirl/swh-environment/.venv/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core', 'console_scripts', 'swh')())
    File "/home/seirl/swh-environment/swh-core/swh/core/cli/__init__.py", line 185, in main
    return swh(auto_envvar_prefix="SWH")
    ...
    • Apr 12 2022, 5:33 PM
    • 49 Lines