In D3073#74823, @olasd wrote:

The only caveat I see from this diff is that it forces people to use tox to build docs reliably.
I don't mind (I almost never use direct commands anymore) but some people might?

Apr 28 2020, 5:08 PM

zack changed the status of T1003: Register a SWH uri scheme with IANA from Open to Work in Progress.

I've prepared a draft IANA registration template. We have some work in flux on the SWHID spec, so I'll submit the registration request once it's done.

Apr 28 2020, 5:01 PM · Data Model

zack committed rDMODd2309384b0a5: SWHID spec: make SWHIDs plural where needed (authored by zack).

SWHID spec: make SWHIDs plural where needed

Apr 28 2020, 4:06 PM

zack committed rDMODf97d2165b2af: SWHID spec: bump version to 1.3 and add last modified date (authored by zack).

SWHID spec: bump version to 1.3 and add last modified date

Apr 28 2020, 4:06 PM

Apr 27 2020

zack committed rDMOD3ef4843c8955: SWHID spec: add support for IRI (authored by zack).

SWHID spec: add support for IRI

Apr 27 2020, 3:33 PM

zack committed rDMOD137938567f6b: SWHID spec: simplify and generalize escaping requirements (authored by zack).

SWHID spec: simplify and generalize escaping requirements

Apr 27 2020, 3:33 PM

zack closed T2379: SWHID: expand spec to allow IRI characters, a subtask of T2262: Deal with IRIs, as Resolved.

Apr 27 2020, 3:33 PM · Storage manager, Data Model

zack closed T2379: SWHID: expand spec to allow IRI characters as Resolved by committing rDMOD3ef4843c8955: SWHID spec: add support for IRI.

Apr 27 2020, 3:33 PM · Storage manager, Data Model

zack closed D3068: SWHID spec: add support for IRI.

Apr 27 2020, 3:33 PM

zack updated the diff for D3068: SWHID spec: add support for IRI.

SWHID spec: simplify and generalize escaping requirements

Apr 27 2020, 3:18 PM

zack added a comment to T2381: Validate extrinsic metadata terminology .

You correctly mention uniformity as a positive thing to have.
But term correctness is also a positive value, in my opinion.

Apr 27 2020, 2:09 PM · Metadata workflow

Apr 26 2020

zack added a comment to D3068: SWHID spec: add support for IRI.

In D3068#74533, @vlorentz wrote:

I wonder if we should require extra escapes, like whitespaces and/or angle brackets (to embed SWHIDs in free text)

(And it's out of scope, but I don't think we need to mention % must me escaped, it's already required by the RFC)

Apr 26 2020, 8:55 PM

zack added a comment to T2379: SWHID: expand spec to allow IRI characters.

Upon (admittedly quick) review, I don't think that anything more than D3068 is needed to address this.
Double-checking/feedback welcome!

Apr 26 2020, 4:46 PM · Storage manager, Data Model

zack created D3068: SWHID spec: add support for IRI.

Apr 26 2020, 4:45 PM

zack added a revision to T2379: SWHID: expand spec to allow IRI characters: D3068: SWHID spec: add support for IRI.

Apr 26 2020, 4:45 PM · Storage manager, Data Model

zack triaged T2380: documentation: nightly buid fails with "RuntimeError: Working outside of request context" as Unbreak Now! priority.

Apr 26 2020, 2:38 PM · Documentation

Apr 25 2020

zack committed rDMOD56cf99aeaf38: SWHID: deal with escaping in origin qualifiers (authored by zack).

SWHID: deal with escaping in origin qualifiers

Apr 25 2020, 8:41 AM

zack closed D3065: SWHID: deal with escaping in origin qualifiers.

Apr 25 2020, 8:41 AM

Apr 24 2020

zack created D3065: SWHID: deal with escaping in origin qualifiers.

Apr 24 2020, 4:59 PM

zack triaged T2379: SWHID: expand spec to allow IRI characters as Normal priority.

Apr 24 2020, 3:32 PM · Storage manager, Data Model

zack renamed T2262: Deal with IRIs from Dealing with IRIs to SWHID: deal with IRIs.

Apr 24 2020, 10:28 AM · Storage manager, Data Model

zack committed rDMOD3f388086cb7b: SWHID doc: improve wording of intrinsic parts v. the rest (authored by zack).

SWHID doc: improve wording of intrinsic parts v. the rest

Apr 24 2020, 10:12 AM

Apr 23 2020

zack added a comment to T2363: scanner: json output should return both known and unknown files/dirs.

In T2363#43710, @DanSeraf wrote:

$ swh scanner scan -f json /tmp/test
{
    "dir1": {
        "children": {
            "subdir1": {
                "children": {
                    "text.txt": {
                        "known": true,
                        "swhid": "swh:1:cnt:ff5b57b7095eb5d168a36db6552ad2ce1f219bf6"
                    }

Apr 23 2020, 10:47 AM · Code scanner

Apr 22 2020

zack retitled D3044: Move get_journal_client function to swh.journal.client from Move get_journal_client function in swh.journal.client to Move get_journal_client function to swh.journal.client.

Apr 22 2020, 5:18 PM

zack added a comment to D3011: dataset: add graph export based on kafka.

it's not a blocker, and can be added in a subsequent commit, but can you please also add documentation of the CLI, in the style of https://docs.softwareheritage.org/devel/swh-model/cli.html ?

Apr 22 2020, 11:10 AM

Apr 21 2020

zack added a comment to T2369: Review IPOL deposit metadata.

If we go for the DOI, shouldn't we converge on it also for other similarly archives, for the sake of uniformity? HAL comes to mind, of course.
(To be clear: I'm not proposing to uniform past deposits, but to change this for the future.)

Apr 21 2020, 8:51 PM · Metadata workflow, SWORD deposit

zack added a comment to T2335: automate handling of hanging/dead/stuck loaders.

In T2335#43554, @olasd wrote:

I then run clush -w <dead_workers> 'systemctl kill --kill-who all --signal 9 swh-worker@loader_git' on pergamon to restart these git loaders

Apr 21 2020, 2:30 PM · Scheduling utilities

zack closed T1195: git loader: fail to ingest our own hello world repository as Resolved.

Apr 21 2020, 11:46 AM · Git loader

Apr 20 2020

zack committed rDGRPH8ba977ed96ba: upgrade old comment-based type annotations to modern ones (authored by zack).

upgrade old comment-based type annotations to modern ones

Apr 20 2020, 2:06 PM

Apr 18 2020

zack committed rDDOC95e17dc37707: developer setup: document workon and generalize prose about venv management (authored by zack).

developer setup: document workon and generalize prose about venv management

Apr 18 2020, 7:36 PM

Apr 17 2020

zack added a reverting change for rDDOCbdefee74d23e: requirements-swh*: add dependencies on swh-scanner: rDDOC72b54026aaa1: Revert "requirements-swh*: add dependencies on swh-scanner".

Apr 17 2020, 9:40 PM

zack committed rDDOC72b54026aaa1: Revert "requirements-swh*: add dependencies on swh-scanner" (authored by zack).

Revert "requirements-swh*: add dependencies on swh-scanner"

Apr 17 2020, 9:40 PM

zack committed rDMODd52549f3b42c: CLI: add test for swh identify w/o args (authored by zack).

CLI: add test for swh identify w/o args

Apr 17 2020, 5:47 PM

zack committed rDMOD7b2cc1fafd91: CLI: require explicit "-" to identify via stdin (authored by zack).

CLI: require explicit "-" to identify via stdin

Apr 17 2020, 5:47 PM

zack closed D3029: CLI: require explicit "-" to identify via stdin.

Apr 17 2020, 5:47 PM

zack created D3029: CLI: require explicit "-" to identify via stdin.

Apr 17 2020, 5:25 PM

zack accepted D3028: Add documentation for command-line tools.

Apr 17 2020, 5:18 PM

zack committed rDMOD6ac6cb751721: SWHID doc: fix minor grammar issue (authored by zack).

SWHID doc: fix minor grammar issue

Apr 17 2020, 5:11 PM

zack committed rDMOD098f76a753f2: SWHID doc: fix link in CISE paper reference (authored by zack).

SWHID doc: fix link in CISE paper reference

Apr 17 2020, 4:42 PM

zack committed rDDOCbdefee74d23e: requirements-swh*: add dependencies on swh-scanner (authored by zack).

requirements-swh*: add dependencies on swh-scanner

Apr 17 2020, 4:39 PM

zack committed rDWAPPS51dabba36f6f: doc: reference SWHID using explicit anchors (authored by zack).

doc: reference SWHID using explicit anchors

Apr 17 2020, 4:29 PM

zack committed rDSTO32b3e9369526: doc: reference SWHID using explicit anchors (authored by zack).

doc: reference SWHID using explicit anchors

Apr 17 2020, 4:28 PM

zack committed rDGRPHc8fd980b2f03: spec: reference SWHID using explicit anchors (authored by zack).

spec: reference SWHID using explicit anchors

Apr 17 2020, 4:27 PM

zack committed rDDEP4ffc7bcb101c: spec: reference SWHID using explicit anchors (authored by zack).

spec: reference SWHID using explicit anchors

Apr 17 2020, 4:27 PM

zack committed rDMOD36f921b1638b: identifiers.py: reference to SWHIDs using explicit anchors (authored by zack).

identifiers.py: reference to SWHIDs using explicit anchors

Apr 17 2020, 4:24 PM

zack committed rDMOD4c78d47944f9: PID doc: embrace the SWHID naming (authored by zack).

PID doc: embrace the SWHID naming

Apr 17 2020, 4:24 PM

zack committed rDMOD94242cafd861: swh identify: embrace SWHID naming in user-facing doc/messages (authored by zack).

swh identify: embrace SWHID naming in user-facing doc/messages

Apr 17 2020, 4:24 PM

zack committed rDMOD0ab482e7b895: PID doc: add reference to CISE paper (authored by zack).

PID doc: add reference to CISE paper

Apr 17 2020, 4:24 PM

zack committed rDMOD2ae347dce241: doc: document identify CLI (authored by zack).

doc: document identify CLI

Apr 17 2020, 1:15 PM

zack closed D3025: doc: document identify CLI.

Apr 17 2020, 1:15 PM

Apr 16 2020

zack committed rDGRPH7a5920220ee2: doc: document CLI (authored by zack).

doc: document CLI

Apr 16 2020, 4:30 PM

zack closed D3024: doc: document CLI.

Apr 16 2020, 4:30 PM

zack closed D3023: sphinx: add support to generate click CLI doc.

Apr 16 2020, 4:26 PM

zack committed rDDOC97941b9cb439: sphinx: add support to generate click CLI doc (authored by zack).

sphinx: add support to generate click CLI doc

Apr 16 2020, 4:26 PM

zack created D3025: doc: document identify CLI.

Apr 16 2020, 4:25 PM

zack retitled D3024: doc: document CLI from doc: document CLI Depends on D3023 to doc: document CLI.

Apr 16 2020, 4:20 PM

zack created D3024: doc: document CLI.

Apr 16 2020, 4:20 PM

zack created D3023: sphinx: add support to generate click CLI doc.

Apr 16 2020, 4:17 PM

Apr 15 2020

zack updated the task description for T2363: scanner: json output should return both known and unknown files/dirs.

Apr 15 2020, 2:07 PM · Code scanner

zack triaged T2365: scanner: add color legend for sunburst output as Low priority.

Apr 15 2020, 1:56 PM · Code scanner

zack reopened T2300: swh-scanner: print a nicer error message when rate limit is hit as "Open".

Reopening, as it looks like this issue is back, e.g.:

$ swh scanner scan -f sunburst . 
Traceback (most recent call last):
  File "/home/zack/.virtualenvs/swh/bin/swh", line 11, in <module>
    load_entry_point('swh.core', 'console_scripts', 'swh')()
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-core/swh/core/cli/__init__.py", line 122, in main
    return swh(auto_envvar_prefix="SWH")
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/cli.py", line 56, in scan
    loop.run_until_complete(run(path, api_url, source_tree))
  File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 138, in run
    await _scan(root, session, api_url, source_tree)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 135, in _scan
    await _scan(path, session, api_url, source_tree)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 135, in _scan
    await _scan(path, session, api_url, source_tree)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 135, in _scan
    await _scan(path, session, api_url, source_tree)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 125, in _scan
    for path, pid, found in await parse_path(root, session, api_url):
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 104, in parse_path
    parsed_pids = await pids_discovery(list(parsed_paths.values()), session, api_url)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 61, in pids_discovery
    return await make_request(pids)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 49, in make_request
    error_response(resp.reason, resp.status, endpoint)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/exceptions.py", line 18, in error_response
    raise APIError(error_msg)
swh.scanner.exceptions.APIError: "429 Too Many Requests: 'https://archive.softwareheritage.org/api/1/known/'"

Apr 15 2020, 1:55 PM · Easy hack, Code scanner

zack triaged T2364: scanner: file browser in the sunburst/dashboard output as Normal priority.

Apr 15 2020, 1:51 PM · Code scanner

zack triaged T2363: scanner: json output should return both known and unknown files/dirs as Normal priority.

Apr 15 2020, 1:45 PM · Code scanner

zack triaged T2362: scanner: aiohttp.client_exceptions.ServerDisconnectedError: None as High priority.

Apr 15 2020, 1:37 PM · Code scanner

zack triaged T2361: WARNING:swh.core.cli:Could not load subcommand dataset: No module named 'swh.dataset.cli' as Normal priority.

Apr 15 2020, 1:32 PM · Datasets

Apr 14 2020

zack accepted D2881: client: add `visits` method to get the list of an origin visits.

Apr 14 2020, 11:40 AM

zack accepted D3002: cli: add a cachemount command.

Apr 14 2020, 11:39 AM

zack retitled D3002: cli: add a cachemount command from cli: add a memcache command to cli: add a cachemount command.

Apr 14 2020, 11:36 AM

Apr 10 2020

zack requested changes to D3002: cli: add a cachemount command.

In D3002#72795, @vlorentz wrote:

Confusing name, I expected this to be a proxy using memcached.

Apr 10 2020, 4:52 PM

Apr 9 2020

zack committed rDENVc215d0e3f12a: Makefile.python: drop unused vars as well as check-staged (authored by zack).

Makefile.python: drop unused vars as well as check-staged

Apr 9 2020, 9:46 AM

zack committed rDENV4fcc945cc859: Makefile.python: make "make check" use pre-commit (authored by zack).

Makefile.python: make "make check" use pre-commit

Apr 9 2020, 9:46 AM

zack closed D2984: Makefile.python: make "make check" use pre-commit.

Apr 9 2020, 9:46 AM

zack added a comment to D2984: Makefile.python: make "make check" use pre-commit.

In D2984#72249, @ardumont wrote:

Please, also clean up the tool environment variables :)

Apr 9 2020, 9:45 AM

zack updated the diff for D2984: Makefile.python: make "make check" use pre-commit.

Makefile.python: drop unused vars as well as check-staged

Apr 9 2020, 9:45 AM

Apr 8 2020

zack created D2984: Makefile.python: make "make check" use pre-commit.

Apr 8 2020, 8:28 PM

zack committed rDENV3c7557920216: Makefile.python: add support black support to "check" target (authored by zack).

Makefile.python: add support black support to "check" target

Apr 8 2020, 8:07 PM

zack committed rDTPL846949bb9ba3: blacken (authored by zack).

blacken

Apr 8 2020, 4:21 PM

Apr 3 2020

zack added a comment to T2317: Redesign landing page.

In T2317#43088, @moranegg wrote:

In the sidebar menu I would keep the API access (unless it is accessible during the navigation - not on the landing page).
Also the access to the API on the side, is very small.

Apr 3 2020, 11:24 AM · UX, 2019 UX audit, Web app