Page MenuHomeSoftware Heritage

zack (Stefano Zacchiroli)
UserAdministrator

User Details

User Since
Sep 7 2015, 3:43 PM (263 w, 4 d)
Roles
Administrator

Recent Activity

Today

zack committed rDFUSE9ced0b454fb9: docs: add design notes (authored by zack).
docs: add design notes
Fri, Sep 25, 4:13 PM
zack closed D4042: docs: add design notes.
Fri, Sep 25, 4:13 PM
zack updated the summary of D4042: docs: add design notes.
Fri, Sep 25, 4:03 PM
zack added a revision to T1926: FUSE filesystem to navigate the archive: D4042: docs: add design notes.
Fri, Sep 25, 4:03 PM · Storage manager, Graph service
zack updated subscribers of T2635: web client: add async API.

As @olasd pointed out, async is a keyword so let's not use it as module name :-)

Fri, Sep 25, 1:19 PM · Web client
zack added a comment to D4046: Fix default config file may be absent in scanner cli.
In D4046#100083, @tenma wrote:

on another note, do I write tests for this cli?

Fri, Sep 25, 11:59 AM
zack added inline comments to D4046: Fix default config file may be absent in scanner cli.
Fri, Sep 25, 11:53 AM
zack added inline comments to D4046: Fix default config file may be absent in scanner cli.
Fri, Sep 25, 11:48 AM
zack edited P778 Masterwork From Distant Lands.
Fri, Sep 25, 11:20 AM
zack committed rDGRPH19e0860848e6: docs/git2graph.md: use myst link to avoid absolute URL references (authored by zack).
docs/git2graph.md: use myst link to avoid absolute URL references
Fri, Sep 25, 10:00 AM
zack edited P777 Masterwork From Distant Lands.
Fri, Sep 25, 8:24 AM
zack updated the diff for D4042: docs: add design notes.

trigger CI build

Fri, Sep 25, 8:23 AM

Yesterday

zack created D4042: docs: add design notes.
Thu, Sep 24, 9:05 PM
zack committed rDDOC1e40dcf8e03b: sphinx: add todo extention, to typeset TODO notes (authored by zack).
sphinx: add todo extention, to typeset TODO notes
Thu, Sep 24, 9:03 PM
zack committed rDDOC1977032c6510: sphinx: use MyST for markdown rendering (authored by zack).
sphinx: use MyST for markdown rendering
Thu, Sep 24, 9:03 PM
zack closed D4041: sphinx: use MyST for markdown rendering.
Thu, Sep 24, 9:03 PM
zack added a comment to D4041: sphinx: use MyST for markdown rendering.

Why do we need markdown support in the docs?

Thu, Sep 24, 8:59 PM
zack created D4041: sphinx: use MyST for markdown rendering.
Thu, Sep 24, 8:55 PM
zack updated subscribers of T2640: Add link from the main archive to the Bitbucket mercurial case (https://bitbucket-archive.softwareheritage.org/).

I agree on linking Gitorious to the wiki page, but we should find a consistency that can generate automatically the coverage page.

Thu, Sep 24, 5:25 PM · Web app
zack added a comment to T2624: Create strategy for documentation with a map or a full table of content.

For whoever who want to work on this (and on doc restructuring in general), I recommend to watch this talk: https://www.youtube.com/watch?v=t4vKPhjcMZg , which provides a good taxonomy of the kind of docs that exist. Our GSoD 2019 application page also contains a lot of useful related work.

Thu, Sep 24, 4:48 PM · Documentation
zack added a project to T2621: running tox fails because C.UTF-8 is not available: Development environment.
Thu, Sep 24, 3:15 PM · Development environment
zack added a comment to D4017: Improve code quality and doc in BufferedProxyStorage.
In D4017#99716, @tenma wrote:

The fact that is not so common a pattern does not make it bad.

Thu, Sep 24, 1:16 PM

Wed, Sep 23

zack edited P776 hackmd2rst.
Wed, Sep 23, 11:14 PM
zack triaged T2635: web client: add async API as Normal priority.
Wed, Sep 23, 9:29 PM · Web client
zack committed rDFUSE607a2be7ebf8: make tox pass (fix deps and leftover boilerplate) (authored by zack).
make tox pass (fix deps and leftover boilerplate)
Wed, Sep 23, 4:02 PM
zack accepted D4021: Add libfuse3-dev in the docker.
Wed, Sep 23, 3:45 PM
zack committed rDFUSEb7fa82e62589: package layout: uniform documentation layout with other modules (authored by zack).
package layout: uniform documentation layout with other modules
Wed, Sep 23, 3:31 PM
zack accepted D4019: Add swh-fuse.
Wed, Sep 23, 2:53 PM
zack committed rDFUSEe324d270c295: initial code skeleton for the FUSE virtual file system (authored by zack).
initial code skeleton for the FUSE virtual file system
Wed, Sep 23, 2:31 PM
zack committed rDFUSEcc4b996913ad: import template from swh-py-template (init-py-repo) (authored by zack).
import template from swh-py-template (init-py-repo)
Wed, Sep 23, 2:31 PM
zack added a project to T2634: swh-core: missing test dependency on requests: Easy hack.
Wed, Sep 23, 2:23 PM · Easy hack, Core & foundations
zack edited projects for T2634: swh-core: missing test dependency on requests, added: Core & foundations; removed Core Loader.
Wed, Sep 23, 2:23 PM · Easy hack, Core & foundations
zack triaged T2634: swh-core: missing test dependency on requests as Low priority.
Wed, Sep 23, 2:23 PM · Easy hack, Core & foundations
zack triaged T2632: swh scanner fail to start when configuration file is missing as High priority.
Wed, Sep 23, 2:12 PM · Code scanner
zack abandoned D3974: WIP: fuse design doc.

we have rewritten this from scratch and we will commit it separately

Wed, Sep 23, 10:43 AM
zack commandeered D3974: WIP: fuse design doc.
Wed, Sep 23, 10:43 AM

Tue, Sep 22

zack accepted D4004: origin_save: Fix typo in a save task status (s/succeed/succeeded/).

Caveat: as this results in an API change (I think), it should be announced on the swh-devel mailing list before deployment.

Tue, Sep 22, 12:59 PM

Mon, Sep 21

zack renamed T2617: Improve UTF8 UnicodeDecodeError handling in JSON conversion layer and update API documentation from Improve UnicodeDecodeError handling in JSON conversion layer and update API documentation to Improve UTF8 UnicodeDecodeError handling in JSON conversion layer and update API documentation.
Mon, Sep 21, 4:48 PM · Web app
zack added a comment to T2373: git loader OOM when loading huge repository.

fwiw, loader-core v0.11.0 deployed in production.

Mon, Sep 21, 2:43 PM · Git loader

Sun, Sep 20

zack added a comment to T2373: git loader OOM when loading huge repository.

I can confirm that with the current master HEAD of swh-loader-core (452fa224f9ca635a979cf1a8e98c88bb560ca98a), loading of the Linux kernel repo no longer OOM.
(It failed after ~24 hours, but apparently for unrelated reasons.)

Sun, Sep 20, 2:31 PM · Git loader

Sat, Sep 19

zack accepted D3987: Fix license statements on db_utils.py.

This is fine, and better than the status quo.
(We should in theory also mention the other applicable copyright notices at the beginning of the file, but we weren't doing that before either, so this change LGTM.)

Sat, Sep 19, 8:13 PM
zack closed T2114: swh-graph API: add ?limit=N method variants to return first N results as Resolved.

this has been fixed a while ago by D2669

Sat, Sep 19, 8:08 PM · Easy hack, Graph service
zack updated the task description for T2615: Web UI: typo in success status for save code now requests: "succeed" -> "succeeded".
Sat, Sep 19, 8:05 PM · Easy hack, Web app
zack triaged T2615: Web UI: typo in success status for save code now requests: "succeed" -> "succeeded" as Low priority.
Sat, Sep 19, 8:05 PM · Easy hack, Web app
zack triaged T2614: Web UI: add status "ongoing" for "save code now" requests as Low priority.
Sat, Sep 19, 8:02 PM · Web app
zack renamed T2613: Web UI: make "save code now" icon consistent (and look like a camera) from Web UI: make "save code now" icon consistent (looking like a camera) to Web UI: make "save code now" icon consistent (and look like a camera).
Sat, Sep 19, 2:48 PM · Easy hack, Web app
zack merged T2281: [proposition] change icon of save code now into T2613: Web UI: make "save code now" icon consistent (and look like a camera).
Sat, Sep 19, 2:47 PM · Easy hack, Web app
zack merged task T2281: [proposition] change icon of save code now into T2613: Web UI: make "save code now" icon consistent (and look like a camera).
Sat, Sep 19, 2:47 PM · Web app
zack triaged T2613: Web UI: make "save code now" icon consistent (and look like a camera) as Low priority.
Sat, Sep 19, 2:46 PM · Easy hack, Web app

Fri, Sep 18

zack committed rDICP54aaadb39c0f: blackify: auto format python code with black (authored by zack).
blackify: auto format python code with black
Fri, Sep 18, 5:21 PM
zack committed rDICPc7e7a75a6625: uniform tox.ini and setup.cfg with python module template (authored by zack).
uniform tox.ini and setup.cfg with python module template
Fri, Sep 18, 5:21 PM
zack closed D3983: blackify: auto format python code with black.
Fri, Sep 18, 5:21 PM
zack added a comment to D3983: blackify: auto format python code with black.

The repository is missing configuration for using black. Could you add the pre-commit hook and the black configuration in pyproject.toml as in other swh repos ?

Fri, Sep 18, 5:18 PM
zack updated the diff for D3983: blackify: auto format python code with black.
  • uniform tox.ini and setup.cfg with python module template
Fri, Sep 18, 5:16 PM
zack retitled D3983: blackify: auto format python code with black from test_vault.py: make black pass again to blackify: auto format python code with black.
Fri, Sep 18, 5:13 PM
zack updated the diff for D3983: blackify: auto format python code with black.

blackify all module, rather than just test_vault.py

Fri, Sep 18, 5:12 PM
zack accepted D3994: Replace deprecated persistent_identifier method.
Fri, Sep 18, 4:40 PM
zack requested changes to D3974: WIP: fuse design doc.
Fri, Sep 18, 2:05 PM
zack added a comment to T2589: expose swh-graph API at archive.s.o/api/1/graph/.

You are right, they are not stored in database but there is a storage.origin_get_by_sha1 method.

Fri, Sep 18, 12:54 PM · System administration, Web app, Graph service
zack edited P771 Masterwork From Distant Lands.
Fri, Sep 18, 10:49 AM
zack created D3983: blackify: auto format python code with black.
Fri, Sep 18, 10:44 AM
zack committed rDWCLI4749ea13cfea: CONTRIBUTORS: add haltode (authored by zack).
CONTRIBUTORS: add haltode
Fri, Sep 18, 10:35 AM
zack added inline comments to D3982: Replace deprecated PersistentID class with SWHID.
Fri, Sep 18, 9:51 AM
zack accepted D3982: Replace deprecated PersistentID class with SWHID.

LGTM

Fri, Sep 18, 9:47 AM
zack edited reviewers for D3982: Replace deprecated PersistentID class with SWHID, added: Reviewers; removed: zack.
Fri, Sep 18, 9:44 AM

Thu, Sep 17

zack added a comment to T2589: expose swh-graph API at archive.s.o/api/1/graph/.
  • We can process swh-graph responses to enrich the data (notably get origin urls from their sha1 or turn swhids into dicts) and returns them in JSON format
Thu, Sep 17, 10:25 PM · System administration, Web app, Graph service
zack retitled D3979: Fix blackified strings with spurious concatenation and use f-strings from Fix blackified strings with spurrious concatenation and use f-strings to Fix blackified strings with spurious concatenation and use f-strings.
Thu, Sep 17, 7:39 PM
zack requested changes to D3974: WIP: fuse design doc.
Thu, Sep 17, 4:31 PM
zack added a comment to T2610: Add isort pre-commit hook and configuration to all repos.

(I was initially surprised by the mixing together of "import" lines with "from ... import" ones, but upon reflection it makes a lot of sense, because one might have to switch between the two forms, and it's silly to have to move the line back and forth between import blocks when that happens.)

Thu, Sep 17, 3:38 PM · Development environment
zack updated subscribers of T1789: batch API to check for the presence of content in the archive.
Thu, Sep 17, 3:00 PM · Web app
zack closed T1789: batch API to check for the presence of content in the archive as Resolved.

this has been addressed, and in a more general way that works for any SWHID, in D2582

Thu, Sep 17, 3:00 PM · Web app
zack merged task T2607: git loader OOM when loading the linux kernel repo into T2373: git loader OOM when loading huge repository.
Thu, Sep 17, 9:53 AM · Git loader
zack merged T2607: git loader OOM when loading the linux kernel repo into T2373: git loader OOM when loading huge repository.
Thu, Sep 17, 9:53 AM · Git loader
zack renamed T2373: git loader OOM when loading huge repository from staging: git loader: failure to ingest huge repository (e.g. nixpkgs) to git loader OOM when loading huge repository.
Thu, Sep 17, 9:53 AM · Git loader
zack changed the status of T1847: fully automate export of the graph dataset, a subtask of T1848: refresh graph dataset export, from Open to Work in Progress.
Thu, Sep 17, 9:04 AM · Datasets
zack changed the status of T1847: fully automate export of the graph dataset from Open to Work in Progress.
Thu, Sep 17, 9:04 AM · Graph service, Datasets
zack renamed T2607: git loader OOM when loading the linux kernel repo from git loader OOM when loading the linux kernel repo (at least in the docker dev environment) to git loader OOM when loading the linux kernel repo.
Thu, Sep 17, 9:03 AM · Git loader
zack raised the priority of T2607: git loader OOM when loading the linux kernel repo from Normal to High.

Very likely the same issue, thanks @ardumont !
Given what @olasd said in that issue (the ingestion logic having remained pretty much the same since ever), and that I can confirm linux.git was loading just fine on my laptop no more than a year ago, the increased memory usage probably comes from elsewhere.
Anyway, it looks like a potentially important issue, so I'm raising priority and also removing the association with the docker env (as you could also reproduce this on staging).

Thu, Sep 17, 9:03 AM · Git loader

Wed, Sep 16

zack removed a parent task for T1848: refresh graph dataset export: T1868: refresh compressed representation of the archive.
Wed, Sep 16, 8:43 PM · Datasets
zack removed a subtask for T1868: refresh compressed representation of the archive: T1848: refresh graph dataset export.
Wed, Sep 16, 8:43 PM · Graph service
zack added a comment to T1847: fully automate export of the graph dataset.

I think this is (reasonably) done now, please check and close it.

Wed, Sep 16, 8:43 PM · Graph service, Datasets
zack raised the priority of T1848: refresh graph dataset export from Normal to High.
Wed, Sep 16, 8:42 PM · Datasets
zack added a comment to T1848: refresh graph dataset export.
Wed, Sep 16, 8:42 PM · Datasets
zack assigned T1868: refresh compressed representation of the archive to seirl.

We have now a newer version of the compressed graph (2020-05-20), but it's not yet running on granet (I *think*, and, lacking T2579, I haven't checked).
Please make granet run that version of this task and close this task. (Or just close this task if it's already done.)

Wed, Sep 16, 8:41 PM · Graph service
zack updated the task description for T2607: git loader OOM when loading the linux kernel repo.
Wed, Sep 16, 8:28 PM · Git loader
zack triaged T2607: git loader OOM when loading the linux kernel repo as Normal priority.
Wed, Sep 16, 8:26 PM · Git loader
zack triaged T2605: Web UI: add a way to browse origins, other than search as Low priority.
Wed, Sep 16, 5:54 PM · Web app

Tue, Sep 15

zack accepted D3945: docs: quickstart: add compression instructions.
Tue, Sep 15, 1:22 PM
zack triaged T2601: create a scratch/temporary postgres DB to experiment with flattened directories as Normal priority.
Tue, Sep 15, 12:55 PM · System administration
zack triaged T2600: SQL storage: experiment with flattened layouts for directory nodes as Normal priority.
Tue, Sep 15, 12:53 PM · Storage manager
zack created P766 current size of directory-related DB entities (swh-replica cluster).
Tue, Sep 15, 12:38 PM
zack resigned from D3945: docs: quickstart: add compression instructions.
Tue, Sep 15, 11:03 AM
zack requested changes to D3945: docs: quickstart: add compression instructions.

looks great in general!
just a few nits here and there (and possibly a separate issue to file for the sane default part)

Tue, Sep 15, 11:02 AM
zack added inline comments to D3945: docs: quickstart: add compression instructions.
Tue, Sep 15, 11:01 AM

Mon, Sep 14

zack renamed T2589: expose swh-graph API at archive.s.o/api/1/graph/ from expose the compressed graph API at archive.s.o/api/1/graph/ to expose swh-graph API at archive.s.o/api/1/graph/.
Mon, Sep 14, 2:37 PM · System administration, Web app, Graph service
zack triaged T2589: expose swh-graph API at archive.s.o/api/1/graph/ as Normal priority.
Mon, Sep 14, 2:37 PM · System administration, Web app, Graph service
zack updated subscribers of T2577: Test gitea lister on staging environment.

An email was sent on the swh-devel mailing list to ask for reviews.
The deployment in production will be performed in the middle of week 38 is no problems are raised.

Mon, Sep 14, 10:22 AM · Lister
zack assigned T1926: FUSE filesystem to navigate the archive to haltode.
Mon, Sep 14, 9:59 AM · Storage manager, Graph service

Thu, Sep 10

zack accepted D3876: readme and cli description update.
Thu, Sep 10, 5:17 PM
zack requested changes to D3876: readme and cli description update.
Thu, Sep 10, 4:34 PM