diff --git a/docs/README.rst b/docs/README.rst
index ef841fb..0e1c863 100644
--- a/docs/README.rst
+++ b/docs/README.rst
@@ -1,7 +1,7 @@
-Software Heritage - Virtual file system
-=======================================
+Software Heritage virtual filesystem (SwhFS)
+============================================
-Virtual file system to browse the
-`Software Heritage `_
-`archive `_,
-using the `FUSE `_ framework.
+Virtual filesystem to browse the `Software Heritage
+`_ `archive
+`_, using the `FUSE
+`_ framework.
diff --git a/docs/design.md b/docs/design.md
index 0a1f890..afdc8c7 100644
--- a/docs/design.md
+++ b/docs/design.md
@@ -1,220 +1,221 @@
-# SWH FUSE — Design notes
+# Software Heritage virtual filesystem (SwhFS) --- Design notes
```{warning}
-this document describes design notes for SWH FUSE, which is still under active
-development and hence **not yet available** for general use.
+this document describes design notes for the Software Heritage virtual
+filesystem (SwhFS), which is still under active development and hence **not yet
+available** for general use.
```
-The [Software Heritage](https://www.softwareheritage.org/)
-{ref}`data model ` is a [Direct Acyclic
-Graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) with nodes of
-different types that correspond to source code artifacts such as directories,
-commits, etc. Using this
-[FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) module (*SWH FUSE*
-for short) you can locally mount, and then navigate as a (virtual) file system,
-parts of the archive identified by
-{ref}`Software Heritage identifiers ` (SWHIDs).
+The [Software Heritage](https://www.softwareheritage.org/) {ref}`data model
+` is
+a [Direct Acyclic Graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph)
+(DAG) with nodes of different types that correspond to source code artifacts
+such as directories, commits, etc. Using
+this [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) module
+(*SwhFS* for short) you can locally mount, and then navigate as a (virtual)
+file system, parts of the archive identified by {ref}`Software Heritage
+identifiers ` (SWHIDs).
-To retrieve information about the source code artifacts the FUSE module
-interacts over the network with the Software Heritage archive via its
-{ref}`Web API `.
+To retrieve information about the source code artifacts, SwhFS interacts over
+the network with the Software Heritage archive via its {ref}`Web API
+`.
## Command-line interface
$ swh fuse mount [SWHID]...
-will mount the Software Heritage archive at the local ``, the *SWH FUSE
-mount point*. From there, the user will be able to lazily load and navigate the
+will mount the Software Heritage archive at the local ``, the *SwhFS mount
+point*. From there, the user will be able to lazily load and navigate the
archive using SWHID at entry points.
If one or more SWHIDs are also specified, the corresponding objects will be pre-
fetched from the archive at mount-time and available at `/archive/`.
For more details see the {ref}`CLI documentation `.
## Mount point
-The SWH FUSE mount point contain:
+The SwhFS mount point contain:
- `archive/`: initially empty, this directory is lazily populated with one entry
per accessed SWHID, having actual SWHIDs as names.
- `meta/`: initially empty, this directory contains one `.json` file for
each `` entry under `archive/`. The JSON file contain all available meta
information about the given SWHID, as returned by the Software Heritage Web API
for that object. Note that, in case of pagination (e.g., snapshot objects with
many branches) the JSON file will contain a complete version with all pages
merged together.
```{todo}
Consider sharding ``/`.json` files under `ab/cd/` dirs to avoid
exploding the number of dir entries under `archive/` and `meta/`
(cf. [T2694](https://forge.softwareheritage.org/T2694))
```
## File system representation
SWHID are represented differently on the file-system depending on the associated
node types in the Software Heritage graph. Details are given below, for each
node type.
### `cnt` nodes (blobs)
Content leaves (AKA blobs) are represented on disks as regular files, containing
the corresponding bytes, as archived.
Note that permissions are associated to blobs only in the context of
directories. Hence, when accessing blobs from the top-level `archive/`
directory, the permissions of the `archive/SWHID` file will be arbitrary and not
meaningful (e.g., `0x644`).
### `dir` nodes (directories)
Directory nodes are represented as directories on the file-system, containing
one entry for each entry of the archived directory. Entry names and other
metadata, including permissions, will correspond to the archived entry metadata.
-Note that the FUSE mount is read-only, no matter what the permissions say. So it
+Note that SwhFS is mounted read-only, no matter what the permissions say. So it
is possible that, in the context of a directory, a file is presented as
writable, whereas actually writing to it will fail with `EPERM`.
### `rev` nodes (commits)
Revision (AKA commit) nodes are represented on the file-system as directories
with the following entries:
- `root`: source tree at the time of the commit, as a symlink pointing into
`archive/`, to a SWHID of type `dir`
- `parents/` (note the plural): a virtual directory containing entries named
`1`, `2`, `3`, etc., one for each parent commit. Each of these entry is a
symlink pointing into `archive/`, to the SWHID file for the given parent commit
- `parent` (note the singular): present if and only if the current commit has a
single parent commit (which is the most common case). When present it is a
symlink pointing into `archive/` to the SWHID for the sole parent commit
- `meta.json`: metadata for the current node, as a symlink pointing to the
relevant `meta/.json` file
### `rel` nodes (releases)
Release nodes are represented on the file-system as directories with the
following entries:
- `target`: target node, as a symlink to `archive/`
- `target_type`: type of the target SWHID, as a 3-letter code
- `root`: present if and only if the release points to something that
(transitively) resolves to a directory. When present it is a symlink pointing
into `archive/` to the SWHID of the given directory
- `meta.json`: metadata for the current node, as a symlink pointing to the
relevant `meta/.json` file
### `snp` nodes (snapshots)
Snapshot nodes are represented on the file-system as directories with on entry
for each branch in the snapshot.
Branch names are mangled by replacing...
```{todo}
decide how to do branch name escaping and describe it here
```
Each entry is a symlink pointing into `archive/` to the branch target SWHID.
## Caching
-SWH FUSE retrieves both metadata and file contents from the Software Heritage
+SwhFS retrieves both metadata and file contents from the Software Heritage
archive via the network. In order to obtain reasonable performances several
caches are used to minimize network transfer.
Caches are stored on disk in SQLite DB(s) located under
`$XDG_CACHE_HOME/swh/fuse/`.
```{todo}
- potential improvement: store blobs larger than a threshold on disk as files
rather than in SQLite, e.g., under `$XDG_CACHE_HOME/swh/fuse/objects/`
```
-All caches are persistent (i.e., they survive the restart of the SWH FUSE
-process) and global (i.e., they are shared by concurrent SWH FUSE processes).
+All caches are persistent (i.e., they survive the restart of the SwhFS process)
+and global (i.e., they are shared by concurrent SwhFS processes).
We assume that no cache *invalidation* is necessary, due to intrinsic properties
of the Software Heritage archive, such as integrity verification and append-only
archive changes. To clean the caches one can just remove the corresponding files
from disk.
### Metadata cache
SWHID → JSON metadata
The metadata cache map each SWHID to the complete metadata of the referenced
object. This is analogous to what is available in `meta/.json` file (and
generally used as data source for returning the content of those files).
Cache location on-disk: `$XDG_CACHE_HOME/swh/fuse/metadata.sqlite`
### Blob cache
cnt SWHID → bytes
The blob cache map SWHIDs of type `cnt` to the bytes of their archived content.
In general, each SWHID that has an entry in the blob cache also has a matching
entry in the metadata cache for other blob attributes (e.g., checksums, size,
etc.).
The blob cache entry for a given content object is populated, at the latest, the
first time the object is `open()`-d. It might be populated earlier on due to
prefetching, e.g., when a directory pointing to the given content is listed for
the first time.
Cache location on-disk: `$XDG_CACHE_HOME/swh/fuse/blob.sqlite`
### Dentry cache
dir SWHID → directory entries
The dentry (directory entry) cache map SWHIDs of type `dir` to the directory
entries they contain. Each entry comes with its name as well as file attributes
(i.e., all its needed to perform a detailed directory listing).
Additional attributes of each directory entry should be looked up on a entry by
entry basis, possibly hitting the metadata cache.
The dentry cache for a given dir is populated, at the latest, when the content
of the directory is listed. More aggressive prefetching might happen. For
instance, when first opening a dir a recursive listing of it can be retrieved
from the remote backend and used to recursively populate the dentry cache for
all (transitive) sub-directories.
### Parents cache
rev SWHID → parent SWHIDs
The parents cache map SWHIDs of type `rev` to the list of their parent commits.
The parents cache for a given rev is populated, at the latest, when the content
of the revision virtual directory is listed. More aggressive prefetching might
happen. For instance, when first opening a rev virtual directory a recursive
listing of all its ancestor can be retrieved from the remote backend and used to
recursively populate the parents cache for all ancestors.
diff --git a/swh/fuse/cache.py b/swh/fuse/cache.py
index bf47451..8e6c199 100644
--- a/swh/fuse/cache.py
+++ b/swh/fuse/cache.py
@@ -1,140 +1,141 @@
# Copyright (C) 2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
from abc import ABC
import json
from pathlib import Path
from typing import Any, AsyncGenerator, Dict, Optional
import aiosqlite
from swh.model.identifiers import SWHID, parse_swhid
from swh.web.client.client import typify_json
class FuseCache:
- """ SWH FUSE retrieves both metadata and file contents from the Software
- Heritage archive via the network. In order to obtain reasonable performances
- several caches are used to minimize network transfer.
+ """SwhFS retrieves both metadata and file contents from the Software Heritage archive
+ via the network. In order to obtain reasonable performances several caches are used
+ to minimize network transfer.
Caches are stored on disk in SQLite databases located at
`$XDG_CACHE_HOME/swh/fuse/`.
- All caches are persistent (i.e., they survive the restart of the SWH FUSE
- process) and global (i.e., they are shared by concurrent SWH FUSE
- processes).
+ All caches are persistent (i.e., they survive the restart of the SwhFS process) and
+ global (i.e., they are shared by concurrent SwhFS processes).
We assume that no cache *invalidation* is necessary, due to intrinsic
properties of the Software Heritage archive, such as integrity verification
and append-only archive changes. To clean the caches one can just remove the
- corresponding files from disk. """
+ corresponding files from disk.
+
+ """
def __init__(self, cache_conf: Dict[str, Any]):
self.cache_conf = cache_conf
async def __aenter__(self):
self.metadata = MetadataCache(self.cache_conf["metadata"])
self.blob = BlobCache(self.cache_conf["blob"])
await self.metadata.__aenter__()
await self.blob.__aenter__()
return self
async def __aexit__(self, type=None, val=None, tb=None) -> None:
await self.metadata.__aexit__()
await self.blob.__aexit__()
async def get_cached_swhids(self) -> AsyncGenerator[SWHID, None]:
""" Return a list of all previously cached SWHID """
# Use the metadata db since it should always contain all accessed SWHIDs
metadata_cursor = await self.metadata.conn.execute(
"select swhid from metadata_cache"
)
swhids = await metadata_cursor.fetchall()
for raw_swhid in swhids:
yield parse_swhid(raw_swhid[0])
class AbstractCache(ABC):
""" Abstract cache implementation to share common behavior between cache
types (such as: YAML config parsing, SQLite context manager) """
def __init__(self, conf: Dict[str, Any]):
self.conf = conf
async def __aenter__(self):
# In-memory (thus temporary) caching is useful for testing purposes
if self.conf.get("in-memory", False):
path = ":memory:"
else:
path = Path(self.conf["path"])
path.parent.mkdir(parents=True, exist_ok=True)
self.conn = await aiosqlite.connect(path)
return self
async def __aexit__(self, type=None, val=None, tb=None) -> None:
await self.conn.close()
class MetadataCache(AbstractCache):
""" The metadata cache map each SWHID to the complete metadata of the
referenced object. This is analogous to what is available in
`meta/.json` file (and generally used as data source for returning
the content of those files). """
async def __aenter__(self):
await super().__aenter__()
await self.conn.execute(
"create table if not exists metadata_cache (swhid, metadata)"
)
return self
async def get(self, swhid: SWHID, typify: bool = True) -> Any:
cursor = await self.conn.execute(
"select metadata from metadata_cache where swhid=?", (str(swhid),)
)
cache = await cursor.fetchone()
if cache:
metadata = json.loads(cache[0])
return typify_json(metadata, swhid.object_type) if typify else metadata
else:
return None
async def set(self, swhid: SWHID, metadata: Any) -> None:
await self.conn.execute(
"insert into metadata_cache values (?, ?)",
(str(swhid), json.dumps(metadata)),
)
class BlobCache(AbstractCache):
""" The blob cache map SWHIDs of type `cnt` to the bytes of their archived
content.
The blob cache entry for a given content object is populated, at the latest,
the first time the object is `read()`-d. It might be populated earlier on
due to prefetching, e.g., when a directory pointing to the given content is
listed for the first time. """
async def __aenter__(self):
await super().__aenter__()
await self.conn.execute("create table if not exists blob_cache (swhid, blob)")
return self
async def get(self, swhid: SWHID) -> Optional[bytes]:
cursor = await self.conn.execute(
"select blob from blob_cache where swhid=?", (str(swhid),)
)
cache = await cursor.fetchone()
if cache:
blob = cache[0]
return blob
else:
return None
async def set(self, swhid: SWHID, blob: bytes) -> None:
await self.conn.execute(
"insert into blob_cache values (?, ?)", (str(swhid), blob)
)