diff --git a/docs/README.rst b/docs/README.rst index 0e1c863..0b32abf 100644 --- a/docs/README.rst +++ b/docs/README.rst @@ -1,7 +1,8 @@ -Software Heritage virtual filesystem (SwhFS) -============================================ +Software Heritage Filesystem (SwhFS) +==================================== -Virtual filesystem to browse the `Software Heritage +User-space POSIX filesystem to browse the `Software Heritage `_ `archive -`_, using the `FUSE -`_ framework. +`_, as if it were locally available. + +Built using the `FUSE `_ framework. diff --git a/docs/configuration.rst b/docs/configuration.rst index e2b94e2..fdeabe8 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -1,59 +1,59 @@ .. _swh-fuse-config: Configuration ============= -The configuration for the Software Heritage virtual file system resides in the +The configuration for the Software Heritage Filesystem resides in the ``swh > fuse`` section of the shared `YAML `_ configuration file used by all Software Heritage tools, located by default at ``~/.config/swh/global.yml``. The configuration file location is subject to the `XDG Base Directory `_ specification as well as explicitly overridden on the :ref:`command line ` via the ``-C/--config-file`` flag. The following sub-sections and fields can be used within the `swh > fuse` stanza: - ``cache``: - ``metadata``: where to store the metadata cache, must have either a ``in-memory`` boolean entry or a ``path`` string entry (with the corresponding disk path) - ``blob``: where to store the blob cache, same entries as the ``metadata`` cache - ``web-api``: - ``url``: archive API URL - ``auth-token``: authentication token used with the API URL If no configuration is given, default values are: - ``cache``: all cache files are stored in ``$XDG_CACHE_HOME/swh/fuse/`` (or ``~/.cache/swh/fuse`` if ``XDG_CACHE_HOME`` is not set) - ``web-api``: default URL is , with no authentication token Example ------- Here is a full ``~/.config/swh/global.yml`` example, showcasing different cache storage strategies (in-memory for metadata and on-disk for blob), using the default Web API service: .. code:: yaml swh: fuse: cache: metadata: in-memory: true blob: path: "/path/to/cache/blob.sqlite" web-api: url: "https://archive.softwareheritage.org/api/1/" auth-token: null diff --git a/docs/design.md b/docs/design.md index a81ec70..ed8e952 100644 --- a/docs/design.md +++ b/docs/design.md @@ -1,235 +1,235 @@ -# Software Heritage virtual filesystem (SwhFS) --- Design notes +# Software Heritage Filesystem (SwhFS) --- Design notes The [Software Heritage](https://www.softwareheritage.org/) {ref}`data model ` is a [Direct Acyclic Graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) (DAG) with nodes of different types that correspond to source code artifacts such as directories, commits, etc. Using this [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) module -(*SwhFS* for short) you can locally mount, and then navigate as a (virtual) -file system, parts of the archive identified by {ref}`Software Heritage +(*SwhFS* for short) you can locally mount, and then navigate as a +filesystem, parts of the archive identified by {ref}`Software Heritage identifiers ` (SWHIDs). To retrieve information about the source code artifacts, SwhFS interacts over the network with the Software Heritage archive via its {ref}`Web API `. ## Architecture SwhFS in context ([C4](https://en.wikipedia.org/wiki/C4_model) context diagram): ```{image} images/arch-context.svg :align: center ``` Main components of SwhFS (C4 container diagram): ```{image} images/arch-container.svg :align: center ``` ## Command-line interface $ swh fs mount [SWHID]... will mount the Software Heritage archive at the local ``, the *SwhFS mount point*. From there, the user will be able to lazily load and navigate the archive using SWHID at entry points. If one or more SWHIDs are also specified, the corresponding objects will be pre- fetched from the archive at mount-time and available at `/archive/`. For more details see the {ref}`CLI documentation `. ## Mount point The SwhFS mount point contain: - `archive/`: initially empty, this directory is lazily populated with one entry per accessed SWHID, having actual SWHIDs as names (possibly sharded into `xy/../SWHID` paths to avoid overcrowding `archive/`). - `meta/`: initially empty, this directory contains one `.json` file for each `` entry under `archive/`. The JSON file contain all available meta information about the given SWHID, as returned by the Software Heritage Web API for that object. Note that, in case of pagination (e.g., snapshot objects with many branches) the JSON file will contain a complete version with all pages merged together. ## File system representation SWHID are represented differently on the file-system depending on the associated node types in the Software Heritage graph. Details are given below, for each node type. ### `cnt` nodes (blobs) Content leaves (AKA blobs) are represented on disks as regular files, containing the corresponding bytes, as archived. Note that permissions are associated to blobs only in the context of directories. Hence, when accessing blobs from the top-level `archive/` directory, the permissions of the `archive/SWHID` file will be arbitrary and not meaningful (e.g., `0x644`). ### `dir` nodes (directories) Directory nodes are represented as directories on the file-system, containing one entry for each entry of the archived directory. Entry names and other metadata, including permissions, will correspond to the archived entry metadata. Note that SwhFS is mounted read-only, no matter what the permissions say. So it is possible that, in the context of a directory, a file is presented as writable, whereas actually writing to it will fail with `EPERM`. ### `rev` nodes (commits) Revision (AKA commit) nodes are represented on the file-system as directories with the following entries: - `root`: source tree at the time of the commit, as a symlink pointing into `archive/`, to a SWHID of type `dir` - `parents/` (note the plural): a virtual directory containing entries named `1`, `2`, `3`, etc., one for each parent commit. Each of these entry is a symlink pointing into `archive/`, to the SWHID file for the given parent commit - `parent` (note the singular): present if and only if the current commit has a single parent commit (which is the most common case). When present it is a symlink pointing into `archive/` to the SWHID for the sole parent commit - `history`: a virtual directory containing all the parents commit until the root commit. Entries are listed as symlinks with the SWHID as directory name, pointing into `archive/SWHID`, and are returned in a topological ordering similar to `git log` ordering. - `meta.json`: metadata for the current node, as a symlink pointing to the relevant `meta/.json` file ### `rel` nodes (releases) Release nodes are represented on the file-system as directories with the following entries: - `target`: target node, as a symlink to `archive/` - `target_type`: type of the target SWHID, as a 3-letter code - `root`: present if and only if the release points to something that (transitively) resolves to a directory. When present it is a symlink pointing into `archive/` to the SWHID of the given directory - `meta.json`: metadata for the current node, as a symlink pointing to the relevant `meta/.json` file ### `snp` nodes (snapshots) Snapshot nodes are represented on the file-system as directories with one entry for each branch in the snapshot. Branch names are subject to URL encoding, in order to avoid problematic characters (e.g., `/` are replaced by `%2F`). Each entry is a symlink named as the branch name, URL encoded (to avoid problematic characters such as `/`, which becomes `%2F`). The symlink target points into `archive/` to the SWHID corresponding to the branch target. ## Caching SwhFS retrieves both metadata and file contents from the Software Heritage archive via the network. In order to obtain reasonable performances several caches are used to minimize network transfer. Caches are stored on disk in SQLite DB(s) located under `$XDG_CACHE_HOME/swh/fuse/`. ```{todo} - potential improvement: store blobs larger than a threshold on disk as files rather than in SQLite, e.g., under `$XDG_CACHE_HOME/swh/fuse/objects/` ``` All caches are persistent (i.e., they survive the restart of the SwhFS process) and global (i.e., they are shared by concurrent SwhFS processes). We assume that no cache *invalidation* is necessary, due to intrinsic properties of the Software Heritage archive, such as integrity verification and append-only archive changes. To clean the caches one can just remove the corresponding files from disk. ### Metadata cache SWHID → JSON metadata The metadata cache map each SWHID to the complete metadata of the referenced object. This is analogous to what is available in `meta/.json` file (and generally used as data source for returning the content of those files). Cache location on-disk: `$XDG_CACHE_HOME/swh/fuse/metadata.sqlite` ### Blob cache cnt SWHID → bytes The blob cache map SWHIDs of type `cnt` to the bytes of their archived content. In general, each SWHID that has an entry in the blob cache also has a matching entry in the metadata cache for other blob attributes (e.g., checksums, size, etc.). The blob cache entry for a given content object is populated, at the latest, the first time the object is `open()`-d. It might be populated earlier on due to prefetching, e.g., when a directory pointing to the given content is listed for the first time. Cache location on-disk: `$XDG_CACHE_HOME/swh/fuse/blob.sqlite` ### Dentry cache dir SWHID → directory entries The dentry (directory entry) cache map SWHIDs of type `dir` to the directory entries they contain. Each entry comes with its name as well as file attributes (i.e., all its needed to perform a detailed directory listing). Additional attributes of each directory entry should be looked up on a entry by entry basis, possibly hitting the metadata cache. The dentry cache for a given dir is populated, at the latest, when the content of the directory is listed. More aggressive prefetching might happen. For instance, when first opening a dir a recursive listing of it can be retrieved from the remote backend and used to recursively populate the dentry cache for all (transitive) sub-directories. ### Parents cache rev SWHID → parent SWHIDs The parents cache map SWHIDs of type `rev` to the list of their parent commits. The parents cache for a given rev is populated, at the latest, when the content of the revision virtual directory is listed. More aggressive prefetching might happen. For instance, when first opening a rev virtual directory a recursive listing of all its ancestor can be retrieved from the remote backend and used to recursively populate the parents cache for all ancestors. ### History cache rev SWHID → ancestor SWHIDs The history cache map SWHIDs of type `rev` to a list of `rev` SWHIDs corresponding to all its revision ancestors, sorted in reverse topological order. As the parents cache, the history cache is lazily populated and can be prefetched. To efficiently store the ancestor lists, the history cache represents ancestors as graph edges (a pair of two SWHID nodes), meaning the history cache is shared amongst all revisions parents. diff --git a/docs/images/arch-container.pdf b/docs/images/arch-container.pdf index b2d6c52..2ec062a 100644 Binary files a/docs/images/arch-container.pdf and b/docs/images/arch-container.pdf differ diff --git a/docs/images/arch-container.puml b/docs/images/arch-container.puml index a1e105a..8c4d645 100644 --- a/docs/images/arch-container.puml +++ b/docs/images/arch-container.puml @@ -1,26 +1,26 @@ @startuml arch-container ' !includeurl https://raw.githubusercontent.com/RicardoNiepel/C4-PlantUML/master/C4_Container.puml !include c4-plantuml/C4_Container.puml -' title Container diagram for the Software Heritage virtual filesystem (SwhFS) +' title Container diagram for the Software Heritage Filesystem (SwhFS) left to right direction Person(user, "Developer") System(tool, "Programming Tool") Container(fuse, "FUSE", "kernel", "Delegates filesystem implementation to user space") System_Boundary(swhfs_bound, "SwhFS") { Container(swh_fuse, "swh.fuse", "pyfuse3, libfuse, FUSE", "Provides a filesystem view of Software Heritage objects") ContainerDb(cache, "Cache", "SQLite or in-memory", "Local cache for objects and metadata") Rel(swh_fuse, cache, "Reads objects and metadata from", "SQL") } System_Ext(archive, "Software Heritage archive") Rel(user, swhfs_bound, "Controls user-space daemon", "Command-line interface") Rel(user, fuse, "Browses", "POSIX filesystem") Rel(tool, fuse, "Accesses", "POSIX filesystem") Rel(fuse, swh_fuse, "Calls user-space daemon", "FUSE API") Rel(swh_fuse, archive, "Retrieves objects and metadata from", "REST API") @enduml diff --git a/docs/images/arch-context.pdf b/docs/images/arch-context.pdf index b4da999..7b72160 100644 Binary files a/docs/images/arch-context.pdf and b/docs/images/arch-context.pdf differ diff --git a/docs/images/arch-context.png b/docs/images/arch-context.png index ad65a76..ca98f3d 100644 Binary files a/docs/images/arch-context.png and b/docs/images/arch-context.png differ diff --git a/docs/images/arch-context.puml b/docs/images/arch-context.puml index 68d120a..8f88858 100644 --- a/docs/images/arch-context.puml +++ b/docs/images/arch-context.puml @@ -1,15 +1,15 @@ @startuml arch-context ' !includeurl https://raw.githubusercontent.com/RicardoNiepel/C4-PlantUML/master/C4_Context.puml !include c4-plantuml/C4_Context.puml -' title System context diagram for the Software Heritage virtual filesystem (SwhFS) +' title System context diagram for the Software Heritage Filesystem (SwhFS) left to right direction Person(user, "User") -System(swhfs, "Software Heritage virtual filesystem\n(SwhFS)") +System(swhfs, "Software Heritage Filesystem\n(SwhFS)") System_Ext(archive, "Software Heritage archive") Rel(user, swhfs, "Accesses", "Filesystem") Rel(swhfs, archive, "Retrieves objects and metadata from", "REST API") @enduml diff --git a/docs/images/arch-context.svg b/docs/images/arch-context.svg index 5a2d8af..f55e19f 100644 --- a/docs/images/arch-context.svg +++ b/docs/images/arch-context.svg @@ -1,136 +1,136 @@ «person»User«system»Software Heritagevirtual filesystem(SwhFS)«system»Software HeritageFilesystem(SwhFS)«external_system»Software HeritagearchiveAccesses[Filesystem]Retrieves objectsand metadata from[REST API]Retrieves objectsand metadata from[REST API] \ No newline at end of file