Page MenuHomeSoftware Heritage
Differential D4064

Early FUSE implementation, with support for blob and directory objects
ClosedPublic
Actions

Authored by haltode on Sep 28 2020, 2:27 PM.

Details

Reviewers
zack
seirl
Group Reviewers
Reviewers
Maniphest Tasks
T1926: FUSE filesystem to navigate the archive
Commits
rDFUSEee058855d1a5: fs: move ArtifactEntry class to artifact.py
rDFUSEc1bfbcece32a: fuse: remove unnecessary fd maps (use inode instead)
rDFUSE0dd35220b085: fuse: remove unnecessary inode2path
rDFUSE3c79873af9ee: cache: use non-typified option of WebAPIClient methods
rDFUSE34c5d37c0d77: fuse: get_blob: use bytes instead of string
rDFUSEf16985f52dab: fuse: add async to the codebase
rDFUSE181b910f7088: fuse: lookup: raise and log exception when name not found
rDFUSE679db87aa057: fuse: read: use `offset` and `length` parameters
rDFUSE2b688f8f3918: cli: use SWHIDParamType from swh.model.cli
rDFUSEf7194e5a192c: fuse: remove internal entry2inode dictionnary
rDFUSEfc0dd584018c: fs: rework code architecture
rDFUSE9417ab147eab: tests: add unit tests framework
rDFUSE9d60949c1295: tox.ini: pin black version to 19.10b0
rDFUSE8c77b5cdc2eb: cli: use a config.yml file to configure cache/web-api
rDFUSE0c131526e2d4: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
rDFUSE4a67601e741e: cache: separate metadata and blob caches
rDFUSE515adac52cde: fuse: use WebAPIClient to retrieve raw content data
rDFUSE1190063fe350: fuse: small code clean up
rDFUSEd2f70eb1f0f4: cli: by default run as daemon (or use -f/--foreground)
rDFUSE3fb6a556154b: fuse: add documentation and type hints
rDFUSE22c4363b9734: fuse: first working proof of concept
Summary

This is an early implementation of the swh-fuse system. What is working right
now:

  • mounting dir/cnt
  • ls/cd/cat
  • disk cache for metadata/blob

What needs to be done before merging:

  • Add documentation
  • Add type hints everywhere
  • Add unit/functional tests

There are still a lot of TODO in the code, but feel free to comment!

Related to T1926

Diff Detail

Repository
rDFUSE FUSE virtual file system
Branch
feature/early-implem
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 15981
Build 24596: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 24595: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
haltode added a comment.Oct 5 2020, 10:08 AM
  • Clean commit history
  • Remove internal entry2inode dictionnary
haltode retitled this revision from WIP: early FUSE implementation to Early FUSE implementation.Oct 5 2020, 10:08 AM
swh-public-ci added a comment.Oct 5 2020, 10:09 AM

Build has FAILED

Patch application report for D4064 (id=14588)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Changes applied before test
commit f6db4a64d8d1885c769d4461cc932aabf8998ab6
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit 6899845367e8865b5b5dcec7f3d13893d9be2f4e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit e6a88521f9dc909f5ba13884205c81a6e5e12941
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit f43c6958ac388efbbbf651b89ec0965afc97ca01
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit ba649b0787a6d12b34e6f8d02a030004dc066976
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 2955a34287645b0c9c1db2ca15880628e89fd778
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit 6a28ec125f358e89d75e1672ddf0340a9b059fc0
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit b8098717058bd98fe8e36e31efd1db6c0a148f5f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 83333b6e2f10338eacbb418262241ba3114e5106
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 43a98afdaf65b2fffd4cd0c2557f6ee02791832d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit a07432d020059039d9acef4af1817f4dd49274e1
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit b5a01874e2663918084e3d46064fe41cff56db7e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/21/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/21/console

Harbormaster failed remote builds in B15869: Diff 14588!Oct 5 2020, 10:09 AM
haltode updated this revision to Diff 14589.Oct 5 2020, 10:11 AM

Use enter_context instead of enter from the contextlib.

swh-public-ci added a comment.Oct 5 2020, 10:12 AM

Build has FAILED

Patch application report for D4064 (id=14589)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Changes applied before test
commit b185e5bdcfc822183359ee0760b0d4464d50457f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit a927aa379e52b5d8a82d770733efa327caabc53d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit c3ed85ffccc4ff430ced133e0565ebe8a3bae472
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit 9ffe74d1e9edef476a94c8450d7134db7a8ed419
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit c8a4b7dc243b823de0460413f28f9f7fcc57fe78
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit d3f0fbb6b25504fecf9eb6c1c278df2e076f7206
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit aef91f49c3f4097b8e40c5280fa9bf93cb0877ef
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit dde7bd2c6a912f3c6276902ec37bef120b8a14db
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 43dd015731b5342273483d2603c6090d9f8b7107
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit b5009ee9c32110f89e87bb8828b510a180f7379c
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit 170f94db841bf976630bc67a5a8355e130477a9d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit ce5da450c256e0e0f4c4c02d733b3ef9fc6f1026
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/22/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/22/console

Harbormaster failed remote builds in B15870: Diff 14589!Oct 5 2020, 10:12 AM
haltode marked 5 inline comments as done.Oct 5 2020, 10:12 AM
haltode added inline comments.Oct 5 2020, 10:31 AM
swh/fuse/cli.py
47

I opened a diff about this: D4139

haltode updated this revision to Diff 14593.Oct 5 2020, 10:39 AM

cli: use SWHIDParamType from swh.model.cli

haltode marked an inline comment as done.Oct 5 2020, 10:39 AM
haltode added inline comments.
swh/fuse/cli.py
47

The diff has been landed and I updated the swh-fuse CLI code.

swh-public-ci added a comment.Oct 5 2020, 10:40 AM

Build has FAILED

Patch application report for D4064 (id=14593)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Changes applied before test
commit 842342046b57f9b680b6b8f30fc28a542dae97aa
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli

commit 62367fcba499ea94fe6c8ddb343d4b5a5cbe97f2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit 2f321f356c7ba362575636ed414b20db54a94c03
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit f23191f426795f8d5b2e72013d02e19f0f83603b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit aac1c780959f841ab7908eb0cc765ba6cef095a0
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit ba58bbcf988655592eea2e8cc796cbd20d668c22
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit a5683e7000c565d7d60cc3255f346b17ecb54ec3
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit 3026c4594422dc035a4075035744fff329831378
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit db67c8d92450c22c9e3e6ddc05b15f193f1dd4eb
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 2d7168f3073d60fa22a8b7e397ffcb2c2e5ca758
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 4befb1063908fe5645f110d7f8dd15d6c559df36
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit fe4b0a35d09a4502a9599ac075b1b8ad2709c9df
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit 6047834275e5a101443434e6954a5d4dfab497c5
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/23/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/23/console

Harbormaster failed remote builds in B15874: Diff 14593!Oct 5 2020, 10:40 AM
haltode updated this revision to Diff 14595.Oct 5 2020, 11:49 AM

Move cache pre-fetching in fuse main

haltode marked 2 inline comments as done.Oct 5 2020, 11:50 AM
swh-public-ci added a comment.Oct 5 2020, 11:50 AM

Build has FAILED

Patch application report for D4064 (id=14595)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Changes applied before test
commit b67b45ca1686b66f9347a1a8026b28589624c3a2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli

commit c1d5e366ff33c990393efa4e8121aa36efc3bc7f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit c60e4965a84a73e2369d958f68ec509174215757
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit 354c113e8da9d9953a67bdd7c1d37cb584302486
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit 88d68558f4b96d56183ba058c2850130b9754edf
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit a12a87c13dc630f2d34dd9a97e10ea3611df65a6
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit c5b56891a9f263e05d9ab590701c56f7715b7191
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit e3040c0c113def93fc2db34c5d5a11282e0e7b13
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit 428a147cb16d8e8d982fa1fc743c8877eb72a02f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 39b57444e7c6821447759acd7131c9ae1945e7ae
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 658c82deb5daee117304855474a465286ede5447
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit c64e482629d1fda6ab845457c4b2090ac37b9101
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit 2feb1be2f53e3628f21abce231057e0250471a7b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/24/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/24/console

Harbormaster failed remote builds in B15876: Diff 14595!Oct 5 2020, 11:50 AM
haltode updated this revision to Diff 14597.Oct 5 2020, 12:02 PM
  • fuse: read: use offset and length parameters
  • fuse: lookup: raise and log exception when name not found
swh-public-ci added a comment.Oct 5 2020, 12:02 PM

Build has FAILED

Patch application report for D4064 (id=14597)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Applying: fuse: read: use `offset` and `length` parameters
Applying: fuse: lookup: raise and log exception when name not found
Changes applied before test
commit 403e7f6fa5a17b09b07e161a236f90f7dff4e622
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit 21c2ed8678698d9de15a3a79e931c6d4f8259dfa
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit 549652d54caaa94ed963ee5fbaae8aac5f0ea893
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit 93a4a002457c7592ef222219f5e084c381ed1091
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit 8d3603c44ac541f9ccfe4392f035805f7233c490
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit aca31e7d012dfa1466f6402b92f3e48b784b20af
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit b985cc7e0d45ed8fc1328bf03582c02fbb95425e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit 3b762a8017c14a7a731801699478769a83395783
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 8b3f741904b14b73b12c803809d83a8432a9b706
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit 7558199dda1713393d0c7ee240db13be7a1e12b8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit bd3af91fe3752be09dde867892e67eb16f84328d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 1795610b7997894ef856531d3712db0a18433978
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 202fb8baaf6d79eeca763c5889a3b47da1aadd82
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit f8d1a006dd34fa0ccd72f76d052215e1ec453a21
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit 9dc222e5e4d61d69468a821de50370d00d4a77bc
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/25/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/25/console

Harbormaster failed remote builds in B15878: Diff 14597!Oct 5 2020, 12:02 PM
zack retitled this revision from Early FUSE implementation to Early FUSE implementation, with support for blob and directory objects.Oct 5 2020, 2:09 PM
swh-public-ci added a comment.Oct 5 2020, 3:55 PM

Build has FAILED

Patch application report for D4064 (id=14597)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Applying: fuse: read: use `offset` and `length` parameters
Applying: fuse: lookup: raise and log exception when name not found
Changes applied before test
commit 75920194af0ceb43f69997e731418eed0370d275
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit f135edb2d6012b28ae399fb95c517e250a74aff8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit 89c5c11adfa0194f6fd56f85d6341a1960128083
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit 29f92840b12d92fbbc0280166b3fc1f5713d5516
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit 2197f9aa3dad2b08143bd76e74f8569d5bbf70c9
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit b8ea3cbb0f81064efbd82a9860dd10a788392639
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit 14b0ca17c1dff7a5e826b104c48030a38e877a8e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit 79e45c6c183a7f00a4f0283179b94f4cf509e573
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 9db99ca68e517e2358a3b783c3f03c8c2fc0f9c2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit dea533ff2ac45a4f6586c13568d8e2e8550f0afa
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit b1a731633e1f5e81084f98771e81d1ab2df49b15
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 4b51a7bdc4157df7bd47d004f58cab8ca945a852
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 07630c286186f3a0883fd60390990fc2e68cbf70
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit 10d71f9ddc8205c13224d1de5104d6f3ca8c8ea4
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit b938dfc32d8a987e60dcb18c9cc0247ffa08301f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/26/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/26/console

swh-public-ci added a comment.Oct 5 2020, 4:01 PM

Build has FAILED

Patch application report for D4064 (id=14597)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Applying: fuse: read: use `offset` and `length` parameters
Applying: fuse: lookup: raise and log exception when name not found
Changes applied before test
commit e4adad1fd9a3a29d62141e6637aaf740ee1fa142
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit 608f8a812f631ae0c29337197d3964c1420891a6
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit 62e9bf252231037b157a915909ac418db9c2ede8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit 3db8c03308cdf70c4d60fc364af59a2c1aa7eee8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit 252070c8c794ad60853395a1dcb380dd27ae4e3c
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit 6156e13c59b6e6e0954696d646cbb59773ca8a7b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit 23e42bafd5248d1a83d7ad22a302096bb16d2483
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit ecb8fbeba253fbb0cffe4d14cb7992789e027f64
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 00bb9347e07268de83360fe528fe6577013e7afe
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit be6deb7797a805323c27f7a22c6545eed92d9f70
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit 5fbc592e50f62727ca6f0f656a0a5756ce085c90
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 2099da5bb98cab68d7baec4e97b345416ad6bd9a
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit b8b2d6c0592eadcf76493d06b42735fb79d4c540
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit 542f20c6fd43a05e9553f48d289d4a43aae1e6d3
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit 5facbb2e94682ae5fdec2c63061156722c0b27c4
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/27/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/27/console

swh-public-ci added a comment.Oct 6 2020, 9:49 AM

Build is green

Patch application report for D4064 (id=14597)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Applying: fuse: read: use `offset` and `length` parameters
Applying: fuse: lookup: raise and log exception when name not found
Changes applied before test
commit dcac4b5213b7c1f6adb80ee402c3ac44fe85d41d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit b3cef928453f81d81109fb521ec4772522c99ba4
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit 4479aa3000f13865c5e9519026dac3a87e402018
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit 8ffee8fc000d505bf78c028980887bbb54c72665
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit 2cbbd754d94b1a2f28bdf15857aed78a4a8639c2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit 08420cee9c8f3b52d226635a51a5f1568afcad18
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit beda93580bdae610c344beb9209a2cda3f4f4745
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit bb81502f33d8385e1205955c3c2903342d4f4697
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 66e61c04275efc03447b1f6bd1d8185cd1aa7df0
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit 2520d186d37d81179515e0d1bd67cb79fbea364d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit c463514457cd2274561dbf43e82a6e3be05146a4
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 26480c686492c6135ae7d0f1b3271e6719217f5c
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 8a3ba0c8832a8e0a9fc09aac4f2f7aee109d7ac8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit fa9c848ffc6dd841e1bedad1a4394b7a40317cae
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit e64da8e5383f8d7a079018b684daec615d1bd29e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/28/ for more details.

swh-public-ci added a comment.Oct 6 2020, 9:51 AM

Build is green

Patch application report for D4064 (id=14597)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Applying: fuse: read: use `offset` and `length` parameters
Applying: fuse: lookup: raise and log exception when name not found
Changes applied before test
commit 83699ba1d2d80c5608a883db93fd25f36366ba4e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit 15b978b0b9d33f63a21be9e93bf78c74fa6a1bb0
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit be97332da10acde14c18f3a99aaa3a1d87feb4ac
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit ad38a02e73827ec7b79644c1f5536d1f46a313c5
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit ef622f23f50ef0d9efa3269e2a47ac181f281a61
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit 56b098d13e7d36f0a9ed5bea9eec687fb2e97bb8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit 9a60190d703cb436d2b3a06a5ae41866a26508e2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit 7797176c41e79e6c8d1728e6a4eec4aabd8ada46
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 099bf4fb40f228f5c2ea7c972ba6b9038bf2a3d9
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit 4d93768506c4694ee6f47d528c01fee53b618f63
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit 1bde390278180a3f7c7c3385071121cc566c4183
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 9e806fe70e8fc949e39f4c15c0fcd1371b8f1181
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 967be1012f6122ea5276eecb7da297cbd79df60e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit 2b44e84547fd1dad2b1d26378e834a7c5cff9356
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit 669ebbdadc01249708ea94e1dd5a4e8620edbe4f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/29/ for more details.

Harbormaster completed remote builds in B15878: Diff 14597.Oct 6 2020, 9:51 AM
haltode updated this revision to Diff 14633.Oct 6 2020, 10:47 AM

Add async to the codebase

swh-public-ci added a comment.Oct 6 2020, 10:48 AM

Build is green

Patch application report for D4064 (id=14633)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Applying: fuse: read: use `offset` and `length` parameters
Applying: fuse: lookup: raise and log exception when name not found
Applying: fuse: add async to the codebase
Changes applied before test
commit ba2ad000e8f644fd52edb498228393b1eddaf3a0
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 10:42:50 2020 +0200

    fuse: add async to the codebase
    
    - Change class interface to match async interface (eg: iter -> aiter)
    - Sync code is now running with `run_in_executor`
    - Use aiosqlite as the async interface for sqlite

commit 827b56305429f02c3e99569898fa706c305ef79b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit 34eb819ec2320cd9f64bf33b7e196428381ac921
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit df34024680de72d576195ccf51083a26a3c06686
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit d5758b5630b1c0d19e6d0c30c24ce02f3264e535
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit 61fa834c5ef54ee4c864a4568419b51dbd38627c
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit 93783d217bec06522859ebd38856c25719429c8f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit e8f00088b6c0230c5cc0102bd885ae800e33fcc9
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit b82d338024cf9bd563ad56097dcaf37a2da50f9e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 4846e33e9c90e5d6960c565c17c5ab50f97b564d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit 4efdf94a0dcc3404b43e9269b416836ab8a2914c
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit 5d83541da008a43e8c60412b6c29b1cd9bdd4134
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 8dbf21ce6c202483959fa2a961f82d86563b9c79
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit fe9f444c9c66cf99f016169eea5df43ffeb57309
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit 45d4614214c0a2bd8af5238c689a2016dfcc4821
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit f83e3d3cf064dcc02c4c5b1ea64674f91ad8dd7f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/30/ for more details.

Harbormaster completed remote builds in B15913: Diff 14633.Oct 6 2020, 10:48 AM
haltode marked 3 inline comments as done.Oct 6 2020, 10:48 AM
haltode added inline comments.
swh/fuse/fuse.py
169–173

This is weird without itertools.islice, maybe you have a better solution?

haltode marked an inline comment as not done.Oct 6 2020, 10:51 AM
haltode updated this revision to Diff 14639.Oct 6 2020, 12:10 PM

fuse: get_blob: use bytes instead of string

swh-public-ci added a comment.Oct 6 2020, 12:11 PM

Build is green

Patch application report for D4064 (id=14639)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Applying: fuse: read: use `offset` and `length` parameters
Applying: fuse: lookup: raise and log exception when name not found
Applying: fuse: add async to the codebase
Applying: fuse: get_blob: use bytes instead of string
Changes applied before test
commit 069b28f5640f0a37c0682074fd1f7d3ae7a206b9
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 12:09:27 2020 +0200

    fuse: get_blob: use bytes instead of string
    
    We do not know the encoding beforehand so manipulate every blob coming
    from the archive as bytes.

commit 301a2848862ce2d44188185ab12722070cbf0aaa
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 10:42:50 2020 +0200

    fuse: add async to the codebase
    
    - Change class interface to match async interface (eg: iter -> aiter)
    - Sync code is now running with `run_in_executor`
    - Use aiosqlite as the async interface for sqlite

commit 90647e2d404665d1ac25dbc0ea60de250d53b5c6
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit eb55aea3c9dba4bba4d71fa6d3b3e321b5431df4
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit 9bed55ef77d0cfd65f35faada285a98389d0c3da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit 9a68db13e0f8c21230d55f9ae0c0cc6310e4a3ca
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit 15a95a7e18ac3b24346b97068303211a3241acab
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit 8318705da1e79ec1efe6d4d916fbc9954b706fa5
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit 542a9143be6ee86c520cb96f01d2159ceffca8ee
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit 00132ed344168a4a4cd89fb2fdd7373e4a03f0d2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 358fd16291a9dee0a9c3655a0a83a5a773c85486
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit 4eb6044261e7bc6bb9885327a356938d7ac8f235
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit 8a40f8788246caa048769e819a0da56b30b1bcfc
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit e979e36f47a025154c17d42645495a5fabfaa868
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit c4b1dfbf6f437b2104f92ad7608965a7edb5d5ac
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit 0bda870bc8bd3460a1288aa85bb6b504eb9177df
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit 281fc73c19b99bae57ad6620db814c4bcdf1fc6d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/31/ for more details.

Harbormaster completed remote builds in B15919: Diff 14639.Oct 6 2020, 12:11 PM
haltode updated this revision to Diff 14667.Oct 6 2020, 4:54 PM

fuse: remove unnecessary fd maps (use inode instead)

swh-public-ci added a comment.Oct 6 2020, 4:55 PM

Build is green

Patch application report for D4064 (id=14667)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Applying: fuse: read: use `offset` and `length` parameters
Applying: fuse: lookup: raise and log exception when name not found
Applying: fuse: add async to the codebase
Applying: fuse: get_blob: use bytes instead of string
Applying: fuse: remove unnecessary fd maps (use inode instead)
Changes applied before test
commit 93d04db8b49a3e0239195fca6b4cefb774179b3f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 16:53:18 2020 +0200

    fuse: remove unnecessary fd maps (use inode instead)

commit 35828f609a30d8aca5a82032cfebf373e6e37ac2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 12:09:27 2020 +0200

    fuse: get_blob: use bytes instead of string
    
    We do not know the encoding beforehand so manipulate every blob coming
    from the archive as bytes.

commit 993893aca103e0989a0b71b7e25eac4d0646d11c
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 10:42:50 2020 +0200

    fuse: add async to the codebase
    
    - Change class interface to match async interface (eg: iter -> aiter)
    - Sync code is now running with `run_in_executor`
    - Use aiosqlite as the async interface for sqlite

commit afadb837a858e869aaa70574dd83d89282e31951
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit 2dc96fc4e9d983fded3ce3706cc675095a2bbb95
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit 8954818e94a689329aa2ed1e283660f1569c16e0
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit 8f69f5a2ab78dd9240f2c6ab920d0a0bffad53e5
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit 06b8e17bf9a18ca3ab511845f3f792274c849a3c
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit b74a0d994dfa4e08cc955b7b914aaa50122e04fb
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit a14aaa71370bc89f394fbeb5d856ad67085bed21
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit 4d3e5bc24b472d5343753fe4b29f0dc2dc2e97f1
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 1f47672672b67dd7024228c5e98045a02b9d342a
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit 3857da64518a803cfbd96b6915e74e0c466bcaca
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit 6818ac95f385e8d1d85ece63d42c1c66a402a664
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit d39d9145ec4e93140a901d61b5315545098348fe
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 90b32f00a081249c7670321f01464f1268484a1d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit 62beb2bba1bb412cc4292b0cf65ce5f0ade6ceb0
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit 3e516c8cce5e88d71e89ac89a7ac5076cb93e8da
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/32/ for more details.

Harbormaster completed remote builds in B15944: Diff 14667.Oct 6 2020, 4:55 PM
seirl requested changes to this revision.EditedOct 6 2020, 6:34 PM

This is looking pretty great. I see three more good refactoring possibilities:

  • drop inode2path, this mapping doesn't seem to serve a purpose anymore
  • the way files are split in fs/ is weird to me, like ArtifactEntry not being in artifact. I don't see an obvious way of splitting them, maybe they just all belong in the same file?
  • I agree that the fuse class <-> entries cycle isn't so elegant. I think we actually don't have to do that. We could create another object, say FuseBackend, that contains the cache, get_blob, get_metadata(), get_cached_swhids(), alloc_inode() and inode2entry(). We can then pass this backend around between the Fuse class and all the entries.

Does that sound better?

swh/fuse/fuse.py
26

navigate*

114

This method should go in the FuseEntry base class.

215

Unlike what you had before, this won't get called if there is an exception before. Maybe use a finally block?

This revision now requires changes to proceed.Oct 6 2020, 6:34 PM
haltode updated this revision to Diff 14678.Oct 7 2020, 9:42 AM
  • fuse: remove unnecessary inode2path
  • cache: use non-typified option of WebAPIClient methods
  • fs: move ArtifactEntry class to artifact.py
haltode marked 2 inline comments as done.Oct 7 2020, 9:43 AM
swh-public-ci added a comment.Oct 7 2020, 9:43 AM

Build has FAILED

Patch application report for D4064 (id=14678)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Applying: fuse: read: use `offset` and `length` parameters
Applying: fuse: lookup: raise and log exception when name not found
Applying: fuse: add async to the codebase
Applying: fuse: get_blob: use bytes instead of string
Applying: fuse: remove unnecessary fd maps (use inode instead)
Applying: fuse: remove unnecessary inode2path
Applying: cache: use non-typified option of WebAPIClient methods
Applying: fs: move ArtifactEntry class to artifact.py
Changes applied before test
commit 21a6a5b5e295c94ae8d3b0911f659e548107cf18
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Oct 7 09:25:07 2020 +0200

    fs: move ArtifactEntry class to artifact.py

commit e38976770588b5a0afc3c12832b966694b691c56
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 17:26:19 2020 +0200

    cache: use non-typified option of WebAPIClient methods

commit efa125fcc904a735a6d07a272478a675203995c3
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 17:38:58 2020 +0200

    fuse: remove unnecessary inode2path

commit 2dcdb26b0ca75f60c9561aeb0a480fc87188b5a1
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 16:53:18 2020 +0200

    fuse: remove unnecessary fd maps (use inode instead)

commit 63186b6782acf5194659524a578972176747d921
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 12:09:27 2020 +0200

    fuse: get_blob: use bytes instead of string
    
    We do not know the encoding beforehand so manipulate every blob coming
    from the archive as bytes.

commit cfc644f278db34d724cd4d648fad4b61db76d0ce
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 10:42:50 2020 +0200

    fuse: add async to the codebase
    
    - Change class interface to match async interface (eg: iter -> aiter)
    - Sync code is now running with `run_in_executor`
    - Use aiosqlite as the async interface for sqlite

commit aa9939a77e612ec932a08b2e0f4c05cc9da9e0a5
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit e84f09c704ee28420c7cd99bf40a8034e2c5be5e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit 8f9da1f5a54fbc71af7ba03b73d03270914cdb73
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit 39ca64b5efd2a5356add2a514b008432c9c0f23e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit 3bf92a765d8a3a309e2e7e8f25ddb46d5f5e5381
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit e355387c752454ad0e0d1f2bbce2e7e5fb38e302
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit 533afeceb41049eb42a0957b5801c8c88a043d3b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit 82aeefaa9b3c4b478e76b7d03a61378700786a7e
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 7b3810f6d05f7768b2a7d12ddde3a2ecfff837c8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit dbab9ef9d5db71269dda7e3ea2acc42527a823d0
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit f0332e05d5b1a839ce072912fd3ba81a801df115
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 0ff2ef87a1659ebca49db6f3662cb70d6b6eb8c1
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 42c072d04b752da1a1e16bef1be71fcba76725db
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit 7c9f8182fbd1cee7514775147ade919dc560cdf6
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit 85955cae761d4569d27a1fd6cd68e3c024a91b3b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/33/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/33/console

Harbormaster failed remote builds in B15955: Diff 14678!Oct 7 2020, 9:43 AM
haltode updated this revision to Diff 14687.Oct 7 2020, 12:09 PM

Small style fix.

swh-public-ci added a comment.Oct 7 2020, 12:10 PM

Build has FAILED

Patch application report for D4064 (id=14687)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Applying: fuse: read: use `offset` and `length` parameters
Applying: fuse: lookup: raise and log exception when name not found
Applying: fuse: add async to the codebase
Applying: fuse: get_blob: use bytes instead of string
Applying: fuse: remove unnecessary fd maps (use inode instead)
Applying: fuse: remove unnecessary inode2path
Applying: cache: use non-typified option of WebAPIClient methods
Applying: fs: move ArtifactEntry class to artifact.py
Changes applied before test
commit 070f8baeedfeb5b94f9347d472b3d00978df75ff
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Oct 7 09:25:07 2020 +0200

    fs: move ArtifactEntry class to artifact.py

commit f148806f98c3e5bc955294f8ad437bf5a9d1bb18
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 17:26:19 2020 +0200

    cache: use non-typified option of WebAPIClient methods

commit c820f9e6956f4592f1c6d2021c74272acbb2fdde
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 17:38:58 2020 +0200

    fuse: remove unnecessary inode2path

commit f7c9e71ecb724067d18a92b340009024e61b0e99
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 16:53:18 2020 +0200

    fuse: remove unnecessary fd maps (use inode instead)

commit ed199813b2ba6e50f3cd644509852138206ee636
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 12:09:27 2020 +0200

    fuse: get_blob: use bytes instead of string
    
    We do not know the encoding beforehand so manipulate every blob coming
    from the archive as bytes.

commit b2780e8e9c68e09c088385c31825338f3b9e275f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 10:42:50 2020 +0200

    fuse: add async to the codebase
    
    - Change class interface to match async interface (eg: iter -> aiter)
    - Sync code is now running with `run_in_executor`
    - Use aiosqlite as the async interface for sqlite

commit 54725ebcfa2235b0f7eadfa221fc94f8630f1424
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit 10b7f2ecc714eda995d8e2bf94c85a6af5e932ea
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit c9749f8ed499192401e3e3c973331396d1cff722
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit 44231919b7deb0dc632ba739c4a54dc795996f62
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit 6a7c3ef6a59ba8889e247ce25d6f60e95e142aa6
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit 1ff24f0f10ee04c4637c98cfb66803ac1677e280
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit 60638a74fdb58f1c76fa2bf12022294a51ab82e0
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit 837188c5dbd0e354f2addcbbb2401b9884af8b33
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit db99111bf0dec1d02241a1b53aa3176d761b4bd3
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit 8c3af24dbf0abc73cfc6321cffa37ceecf8799a8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit 79e0b54077988f7de02cdadc13fa6511eec3489a
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit a4a8e4028935b8f15637968053629f2fcfe4d655
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 273fe19265f36f7f10ca1a4319cfd8e9af23e78b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit 3d7703ee552fca525c3fc48f81cf2aaf1ff6e7f3
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit ed78230d15c7c7e568cb020a4e325b771fb19187
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

Link to build: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/34/
See console output for more information: https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/34/console

Harbormaster failed remote builds in B15964: Diff 14687!Oct 7 2020, 12:10 PM
haltode updated this revision to Diff 14691.Oct 7 2020, 12:56 PM

Update swh.model version to 0.7.0 in requirements

swh-public-ci added a comment.Oct 7 2020, 12:56 PM

Build is green

Patch application report for D4064 (id=14691)

Rebasing onto 9ced0b454f...

First, rewinding head to replay your work on top of it...
Applying: fuse: first working proof of concept
Applying: fuse: add documentation and type hints
Applying: fuse: small code clean up
Applying: cli: by default run as daemon (or use -f/--foreground)
Applying: fuse: use WebAPIClient to retrieve raw content data
Applying: fuse: lookup: now handle mountpoint, archive/ and meta/ entries
Applying: cache: separate metadata and blob caches
Applying: cli: use a config.yml file to configure cache/web-api
Applying: tests: add unit tests framework
Applying: tox.ini: pin black version to 19.10b0
Applying: fs: rework code architecture
Applying: fuse: remove internal entry2inode dictionnary
Applying: cli: use SWHIDParamType from swh.model.cli
Applying: fuse: read: use `offset` and `length` parameters
Applying: fuse: lookup: raise and log exception when name not found
Applying: fuse: add async to the codebase
Applying: fuse: get_blob: use bytes instead of string
Applying: fuse: remove unnecessary fd maps (use inode instead)
Applying: fuse: remove unnecessary inode2path
Applying: cache: use non-typified option of WebAPIClient methods
Applying: fs: move ArtifactEntry class to artifact.py
Changes applied before test
commit f514d8fc96a9f4ea101f22dc66283023a08d5bb2
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Oct 7 09:25:07 2020 +0200

    fs: move ArtifactEntry class to artifact.py

commit 78a238e614b509ac71f6b711f046ce698c0a89df
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Oct 7 12:50:27 2020 +0200

    cache: use non-typified option of WebAPIClient methods

commit 510304d03a8ebf1f0302a6fe19a45856ddc79f1d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 17:38:58 2020 +0200

    fuse: remove unnecessary inode2path

commit 55b3ff83c4f1d676c92de8a247f4ed6019be1677
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 16:53:18 2020 +0200

    fuse: remove unnecessary fd maps (use inode instead)

commit aff239422c67f14665397dbdcb8011b245e15a28
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 12:09:27 2020 +0200

    fuse: get_blob: use bytes instead of string
    
    We do not know the encoding beforehand so manipulate every blob coming
    from the archive as bytes.

commit a76eaa2f6b748ee1c7f10b7bba4e9ee44fed1592
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 10:42:50 2020 +0200

    fuse: add async to the codebase
    
    - Change class interface to match async interface (eg: iter -> aiter)
    - Sync code is now running with `run_in_executor`
    - Use aiosqlite as the async interface for sqlite

commit 72ab2aefef785418c166a08fb6d87448536d9ec5
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit fde55b83c138b082c83f57c64f01924a53d6a5df
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit 0a2944273fa8bc1d297b3df0e74aeb6debdbd502
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit 4911f440c52f92c984d3918550fa2b82c7c011e4
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit c8d1fb7328f81fde1ab256a6b04cb2bc4e14912f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit ec5302161f5f535023fbf2c37e322c6ce0dd2946
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit 7054010bbcb0f8b11aa026db45fb35e37366b3b0
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit 44e0a753a8ccd6b9c0015126992b5cb5d98522e0
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 64e319b36fecd7b4e1128465e4f66365bb56b9be
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit e7d2bb24cd2abb4917d622ca2156f767a0d3f992
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit e239b15e491aad90904c398f3ff7f245d6060938
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit 2c4604903e669b7865d2ce9bf6da8ca83e8b1f49
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 047212395c05292b997d7876fce1747e2962f27f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit a518bd3f932fc2ea1115531eefcb264ea3fce6b4
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit cb6c3003815ee3bcc2bc9da4cc8ce05dd8b7c9be
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/35/ for more details.

Harbormaster completed remote builds in B15968: Diff 14691.Oct 7 2020, 12:56 PM
seirl accepted this revision.Oct 7 2020, 3:06 PM
zack accepted this revision.Oct 7 2020, 3:09 PM
This revision is now accepted and ready to land.Oct 7 2020, 3:09 PM
haltode updated this revision to Diff 14704.Oct 7 2020, 3:10 PM

Rebasing on master.

swh-public-ci added a comment.Oct 7 2020, 3:11 PM

Build is green

Patch application report for D4064 (id=14704)

Rebasing onto 9ced0b454f...

Current branch diff-target is up to date.
Changes applied before test
commit ee058855d1a55a7bde98227c963400d137eab0e8
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Oct 7 09:25:07 2020 +0200

    fs: move ArtifactEntry class to artifact.py

commit 3c79873af9ee4f250d043b3443fc0014e240d1ff
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Oct 7 12:50:27 2020 +0200

    cache: use non-typified option of WebAPIClient methods

commit 0dd35220b0856ef81e402f26eed1ce797ec794cc
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 17:38:58 2020 +0200

    fuse: remove unnecessary inode2path

commit c1bfbcece32aada7b4992f3804a10e66b690effa
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 16:53:18 2020 +0200

    fuse: remove unnecessary fd maps (use inode instead)

commit 34c5d37c0d77da24a9dee4ad06986b058618a469
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 12:09:27 2020 +0200

    fuse: get_blob: use bytes instead of string
    
    We do not know the encoding beforehand so manipulate every blob coming
    from the archive as bytes.

commit f16985f52dab591b7eb6cd567ef772e3d77010df
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Oct 6 10:42:50 2020 +0200

    fuse: add async to the codebase
    
    - Change class interface to match async interface (eg: iter -> aiter)
    - Sync code is now running with `run_in_executor`
    - Use aiosqlite as the async interface for sqlite

commit 181b910f708868fdbfa312a05fc12d5e3d1e09b9
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 12:01:32 2020 +0200

    fuse: lookup: raise and log exception when name not found

commit 679db87aa05720da425f1f3b0df7fd731e06f513
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 11:57:07 2020 +0200

    fuse: read: use `offset` and `length` parameters
    
    This is necessary to read very long files, or use a terminal pager.

commit 2b688f8f3918ed5838151d3bd6eafd4c761ccc44
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:39:26 2020 +0200

    cli: use SWHIDParamType from swh.model.cli
    
    See D4139.

commit f7194e5a192c534c1ce8dcac1a048627ee17107a
Author: Thibault Allançon <haltode@gmail.com>
Date:   Mon Oct 5 10:05:15 2020 +0200

    fuse: remove internal entry2inode dictionnary

commit fc0dd584018c3fbc2b08405d962c7773766427d1
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 15:39:15 2020 +0200

    fs: rework code architecture
    
    Better class abstraction. Now the information is less scattered around
    the code (especially for fuse.py). Each FUSE entry virtual type is
    self-contained and defines entirely how it is used, listed, read, etc.

commit 9d60949c1295ef666f4c1e07042e16053614ec9d
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Oct 2 16:16:36 2020 +0200

    tox.ini: pin black version to 19.10b0

commit 9417ab147eab9520d40b624be7aa6990176614ed
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:46:12 2020 +0200

    tests: add unit tests framework
    
    - Create a pytest fixture to run swh-fuse in a separate process and
      setup an offline mock web API
    - Add basic cnt/dir tests

commit 8c77b5cdc2eb20b500ffa908d03385c1ff6a746b
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 15:27:48 2020 +0200

    cli: use a config.yml file to configure cache/web-api
    
    More flexible and easier to tweak settings. For example we can now have,
    the metadata cache in-memory while the blob cache is on-disk:
    
      cache:
        metadata:
          in-memory: true
        blob:
          path: /cache/dir/blob.sqlite

commit 4a67601e741eed815f868ae30d46b049dc0b292a
Author: Thibault Allançon <haltode@gmail.com>
Date:   Thu Oct 1 11:13:26 2020 +0200

    cache: separate metadata and blob caches
    
    Rework the cache architecture. The cache code is now separated into
    multiple classes (one for each cache type, and one global). Every cache
    is now used as a context manager and must implement `__getitem__` and
    `__setitem__` to access/update the underlying cache information.

commit 0c131526e2d4cb164171a771de82cdac41dbc447
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:43:15 2020 +0200

    fuse: lookup: now handle mountpoint, archive/ and meta/ entries

commit 515adac52cde80ca51c76c873a8574467487c475
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:41:08 2020 +0200

    fuse: use WebAPIClient to retrieve raw content data

commit d2f70eb1f0f40aeb5a43d1e842357dc2bf27de4f
Author: Thibault Allançon <haltode@gmail.com>
Date:   Wed Sep 30 16:37:35 2020 +0200

    cli: by default run as daemon (or use -f/--foreground)
    
    To stop the process, run: `fusermount -u MNTDIR`.

commit 1190063fe350d4316ac0e2493f394a5c2759fbf7
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:28:58 2020 +0200

    fuse: small code clean up
    
    - Variables/comments rewording
    - Use SQLite context manager instead of manual commit
    - Get a list of swhids as input in the CLI

commit 3fb6a556154b8e79573ebe6680c7dc1a3a0c38bd
Author: Thibault Allançon <haltode@gmail.com>
Date:   Tue Sep 29 10:14:51 2020 +0200

    fuse: add documentation and type hints

commit 22c4363b9734916d6acba0d127344abc62f85043
Author: Thibault Allançon <haltode@gmail.com>
Date:   Fri Sep 25 16:00:00 2020 +0200

    fuse: first working proof of concept
    
    FUSE bindings to locally mount and navigate (as a virtual file system)
    the Software Heritage archive.
    
    - Basic CLI (usage: swh fuse mount SWHID PATH)
    - Handle `cnt` and `dir` artifacts
    - User-level commands working: ls, cd, cat
    - Metadata and blob are cached using SQLite

See https://jenkins.softwareheritage.org/job/DFUSE/job/tests-on-diff/36/ for more details.

Harbormaster completed remote builds in B15981: Diff 14704.Oct 7 2020, 3:11 PM
Closed by commit rDFUSE22c4363b9734: fuse: first working proof of concept (authored by haltode). · Explain WhyOct 7 2020, 3:12 PM
This revision was automatically updated to reflect the committed changes.
haltode added a commit: rDFUSE22c4363b9734: fuse: first working proof of concept.
haltode added a commit: rDFUSE3fb6a556154b: fuse: add documentation and type hints.
haltode added a commit: rDFUSE1190063fe350: fuse: small code clean up.
haltode added a commit: rDFUSEd2f70eb1f0f4: cli: by default run as daemon (or use -f/--foreground).
haltode added a commit: rDFUSE515adac52cde: fuse: use WebAPIClient to retrieve raw content data.
haltode added a commit: rDFUSE0c131526e2d4: fuse: lookup: now handle mountpoint, archive/ and meta/ entries.
haltode added a commit: rDFUSE4a67601e741e: cache: separate metadata and blob caches.
haltode added a commit: rDFUSE8c77b5cdc2eb: cli: use a config.yml file to configure cache/web-api.
haltode added a commit: rDFUSE9d60949c1295: tox.ini: pin black version to 19.10b0.
haltode added a commit: rDFUSE9417ab147eab: tests: add unit tests framework.
haltode added a commit: rDFUSEfc0dd584018c: fs: rework code architecture.
haltode added a commit: rDFUSEf7194e5a192c: fuse: remove internal entry2inode dictionnary.
haltode added a commit: rDFUSE2b688f8f3918: cli: use SWHIDParamType from swh.model.cli.
haltode added a commit: rDFUSE679db87aa057: fuse: read: use `offset` and `length` parameters.
haltode added a commit: rDFUSE181b910f7088: fuse: lookup: raise and log exception when name not found.
haltode added a commit: rDFUSEf16985f52dab: fuse: add async to the codebase.
haltode added a commit: rDFUSE34c5d37c0d77: fuse: get_blob: use bytes instead of string.
haltode added a commit: rDFUSEc1bfbcece32a: fuse: remove unnecessary fd maps (use inode instead).
haltode added a commit: rDFUSE0dd35220b085: fuse: remove unnecessary inode2path.
haltode added a commit: rDFUSE3c79873af9ee: cache: use non-typified option of WebAPIClient methods.
haltode added a commit: rDFUSEee058855d1a5: fs: move ArtifactEntry class to artifact.py.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
13 lines--
4 lines--
1 line--
2 lines--
swh/
fuse/
138 lines95%
Loading...
70 lines86%
Loading...
fs/
93 lines92%
Loading...
48 lines88%
Loading...
77 lines77%
Loading...
216 lines93%
Loading...
tests/
445 lines100%
Loading...
66 lines91%
Loading...
51 lines100%
Loading...
20 lines100%
Loading...
16 lines100%
Loading...
14 lines100%
Loading...
2 lines--

Diff 14704

View Options

mypy.ini

Loading...
View Options

requirements-swh.txt

Loading...
View Options

requirements-test.txt

Loading...
View Options

requirements.txt

Loading...
View Options

swh/fuse/cache.py

Loading...
View Options

swh/fuse/cli.py

Loading...
View Options

swh/fuse/fs/artifact.py

Loading...
View Options

swh/fuse/fs/entry.py

Loading...
View Options

swh/fuse/fs/mountpoint.py

Loading...
View Options

swh/fuse/fuse.py

Loading...
View Options

swh/fuse/tests/api_data.py

Loading...
View Options

swh/fuse/tests/conftest.py

Loading...
View Options

swh/fuse/tests/gen-api-data.py

Loading...
View Options

swh/fuse/tests/test_cli.py

Loading...
View Options

swh/fuse/tests/test_content.py

Loading...
View Options

swh/fuse/tests/test_directory.py

Loading...
View Options

tox.ini

Loading...
About · Developer information · Legal