I experimented a bit with the streaming idea, here is what we could do:

The visibility part of this task is done, I filed a new task about supporting rm cache/{...} syntax in T2889.

My API idea was to simply have something like ENTRIES_REGEXP = r'^.*:.*$' as a class attribute of each type of directory, and a validate_entry(self, name: str) method which, by default, just checks that it matches the regexp.

good catch!, a broken symlink would be preferable over omitting the entry

In T2771#53972, @seirl wrote:

We also need to discuss what exactly we put in cache/. I thought about symlinks to archive/ and meta/, what do you think? Removing the symlinks also means removing the data from the cache.

We also need to discuss what exactly we put in cache/. I thought about symlinks to archive/ and meta/, what do you think? Removing the symlinks also means removing the data from the cache.

New proposal (lather, rinse, repeat…) based on an idea from @seirl:

Yes, this is actually documented (https://docs.softwareheritage.org/devel/swh-fuse/design.html#ori-nodes-origins) and logged (in DEBUG level) when there is a date conflict. We will probably decide on a more fine-grained layout in the future to account for this instead of simply ignoring and picking the first one.

@vlorentz: can we have the logs please?
Run mount with the "--foreground" option and/or check your user log in "journalctl --user".
TIA

See D4636.

The difficulty with this one is deciding when to re-query the backend to check if there are new visits. Doing it too often will make the cache of visit metadata useless. Doing it too seldomly will make you miss new visits. Either way, we probably need to add a timestamp somewhere in the cache to note down when the metadata have been fetched last (!= most recent visit timestamp).

nice catch, thanks @vlorentz.