Some repositories @olasd mentioned to me that qualifies as gitlab repositories (in parenthesis, their current size in term of repositories):
The general problem (see below for the deposit-specific case) is indeed complex to deal with (both conceptually in a pure Merkle setting and practically due to the existence of zip bombs). I think a workable solution might be ingest the archive as is and also ingest a separate directory corresponding to the archive content, with some metadata linking the two. That way by default we will only return what we have ingested (without recursion), but we will offer ways to dig-in recursively, e.g., in the web app. There will be plenty of devils in plenty of details for this though.
I've generalized the title of this task, will add sub-tasks for the specific features that are still missing to complete this.
I completely agree that 'filename' is not enough and adding each time a new piece of context isn't a good solution.
Both path strategies (integers vs identifiers) are interesting.
Here is a concrete proposal for the path language:
Well, there are other scenarios: like us being forced to remove content for legal reasons. But note that I'm not arguing against the path-based approach. The risk exists only for path encoded using *integers*, because they're by construction relative to the object you traverse. You can have paths that contain the full-step information (e.g., a file/directory name, or a commit identifier), and those paths would be resolvable even if you lose access to intermediate objects. The problem with those kind of paths is that they are much longer than the integer-based ones. That robustness-v-compactness trade-off is the though one I was referring to.
I see your point, but let's remember that here we want to provide a means for a user A to encode efficiently the context information necessary for another user B to be shown the same view of the archive as the one A has.
It just occurred to me that this works (in the sense that the paths will be resolvable) only if we have all the objects in the path from the snapshot down to the pointed object, which is not something we can guarantee in general — e.g., we might have archived a repository which had missing objects in the first place.
It is all contextual information which would not make it impossible to see the final object you're pointing too. But this issue calls into question the robustness of integer-based paths for our purposes here. For instance, an fpath based on actual file/directory names will always be displayable, one based on integers will not be.
Though trade-off…
Actually, we can generalize the approach even a bit more.
Thanks for starting this... it's an important discussion, and it goes quite beyond the need of a "filename" attribute in our family of context attributes :-)
Seems a nice way to go: we would also need some easy to use interface to
edit the "visibility" bit too...
The simplest approximation of this that I can see is adding a visibility column to the origin table, and tweaking that manually when we get a request.
(tagging as General, while we discuss it)
closing, now that all sub-tasks have been completed
I agree with your proposal.
So I think the best option here is to used named parameters as optional parts in the identifiers. This will give us some flexibility regarding the adding of new ones in the future. Regarding the separator, we could either used \ or | as they should not interfere with origin urls to extract.
Thanks for the clear explanation.
the problems I see with optional URL parameters instead of modifying the identifiers themselves are the following:
As I am currently implementing the task, I am wondering if adding optional parts to a swh identifier v1 is the adequate solution.
@ardumont, I think you can resolve this one ;-)
relevant highlights:
update from joeyh, there is no need for any specific hack to maintain a local mirror, it is just an undocumented feature:
swh-loader-git and swh-loader-debian have now been migrated to snapshots as well, and restarted.
Current status on the development migration towards snapshot (branch wip/snapshot(s)) as far as I know:
yeah, i was thinking about it while running earlier on today :) i'm not yet sure if i'll specify the meaning of the sha1 of each object here, or just say that the sha1 is the primary key of the object and refer to swh-model, we'll see
When writing the documentation, please be sure to be explicit whether content identifier its sha1 or its salted sha1_git, because that's not clear which it is from this discussion :)
In T335#16990, @zack wrote:identifier = "swh" ":" scheme_version ":" obj_type ":" obj_id ; scheme_version = "1" ; obj_type = "snp" # snapshot | "rel" # release | "rev" # revision | "dir" # directory | "cnt" # content ; obj_id = object sha1, hex-encoded with (lowercase) ASCII characters ;
in the future, if we switch to blake2/256 (or equivalent length checksums), the examples would become something like:
concrete, tentative proposal (EBNF):
identifier = "swh" ":" scheme_version ":" obj_type ":" obj_id ;
scheme_version = "1" ;
obj_type =
"snp" # snapshot
| "rel" # release
| "rev" # revision
| "dir" # directory
| "cnt" # content
;
obj_id = object sha1, hex-encoded with (lowercase) ASCII characters ;This is already well beyond a prototype, so I'm closing this task.
As the "prototype" part of the vault is definitely over, I'm closing this task.
We can always allow people to truncate the identifier to some arbitrary (shorter) length. The canonical URI would be the full identifier, but our URI resolver can recognize shortened identifiers and point to a disambiguation page with all the objects whose identifier starts with the given string.
I agree with all the suggestions: the full id should definitely contain all
this information.
Nevertheless, the sheer length of the result *may* turn out to be a blocker
for adoption as a reference to software in the academic publishing
framework. We can propose this, and see if we need to also provide a
shorter backup if really there is a strong negative feedback.
I'm not opposed to having explicit hash scheme names in the IDs—it is a good idea, only to be weighed against the cost in terms of length.
But we should also have schema version numbers, in case more radical changes will be needed in the future, e.g., renaming the object types in the graph.
If we retain both suggestions, that would give:
I've been thinking about this in relation to T836.
we're taking a different route for this now, based on @grouss WIP