In T421#21642, @ardumont wrote:

The pypi api provides already quite the information (P288, P289 for examples).
For now, the current implementation leverages it.

The basic loader will be the tarball loader, yes. In addition to that there are two aspects to be defined:

1. the stack of objects to be added to the DAG
2. the metadata to extract

First of all, thanks for this design document. I have read through it (although not verified the details of offsets, sizes, etc, mind you :-)) and it looks reasonable to me. A few questions/comments below:

good catch !

In T1158#21531, @ardumont wrote:

@zack or @olasd if you have some time to review P286 at one point in time, that would be awesome.

In T960#21455, @moranegg wrote:

Regarding implementation, no plans of implementing it are on the horizon, it is something to consider for the priority/yearly planning.
I can also open a review documentation subtask.

In T422#21473, @ardumont wrote:

If your consumer is actually an organization or service that will be downloading a lot of packages from PyPI, consider using your own index mirror or cache.

That's not a sustainable way. If we choose that path for all the forges we need to archive... that will be difficult in terms of infrastructure and maintenance.

better LWN link to the actual article covering this: https://lwn.net/Articles/751458/

In T420#21471, @ardumont wrote:

Looking at the faq [4], they also (now?) recommend bandersnatch. Quoting it:

I'll be AFK for a while, so I can't check the diff, but if you (@moranegg ) can point me to the current version (on docs.s.o?, if it's deployed), I'll be happy to have a look before it's implemented

In T336#21437, @ardumont wrote:

E.g., you don't "schedule" the addition of an entire forge as a single task,

Yes, there are 2 tasks for now (incremental, full) but if we also hide that detail within T1157... Then that could be a win, i think ;)

In T336#21431, @ardumont wrote:

Does adding a supported forge (e.g gitlab instance) considered a possible save now request?

Thanks for spotting. We also need a separate task to correct the
revisions that were already loaded in the archive. Can you please file
it? (tag "archive content")

Good idea!

With that, I do think it is important to have the metadata accessible and keep in mind that with the contextual URL which is used by HAL, the metadata is easily found !

• nobody (not even HAL, or any other depositor, including the ones concerned by the compliance use cases) can recompute independently from us this swh-id SR, because it depends not only on the metadata added, but also on the particular mangling of this metadata done during the ingestion, that may well change over time; providing only SR as a swh-id for such a deposit makes it impossible for somebody that may have a copy of the same code and an article mentioning the swh-id SR to check that the code is the same withouth accessing SWH: that would make us a middle man and for our long term strategy we do not want middle men, not even us

TL;DR: by ingesting a revision and not returning its ID, we will have a protocol that — at the protocol level — loses information, and that is a bad idea.

In T1152#21324, @rdicosmo wrote:

It is essential for reproducibility that the shw-id offered to researchers
to reference a deposited piece of software depend only on the software
deposited itself: if three papers use the same software tree, they must
show the same swh-id, no matter whether this software tree has been
deposited once, twice, or three times.

In the case of .zip/.tar files this is the swh-id of the root directory,
not the shw-id of the synthetic commit.

Can you (and/or @rdicosmo ) elaborate on the rationale for this?

great, thanks for working on this!

The general problem (see below for the deposit-specific case) is indeed complex to deal with (both conceptually in a pure Merkle setting and practically due to the existence of zip bombs). I think a workable solution might be ingest the archive as is and also ingest a separate directory corresponding to the archive content, with some metadata linking the two. That way by default we will only return what we have ingested (without recursion), but we will offer ways to dig-in recursively, e.g., in the web app. There will be plenty of devils in plenty of details for this though.

just as an idea for the UI for whitelist/blacklist URL patterns, here is what adblock does, which is quite nice:

I've generalized the title of this task, will add sub-tasks for the specific features that are still missing to complete this.

I agree we need a more user friendly way of resolving IDs. (And, in passing, I think we also need an API method /resolve for programmatically resolving PIDs.)
But rather than adding a separate search form, I think we should generalize the current one, to be a Google-style, catch-all search box.

[ Aside on the actual bug here. @rdicosmo can you change the edit policy of this task to "public"? It's the default and it's generally the right one as it allows to do stuff like change task tags and the like. ]

it's here https://forge.softwareheritage.org/source/swh-web/

In T1104#20616, @ardumont wrote:

I recall some remarks about the persistent identifier representation being too simple or something.

I don't know what's wrong with that simple representation as:

• everyone can manipulate dict

yes! +1 :-)