GitLab returns very little data while logged out, so we won't be able to collect much. This seems to differ from their documentation, so I opened a ticket https://gitlab.com/gitlab-org/gitlab/-/issues/361952

replaced loader-based metadata loading (T4188 / T4186)

Currently can't do it on GitLab while logged out: https://gitlab.com/gitlab-org/gitlab/-/issues/361952

WIP

Removing this from the subtasks of T1740; given that we decided in T1739 to query the API at the same time we load origins

I started working this design. We'll see if it needs to change later

In summary, we would archive everything with priority "high" or "mid", as well as the "license" and "main language" fields, as they are all easy to fetch and store

In T1739#82939, @olasd wrote:

Yes, all these are good points. As long as forges don't provide a way of loading the metadata in bulk, it makes sense to do it at the same time as loading.

In T1739#82920, @vlorentz wrote:

The original idea for this was to have separate tasks to fetch metadata, so that loaders did not have forge-specific code to fetch metadata.

However, the idea of loading metadata from loader is more appealing the more I think about it:

1. Metadata are fetched at about the same time as we snapshot code; which would allow showing more consistent states of repositories
2. Active repositories automatically have their metadata fetched more often than inactive ones
3. We don't have one more moving part to monitor and schedule
4. This allows the Git loader to know a new repo is a "forge fork" of another one before it starts loading, so it can do an incremental load

To me the advantages are strong ,especially point 1 and 4.

The original idea for this was to have separate tasks to fetch metadata, so that loaders did not have forge-specific code to fetch metadata.

Looks like *what* we want to collect is a solved issue.

I think we (or I should say- I) missed ambiguity of the concept origin.

(this task is a dependency of T3638, because author != committer in revisions created by opam, and releases don't have a "committer" field, so switching to releases would lose this data)

I updated the task with a breakdown of the cost of getting each info.