This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

swh.loader.git 2.1.0 has now been deployed on all workers.

I did some profiling early this week, and found that when incrementally loading a linux fork we already visited:

In T4219#84994, @vlorentz wrote:

This indicates we should load incrementally from the last snapshot of the origin AND the last snapshot of its parent, so we would capture these new commits without reloading half of the parent's history. As @olasd puts it, "that's a (very) lightweight way of doing global deduplication".

https://grafana.softwareheritage.org/d/FqGC4zu7z/vlorentz-loader-metrics?orgId=1&var-environment=production&var-interval=1h&var-visit_type=git&var-has_parent_origins=True shows we spend a considerable amount of time loading data from git repositories with an existing visit + a parent:

Currently can't do it on GitLab while logged out: https://gitlab.com/gitlab-org/gitlab/-/issues/361952

I'm closing this. I've submitted T4216 to track the actual packfile limit issue.

In summary, we would archive everything with priority "high" or "mid", as well as the "license" and "main language" fields, as they are all easy to fetch and store

dealt with (at least in terms of only using https for clones)

Looks like *what* we want to collect is a solved issue.

I guess this is then related to T3653 somehow

Oh, and now that we've moved workers to have a large swap space, the issue of downloading the full packfile in ram before rejecting it should be less disruptive than it's been in the past (where the whole worker would get killed because it ran out of its memory allocation).

For now, I've disabled our hardcoding of the TCP transport for GitHub origins.

The first github milestone was reached today.
All the github loading are failing since 00:00 UTC.

I updated the task with a breakdown of the cost of getting each info.

In T3544#69746, @olasd wrote:

I can see a few alternatives to using git:// over tcp:

• Give our swh bot accounts SSH keys, and use that to clone from GitHub over ssh.

The dulwich HTTP(s) support is implemented on top of urllib(3?).

no and yes, respectively

do we need the "list of forks" if we keep the "fork of what"? I mean these are the 2 ends of the fork relation, right?

"topics" (these are the "tags", right?)

Here's an opinionated and prioritized list.

In T3542#69656, @moranegg wrote:

At the moment, I think that all the properties you have selected in the task are needed.

At the moment, I think that all the properties you have selected in the task are needed.
+1 for License (it is something they show on the interface even if it is based on a heuristic).