Feed Advanced Search

Advanced Search
Use Results
Edit Query
Hide Query

	Include stories about projects I am a member of.

Oct 14 2021

olasd added a parent task for T3635: git loader: enable "partial" global deduplication of revisions via the extid mapping table: T3655: loader git: enable global deduplication of head branches before fetching them.

Oct 14 2021, 11:18 AM · Git loader

olasd added subtasks for T3655: loader git: enable global deduplication of head branches before fetching them: T3635: git loader: enable "partial" global deduplication of revisions via the extid mapping table, T3654: loader git: load revisions in topological order.

Oct 14 2021, 11:18 AM · Git loader

olasd added a parent task for T3654: loader git: load revisions in topological order: T3655: loader git: enable global deduplication of head branches before fetching them.

Oct 14 2021, 11:18 AM · Git loader

olasd triaged T3655: loader git: enable global deduplication of head branches before fetching them as Normal priority.

Oct 14 2021, 11:18 AM · Git loader

olasd renamed T3635: git loader: enable "partial" global deduplication of revisions via the extid mapping table from Reduce git loader work (use extid mapping table) to git loader: enable "partial" global deduplication of revisions via the extid mapping table.

Oct 14 2021, 11:15 AM · Git loader

olasd added a comment to T3654: loader git: load revisions in topological order.

(I've removed T3653 as parent as this is a somewhat longer term endeavour. Not the topological sorting itself, but making sure that (most) existing revisions aren't dangling, before we can use this topological guarantee)

Oct 14 2021, 11:13 AM · Git loader

olasd removed a parent task for T3654: loader git: load revisions in topological order: T3653: Stabilize loader git.

Oct 14 2021, 11:12 AM · Git loader

olasd removed a subtask for T3653: Stabilize loader git: T3654: loader git: load revisions in topological order.

Oct 14 2021, 11:12 AM · Git loader

olasd triaged T3654: loader git: load revisions in topological order as Low priority.

Oct 14 2021, 11:11 AM · Git loader

ardumont updated the task description for T3653: Stabilize loader git.

Oct 14 2021, 10:46 AM · Git loader

ardumont added a subtask for T3653: Stabilize loader git: T3652: Cannot ingest git repositories with (too) large packfiles.

Oct 14 2021, 10:44 AM · Git loader

ardumont added a parent task for T3652: Cannot ingest git repositories with (too) large packfiles: T3653: Stabilize loader git.

Oct 14 2021, 10:44 AM · Git loader

ardumont updated the task description for T3653: Stabilize loader git.

Oct 14 2021, 10:41 AM · Git loader

ardumont updated the task description for T3653: Stabilize loader git.

Oct 14 2021, 10:39 AM · Git loader

ardumont closed T3625: Reduce git loader memory footprint as Resolved.

Actually deployed and the number of oom actually decreased.

Oct 14 2021, 10:38 AM · Git loader

ardumont closed T3625: Reduce git loader memory footprint, a subtask of T3653: Stabilize loader git, as Resolved.

Oct 14 2021, 10:38 AM · Git loader

ardumont updated the task description for T3653: Stabilize loader git.

Oct 14 2021, 10:38 AM · Git loader

ardumont added a parent task for T3625: Reduce git loader memory footprint: T3653: Stabilize loader git.

Oct 14 2021, 10:37 AM · Git loader

ardumont added a parent task for T3635: git loader: enable "partial" global deduplication of revisions via the extid mapping table: T3653: Stabilize loader git.

Oct 14 2021, 10:37 AM · Git loader

ardumont added subtasks for T3653: Stabilize loader git: T3625: Reduce git loader memory footprint, T3635: git loader: enable "partial" global deduplication of revisions via the extid mapping table, T3640: Make long running task stop fast when warm shutdown is triggered.

Oct 14 2021, 10:37 AM · Git loader

ardumont triaged T3653: Stabilize loader git as Normal priority.

Oct 14 2021, 10:36 AM · Git loader

ardumont triaged T3652: Cannot ingest git repositories with (too) large packfiles as Normal priority.

Oct 14 2021, 10:29 AM · Git loader

Oct 11 2021

ardumont added a comment to T3625: Reduce git loader memory footprint.

Deploy storage v0.38 on worker (proxy buffer/filter adaptations client/loader side).
Restarted all loaders with it.

Oct 11 2021, 6:19 PM · Git loader

anlambert closed T3618: Reschedule loading of dumb git origins submitted to "Save code now" service as Resolved.

Issues related to git dumb loading have been handled and the 5 dumb origins that were failing have been resubmitted through save code now and successfully loaded, closing this.

Oct 11 2021, 1:51 PM · Save Code Now, Archive coverage, Git loader

douardda added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

In T3627#71819, @vlorentz wrote:

An alternative to annotating synthetic refs: add a "type" or "forge_type" attribute to snapshots.

Oct 11 2021, 12:33 PM · Git loader

Oct 8 2021

ardumont changed the status of T3625: Reduce git loader memory footprint from Open to Work in Progress.

btw ^

Oct 8 2021, 5:55 PM · Git loader

olasd added a comment to T3625: Reduce git loader memory footprint.

In T3625#71799, @olasd wrote:

While we're at it, we should probably be adding some thresholds in the buffer proxy for:

cumulated length of messages for revisions and releases

Oct 8 2021, 4:02 PM · Git loader

olasd added a revision to T3625: Reduce git loader memory footprint: D6445: buffer: add a threshold for the number of revision parents in one batch.

Oct 8 2021, 4:01 PM · Git loader

olasd added a revision to T3625: Reduce git loader memory footprint: D6446: buffer: add a threshold for the estimated size of revision and release batches.

Oct 8 2021, 3:58 PM · Git loader

olasd added a revision to T3625: Reduce git loader memory footprint: D6443: buffer: add a threshold for the number of directory entries in one batch.

Oct 8 2021, 3:06 PM · Git loader

ardumont added a project to T3635: git loader: enable "partial" global deduplication of revisions via the extid mapping table: Git loader.

Oct 8 2021, 2:17 PM · Git loader

ardumont added a comment to T3625: Reduce git loader memory footprint.

I concur with this analysis btw

Oct 8 2021, 2:00 PM · Git loader

Oct 7 2021

vlorentz added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

An alternative to annotating synthetic refs: add a "type" or "forge_type" attribute to snapshots.

Oct 7 2021, 2:10 PM · Git loader

olasd added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

Ah, another question I've been thinking about: should we go back to existing visits of git repositories and give them a new, pruned snapshot? Our data model now allows it: we can just append a new final OriginVisitStatus pointing at a pruned snapshot.

Oct 7 2021, 12:46 PM · Git loader

olasd added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

In T3627#71809, @zack wrote:

Thanks for your feedback @olasd. I see three main arguments raised there: (1) the raciness of archiving those data via other means (= related forks), (2) the completeness of our canvassing of synthetic refs, (3) annotating rather than not archiving "synthetic" refs.

For (1), sure, it's racy, hence we could lose stuff that gets removed from GitHub before we have the time to archive it. But this is a drop in the ocean in comparison with our lag/backlog.

Oct 7 2021, 12:15 PM · Git loader

zack added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

Thanks for your feedback @olasd. I see three main arguments raised there: (1) the raciness of archiving those data via other means (= related forks), (2) the completeness of our canvassing of synthetic refs, (3) annotating rather than not archiving "synthetic" refs.

Oct 7 2021, 10:54 AM · Git loader

olasd added a comment to T3625: Reduce git loader memory footprint.

While we're at it, we should probably be adding some thresholds in the buffer proxy for:

cumulated length of messages for revisions and releases
cumulated number of parents for revisions

Oct 7 2021, 10:11 AM · Git loader

olasd added a comment to T3625: Reduce git loader memory footprint.

(this also matches the fact that we've seen, on our main ingestion database, directory_add operations that would take multiple hours, and have knock-on effects on backups and replications because of the long-running insertion transactions)

Oct 7 2021, 10:09 AM · Git loader

olasd added a comment to T3625: Reduce git loader memory footprint.

So, after doing some more analysis of memory usage patterns on these edge case repositories, my suspicion is that the high memory usage is generally being caused by the loader processing batches of large directories, closely packed together, at the same time.

Oct 7 2021, 10:08 AM · Git loader

olasd added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

In T3627#71790, @rdicosmo wrote:

Yes, we must filter this stuff out (we discussed this issue with @zack some time ago)

Oct 7 2021, 9:53 AM · Git loader

olasd added a revision to T3625: Reduce git loader memory footprint: D6427: swh.storage filter/buffer improvements.

Oct 7 2021, 9:23 AM · Git loader

Oct 6 2021

rdicosmo added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

Yes, we must filter this stuff out (we discussed this issue with @zack some time ago, and you may see Torvalds' opinion too https://www.zdnet.com/article/linux-boosts-microsoft-ntfs-support-as-linus-torvalds-complains-about-github-merges/ )

Oct 6 2021, 10:23 PM · Git loader

douardda added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

FTR without D6401, the packfile received from GH for the CocoaPods/Specs repo contains 21162 references, 21146 of which are starting with /refs/pull/ and 7126 are ending with /merge (even if those have been explicitly not asked thanks to the filtering in RepoRepresentation.determine_wanted().
When D6401 is applied, we only get the 20-ish references that are not pull request related.

Oct 6 2021, 2:56 PM · Git loader

ardumont added a revision to T3625: Reduce git loader memory footprint: D6419: Add some tracemalloc hooks.

Oct 6 2021, 2:19 PM · Git loader

ardumont updated subscribers of T3627: Consider dropping pull request references from the git loader ingestion.

So I'm actually proposing that we filter out all branches whose name start with refs/pulls (with no other conditions attached).

Oct 6 2021, 2:18 PM · Git loader

Oct 5 2021

ardumont added a comment to T3625: Reduce git loader memory footprint.

Trigger other runs with memory-profiler instead. [1]
It's not perfect though. I cannot find the proper way to actually have the
legends as they described in their documentation. [2]

Oct 5 2021, 6:07 PM · Git loader

ardumont added a comment to T3625: Reduce git loader memory footprint.

I'm actually not sure yet.
So i gave a spin to [1] triggering a run on the most heavy and problematic origins with it (on
production nodes keybase/client and CocoaPods/Specs).
Let's see what result that gives (ongoing).

[1] https://github.com/pythonspeed/filprofiler

Oct 5 2021, 5:52 PM · Git loader

anlambert added a revision to T3618: Reschedule loading of dumb git origins submitted to "Save code now" service: D6416: dumb: Handle missing or corrupted pack file.

Oct 5 2021, 4:14 PM · Save Code Now, Archive coverage, Git loader

anlambert added a revision to T3618: Reschedule loading of dumb git origins submitted to "Save code now" service: D6415: dumb: Handle empty repository edge case.

Oct 5 2021, 3:37 PM · Save Code Now, Archive coverage, Git loader

Oct 4 2021

ardumont updated subscribers of T3625: Reduce git loader memory footprint.

One good question got raised on the mailing list thread both by @douardda and @stsp.

Oct 4 2021, 6:45 PM · Git loader

zack added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

In T3627#71640, @ardumont wrote:

No, the snippet mentioned filters out refs whose name starts with refs/pulls and finishes with /merge
(i realize i made a typo in the main description..., it's now fixed)

Oct 4 2021, 4:37 PM · Git loader

ardumont added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

@zack Maybe the diff ^ will clarify a bit ;)

Oct 4 2021, 4:20 PM · Git loader

ardumont added a revision to T3627: Consider dropping pull request references from the git loader ingestion: D6401: Filter out pull request related branches.

Oct 4 2021, 4:18 PM · Git loader

ardumont added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

According to the snippet referenced by @ardumont, all branch names starting with refs/pull/ should be filtered out.

Oct 4 2021, 4:11 PM · Git loader

ardumont updated the task description for T3627: Consider dropping pull request references from the git loader ingestion.

Oct 4 2021, 4:11 PM · Git loader

ardumont updated the task description for T3627: Consider dropping pull request references from the git loader ingestion.

Oct 4 2021, 4:09 PM · Git loader

zack added a comment to T3627: Consider dropping pull request references from the git loader ingestion.

According to the snippet referenced by @ardumont, all branch names starting with refs/pull/ should be filtered out.
But in the recent snapshot of torvalds/linux there are a lot of branch names like that.
How come?

Oct 4 2021, 2:01 PM · Git loader

ardumont updated the task description for T3625: Reduce git loader memory footprint.

Oct 4 2021, 1:19 PM · Git loader

ardumont added a revision to T3025: git loaders are getting oom-killed repeatedly in prod: D5657: Spool large packfiles to disk instead of consuming tons of memory.

Oct 4 2021, 1:15 PM · Git loader, System administration

ardumont updated the task description for T3457: Some git repositories are failing to be ingested because of MemoryError.

Oct 4 2021, 1:15 PM · Git loader

ardumont added a project to T3627: Consider dropping pull request references from the git loader ingestion: Git loader.

Oct 4 2021, 1:10 PM · Git loader

Oct 3 2021

ardumont added a comment to T3625: Reduce git loader memory footprint.

All runs done from medium to large repositories.
No diverging hash and consistently the loader-git ran with the patched version uses less memory.

Oct 3 2021, 5:44 PM · Git loader

Oct 2 2021

ardumont added a comment to T3625: Reduce git loader memory footprint.

A draft note to send to the #swh-devel ml is been drafted [1]
Open as draft for review first.

Oct 2 2021, 11:05 AM · Git loader

Oct 1 2021

ardumont added a comment to T3625: Reduce git loader memory footprint.

[3] Another idea that was only discussed would be to make certain we first start by
ingesting in order tag references (under the assumption that we will then ingest mostly
in natural order the repository). Then focus on the remaining references (because mostly
there is a high probability that if we start with HEAD and/or master at firstz, we will
end up with the overall repository).

Oct 1 2021, 6:56 PM · Git loader

ardumont added a revision to T3625: Reduce git loader memory footprint: D6392: git: Ingest ordered tags then ordered branches references.

Oct 1 2021, 6:55 PM · Git loader

ardumont updated subscribers of T3625: Reduce git loader memory footprint.

D6377 actually increased the memory footprint to the point of getting ingestion killed
fast. So closed!

Oct 1 2021, 6:24 PM · Git loader

ardumont added a revision to T3625: Reduce git loader memory footprint: D6377: wip: git: Group objects per type early to drop the packfile reference asap.

Oct 1 2021, 6:07 PM · Git loader

ardumont renamed T3625: Reduce git loader memory footprint from Improve git loader memory footprint to Reduce git loader memory footprint.

Oct 1 2021, 6:06 PM · Git loader

ardumont updated the task description for T3625: Reduce git loader memory footprint.

Oct 1 2021, 6:06 PM · Git loader

ardumont added a revision to T3625: Reduce git loader memory footprint: D6386: git: Load git repository through multiple packfiles fetch operations.

Oct 1 2021, 2:09 PM · Git loader

Sep 30 2021

ardumont added a revision to T3625: Reduce git loader memory footprint: D6380: Allow partial snapshot creation during ingestion.

Sep 30 2021, 6:01 PM · Git loader

ardumont triaged T3625: Reduce git loader memory footprint as High priority.

Sep 30 2021, 6:01 PM · Git loader

Sep 29 2021

anlambert added a comment to T3618: Reschedule loading of dumb git origins submitted to "Save code now" service.

All save requests have been processed, 127 succeeded and 5 failed due to the following issues:

https://git.qsomula.top/SUGAR__STRAWBERRIES/SUGAR__AND__STRAWBERRIES.git is an empty repo, that edge case is not properly handled
4 git repositories whose URLs are prefixed by https://git.zap.org.au/ reference pack files that are no longer hosted on the server, skipping those missing pack files makes the loadings succeeded

Sep 29 2021, 2:53 PM · Save Code Now, Archive coverage, Git loader

anlambert added a comment to T3618: Reschedule loading of dumb git origins submitted to "Save code now" service.

This is the execution result of the script above:

anlambert@carnavalet:~/tmp$ python3 reload_dumb_git_origins.py
{'loading_task_id': 400298414,
 'origin_url': 'https://git.pengutronix.de/cgit/tools/libsocketcan',
 'save_request_date': '2021-09-29T10:07:54.080280+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298415,
 'origin_url': 'https://git.stoutner.com/UltraPrivacy.git',
 'save_request_date': '2021-09-29T10:07:54.507288+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298416,
 'origin_url': 'https://git.stoutner.com/UltraList.git',
 'save_request_date': '2021-09-29T10:07:54.795405+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298417,
 'origin_url': 'https://git.stoutner.com/PrivacyCell.git',
 'save_request_date': '2021-09-29T10:07:54.979121+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298418,
 'origin_url': 'https://git.stoutner.com/PrivacyBrowserROMIntegration.git',
 'save_request_date': '2021-09-29T10:07:55.205037+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298419,
 'origin_url': 'https://git.stoutner.com/PrivacyBrowser.git',
 'save_request_date': '2021-09-29T10:07:55.404696+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298420,
 'origin_url': 'https://git.raptorengineering.com/git/hqemu',
 'save_request_date': '2021-09-29T10:07:55.627926+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298421,
 'origin_url': 'https://forge.touhey.org/casio/packaging.git',
 'save_request_date': '2021-09-29T10:07:55.919487+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298422,
 'origin_url': 'https://forge.touhey.org/casio/libcasio.git',
 'save_request_date': '2021-09-29T10:07:56.202117+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298423,
 'origin_url': 'https://git.tuxfamily.org/harmonist/harmonist.git',
 'save_request_date': '2021-09-29T10:07:56.482737+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298424,
 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tlibs.git',
 'save_request_date': '2021-09-29T10:07:57.516829+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298425,
 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git',
 'save_request_date': '2021-09-29T10:07:57.811821+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298424,
 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tlibs.git',
 'save_request_date': '2021-09-29T10:07:57.516829+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298426,
 'origin_url': 'https://depp.brause.cc/nov.el.git',
 'save_request_date': '2021-09-29T10:07:58.360459+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298427,
 'origin_url': 'https://www.6809.org.uk/git/asm6809.git',
 'save_request_date': '2021-09-29T10:07:58.925928+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298428,
 'origin_url': 'http://git.ghostscript.com/mupdf.git',
 'save_request_date': '2021-09-29T10:07:59.179198+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298429,
 'origin_url': 'https://git.joeyh.name/git/oldusenet.git',
 'save_request_date': '2021-09-29T10:07:59.423211+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298430,
 'origin_url': 'https://git.drobilla.net/ganv.git',
 'save_request_date': '2021-09-29T10:07:59.660229+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298431,
 'origin_url': 'https://genome-source.gi.ucsc.edu/samtabix.git',
 'save_request_date': '2021-09-29T10:07:59.901717+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298432,
 'origin_url': 'http://software.legiasoft.com/git/quickcheck.git',
 'save_request_date': '2021-09-29T10:08:00.111266+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298433,
 'origin_url': 'https://git.progress-linux.org/users/daniel.baumann/debian/packages/lziprecover',
 'save_request_date': '2021-09-29T10:08:00.336308+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298434,
 'origin_url': 'https://git.progress-linux.org/users/daniel.baumann/debian/packages/plzip',
 'save_request_date': '2021-09-29T10:08:00.538700+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298435,
 'origin_url': 'https://git.progress-linux.org/users/daniel.baumann/debian/packages/tarlz',
 'save_request_date': '2021-09-29T10:08:00.758586+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298436,
 'origin_url': 'https://git.progress-linux.org/users/daniel.baumann/debian/packages/lzip',
 'save_request_date': '2021-09-29T10:08:00.976973+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298437,
 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/takin-data.git',
 'save_request_date': '2021-09-29T10:08:01.213923+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298439,
 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tastools.git',
 'save_request_date': '2021-09-29T10:08:01.458344+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298440,
 'origin_url': 'https://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils',
 'save_request_date': '2021-09-29T10:08:01.671555+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298441,
 'origin_url': 'https://c9x.me/git/qbe.git',
 'save_request_date': '2021-09-29T10:08:01.891677+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298442,
 'origin_url': 'https://git.tuxfamily.org/ostorybook/code.git',
 'save_request_date': '2021-09-29T10:08:02.096468+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298444,
 'origin_url': 'https://thingshare.ion.nu/thingshare.git',
 'save_request_date': '2021-09-29T10:08:02.333379+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298445,
 'origin_url': 'https://inqlab.net/git/guile-sodium.git',
 'save_request_date': '2021-09-29T10:08:02.603023+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298446,
 'origin_url': 'https://git.systemreboot.net/guile-xapian',
 'save_request_date': '2021-09-29T10:08:02.980312+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298447,
 'origin_url': 'https://git.netfilter.org/iptables',
 'save_request_date': '2021-09-29T10:08:03.692976+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298448,
 'origin_url': 'https://inqlab.net/git/guile-cbor.git',
 'save_request_date': '2021-09-29T10:08:03.966797+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298449,
 'origin_url': 'https://git.openembedded.org/openembedded-core',
 'save_request_date': '2021-09-29T10:08:04.228102+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298451,
 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/mcstas.git',
 'save_request_date': '2021-09-29T10:08:04.444651+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298437,
 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/takin-data.git',
 'save_request_date': '2021-09-29T10:08:01.213923+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298439,
 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tastools.git',
 'save_request_date': '2021-09-29T10:08:01.458344+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298425,
 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git',
 'save_request_date': '2021-09-29T10:07:57.811821+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298452,
 'origin_url': 'https://git.beyermatthi.as/prometheus-mpd-exporter',
 'save_request_date': '2021-09-29T10:08:05.289687+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298453,
 'origin_url': 'https://git.joeyh.name/git/arduino-copilot.git',
 'save_request_date': '2021-09-29T10:08:05.547289+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298454,
 'origin_url': 'https://git.netfilter.org/libmnl',
 'save_request_date': '2021-09-29T10:08:05.849481+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298455,
 'origin_url': 'https://jff.email/cgit/libunistring.git',
 'save_request_date': '2021-09-29T10:08:06.257710+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298456,
 'origin_url': 'https://inqlab.net/git/eris.git',
 'save_request_date': '2021-09-29T10:08:06.494460+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298457,
 'origin_url': 'https://jff.email/cgit/dmidecode.git',
 'save_request_date': '2021-09-29T10:08:06.776931+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298697,
 'origin_url': 'https://git.tukaani.org/xz.git',
 'save_request_date': '2021-09-29T10:08:06.996597+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298458,
 'origin_url': 'https://git.dthompson.us/guile-parser-combinators.git',
 'save_request_date': '2021-09-29T10:08:07.200536+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298459,
 'origin_url': 'http://git.foldling.org/vim-scheme.git',
 'save_request_date': '2021-09-29T10:08:07.417546+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298460,
 'origin_url': 'https://git.opensvc.com/multipath-tools/.git',
 'save_request_date': '2021-09-29T10:08:07.656692+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298461,
 'origin_url': 'https://git.osmocom.org/rtl-sdr',
 'save_request_date': '2021-09-29T10:08:08.073457+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298463,
 'origin_url': 'http://git.kpe.io/uffi.git',
 'save_request_date': '2021-09-29T10:08:08.438223+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298465,
 'origin_url': 'http://git.kpe.io/clsql.git',
 'save_request_date': '2021-09-29T10:08:09.662233+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298466,
 'origin_url': 'http://git.kpe.io/ptester.git',
 'save_request_date': '2021-09-29T10:08:11.056976+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298467,
 'origin_url': 'https://git.osmocom.org/gr-osmosdr',
 'save_request_date': '2021-09-29T10:08:11.892136+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298468,
 'origin_url': 'https://git.osmocom.org/libosmo-dsp',
 'save_request_date': '2021-09-29T10:08:12.314649+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298469,
 'origin_url': 'http://git.kpe.io/xlunit.git',
 'save_request_date': '2021-09-29T10:08:12.602773+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298470,
 'origin_url': 'http://git.kpe.io/kmrcl.git',
 'save_request_date': '2021-09-29T10:08:13.082206+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298471,
 'origin_url': 'https://git.open-music-kontrollers.ch/lv2/lv2lint',
 'save_request_date': '2021-09-29T10:08:13.630219+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298472,
 'origin_url': 'https://git.systemreboot.net/exiftool.el',
 'save_request_date': '2021-09-29T10:08:14.011802+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298473,
 'origin_url': 'http://git.kpe.io/cl-base64.git',
 'save_request_date': '2021-09-29T10:08:14.521164+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298475,
 'origin_url': 'http://git.savannah.gnu.org/r/gtick.git',
 'save_request_date': '2021-09-29T10:08:15.004437+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298476,
 'origin_url': 'http://git.sv.gnu.org/r/gtick.git',
 'save_request_date': '2021-09-29T10:08:15.279068+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298477,
 'origin_url': 'https://git.osmocom.org/gr-iqbal',
 'save_request_date': '2021-09-29T10:08:15.475285+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298479,
 'origin_url': 'https://git.savannah.gnu.org/r/guix.git',
 'save_request_date': '2021-09-29T10:08:15.755672+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298480,
 'origin_url': 'https://www.cl.cam.ac.uk/~mgk25/git/tripos',
 'save_request_date': '2021-09-29T10:08:16.222379+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298481,
 'origin_url': 'https://www.cl.cam.ac.uk/~mgk25/git/jbigkit',
 'save_request_date': '2021-09-29T10:08:16.425781+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298482,
 'origin_url': 'https://git.rockbox.org/cgit/sandbox.git',
 'save_request_date': '2021-09-29T10:08:16.631493+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298483,
 'origin_url': 'https://git.rockbox.org/cgit/themesite.git',
 'save_request_date': '2021-09-29T10:08:16.856420+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298484,
 'origin_url': 'https://git.rockbox.org/cgit/translate.git',
 'save_request_date': '2021-09-29T10:08:17.059833+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298485,
 'origin_url': 'https://git.rockbox.org/cgit/www.git',
 'save_request_date': '2021-09-29T10:08:17.277452+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298486,
 'origin_url': 'https://git.rockbox.org/cgit/rockbox.git',
 'save_request_date': '2021-09-29T10:08:17.453703+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298487,
 'origin_url': 'https://git.qsomula.top/SUGAR__STRAWBERRIES/SUGAR__AND__STRAWBERRIES.git',
 'save_request_date': '2021-09-29T10:08:17.696238+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298488,
 'origin_url': 'https://source.netsurf-browser.org/alphagen.git',
 'save_request_date': '2021-09-29T10:08:17.905282+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298489,
 'origin_url': 'https://source.netsurf-browser.org/ttf2f.git',
 'save_request_date': '2021-09-29T10:08:18.124307+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298490,
 'origin_url': 'https://source.netsurf-browser.org/toolchains.git',
 'save_request_date': '2021-09-29T10:08:18.332149+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298491,
 'origin_url': 'https://source.netsurf-browser.org/squeeze.git',
 'save_request_date': '2021-09-29T10:08:18.541807+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298492,
 'origin_url': 'https://source.netsurf-browser.org/packaging/fedora.git',
 'save_request_date': '2021-09-29T10:08:18.775088+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298493,
 'origin_url': 'https://source.netsurf-browser.org/packaging/debian.git',
 'save_request_date': '2021-09-29T10:08:18.958225+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298494,
 'origin_url': 'https://source.netsurf-browser.org/nstheme.git',
 'save_request_date': '2021-09-29T10:08:19.163989+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298495,
 'origin_url': 'https://source.netsurf-browser.org/nsgenbind.git',
 'save_request_date': '2021-09-29T10:08:19.383856+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298496,
 'origin_url': 'https://source.netsurf-browser.org/netsurf.git',
 'save_request_date': '2021-09-29T10:08:19.593296+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298497,
 'origin_url': 'https://source.netsurf-browser.org/netsurf-wiki.git',
 'save_request_date': '2021-09-29T10:08:19.800945+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298498,
 'origin_url': 'https://source.netsurf-browser.org/netsurf-website.git',
 'save_request_date': '2021-09-29T10:08:19.997824+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298500,
 'origin_url': 'https://source.netsurf-browser.org/netsurf-test.git',
 'save_request_date': '2021-09-29T10:08:20.747537+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298501,
 'origin_url': 'https://source.netsurf-browser.org/netsurf-all.git',
 'save_request_date': '2021-09-29T10:08:20.948499+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298502,
 'origin_url': 'https://source.netsurf-browser.org/makerun.git',
 'save_request_date': '2021-09-29T10:08:21.168633+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298503,
 'origin_url': 'https://source.netsurf-browser.org/libwapcaplet.git',
 'save_request_date': '2021-09-29T10:08:21.391437+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298504,
 'origin_url': 'https://source.netsurf-browser.org/libutf8proc.git',
 'save_request_date': '2021-09-29T10:08:21.575900+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298506,
 'origin_url': 'https://source.netsurf-browser.org/libsvgtiny.git',
 'save_request_date': '2021-09-29T10:08:21.766147+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298507,
 'origin_url': 'https://source.netsurf-browser.org/librufl.git',
 'save_request_date': '2021-09-29T10:08:21.971342+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298508,
 'origin_url': 'https://source.netsurf-browser.org/librosprite.git',
 'save_request_date': '2021-09-29T10:08:22.163567+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298509,
 'origin_url': 'https://source.netsurf-browser.org/libpencil.git',
 'save_request_date': '2021-09-29T10:08:22.373564+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298510,
 'origin_url': 'https://source.netsurf-browser.org/libparserutils.git',
 'save_request_date': '2021-09-29T10:08:22.544880+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298511,
 'origin_url': 'https://source.netsurf-browser.org/libnsutils.git',
 'save_request_date': '2021-09-29T10:08:22.742637+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298512,
 'origin_url': 'https://source.netsurf-browser.org/libnspsl.git',
 'save_request_date': '2021-09-29T10:08:22.962451+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298513,
 'origin_url': 'https://source.netsurf-browser.org/libnspdf.git',
 'save_request_date': '2021-09-29T10:08:23.146196+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298514,
 'origin_url': 'https://source.netsurf-browser.org/libnslog.git',
 'save_request_date': '2021-09-29T10:08:23.341888+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298515,
 'origin_url': 'https://source.netsurf-browser.org/libnslayout.git',
 'save_request_date': '2021-09-29T10:08:23.541333+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298516,
 'origin_url': 'https://source.netsurf-browser.org/libnsgif.git',
 'save_request_date': '2021-09-29T10:08:23.747382+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298517,
 'origin_url': 'https://source.netsurf-browser.org/libnsfb.git',
 'save_request_date': '2021-09-29T10:08:23.950486+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298518,
 'origin_url': 'https://source.netsurf-browser.org/libnsbmp.git',
 'save_request_date': '2021-09-29T10:08:24.128810+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298519,
 'origin_url': 'https://source.netsurf-browser.org/libhubbub.git',
 'save_request_date': '2021-09-29T10:08:24.335594+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298520,
 'origin_url': 'https://source.netsurf-browser.org/libdom.git',
 'save_request_date': '2021-09-29T10:08:24.557136+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298521,
 'origin_url': 'https://source.netsurf-browser.org/libcss.git',
 'save_request_date': '2021-09-29T10:08:24.760323+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298522,
 'origin_url': 'https://source.netsurf-browser.org/json-c.git',
 'save_request_date': '2021-09-29T10:08:24.999232+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298523,
 'origin_url': 'https://source.netsurf-browser.org/iconv.git',
 'save_request_date': '2021-09-29T10:08:25.178058+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298524,
 'origin_url': 'https://source.netsurf-browser.org/ci-tools.git',
 'save_request_date': '2021-09-29T10:08:25.398196+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298525,
 'origin_url': 'https://source.netsurf-browser.org/buildsystem.git',
 'save_request_date': '2021-09-29T10:08:25.568154+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298526,
 'origin_url': 'https://source.netsurf-browser.org/art.git',
 'save_request_date': '2021-09-29T10:08:25.759202+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298527,
 'origin_url': 'http://galexander.org/git/simplesshd.git',
 'save_request_date': '2021-09-29T10:08:26.002317+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298528,
 'origin_url': 'https://tinc-vpn.org/git/tinc',
 'save_request_date': '2021-09-29T10:08:26.228797+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298529,
 'origin_url': 'https://git.joeyh.name/git/concurrent-output.git',
 'save_request_date': '2021-09-29T10:08:26.428783+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298530,
 'origin_url': 'https://git.joeyh.name/git/brainfuck-monad.git',
 'save_request_date': '2021-09-29T10:08:26.652730+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298532,
 'origin_url': 'https://git.neodarz.net/pro/umosapicpp.git',
 'save_request_date': '2021-09-29T10:08:26.890221+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298533,
 'origin_url': 'https://git.neodarz.net/pro/umosapi.git',
 'save_request_date': '2021-09-29T10:08:27.129709+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298534,
 'origin_url': 'https://git.neodarz.net/neodarz/waveshare-7inch-touchscreen-driver.git',
 'save_request_date': '2021-09-29T10:08:27.295848+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298535,
 'origin_url': 'https://git.neodarz.net/neodarz/scripts/feed_finder.git',
 'save_request_date': '2021-09-29T10:08:27.489133+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298536,
 'origin_url': 'https://git.neodarz.net/neodarz/scripts/convert.py.git',
 'save_request_date': '2021-09-29T10:08:27.669389+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298537,
 'origin_url': 'https://git.neodarz.net/neodarz/scripts/bandcamp-dl_script.git',
 'save_request_date': '2021-09-29T10:08:27.843411+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298538,
 'origin_url': 'https://git.neodarz.net/neodarz/scripts/arte.tv-dl.git',
 'save_request_date': '2021-09-29T10:08:28.021717+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298539,
 'origin_url': 'https://git.neodarz.net/neodarz/scripts/archlinux_iso_checker.git',
 'save_request_date': '2021-09-29T10:08:28.184597+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298540,
 'origin_url': 'https://git.neodarz.net/neodarz/scripts/SimplyQrGen.git',
 'save_request_date': '2021-09-29T10:08:28.359197+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298541,
 'origin_url': 'https://git.neodarz.net/neodarz/scripts/EphermalVPN.git',
 'save_request_date': '2021-09-29T10:08:28.549753+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298542,
 'origin_url': 'https://git.neodarz.net/neodarz/school/the_transcriber_backend.git',
 'save_request_date': '2021-09-29T10:08:28.711372+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298543,
 'origin_url': 'https://git.neodarz.net/neodarz/resumejson_converter.git',
 'save_request_date': '2021-09-29T10:08:28.875127+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298544,
 'origin_url': 'https://git.neodarz.net/code_archive/freetorrent.git',
 'save_request_date': '2021-09-29T10:08:29.058812+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298545,
 'origin_url': 'https://git.e2factory.org/git/e2factory.git',
 'save_request_date': '2021-09-29T10:08:29.235115+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298546,
 'origin_url': 'https://git.zap.org.au/git/console-fonts-utils.git',
 'save_request_date': '2021-09-29T10:08:29.485093+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298547,
 'origin_url': 'https://git.zap.org.au/git/console-fonts-lucida.git',
 'save_request_date': '2021-09-29T10:08:29.705671+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298548,
 'origin_url': 'https://git.zap.org.au/git/console-fonts-zap.git',
 'save_request_date': '2021-09-29T10:08:29.896133+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298549,
 'origin_url': 'https://git.zap.org.au/git/trader.git',
 'save_request_date': '2021-09-29T10:08:30.119825+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}
{'loading_task_id': 400298550,
 'origin_url': 'http://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git',
 'save_request_date': '2021-09-29T10:08:30.279433+00:00',
 'save_request_status': 'accepted',
 'save_task_status': 'not yet scheduled',
 'visit_date': None,
 'visit_status': None,
 'visit_type': 'git'}

Sep 29 2021, 12:20 PM · Save Code Now, Archive coverage, Git loader

anlambert added a comment to T3618: Reschedule loading of dumb git origins submitted to "Save code now" service.

We will use the following script to create new save requests for those origins:

reload_dumb_git_origins.py

import os
from pprint import pprint

Sep 29 2021, 12:17 PM · Save Code Now, Archive coverage, Git loader

anlambert added a comment to T3618: Reschedule loading of dumb git origins submitted to "Save code now" service.

This is the result of the above script execution:

dumb_git_origins

anlambert@carnavalet:~/tmp$ python3 get_save_code_now_dumb_git_repos.py 2>/dev/null
https://git.pengutronix.de/cgit/tools/libsocketcan
https://git.stoutner.com/UltraPrivacy.git
https://git.stoutner.com/UltraList.git
https://git.stoutner.com/PrivacyCell.git
https://git.stoutner.com/PrivacyBrowserROMIntegration.git
https://git.stoutner.com/PrivacyBrowser.git
https://git.raptorengineering.com/git/hqemu/
https://forge.touhey.org/casio/packaging.git
https://forge.touhey.org/casio/libcasio.git
https://git.tuxfamily.org/harmonist/harmonist.git
https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tlibs.git
https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git
https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tlibs.git/
https://depp.brause.cc/nov.el.git
https://www.6809.org.uk/git/asm6809.git
http://git.ghostscript.com/mupdf.git
https://git.joeyh.name/git/oldusenet.git
https://git.drobilla.net/ganv.git
https://genome-source.gi.ucsc.edu/samtabix.git
http://software.legiasoft.com/git/quickcheck.git
https://git.progress-linux.org/users/daniel.baumann/debian/packages/lziprecover
https://git.progress-linux.org/users/daniel.baumann/debian/packages/plzip
https://git.progress-linux.org/users/daniel.baumann/debian/packages/tarlz
https://git.progress-linux.org/users/daniel.baumann/debian/packages/lzip
https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/takin-data.git
https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tastools.git
https://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils
https://c9x.me/git/qbe.git
https://git.tuxfamily.org/ostorybook/code.git
https://thingshare.ion.nu/thingshare.git/
https://inqlab.net/git/guile-sodium.git
https://git.systemreboot.net/guile-xapian
https://git.netfilter.org/iptables
https://inqlab.net/git/guile-cbor.git
https://git.openembedded.org/openembedded-core
https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/mcstas.git/
https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/takin-data.git/
https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tastools.git/
https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git/
https://git.beyermatthi.as/prometheus-mpd-exporter
https://git.joeyh.name/git/arduino-copilot.git
https://git.netfilter.org/libmnl/
https://jff.email/cgit/libunistring.git
https://inqlab.net/git/eris.git
https://jff.email/cgit/dmidecode.git
https://git.tukaani.org/xz.git
https://git.dthompson.us/guile-parser-combinators.git
http://git.foldling.org/vim-scheme.git
https://git.opensvc.com/multipath-tools/.git
https://git.osmocom.org/rtl-sdr
http://git.kpe.io/uffi.git
http://git.kpe.io/clsql.git
http://git.kpe.io/ptester.git
https://git.osmocom.org/gr-osmosdr
https://git.osmocom.org/libosmo-dsp
http://git.kpe.io/xlunit.git
http://git.kpe.io/kmrcl.git
https://git.open-music-kontrollers.ch/lv2/lv2lint
https://git.systemreboot.net/exiftool.el
http://git.kpe.io/cl-base64.git
http://git.savannah.gnu.org/r/gtick.git
http://git.sv.gnu.org/r/gtick.git
https://git.osmocom.org/gr-iqbal
https://git.savannah.gnu.org/r/guix.git
https://www.cl.cam.ac.uk/~mgk25/git/tripos/
https://www.cl.cam.ac.uk/~mgk25/git/jbigkit
https://git.rockbox.org/cgit/sandbox.git/
https://git.rockbox.org/cgit/themesite.git/
https://git.rockbox.org/cgit/translate.git/
https://git.rockbox.org/cgit/www.git/
https://git.rockbox.org/cgit/rockbox.git/
https://git.qsomula.top/SUGAR__STRAWBERRIES/SUGAR__AND__STRAWBERRIES.git
https://source.netsurf-browser.org/alphagen.git/
https://source.netsurf-browser.org/ttf2f.git/
https://source.netsurf-browser.org/toolchains.git/
https://source.netsurf-browser.org/squeeze.git/
https://source.netsurf-browser.org/packaging/fedora.git/
https://source.netsurf-browser.org/packaging/debian.git/
https://source.netsurf-browser.org/nstheme.git/
https://source.netsurf-browser.org/nsgenbind.git/
https://source.netsurf-browser.org/netsurf.git/
https://source.netsurf-browser.org/netsurf-wiki.git/
https://source.netsurf-browser.org/netsurf-website.git/
https://source.netsurf-browser.org/netsurf-test.git/
https://source.netsurf-browser.org/netsurf-all.git/
https://source.netsurf-browser.org/makerun.git/
https://source.netsurf-browser.org/libwapcaplet.git/
https://source.netsurf-browser.org/libutf8proc.git/
https://source.netsurf-browser.org/libsvgtiny.git/
https://source.netsurf-browser.org/librufl.git/
https://source.netsurf-browser.org/librosprite.git/
https://source.netsurf-browser.org/libpencil.git/
https://source.netsurf-browser.org/libparserutils.git/
https://source.netsurf-browser.org/libnsutils.git/
https://source.netsurf-browser.org/libnspsl.git/
https://source.netsurf-browser.org/libnspdf.git/
https://source.netsurf-browser.org/libnslog.git/
https://source.netsurf-browser.org/libnslayout.git/
https://source.netsurf-browser.org/libnsgif.git/
https://source.netsurf-browser.org/libnsfb.git/
https://source.netsurf-browser.org/libnsbmp.git/
https://source.netsurf-browser.org/libhubbub.git/
https://source.netsurf-browser.org/libdom.git/
https://source.netsurf-browser.org/libcss.git/
https://source.netsurf-browser.org/json-c.git/
https://source.netsurf-browser.org/iconv.git/
https://source.netsurf-browser.org/ci-tools.git/
https://source.netsurf-browser.org/buildsystem.git/
https://source.netsurf-browser.org/art.git/
http://galexander.org/git/simplesshd.git
https://tinc-vpn.org/git/tinc
https://git.joeyh.name/git/concurrent-output.git
https://git.joeyh.name/git/brainfuck-monad.git
https://git.neodarz.net/pro/umosapicpp.git
https://git.neodarz.net/pro/umosapi.git
https://git.neodarz.net/neodarz/waveshare-7inch-touchscreen-driver.git
https://git.neodarz.net/neodarz/scripts/feed_finder.git
https://git.neodarz.net/neodarz/scripts/convert.py.git
https://git.neodarz.net/neodarz/scripts/bandcamp-dl_script.git
https://git.neodarz.net/neodarz/scripts/arte.tv-dl.git
https://git.neodarz.net/neodarz/scripts/archlinux_iso_checker.git
https://git.neodarz.net/neodarz/scripts/SimplyQrGen.git
https://git.neodarz.net/neodarz/scripts/EphermalVPN.git
https://git.neodarz.net/neodarz/school/the_transcriber_backend.git
https://git.neodarz.net/neodarz/resumejson_converter.git
https://git.neodarz.net/code_archive/freetorrent.git
https://git.e2factory.org/git/e2factory.git
https://git.zap.org.au/git/console-fonts-utils.git
https://git.zap.org.au/git/console-fonts-lucida.git
https://git.zap.org.au/git/console-fonts-zap.git
https://git.zap.org.au/git/trader.git
http://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git/

Sep 29 2021, 11:57 AM · Save Code Now, Archive coverage, Git loader

anlambert added a comment to T3618: Reschedule loading of dumb git origins submitted to "Save code now" service.

Afterwards, we use the following script to identify the dumb git origins that can be reloaded:

get_save_code_now_dumb_git_repos.py

import io
import os

Sep 29 2021, 11:54 AM · Save Code Now, Archive coverage, Git loader

anlambert added a comment to T3618: Reschedule loading of dumb git origins submitted to "Save code now" service.

Then we copy that dump locally and load it into our development database:

anlambert@carnavalet:~/tmp$ scp anlambert@moma:swh-web-db.json .
swh-web-db.json                                                                                                                                                                                                                                        100%   33MB   5.6MB/s   00:05    
anlambert@carnavalet:~/tmp$ django-admin loaddata --settings swh.web.settings.development swh-web-db.json
Installed 93893 object(s) from 1 fixture(s)
anlambert@carnavalet:~/tmp$

Sep 29 2021, 11:51 AM · Save Code Now, Archive coverage, Git loader

anlambert added a comment to T3618: Reschedule loading of dumb git origins submitted to "Save code now" service.

First we need a JSON dump of production database:

anlambert@carnavalet:~/tmp$ ssh anlambert@moma
Linux moma 5.10.0-0.bpo.5-amd64 #1 SMP Debian 5.10.24-1~bpo10+1 (2021-03-29) x86_64

Sep 29 2021, 11:49 AM · Save Code Now, Archive coverage, Git loader

anlambert triaged T3618: Reschedule loading of dumb git origins submitted to "Save code now" service as Normal priority.

Sep 29 2021, 11:21 AM · Save Code Now, Archive coverage, Git loader

anlambert closed T2489: Git origin without smart transfer protocol support cannot be loaded as Resolved.

Dumb protocol support has been implemented in the git loader and deployed to production (see that origin example and its list of all visits), closing this.

Sep 29 2021, 11:09 AM · Git loader

Sep 28 2021

ardumont closed T3614: Deploy swh.loader.git v1.1.0 and swh.model v3.0 as Resolved.

Sep 28 2021, 7:23 PM · System administration, Git loader

ardumont moved T3614: Deploy swh.loader.git v1.1.0 and swh.model v3.0 from code-review/await-feedback/pause to deployed/landed/monitoring on the System administration board.

Sep 28 2021, 6:50 PM · System administration, Git loader

ardumont moved T3614: Deploy swh.loader.git v1.1.0 and swh.model v3.0 from in-progress to code-review/await-feedback/pause on the System administration board.

Sep 28 2021, 6:50 PM · System administration, Git loader

ardumont changed the status of T3614: Deploy swh.loader.git v1.1.0 and swh.model v3.0 from Open to Work in Progress.

Sep 28 2021, 5:51 PM · System administration, Git loader

ardumont moved T3614: Deploy swh.loader.git v1.1.0 and swh.model v3.0 from Backlog to Weekly backlog on the System administration board.

Sep 28 2021, 5:51 PM · System administration, Git loader

ardumont added a project to T3614: Deploy swh.loader.git v1.1.0 and swh.model v3.0: System administration.

Sep 28 2021, 5:51 PM · System administration, Git loader

ardumont renamed T3614: Deploy swh.loader.git v1.1.0 and swh.model v3.0 from Deploy swh.loader.git v1.1.0 to Deploy swh.loader.git v1.1.0 and swh.model v3.0.

Sep 28 2021, 5:47 PM · System administration, Git loader

ardumont triaged T3614: Deploy swh.loader.git v1.1.0 and swh.model v3.0 as Normal priority.

Sep 28 2021, 3:39 PM · System administration, Git loader

ardumont closed T3457: Some git repositories are failing to be ingested because of MemoryError as Resolved.

I've deployed dulwich on our workers.
As a bonus, upstream merged the patch \o/.

Sep 28 2021, 11:42 AM · Git loader

Sep 27 2021

ardumont added a comment to T3457: Some git repositories are failing to be ingested because of MemoryError.

I made our jenkins ci build the patched dulwich with the fix discussed here.
It's currently uploaded in the swh debian repository [1]
I've deployed this on staging workers and trigger another run to
ensure everything is fine with it (again). If it is, i'll deploy on other workers tomorrow.

Sep 27 2021, 6:27 PM · Git loader

ardumont added a comment to T3457: Some git repositories are failing to be ingested because of MemoryError.

To ensure everything is working well with that patch, we executed multiple ingestions with and without the patched [1] dulwich version.

Sep 27 2021, 1:38 PM · Git loader

ardumont added a comment to T3457: Some git repositories are failing to be ingested because of MemoryError.

I forgot to mention that the patched dulwich locally and then the loader-git's tests are fine as well.

Sep 27 2021, 12:03 PM · Git loader

ardumont added a comment to T3457: Some git repositories are failing to be ingested because of MemoryError.

Draft analysis [1]
tl; dr: So far so good, the staging workers are reliably (no hash mismatch)
finishing their ingestion with their patched dulwich.

Sep 27 2021, 10:04 AM · Git loader