Sent a summary of this discussion to the swh-devel list for input:
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Oct 26 2021
Oct 25 2021
Oct 19 2021
Oct 18 2021
In T3627#72544, @douardda wrote:B3 I am not convinced a "synthetic" flag on the Snapshot branch makes sense, or at least I find this name confusing, especially considering we already have a synthetic flag on Revision: it's not synthetic in the sense of it's not object crafted by SWH, it comes from the origin.
B3 I am not convinced a "synthetic" flag on the Snapshot branch makes sense, or at least I find this name confusing, especially considering we already have a synthetic flag on Revision: it's not synthetic in the sense of it's not object crafted by SWH, it comes from the origin.
I would like us to conclude this discussion soon.
Oct 15 2021
Now, I still don't understand what mapping is to be stored in the extid table. What is
meaning of (version 0, sha1-git of the commit/tag, revision/release id) above? (I
expect a mapping to be a couple).
Oct 14 2021
In T3635#72206, @douardda wrote:Then I don't really get how this can help if we don't load revisions in topological order.
Ok I think what puzzle me in this description is the fact the 2 first bullets of the "git loader adaptations" are actually only one point: at the end of a successful loading, store a mapping in the extid table.
(I've removed T3653 as parent as this is a somewhat longer term endeavour. Not the topological sorting itself, but making sure that (most) existing revisions aren't dangling, before we can use this topological guarantee)
Actually deployed and the number of oom actually decreased.
Oct 11 2021
Deploy storage v0.38 on worker (proxy buffer/filter adaptations client/loader side).
Restarted all loaders with it.
Issues related to git dumb loading have been handled and the 5 dumb origins that were failing have been resubmitted through save code now and successfully loaded, closing this.
In T3627#71819, @vlorentz wrote:An alternative to annotating synthetic refs: add a "type" or "forge_type" attribute to snapshots.
Oct 8 2021
btw ^
In T3625#71799, @olasd wrote:While we're at it, we should probably be adding some thresholds in the buffer proxy for:
- cumulated length of messages for revisions and releases
I concur with this analysis btw
Oct 7 2021
An alternative to annotating synthetic refs: add a "type" or "forge_type" attribute to snapshots.
Ah, another question I've been thinking about: should we go back to existing visits of git repositories and give them a new, pruned snapshot? Our data model now allows it: we can just append a new final OriginVisitStatus pointing at a pruned snapshot.
In T3627#71809, @zack wrote:Thanks for your feedback @olasd. I see three main arguments raised there: (1) the raciness of archiving those data via other means (= related forks), (2) the completeness of our canvassing of synthetic refs, (3) annotating rather than not archiving "synthetic" refs.
For (1), sure, it's racy, hence we could lose stuff that gets removed from GitHub before we have the time to archive it. But this is a drop in the ocean in comparison with our lag/backlog.
Thanks for your feedback @olasd. I see three main arguments raised there: (1) the raciness of archiving those data via other means (= related forks), (2) the completeness of our canvassing of synthetic refs, (3) annotating rather than not archiving "synthetic" refs.
While we're at it, we should probably be adding some thresholds in the buffer proxy for:
- cumulated length of messages for revisions and releases
- cumulated number of parents for revisions
(this also matches the fact that we've seen, on our main ingestion database, directory_add operations that would take multiple hours, and have knock-on effects on backups and replications because of the long-running insertion transactions)
So, after doing some more analysis of memory usage patterns on these edge case repositories, my suspicion is that the high memory usage is generally being caused by the loader processing batches of large directories, closely packed together, at the same time.
In T3627#71790, @rdicosmo wrote:Yes, we must filter this stuff out (we discussed this issue with @zack some time ago)
Oct 6 2021
Yes, we must filter this stuff out (we discussed this issue with @zack some time ago, and you may see Torvalds' opinion too https://www.zdnet.com/article/linux-boosts-microsoft-ntfs-support-as-linus-torvalds-complains-about-github-merges/ )
FTR without D6401, the packfile received from GH for the CocoaPods/Specs repo contains 21162 references, 21146 of which are starting with /refs/pull/ and 7126 are ending with /merge (even if those have been explicitly not asked thanks to the filtering in RepoRepresentation.determine_wanted().
When D6401 is applied, we only get the 20-ish references that are not pull request related.
So I'm actually proposing that we filter out all branches whose name start with refs/pulls (with no other conditions attached).
Oct 5 2021
Trigger other runs with memory-profiler instead. [1]
It's not perfect though. I cannot find the proper way to actually have the
legends as they described in their documentation. [2]
I'm actually not sure yet.
So i gave a spin to [1] triggering a run on the most heavy and problematic origins with it (on
production nodes keybase/client and CocoaPods/Specs).
Let's see what result that gives (ongoing).
Oct 4 2021
In T3627#71640, @ardumont wrote:No, the snippet mentioned filters out refs whose name starts with refs/pulls and finishes with /merge
(i realize i made a typo in the main description..., it's now fixed)
@zack Maybe the diff ^ will clarify a bit ;)
According to the snippet referenced by @ardumont, all branch names starting with refs/pull/ should be filtered out.
According to the snippet referenced by @ardumont, all branch names starting with refs/pull/ should be filtered out.
But in the recent snapshot of torvalds/linux there are a lot of branch names like that.
How come?
Oct 3 2021
All runs done from medium to large repositories.
No diverging hash and consistently the loader-git ran with the patched version uses less memory.
Oct 2 2021
A draft note to send to the #swh-devel ml is been drafted [1]
Open as draft for review first.
Oct 1 2021
[3] Another idea that was only discussed would be to make certain we first start by
ingesting in order tag references (under the assumption that we will then ingest mostly
in natural order the repository). Then focus on the remaining references (because mostly
there is a high probability that if we start with HEAD and/or master at firstz, we will
end up with the overall repository).
D6377 actually increased the memory footprint to the point of getting ingestion killed
fast. So closed!
Sep 30 2021
Sep 29 2021
All save requests have been processed, 127 succeeded and 5 failed due to the following issues:
- https://git.qsomula.top/SUGAR__STRAWBERRIES/SUGAR__AND__STRAWBERRIES.git is an empty repo, that edge case is not properly handled
- 4 git repositories whose URLs are prefixed by https://git.zap.org.au/ reference pack files that are no longer hosted on the server, skipping those missing pack files makes the loadings succeeded
This is the execution result of the script above:
anlambert@carnavalet:~/tmp$ python3 reload_dumb_git_origins.py {'loading_task_id': 400298414, 'origin_url': 'https://git.pengutronix.de/cgit/tools/libsocketcan', 'save_request_date': '2021-09-29T10:07:54.080280+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298415, 'origin_url': 'https://git.stoutner.com/UltraPrivacy.git', 'save_request_date': '2021-09-29T10:07:54.507288+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298416, 'origin_url': 'https://git.stoutner.com/UltraList.git', 'save_request_date': '2021-09-29T10:07:54.795405+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298417, 'origin_url': 'https://git.stoutner.com/PrivacyCell.git', 'save_request_date': '2021-09-29T10:07:54.979121+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298418, 'origin_url': 'https://git.stoutner.com/PrivacyBrowserROMIntegration.git', 'save_request_date': '2021-09-29T10:07:55.205037+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298419, 'origin_url': 'https://git.stoutner.com/PrivacyBrowser.git', 'save_request_date': '2021-09-29T10:07:55.404696+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298420, 'origin_url': 'https://git.raptorengineering.com/git/hqemu', 'save_request_date': '2021-09-29T10:07:55.627926+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298421, 'origin_url': 'https://forge.touhey.org/casio/packaging.git', 'save_request_date': '2021-09-29T10:07:55.919487+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298422, 'origin_url': 'https://forge.touhey.org/casio/libcasio.git', 'save_request_date': '2021-09-29T10:07:56.202117+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298423, 'origin_url': 'https://git.tuxfamily.org/harmonist/harmonist.git', 'save_request_date': '2021-09-29T10:07:56.482737+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298424, 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tlibs.git', 'save_request_date': '2021-09-29T10:07:57.516829+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298425, 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git', 'save_request_date': '2021-09-29T10:07:57.811821+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298424, 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tlibs.git', 'save_request_date': '2021-09-29T10:07:57.516829+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298426, 'origin_url': 'https://depp.brause.cc/nov.el.git', 'save_request_date': '2021-09-29T10:07:58.360459+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298427, 'origin_url': 'https://www.6809.org.uk/git/asm6809.git', 'save_request_date': '2021-09-29T10:07:58.925928+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298428, 'origin_url': 'http://git.ghostscript.com/mupdf.git', 'save_request_date': '2021-09-29T10:07:59.179198+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298429, 'origin_url': 'https://git.joeyh.name/git/oldusenet.git', 'save_request_date': '2021-09-29T10:07:59.423211+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298430, 'origin_url': 'https://git.drobilla.net/ganv.git', 'save_request_date': '2021-09-29T10:07:59.660229+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298431, 'origin_url': 'https://genome-source.gi.ucsc.edu/samtabix.git', 'save_request_date': '2021-09-29T10:07:59.901717+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298432, 'origin_url': 'http://software.legiasoft.com/git/quickcheck.git', 'save_request_date': '2021-09-29T10:08:00.111266+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298433, 'origin_url': 'https://git.progress-linux.org/users/daniel.baumann/debian/packages/lziprecover', 'save_request_date': '2021-09-29T10:08:00.336308+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298434, 'origin_url': 'https://git.progress-linux.org/users/daniel.baumann/debian/packages/plzip', 'save_request_date': '2021-09-29T10:08:00.538700+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298435, 'origin_url': 'https://git.progress-linux.org/users/daniel.baumann/debian/packages/tarlz', 'save_request_date': '2021-09-29T10:08:00.758586+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298436, 'origin_url': 'https://git.progress-linux.org/users/daniel.baumann/debian/packages/lzip', 'save_request_date': '2021-09-29T10:08:00.976973+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298437, 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/takin-data.git', 'save_request_date': '2021-09-29T10:08:01.213923+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298439, 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tastools.git', 'save_request_date': '2021-09-29T10:08:01.458344+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298440, 'origin_url': 'https://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils', 'save_request_date': '2021-09-29T10:08:01.671555+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298441, 'origin_url': 'https://c9x.me/git/qbe.git', 'save_request_date': '2021-09-29T10:08:01.891677+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298442, 'origin_url': 'https://git.tuxfamily.org/ostorybook/code.git', 'save_request_date': '2021-09-29T10:08:02.096468+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298444, 'origin_url': 'https://thingshare.ion.nu/thingshare.git', 'save_request_date': '2021-09-29T10:08:02.333379+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298445, 'origin_url': 'https://inqlab.net/git/guile-sodium.git', 'save_request_date': '2021-09-29T10:08:02.603023+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298446, 'origin_url': 'https://git.systemreboot.net/guile-xapian', 'save_request_date': '2021-09-29T10:08:02.980312+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298447, 'origin_url': 'https://git.netfilter.org/iptables', 'save_request_date': '2021-09-29T10:08:03.692976+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298448, 'origin_url': 'https://inqlab.net/git/guile-cbor.git', 'save_request_date': '2021-09-29T10:08:03.966797+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298449, 'origin_url': 'https://git.openembedded.org/openembedded-core', 'save_request_date': '2021-09-29T10:08:04.228102+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298451, 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/mcstas.git', 'save_request_date': '2021-09-29T10:08:04.444651+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298437, 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/takin-data.git', 'save_request_date': '2021-09-29T10:08:01.213923+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298439, 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tastools.git', 'save_request_date': '2021-09-29T10:08:01.458344+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298425, 'origin_url': 'https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git', 'save_request_date': '2021-09-29T10:07:57.811821+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298452, 'origin_url': 'https://git.beyermatthi.as/prometheus-mpd-exporter', 'save_request_date': '2021-09-29T10:08:05.289687+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298453, 'origin_url': 'https://git.joeyh.name/git/arduino-copilot.git', 'save_request_date': '2021-09-29T10:08:05.547289+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298454, 'origin_url': 'https://git.netfilter.org/libmnl', 'save_request_date': '2021-09-29T10:08:05.849481+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298455, 'origin_url': 'https://jff.email/cgit/libunistring.git', 'save_request_date': '2021-09-29T10:08:06.257710+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298456, 'origin_url': 'https://inqlab.net/git/eris.git', 'save_request_date': '2021-09-29T10:08:06.494460+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298457, 'origin_url': 'https://jff.email/cgit/dmidecode.git', 'save_request_date': '2021-09-29T10:08:06.776931+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298697, 'origin_url': 'https://git.tukaani.org/xz.git', 'save_request_date': '2021-09-29T10:08:06.996597+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298458, 'origin_url': 'https://git.dthompson.us/guile-parser-combinators.git', 'save_request_date': '2021-09-29T10:08:07.200536+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298459, 'origin_url': 'http://git.foldling.org/vim-scheme.git', 'save_request_date': '2021-09-29T10:08:07.417546+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298460, 'origin_url': 'https://git.opensvc.com/multipath-tools/.git', 'save_request_date': '2021-09-29T10:08:07.656692+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298461, 'origin_url': 'https://git.osmocom.org/rtl-sdr', 'save_request_date': '2021-09-29T10:08:08.073457+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298463, 'origin_url': 'http://git.kpe.io/uffi.git', 'save_request_date': '2021-09-29T10:08:08.438223+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298465, 'origin_url': 'http://git.kpe.io/clsql.git', 'save_request_date': '2021-09-29T10:08:09.662233+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298466, 'origin_url': 'http://git.kpe.io/ptester.git', 'save_request_date': '2021-09-29T10:08:11.056976+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298467, 'origin_url': 'https://git.osmocom.org/gr-osmosdr', 'save_request_date': '2021-09-29T10:08:11.892136+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298468, 'origin_url': 'https://git.osmocom.org/libosmo-dsp', 'save_request_date': '2021-09-29T10:08:12.314649+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298469, 'origin_url': 'http://git.kpe.io/xlunit.git', 'save_request_date': '2021-09-29T10:08:12.602773+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298470, 'origin_url': 'http://git.kpe.io/kmrcl.git', 'save_request_date': '2021-09-29T10:08:13.082206+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298471, 'origin_url': 'https://git.open-music-kontrollers.ch/lv2/lv2lint', 'save_request_date': '2021-09-29T10:08:13.630219+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298472, 'origin_url': 'https://git.systemreboot.net/exiftool.el', 'save_request_date': '2021-09-29T10:08:14.011802+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298473, 'origin_url': 'http://git.kpe.io/cl-base64.git', 'save_request_date': '2021-09-29T10:08:14.521164+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298475, 'origin_url': 'http://git.savannah.gnu.org/r/gtick.git', 'save_request_date': '2021-09-29T10:08:15.004437+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298476, 'origin_url': 'http://git.sv.gnu.org/r/gtick.git', 'save_request_date': '2021-09-29T10:08:15.279068+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298477, 'origin_url': 'https://git.osmocom.org/gr-iqbal', 'save_request_date': '2021-09-29T10:08:15.475285+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298479, 'origin_url': 'https://git.savannah.gnu.org/r/guix.git', 'save_request_date': '2021-09-29T10:08:15.755672+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298480, 'origin_url': 'https://www.cl.cam.ac.uk/~mgk25/git/tripos', 'save_request_date': '2021-09-29T10:08:16.222379+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298481, 'origin_url': 'https://www.cl.cam.ac.uk/~mgk25/git/jbigkit', 'save_request_date': '2021-09-29T10:08:16.425781+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298482, 'origin_url': 'https://git.rockbox.org/cgit/sandbox.git', 'save_request_date': '2021-09-29T10:08:16.631493+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298483, 'origin_url': 'https://git.rockbox.org/cgit/themesite.git', 'save_request_date': '2021-09-29T10:08:16.856420+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298484, 'origin_url': 'https://git.rockbox.org/cgit/translate.git', 'save_request_date': '2021-09-29T10:08:17.059833+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298485, 'origin_url': 'https://git.rockbox.org/cgit/www.git', 'save_request_date': '2021-09-29T10:08:17.277452+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298486, 'origin_url': 'https://git.rockbox.org/cgit/rockbox.git', 'save_request_date': '2021-09-29T10:08:17.453703+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298487, 'origin_url': 'https://git.qsomula.top/SUGAR__STRAWBERRIES/SUGAR__AND__STRAWBERRIES.git', 'save_request_date': '2021-09-29T10:08:17.696238+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298488, 'origin_url': 'https://source.netsurf-browser.org/alphagen.git', 'save_request_date': '2021-09-29T10:08:17.905282+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298489, 'origin_url': 'https://source.netsurf-browser.org/ttf2f.git', 'save_request_date': '2021-09-29T10:08:18.124307+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298490, 'origin_url': 'https://source.netsurf-browser.org/toolchains.git', 'save_request_date': '2021-09-29T10:08:18.332149+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298491, 'origin_url': 'https://source.netsurf-browser.org/squeeze.git', 'save_request_date': '2021-09-29T10:08:18.541807+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298492, 'origin_url': 'https://source.netsurf-browser.org/packaging/fedora.git', 'save_request_date': '2021-09-29T10:08:18.775088+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298493, 'origin_url': 'https://source.netsurf-browser.org/packaging/debian.git', 'save_request_date': '2021-09-29T10:08:18.958225+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298494, 'origin_url': 'https://source.netsurf-browser.org/nstheme.git', 'save_request_date': '2021-09-29T10:08:19.163989+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298495, 'origin_url': 'https://source.netsurf-browser.org/nsgenbind.git', 'save_request_date': '2021-09-29T10:08:19.383856+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298496, 'origin_url': 'https://source.netsurf-browser.org/netsurf.git', 'save_request_date': '2021-09-29T10:08:19.593296+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298497, 'origin_url': 'https://source.netsurf-browser.org/netsurf-wiki.git', 'save_request_date': '2021-09-29T10:08:19.800945+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298498, 'origin_url': 'https://source.netsurf-browser.org/netsurf-website.git', 'save_request_date': '2021-09-29T10:08:19.997824+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298500, 'origin_url': 'https://source.netsurf-browser.org/netsurf-test.git', 'save_request_date': '2021-09-29T10:08:20.747537+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298501, 'origin_url': 'https://source.netsurf-browser.org/netsurf-all.git', 'save_request_date': '2021-09-29T10:08:20.948499+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298502, 'origin_url': 'https://source.netsurf-browser.org/makerun.git', 'save_request_date': '2021-09-29T10:08:21.168633+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298503, 'origin_url': 'https://source.netsurf-browser.org/libwapcaplet.git', 'save_request_date': '2021-09-29T10:08:21.391437+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298504, 'origin_url': 'https://source.netsurf-browser.org/libutf8proc.git', 'save_request_date': '2021-09-29T10:08:21.575900+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298506, 'origin_url': 'https://source.netsurf-browser.org/libsvgtiny.git', 'save_request_date': '2021-09-29T10:08:21.766147+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298507, 'origin_url': 'https://source.netsurf-browser.org/librufl.git', 'save_request_date': '2021-09-29T10:08:21.971342+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298508, 'origin_url': 'https://source.netsurf-browser.org/librosprite.git', 'save_request_date': '2021-09-29T10:08:22.163567+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298509, 'origin_url': 'https://source.netsurf-browser.org/libpencil.git', 'save_request_date': '2021-09-29T10:08:22.373564+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298510, 'origin_url': 'https://source.netsurf-browser.org/libparserutils.git', 'save_request_date': '2021-09-29T10:08:22.544880+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298511, 'origin_url': 'https://source.netsurf-browser.org/libnsutils.git', 'save_request_date': '2021-09-29T10:08:22.742637+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298512, 'origin_url': 'https://source.netsurf-browser.org/libnspsl.git', 'save_request_date': '2021-09-29T10:08:22.962451+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298513, 'origin_url': 'https://source.netsurf-browser.org/libnspdf.git', 'save_request_date': '2021-09-29T10:08:23.146196+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298514, 'origin_url': 'https://source.netsurf-browser.org/libnslog.git', 'save_request_date': '2021-09-29T10:08:23.341888+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298515, 'origin_url': 'https://source.netsurf-browser.org/libnslayout.git', 'save_request_date': '2021-09-29T10:08:23.541333+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298516, 'origin_url': 'https://source.netsurf-browser.org/libnsgif.git', 'save_request_date': '2021-09-29T10:08:23.747382+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298517, 'origin_url': 'https://source.netsurf-browser.org/libnsfb.git', 'save_request_date': '2021-09-29T10:08:23.950486+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298518, 'origin_url': 'https://source.netsurf-browser.org/libnsbmp.git', 'save_request_date': '2021-09-29T10:08:24.128810+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298519, 'origin_url': 'https://source.netsurf-browser.org/libhubbub.git', 'save_request_date': '2021-09-29T10:08:24.335594+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298520, 'origin_url': 'https://source.netsurf-browser.org/libdom.git', 'save_request_date': '2021-09-29T10:08:24.557136+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298521, 'origin_url': 'https://source.netsurf-browser.org/libcss.git', 'save_request_date': '2021-09-29T10:08:24.760323+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298522, 'origin_url': 'https://source.netsurf-browser.org/json-c.git', 'save_request_date': '2021-09-29T10:08:24.999232+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298523, 'origin_url': 'https://source.netsurf-browser.org/iconv.git', 'save_request_date': '2021-09-29T10:08:25.178058+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298524, 'origin_url': 'https://source.netsurf-browser.org/ci-tools.git', 'save_request_date': '2021-09-29T10:08:25.398196+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298525, 'origin_url': 'https://source.netsurf-browser.org/buildsystem.git', 'save_request_date': '2021-09-29T10:08:25.568154+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298526, 'origin_url': 'https://source.netsurf-browser.org/art.git', 'save_request_date': '2021-09-29T10:08:25.759202+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298527, 'origin_url': 'http://galexander.org/git/simplesshd.git', 'save_request_date': '2021-09-29T10:08:26.002317+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298528, 'origin_url': 'https://tinc-vpn.org/git/tinc', 'save_request_date': '2021-09-29T10:08:26.228797+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298529, 'origin_url': 'https://git.joeyh.name/git/concurrent-output.git', 'save_request_date': '2021-09-29T10:08:26.428783+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298530, 'origin_url': 'https://git.joeyh.name/git/brainfuck-monad.git', 'save_request_date': '2021-09-29T10:08:26.652730+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298532, 'origin_url': 'https://git.neodarz.net/pro/umosapicpp.git', 'save_request_date': '2021-09-29T10:08:26.890221+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298533, 'origin_url': 'https://git.neodarz.net/pro/umosapi.git', 'save_request_date': '2021-09-29T10:08:27.129709+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298534, 'origin_url': 'https://git.neodarz.net/neodarz/waveshare-7inch-touchscreen-driver.git', 'save_request_date': '2021-09-29T10:08:27.295848+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298535, 'origin_url': 'https://git.neodarz.net/neodarz/scripts/feed_finder.git', 'save_request_date': '2021-09-29T10:08:27.489133+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298536, 'origin_url': 'https://git.neodarz.net/neodarz/scripts/convert.py.git', 'save_request_date': '2021-09-29T10:08:27.669389+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298537, 'origin_url': 'https://git.neodarz.net/neodarz/scripts/bandcamp-dl_script.git', 'save_request_date': '2021-09-29T10:08:27.843411+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298538, 'origin_url': 'https://git.neodarz.net/neodarz/scripts/arte.tv-dl.git', 'save_request_date': '2021-09-29T10:08:28.021717+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298539, 'origin_url': 'https://git.neodarz.net/neodarz/scripts/archlinux_iso_checker.git', 'save_request_date': '2021-09-29T10:08:28.184597+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298540, 'origin_url': 'https://git.neodarz.net/neodarz/scripts/SimplyQrGen.git', 'save_request_date': '2021-09-29T10:08:28.359197+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298541, 'origin_url': 'https://git.neodarz.net/neodarz/scripts/EphermalVPN.git', 'save_request_date': '2021-09-29T10:08:28.549753+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298542, 'origin_url': 'https://git.neodarz.net/neodarz/school/the_transcriber_backend.git', 'save_request_date': '2021-09-29T10:08:28.711372+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298543, 'origin_url': 'https://git.neodarz.net/neodarz/resumejson_converter.git', 'save_request_date': '2021-09-29T10:08:28.875127+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298544, 'origin_url': 'https://git.neodarz.net/code_archive/freetorrent.git', 'save_request_date': '2021-09-29T10:08:29.058812+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298545, 'origin_url': 'https://git.e2factory.org/git/e2factory.git', 'save_request_date': '2021-09-29T10:08:29.235115+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298546, 'origin_url': 'https://git.zap.org.au/git/console-fonts-utils.git', 'save_request_date': '2021-09-29T10:08:29.485093+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298547, 'origin_url': 'https://git.zap.org.au/git/console-fonts-lucida.git', 'save_request_date': '2021-09-29T10:08:29.705671+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298548, 'origin_url': 'https://git.zap.org.au/git/console-fonts-zap.git', 'save_request_date': '2021-09-29T10:08:29.896133+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298549, 'origin_url': 'https://git.zap.org.au/git/trader.git', 'save_request_date': '2021-09-29T10:08:30.119825+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'} {'loading_task_id': 400298550, 'origin_url': 'http://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git', 'save_request_date': '2021-09-29T10:08:30.279433+00:00', 'save_request_status': 'accepted', 'save_task_status': 'not yet scheduled', 'visit_date': None, 'visit_status': None, 'visit_type': 'git'}
We will use the following script to create new save requests for those origins:
import os from pprint import pprint
This is the result of the above script execution:
anlambert@carnavalet:~/tmp$ python3 get_save_code_now_dumb_git_repos.py 2>/dev/null https://git.pengutronix.de/cgit/tools/libsocketcan https://git.stoutner.com/UltraPrivacy.git https://git.stoutner.com/UltraList.git https://git.stoutner.com/PrivacyCell.git https://git.stoutner.com/PrivacyBrowserROMIntegration.git https://git.stoutner.com/PrivacyBrowser.git https://git.raptorengineering.com/git/hqemu/ https://forge.touhey.org/casio/packaging.git https://forge.touhey.org/casio/libcasio.git https://git.tuxfamily.org/harmonist/harmonist.git https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tlibs.git https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tlibs.git/ https://depp.brause.cc/nov.el.git https://www.6809.org.uk/git/asm6809.git http://git.ghostscript.com/mupdf.git https://git.joeyh.name/git/oldusenet.git https://git.drobilla.net/ganv.git https://genome-source.gi.ucsc.edu/samtabix.git http://software.legiasoft.com/git/quickcheck.git https://git.progress-linux.org/users/daniel.baumann/debian/packages/lziprecover https://git.progress-linux.org/users/daniel.baumann/debian/packages/plzip https://git.progress-linux.org/users/daniel.baumann/debian/packages/tarlz https://git.progress-linux.org/users/daniel.baumann/debian/packages/lzip https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/takin-data.git https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tastools.git https://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils https://c9x.me/git/qbe.git https://git.tuxfamily.org/ostorybook/code.git https://thingshare.ion.nu/thingshare.git/ https://inqlab.net/git/guile-sodium.git https://git.systemreboot.net/guile-xapian https://git.netfilter.org/iptables https://inqlab.net/git/guile-cbor.git https://git.openembedded.org/openembedded-core https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/mcstas.git/ https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/takin-data.git/ https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tastools.git/ https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git/ https://git.beyermatthi.as/prometheus-mpd-exporter https://git.joeyh.name/git/arduino-copilot.git https://git.netfilter.org/libmnl/ https://jff.email/cgit/libunistring.git https://inqlab.net/git/eris.git https://jff.email/cgit/dmidecode.git https://git.tukaani.org/xz.git https://git.dthompson.us/guile-parser-combinators.git http://git.foldling.org/vim-scheme.git https://git.opensvc.com/multipath-tools/.git https://git.osmocom.org/rtl-sdr http://git.kpe.io/uffi.git http://git.kpe.io/clsql.git http://git.kpe.io/ptester.git https://git.osmocom.org/gr-osmosdr https://git.osmocom.org/libosmo-dsp http://git.kpe.io/xlunit.git http://git.kpe.io/kmrcl.git https://git.open-music-kontrollers.ch/lv2/lv2lint https://git.systemreboot.net/exiftool.el http://git.kpe.io/cl-base64.git http://git.savannah.gnu.org/r/gtick.git http://git.sv.gnu.org/r/gtick.git https://git.osmocom.org/gr-iqbal https://git.savannah.gnu.org/r/guix.git https://www.cl.cam.ac.uk/~mgk25/git/tripos/ https://www.cl.cam.ac.uk/~mgk25/git/jbigkit https://git.rockbox.org/cgit/sandbox.git/ https://git.rockbox.org/cgit/themesite.git/ https://git.rockbox.org/cgit/translate.git/ https://git.rockbox.org/cgit/www.git/ https://git.rockbox.org/cgit/rockbox.git/ https://git.qsomula.top/SUGAR__STRAWBERRIES/SUGAR__AND__STRAWBERRIES.git https://source.netsurf-browser.org/alphagen.git/ https://source.netsurf-browser.org/ttf2f.git/ https://source.netsurf-browser.org/toolchains.git/ https://source.netsurf-browser.org/squeeze.git/ https://source.netsurf-browser.org/packaging/fedora.git/ https://source.netsurf-browser.org/packaging/debian.git/ https://source.netsurf-browser.org/nstheme.git/ https://source.netsurf-browser.org/nsgenbind.git/ https://source.netsurf-browser.org/netsurf.git/ https://source.netsurf-browser.org/netsurf-wiki.git/ https://source.netsurf-browser.org/netsurf-website.git/ https://source.netsurf-browser.org/netsurf-test.git/ https://source.netsurf-browser.org/netsurf-all.git/ https://source.netsurf-browser.org/makerun.git/ https://source.netsurf-browser.org/libwapcaplet.git/ https://source.netsurf-browser.org/libutf8proc.git/ https://source.netsurf-browser.org/libsvgtiny.git/ https://source.netsurf-browser.org/librufl.git/ https://source.netsurf-browser.org/librosprite.git/ https://source.netsurf-browser.org/libpencil.git/ https://source.netsurf-browser.org/libparserutils.git/ https://source.netsurf-browser.org/libnsutils.git/ https://source.netsurf-browser.org/libnspsl.git/ https://source.netsurf-browser.org/libnspdf.git/ https://source.netsurf-browser.org/libnslog.git/ https://source.netsurf-browser.org/libnslayout.git/ https://source.netsurf-browser.org/libnsgif.git/ https://source.netsurf-browser.org/libnsfb.git/ https://source.netsurf-browser.org/libnsbmp.git/ https://source.netsurf-browser.org/libhubbub.git/ https://source.netsurf-browser.org/libdom.git/ https://source.netsurf-browser.org/libcss.git/ https://source.netsurf-browser.org/json-c.git/ https://source.netsurf-browser.org/iconv.git/ https://source.netsurf-browser.org/ci-tools.git/ https://source.netsurf-browser.org/buildsystem.git/ https://source.netsurf-browser.org/art.git/ http://galexander.org/git/simplesshd.git https://tinc-vpn.org/git/tinc https://git.joeyh.name/git/concurrent-output.git https://git.joeyh.name/git/brainfuck-monad.git https://git.neodarz.net/pro/umosapicpp.git https://git.neodarz.net/pro/umosapi.git https://git.neodarz.net/neodarz/waveshare-7inch-touchscreen-driver.git https://git.neodarz.net/neodarz/scripts/feed_finder.git https://git.neodarz.net/neodarz/scripts/convert.py.git https://git.neodarz.net/neodarz/scripts/bandcamp-dl_script.git https://git.neodarz.net/neodarz/scripts/arte.tv-dl.git https://git.neodarz.net/neodarz/scripts/archlinux_iso_checker.git https://git.neodarz.net/neodarz/scripts/SimplyQrGen.git https://git.neodarz.net/neodarz/scripts/EphermalVPN.git https://git.neodarz.net/neodarz/school/the_transcriber_backend.git https://git.neodarz.net/neodarz/resumejson_converter.git https://git.neodarz.net/code_archive/freetorrent.git https://git.e2factory.org/git/e2factory.git https://git.zap.org.au/git/console-fonts-utils.git https://git.zap.org.au/git/console-fonts-lucida.git https://git.zap.org.au/git/console-fonts-zap.git https://git.zap.org.au/git/trader.git http://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/miezetools.git/
Afterwards, we use the following script to identify the dumb git origins that can be reloaded:
import io import os
Then we copy that dump locally and load it into our development database:
anlambert@carnavalet:~/tmp$ scp anlambert@moma:swh-web-db.json . swh-web-db.json 100% 33MB 5.6MB/s 00:05 anlambert@carnavalet:~/tmp$ django-admin loaddata --settings swh.web.settings.development swh-web-db.json Installed 93893 object(s) from 1 fixture(s) anlambert@carnavalet:~/tmp$
First we need a JSON dump of production database:
anlambert@carnavalet:~/tmp$ ssh anlambert@moma Linux moma 5.10.0-0.bpo.5-amd64 #1 SMP Debian 5.10.24-1~bpo10+1 (2021-03-29) x86_64
Dumb protocol support has been implemented in the git loader and deployed to production (see that origin example and its list of all visits), closing this.