The following repositories failed to import. Their on-disk structure is either completely empty, or only contains refs (no actual git objects stored):

May 30 2020, 12:58 PM · Git loader, Origin-Gitorious

May 29 2020

olasd added a comment to T2410: Check and complete the gitorious.org import.

After the first (naive, I guess) pass, 1470 repositories are still missing.

May 29 2020, 5:16 PM · Git loader, Origin-Gitorious

May 19 2020

olasd changed the status of T2410: Check and complete the gitorious.org import from Open to Work in Progress.

The code for loading git repositories from disk hasn't been run in production in a while, so I've decided to run the imports of the missing repos manually.

May 19 2020, 5:02 PM · Git loader, Origin-Gitorious

olasd added a comment to T2410: Check and complete the gitorious.org import.

We also have a single origin with no full visit:

May 19 2020, 12:07 PM · Git loader, Origin-Gitorious

olasd added a comment to T2410: Check and complete the gitorious.org import.

After dumping all origins starting with https://gitorious.org/ in the archive:

May 19 2020, 12:04 PM · Git loader, Origin-Gitorious

rdicosmo triaged T2410: Check and complete the gitorious.org import as High priority.

May 19 2020, 9:49 AM · Git loader, Origin-Gitorious

Jun 19 2018

zack edited projects for T312: Gitorious import: ingest repositories, added: Archive coverage; removed Archive content.

Jun 19 2018, 3:27 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Apr 12 2018

ardumont closed T312: Gitorious import: ingest repositories as Resolved.

Apr 12 2018, 2:05 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

ardumont closed T674: Gitorious import: Examine ingestion logs for errors and list them if any, a subtask of T312: Gitorious import: ingest repositories, as Resolved.

Apr 12 2018, 2:04 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

ardumont closed T674: Gitorious import: Examine ingestion logs for errors and list them if any as Resolved.

Last origin rescheduled and injected.

Apr 12 2018, 2:04 PM · Origin-Gitorious, Format-Git

ardumont closed T911: gitorious import: UnicodeDecodeError when reading references, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.

Apr 12 2018, 2:02 PM · Origin-Gitorious, Format-Git

ardumont closed T911: gitorious import: UnicodeDecodeError when reading references as Resolved.

python3-dulwich (fix included) packaged and pushed to our debian repository.

Apr 12 2018, 2:02 PM · Origin-Gitorious, Format-Git

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

After discussion with jelmer (dulwich's author), he proposed and implemented the real solution, deal with bytes (avoiding altogether encoding water mudding ;)
It's landed in dulwich/dulwich's master branch \m/.

Apr 12 2018, 9:23 AM · Origin-Gitorious, Format-Git

Apr 11 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

Patching dulwich to try and detect the encoding (when the problem arose) seems to do the trick:

Apr 11 2018, 6:52 PM · Origin-Gitorious, Format-Git

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

With latest dulwich (> 0.19.1, current head) we break somewhere else now, still encoding related:

Apr 11 2018, 4:20 PM · Origin-Gitorious, Format-Git

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

I opened a discussion at at https://github.com/jelmer/dulwich/issues/608 about this case.

Apr 11 2018, 2:24 PM · Origin-Gitorious, Format-Git

Jan 19 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

I was initially opened to clean up the repository because i thought it was some form of corruption.
But now, i no longer think that's the case. And don't want to tamper with sources.

Jan 19 2018, 11:05 AM · Origin-Gitorious, Format-Git

Jan 18 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

After some digging, it seems an encoding problem:

Jan 18 2018, 6:54 PM · Origin-Gitorious, Format-Git

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

Trying to analyze a bit further that repository, we can see this:

Jan 18 2018, 6:31 PM · Origin-Gitorious, Format-Git

Dec 21 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

The sql error was sheer bad luck, tested locally and no problem, so it was rescheduled, loaded successfully.

Dec 21 2017, 11:42 AM · Origin-Gitorious, Format-Git

ardumont created T911: gitorious import: UnicodeDecodeError when reading references.

Dec 21 2017, 11:37 AM · Origin-Gitorious, Format-Git

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

Only 2 errors left:

1 about bad transaction in db
1 about unicode error:

Dec 21 2017, 11:23 AM · Origin-Gitorious, Format-Git

Dec 19 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

Updated and scheduled the last 170 repositories.
Now, remains those to be checked for errors.

Dec 19 2017, 6:19 PM · Origin-Gitorious, Format-Git

Dec 15 2017

ardumont closed T816: Gitorious import: loose object parsing error with corrupted file as empty one as Resolved.

Dec 15 2017, 7:43 PM · Git loader, Origin-Gitorious

ardumont closed T816: Gitorious import: loose object parsing error with corrupted file as empty one, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.

Dec 15 2017, 7:43 PM · Origin-Gitorious, Format-Git

ardumont added a comment to T816: Gitorious import: loose object parsing error with corrupted file as empty one.

Fixed with that latest version package:

Dec 15 2017, 7:43 PM · Git loader, Origin-Gitorious

ardumont added a comment to T816: Gitorious import: loose object parsing error with corrupted file as empty one.

Packaged it and pushed to our own repository.

Dec 15 2017, 7:37 PM · Git loader, Origin-Gitorious

ardumont added a comment to T816: Gitorious import: loose object parsing error with corrupted file as empty one.

Update on this:

Issue opened.
Pull Request (PR) proposed and merged.

Dec 15 2017, 1:43 PM · Git loader, Origin-Gitorious

Nov 13 2017

ardumont closed T823: Gitorious import: Overflow error in revision time as Resolved by committing rDLDG120f23dd0bf2: swh.loader.git.disk: Force further checks on objects.

Nov 13 2017, 6:40 PM · Origin-Gitorious, Storage manager, Git loader

ardumont added a comment to T816: Gitorious import: loose object parsing error with corrupted file as empty one.

This error slipped under my radar last week.
I opened a related issue in dulwich since it should be handled upstream.

Nov 13 2017, 2:53 PM · Git loader, Origin-Gitorious

Nov 10 2017

ardumont added a comment to T823: Gitorious import: Overflow error in revision time.

PR got merged \m/

Nov 10 2017, 6:29 PM · Origin-Gitorious, Storage manager, Git loader

Nov 7 2017

ardumont closed T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?) as Resolved by committing rDLDGfece2335e246: swh.loader.git.loader: Warn when object malformed and continue.

Nov 7 2017, 6:22 PM · Origin-Gitorious, Git loader

ardumont closed T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?), a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.

Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git

ardumont closed T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.

Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git

ardumont closed T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository as Resolved by committing rDLDGfece2335e246: swh.loader.git.loader: Warn when object malformed and continue.

Nov 7 2017, 6:22 PM · Origin-Gitorious, Git loader

ardumont closed T814: Gitorious import: unexisting object retrieval makes the loading fail, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.

Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git

ardumont closed T814: Gitorious import: unexisting object retrieval makes the loading fail as Resolved by committing rDLDG5e2d236b6a3f: swh.loader.git.loader: Trap missing object id and continue.

Nov 7 2017, 6:22 PM · Origin-Gitorious, Git loader

Nov 4 2017

ardumont added a comment to T823: Gitorious import: Overflow error in revision time.

PR got merged \m/

Nov 4 2017, 12:58 PM · Origin-Gitorious, Storage manager, Git loader

Oct 31 2017

ardumont added a comment to T823: Gitorious import: Overflow error in revision time.

Follow up on this:

Oct 31 2017, 3:38 PM · Origin-Gitorious, Storage manager, Git loader

Oct 27 2017

ardumont triaged T823: Gitorious import: Overflow error in revision time as Normal priority.

Oct 27 2017, 2:26 PM · Origin-Gitorious, Storage manager, Git loader

ardumont added a comment to T823: Gitorious import: Overflow error in revision time.

The revision in question is:

Oct 27 2017, 2:18 PM · Origin-Gitorious, Storage manager, Git loader

ardumont added a comment to T823: Gitorious import: Overflow error in revision time.

Debugging some more, the date generating this error is the following, which raises indeed the initial overflow error:

Oct 27 2017, 2:10 PM · Origin-Gitorious, Storage manager, Git loader

ardumont created T823: Gitorious import: Overflow error in revision time.

Oct 27 2017, 2:09 PM · Origin-Gitorious, Storage manager, Git loader

ardumont added a comment to T814: Gitorious import: unexisting object retrieval makes the loading fail.

Possibly related error.

Oct 27 2017, 1:36 PM · Origin-Gitorious, Git loader

ardumont updated the task description for T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?).

Oct 27 2017, 12:25 PM · Origin-Gitorious, Git loader

ardumont added a comment to T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?).

Debugging problematic object shows 1e82c9224b8898672b3b6fe8b6b737f7eed24cf6 which git fsck references as well.
Turns out it's a badly formatted tag:

Oct 27 2017, 11:52 AM · Origin-Gitorious, Git loader

ardumont added a parent task for T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?): T674: Gitorious import: Examine ingestion logs for errors and list them if any.

Oct 27 2017, 11:00 AM · Origin-Gitorious, Git loader

ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?).

Oct 27 2017, 11:00 AM · Origin-Gitorious, Format-Git

ardumont renamed T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?) from Gitorious import: ObjectFormatException raised when badly formatted object to Gitorious import: ObjectFormatException raised when badly formatted object (around date?).

Oct 27 2017, 10:59 AM · Origin-Gitorious, Git loader

ardumont created T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?).

Oct 27 2017, 10:59 AM · Origin-Gitorious, Git loader

Oct 26 2017

ardumont renamed T816: Gitorious import: loose object parsing error with corrupted file as empty one from Gitorious import: loose object parsing error with the empty file to Gitorious import: loose object parsing error with corrupted file as empty one.

Oct 26 2017, 4:14 PM · Git loader, Origin-Gitorious

ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository.

Oct 26 2017, 4:07 PM · Origin-Gitorious, Format-Git

ardumont added a parent task for T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository: T674: Gitorious import: Examine ingestion logs for errors and list them if any.

Oct 26 2017, 4:07 PM · Origin-Gitorious, Git loader

ardumont renamed T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository from Gitorious import: ObjectFormatException on what looks like a date field to Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository.

Oct 26 2017, 4:07 PM · Origin-Gitorious, Git loader

ardumont added a comment to T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository.

Patching the version to print the identifier in error, i retrieve the following object ae51106031a0bb39a8def57a8592f70116487eab (which is amongst the badly formatted tags listed by git fsck below).

Oct 26 2017, 4:02 PM · Origin-Gitorious, Git loader

ardumont updated the task description for T815: Gitorious import: Release time conversion issue when no release date is provided.

Oct 26 2017, 3:48 PM · Origin-Gitorious, Git loader

ardumont updated the task description for T816: Gitorious import: loose object parsing error with corrupted file as empty one.

Oct 26 2017, 3:48 PM · Git loader, Origin-Gitorious

ardumont renamed T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository from Gitorious import: to Gitorious import: ObjectFormatException on what looks like a date field.

Oct 26 2017, 3:48 PM · Origin-Gitorious, Git loader

ardumont updated the task description for T814: Gitorious import: unexisting object retrieval makes the loading fail.

Oct 26 2017, 3:45 PM · Origin-Gitorious, Git loader

ardumont closed T815: Gitorious import: Release time conversion issue when no release date is provided as Resolved by committing rDLDG2c91a6feb6f0: converters: Fix release time conversion issue when no date provided.

Oct 26 2017, 1:11 PM · Origin-Gitorious, Git loader

ardumont closed T815: Gitorious import: Release time conversion issue when no release date is provided, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.

Oct 26 2017, 1:11 PM · Origin-Gitorious, Format-Git

ardumont renamed T815: Gitorious import: Release time conversion issue when no release date is provided from Gitorious import: Release time conversion issue when none is provided to Gitorious import: Release time conversion issue when no release date is provided.

Oct 26 2017, 1:09 PM · Origin-Gitorious, Git loader

ardumont renamed T816: Gitorious import: loose object parsing error with corrupted file as empty one from Gitorious import: loose object parsing error to Gitorious import: loose object parsing error with the empty file.

Oct 26 2017, 11:55 AM · Git loader, Origin-Gitorious

ardumont renamed T815: Gitorious import: Release time conversion issue when no release date is provided from Gitorious import: Time conversion issue to Gitorious import: Release time conversion issue when none is provided.

Oct 26 2017, 11:53 AM · Origin-Gitorious, Git loader

ardumont added a comment to T815: Gitorious import: Release time conversion issue when no release date is provided.

In that particular repository, the tag has no time (tag.tag_time and tag.tag_timezone are None, tag._tag_timezone_neg_utc is False - those are the default values for that object).
But the swh-loader-git's code expects those values to exist.
In our model though, we are ok with that date not being provided.

Oct 26 2017, 11:53 AM · Origin-Gitorious, Git loader

ardumont added a comment to T816: Gitorious import: loose object parsing error with corrupted file as empty one.

Tweaking the loader git to print the actual sha1:

Oct 26 2017, 11:36 AM · Git loader, Origin-Gitorious

ardumont updated the task description for T816: Gitorious import: loose object parsing error with corrupted file as empty one.

Oct 26 2017, 11:21 AM · Git loader, Origin-Gitorious

ardumont added a parent task for T816: Gitorious import: loose object parsing error with corrupted file as empty one: T674: Gitorious import: Examine ingestion logs for errors and list them if any.

Oct 26 2017, 11:20 AM · Git loader, Origin-Gitorious

ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T816: Gitorious import: loose object parsing error with corrupted file as empty one.

Oct 26 2017, 11:20 AM · Origin-Gitorious, Format-Git

ardumont renamed T816: Gitorious import: loose object parsing error with corrupted file as empty one from Gitorious import: to Gitorious import: loose object parsing error.

Oct 26 2017, 11:20 AM · Git loader, Origin-Gitorious