Page MenuHomeSoftware Heritage

Origin-GitoriousTag
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Recent Activity

Jun 19 2020

olasd closed T2410: Check and complete the gitorious.org import as Resolved.

We still need to try to ingest the zeq2 repo, but that can be done in a followup task.

Jun 19 2020, 10:20 AM · Git loader, Origin-Gitorious

May 30 2020

olasd added a comment to T2410: Check and complete the gitorious.org import.

The following repositories failed to import. Their on-disk structure is either completely empty, or only contains refs (no actual git objects stored):

May 30 2020, 12:58 PM · Git loader, Origin-Gitorious

May 29 2020

olasd added a comment to T2410: Check and complete the gitorious.org import.

After the first (naive, I guess) pass, 1470 repositories are still missing.

May 29 2020, 5:16 PM · Git loader, Origin-Gitorious

May 19 2020

olasd changed the status of T2410: Check and complete the gitorious.org import from Open to Work in Progress.

The code for loading git repositories from disk hasn't been run in production in a while, so I've decided to run the imports of the missing repos manually.

May 19 2020, 5:02 PM · Git loader, Origin-Gitorious
olasd added a comment to T2410: Check and complete the gitorious.org import.

We also have a single origin with no full visit:

May 19 2020, 12:07 PM · Git loader, Origin-Gitorious
olasd added a comment to T2410: Check and complete the gitorious.org import.

After dumping all origins starting with https://gitorious.org/ in the archive:

May 19 2020, 12:04 PM · Git loader, Origin-Gitorious
rdicosmo triaged T2410: Check and complete the gitorious.org import as High priority.
May 19 2020, 9:49 AM · Git loader, Origin-Gitorious

Jun 19 2018

zack edited projects for T312: Gitorious import: ingest repositories, added: Archive coverage; removed Archive content.
Jun 19 2018, 3:27 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Apr 12 2018

ardumont closed T312: Gitorious import: ingest repositories as Resolved.
Apr 12 2018, 2:05 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont closed T674: Gitorious import: Examine ingestion logs for errors and list them if any, a subtask of T312: Gitorious import: ingest repositories, as Resolved.
Apr 12 2018, 2:04 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont closed T674: Gitorious import: Examine ingestion logs for errors and list them if any as Resolved.

Last origin rescheduled and injected.

Apr 12 2018, 2:04 PM · Origin-Gitorious, Format-Git
ardumont closed T911: gitorious import: UnicodeDecodeError when reading references, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Apr 12 2018, 2:02 PM · Origin-Gitorious, Format-Git
ardumont closed T911: gitorious import: UnicodeDecodeError when reading references as Resolved.

python3-dulwich (fix included) packaged and pushed to our debian repository.

Apr 12 2018, 2:02 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

After discussion with jelmer (dulwich's author), he proposed and implemented the real solution, deal with bytes (avoiding altogether encoding water mudding ;)
It's landed in dulwich/dulwich's master branch \m/.

Apr 12 2018, 9:23 AM · Origin-Gitorious, Format-Git

Apr 11 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

Patching dulwich to try and detect the encoding (when the problem arose) seems to do the trick:

Apr 11 2018, 6:52 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

With latest dulwich (> 0.19.1, current head) we break somewhere else now, still encoding related:

Apr 11 2018, 4:20 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

I opened a discussion at at https://github.com/jelmer/dulwich/issues/608 about this case.

Apr 11 2018, 2:24 PM · Origin-Gitorious, Format-Git

Jan 19 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

I was initially opened to clean up the repository because i thought it was some form of corruption.
But now, i no longer think that's the case. And don't want to tamper with sources.

Jan 19 2018, 11:05 AM · Origin-Gitorious, Format-Git

Jan 18 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

After some digging, it seems an encoding problem:

Jan 18 2018, 6:54 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

Trying to analyze a bit further that repository, we can see this:

Jan 18 2018, 6:31 PM · Origin-Gitorious, Format-Git

Dec 21 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

The sql error was sheer bad luck, tested locally and no problem, so it was rescheduled, loaded successfully.

Dec 21 2017, 11:42 AM · Origin-Gitorious, Format-Git
ardumont created T911: gitorious import: UnicodeDecodeError when reading references.
Dec 21 2017, 11:37 AM · Origin-Gitorious, Format-Git
ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

Only 2 errors left:

  • 1 about bad transaction in db
  • 1 about unicode error:
Dec 21 2017, 11:23 AM · Origin-Gitorious, Format-Git

Dec 19 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

Updated and scheduled the last 170 repositories.
Now, remains those to be checked for errors.

Dec 19 2017, 6:19 PM · Origin-Gitorious, Format-Git

Dec 15 2017

ardumont closed T816: Gitorious import: loose object parsing error with corrupted file as empty one as Resolved.
Dec 15 2017, 7:43 PM · Git loader, Origin-Gitorious
ardumont closed T816: Gitorious import: loose object parsing error with corrupted file as empty one, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Dec 15 2017, 7:43 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T816: Gitorious import: loose object parsing error with corrupted file as empty one.

Fixed with that latest version package:

Dec 15 2017, 7:43 PM · Git loader, Origin-Gitorious
ardumont added a comment to T816: Gitorious import: loose object parsing error with corrupted file as empty one.

Packaged it and pushed to our own repository.

Dec 15 2017, 7:37 PM · Git loader, Origin-Gitorious
ardumont added a comment to T816: Gitorious import: loose object parsing error with corrupted file as empty one.

Update on this:

  • Issue opened.
  • Pull Request (PR) proposed and merged.
Dec 15 2017, 1:43 PM · Git loader, Origin-Gitorious

Nov 13 2017

ardumont closed T823: Gitorious import: Overflow error in revision time as Resolved by committing rDLDG120f23dd0bf2: swh.loader.git.disk: Force further checks on objects.
Nov 13 2017, 6:40 PM · Origin-Gitorious, Storage manager, Git loader
ardumont added a comment to T816: Gitorious import: loose object parsing error with corrupted file as empty one.

This error slipped under my radar last week.
I opened a related issue in dulwich since it should be handled upstream.

Nov 13 2017, 2:53 PM · Git loader, Origin-Gitorious

Nov 10 2017

ardumont added a comment to T823: Gitorious import: Overflow error in revision time.

PR got merged \m/

Nov 10 2017, 6:29 PM · Origin-Gitorious, Storage manager, Git loader

Nov 7 2017

ardumont closed T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?) as Resolved by committing rDLDGfece2335e246: swh.loader.git.loader: Warn when object malformed and continue.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Git loader
ardumont closed T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?), a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git
ardumont closed T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git
ardumont closed T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository as Resolved by committing rDLDGfece2335e246: swh.loader.git.loader: Warn when object malformed and continue.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Git loader
ardumont closed T814: Gitorious import: unexisting object retrieval makes the loading fail, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git
ardumont closed T814: Gitorious import: unexisting object retrieval makes the loading fail as Resolved by committing rDLDG5e2d236b6a3f: swh.loader.git.loader: Trap missing object id and continue.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Git loader

Nov 4 2017

ardumont added a comment to T823: Gitorious import: Overflow error in revision time.

PR got merged \m/

Nov 4 2017, 12:58 PM · Origin-Gitorious, Storage manager, Git loader

Oct 31 2017

ardumont added a comment to T823: Gitorious import: Overflow error in revision time.

Follow up on this:

Oct 31 2017, 3:38 PM · Origin-Gitorious, Storage manager, Git loader

Oct 27 2017

ardumont triaged T823: Gitorious import: Overflow error in revision time as Normal priority.
Oct 27 2017, 2:26 PM · Origin-Gitorious, Storage manager, Git loader
ardumont added a comment to T823: Gitorious import: Overflow error in revision time.

The revision in question is:

Oct 27 2017, 2:18 PM · Origin-Gitorious, Storage manager, Git loader
ardumont added a comment to T823: Gitorious import: Overflow error in revision time.

Debugging some more, the date generating this error is the following, which raises indeed the initial overflow error:

Oct 27 2017, 2:10 PM · Origin-Gitorious, Storage manager, Git loader
ardumont created T823: Gitorious import: Overflow error in revision time.
Oct 27 2017, 2:09 PM · Origin-Gitorious, Storage manager, Git loader
ardumont added a comment to T814: Gitorious import: unexisting object retrieval makes the loading fail.

Possibly related error.

Oct 27 2017, 1:36 PM · Origin-Gitorious, Git loader
ardumont updated the task description for T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?).
Oct 27 2017, 12:25 PM · Origin-Gitorious, Git loader
ardumont added a comment to T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?).

Debugging problematic object shows 1e82c9224b8898672b3b6fe8b6b737f7eed24cf6 which git fsck references as well.
Turns out it's a badly formatted tag:

Oct 27 2017, 11:52 AM · Origin-Gitorious, Git loader
ardumont added a parent task for T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?): T674: Gitorious import: Examine ingestion logs for errors and list them if any.
Oct 27 2017, 11:00 AM · Origin-Gitorious, Git loader
ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?).
Oct 27 2017, 11:00 AM · Origin-Gitorious, Format-Git
ardumont renamed T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?) from Gitorious import: ObjectFormatException raised when badly formatted object to Gitorious import: ObjectFormatException raised when badly formatted object (around date?).
Oct 27 2017, 10:59 AM · Origin-Gitorious, Git loader