Page MenuHomeSoftware Heritage

Format-GitTag
ActivePublic

Members

  • This project does not have any members.

Watchers

  • This project does not have any watchers.

Details

Description

Projects related to the Git VCS

Recent Activity

Jun 19 2018

zack edited projects for T312: Gitorious import: ingest repositories, added: Archive coverage; removed Archive content.
Jun 19 2018, 3:27 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Apr 12 2018

ardumont closed T312: Gitorious import: ingest repositories as Resolved.
Apr 12 2018, 2:05 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont closed T674: Gitorious import: Examine ingestion logs for errors and list them if any, a subtask of T312: Gitorious import: ingest repositories, as Resolved.
Apr 12 2018, 2:04 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont closed T674: Gitorious import: Examine ingestion logs for errors and list them if any as Resolved.

Last origin rescheduled and injected.

Apr 12 2018, 2:04 PM · Origin-Gitorious, Format-Git
ardumont closed T911: gitorious import: UnicodeDecodeError when reading references, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Apr 12 2018, 2:02 PM · Origin-Gitorious, Format-Git
ardumont closed T911: gitorious import: UnicodeDecodeError when reading references as Resolved.

python3-dulwich (fix included) packaged and pushed to our debian repository.

Apr 12 2018, 2:02 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

After discussion with jelmer (dulwich's author), he proposed and implemented the real solution, deal with bytes (avoiding altogether encoding water mudding ;)
It's landed in dulwich/dulwich's master branch \m/.

Apr 12 2018, 9:23 AM · Origin-Gitorious, Format-Git

Apr 11 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

Patching dulwich to try and detect the encoding (when the problem arose) seems to do the trick:

Apr 11 2018, 6:52 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

With latest dulwich (> 0.19.1, current head) we break somewhere else now, still encoding related:

Apr 11 2018, 4:20 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

I opened a discussion at at https://github.com/jelmer/dulwich/issues/608 about this case.

Apr 11 2018, 2:24 PM · Origin-Gitorious, Format-Git

Jan 19 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

I was initially opened to clean up the repository because i thought it was some form of corruption.
But now, i no longer think that's the case. And don't want to tamper with sources.

Jan 19 2018, 11:05 AM · Origin-Gitorious, Format-Git

Jan 18 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

After some digging, it seems an encoding problem:

Jan 18 2018, 6:54 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

Trying to analyze a bit further that repository, we can see this:

Jan 18 2018, 6:31 PM · Origin-Gitorious, Format-Git

Dec 21 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

The sql error was sheer bad luck, tested locally and no problem, so it was rescheduled, loaded successfully.

Dec 21 2017, 11:42 AM · Origin-Gitorious, Format-Git
ardumont created T911: gitorious import: UnicodeDecodeError when reading references.
Dec 21 2017, 11:37 AM · Origin-Gitorious, Format-Git
ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

Only 2 errors left:

  • 1 about bad transaction in db
  • 1 about unicode error:
Dec 21 2017, 11:23 AM · Origin-Gitorious, Format-Git

Dec 19 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

Updated and scheduled the last 170 repositories.
Now, remains those to be checked for errors.

Dec 19 2017, 6:19 PM · Origin-Gitorious, Format-Git

Dec 15 2017

ardumont closed T816: Gitorious import: loose object parsing error with corrupted file as empty one, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Dec 15 2017, 7:43 PM · Origin-Gitorious, Format-Git

Nov 7 2017

ardumont closed T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?), a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git
ardumont closed T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git
ardumont closed T814: Gitorious import: unexisting object retrieval makes the loading fail, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git

Oct 27 2017

ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?).
Oct 27 2017, 11:00 AM · Origin-Gitorious, Format-Git

Oct 26 2017

ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository.
Oct 26 2017, 4:07 PM · Origin-Gitorious, Format-Git
ardumont closed T815: Gitorious import: Release time conversion issue when no release date is provided, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Oct 26 2017, 1:11 PM · Origin-Gitorious, Format-Git
ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T816: Gitorious import: loose object parsing error with corrupted file as empty one.
Oct 26 2017, 11:20 AM · Origin-Gitorious, Format-Git
ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T814: Gitorious import: unexisting object retrieval makes the loading fail.
Oct 26 2017, 11:11 AM · Origin-Gitorious, Format-Git
ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T815: Gitorious import: Release time conversion issue when no release date is provided.
Oct 26 2017, 11:11 AM · Origin-Gitorious, Format-Git

Oct 3 2017

ardumont renamed T312: Gitorious import: ingest repositories from ingest Gitorious repositories to Gitorious import: ingest repositories.
Oct 3 2017, 10:14 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont added a subtask for T312: Gitorious import: ingest repositories: T674: Gitorious import: Examine ingestion logs for errors and list them if any.
Oct 3 2017, 10:14 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont added a parent task for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T312: Gitorious import: ingest repositories.
Oct 3 2017, 10:14 AM · Origin-Gitorious, Format-Git
ardumont removed a parent task for T312: Gitorious import: ingest repositories: T674: Gitorious import: Examine ingestion logs for errors and list them if any.
Oct 3 2017, 10:13 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont removed a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T312: Gitorious import: ingest repositories.
Oct 3 2017, 10:13 AM · Origin-Gitorious, Format-Git

Jul 28 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

For information, the last injection has been done. The remaining errors:

Jul 28 2017, 11:11 AM · Origin-Gitorious, Format-Git
ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.
(but we should have a list of those repos, for posterity).
Jul 28 2017, 10:29 AM · Origin-Gitorious, Format-Git

Jul 27 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

These should be rescheduled and driven to successful completion.

Jul 27 2017, 3:03 PM · Origin-Gitorious, Format-Git
zack added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.
  • we send something that was not a git repository.
  • integrity error (which is expected for now)
Jul 27 2017, 12:02 PM · Origin-Gitorious, Format-Git

Jul 26 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

After much learning on how to read and extract logs from our kibana instance, here is the error repartition.

Jul 26 2017, 12:51 PM · Origin-Gitorious, Format-Git

Jun 6 2017

ardumont added a comment to T312: Gitorious import: ingest repositories.

As of now, ingestion, after multiple (re)schedulings, has been done.

Jun 6 2017, 1:34 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Apr 26 2017

ardumont added a comment to T312: Gitorious import: ingest repositories.

Update on this.

Apr 26 2017, 10:41 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Apr 7 2017

zack added a project to T312: Gitorious import: ingest repositories: Archive content.
Apr 7 2017, 11:00 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Feb 15 2017

ardumont added a comment to T312: Gitorious import: ingest repositories.

Visit dates have been fixed for the origins already injected.

Feb 15 2017, 7:34 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack added a parent task for T312: Gitorious import: ingest repositories: T674: Gitorious import: Examine ingestion logs for errors and list them if any.
Feb 15 2017, 4:13 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T312: Gitorious import: ingest repositories.
Feb 15 2017, 4:13 PM · Origin-Gitorious, Format-Git
zack removed a subtask for T312: Gitorious import: ingest repositories: T674: Gitorious import: Examine ingestion logs for errors and list them if any.
Feb 15 2017, 4:13 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack removed a parent task for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T312: Gitorious import: ingest repositories.
Feb 15 2017, 4:13 PM · Origin-Gitorious, Format-Git

Feb 12 2017

zack renamed T312: Gitorious import: ingest repositories from ingest gitorious repositories to ingest Gitorious repositories.
Feb 12 2017, 6:37 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack moved T312: Gitorious import: ingest repositories from Restricted Project Column to Restricted Project Column on the Restricted Project board.
Feb 12 2017, 6:37 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack added a project to T312: Gitorious import: ingest repositories: Restricted Project.
Feb 12 2017, 6:13 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Feb 11 2017

ardumont added a comment to T312: Gitorious import: ingest repositories.

Command to trigger the messages (from worker01):

cat /srv/storage/space/mirrors/gitorious.org/full_mapping.txt | SWH_WORKER_INSTANCE=swh_loader_git_disk ./load_gitorious.py --root-repositories /srv/storage/space/mirrors/gitorious.org/mnt/repositories

(The script defaults to use the right queue 'swh_loader_git_express' and the right origin-date 'Wed, 30 Mar 2016 09:40:04 +0200')

Feb 11 2017, 2:11 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Feb 10 2017

ardumont changed the status of T312: Gitorious import: ingest repositories from Open to Work in Progress.
Feb 10 2017, 8:15 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git