Page MenuHomeSoftware Heritage
Feed Advanced Search

Jun 19 2018

zack edited projects for T312: Gitorious import: ingest repositories, added: Archive coverage; removed Archive content.
Jun 19 2018, 3:27 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Apr 12 2018

ardumont closed T312: Gitorious import: ingest repositories as Resolved.
Apr 12 2018, 2:05 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont closed T674: Gitorious import: Examine ingestion logs for errors and list them if any, a subtask of T312: Gitorious import: ingest repositories, as Resolved.
Apr 12 2018, 2:04 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont closed T674: Gitorious import: Examine ingestion logs for errors and list them if any as Resolved.

Last origin rescheduled and injected.

Apr 12 2018, 2:04 PM · Origin-Gitorious, Format-Git
ardumont closed T911: gitorious import: UnicodeDecodeError when reading references, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Apr 12 2018, 2:02 PM · Origin-Gitorious, Format-Git
ardumont closed T911: gitorious import: UnicodeDecodeError when reading references as Resolved.

python3-dulwich (fix included) packaged and pushed to our debian repository.

Apr 12 2018, 2:02 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

After discussion with jelmer (dulwich's author), he proposed and implemented the real solution, deal with bytes (avoiding altogether encoding water mudding ;)
It's landed in dulwich/dulwich's master branch \m/.

Apr 12 2018, 9:23 AM · Origin-Gitorious, Format-Git

Apr 11 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

Patching dulwich to try and detect the encoding (when the problem arose) seems to do the trick:

Apr 11 2018, 6:52 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

With latest dulwich (> 0.19.1, current head) we break somewhere else now, still encoding related:

Apr 11 2018, 4:20 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

I opened a discussion at at https://github.com/jelmer/dulwich/issues/608 about this case.

Apr 11 2018, 2:24 PM · Origin-Gitorious, Format-Git

Jan 19 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

I was initially opened to clean up the repository because i thought it was some form of corruption.
But now, i no longer think that's the case. And don't want to tamper with sources.

Jan 19 2018, 11:05 AM · Origin-Gitorious, Format-Git

Jan 18 2018

ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

After some digging, it seems an encoding problem:

Jan 18 2018, 6:54 PM · Origin-Gitorious, Format-Git
ardumont added a comment to T911: gitorious import: UnicodeDecodeError when reading references.

Trying to analyze a bit further that repository, we can see this:

Jan 18 2018, 6:31 PM · Origin-Gitorious, Format-Git

Dec 21 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

The sql error was sheer bad luck, tested locally and no problem, so it was rescheduled, loaded successfully.

Dec 21 2017, 11:42 AM · Origin-Gitorious, Format-Git
ardumont created T911: gitorious import: UnicodeDecodeError when reading references.
Dec 21 2017, 11:37 AM · Origin-Gitorious, Format-Git
ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

Only 2 errors left:

  • 1 about bad transaction in db
  • 1 about unicode error:
Dec 21 2017, 11:23 AM · Origin-Gitorious, Format-Git

Dec 19 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

Updated and scheduled the last 170 repositories.
Now, remains those to be checked for errors.

Dec 19 2017, 6:19 PM · Origin-Gitorious, Format-Git

Dec 15 2017

ardumont closed T816: Gitorious import: loose object parsing error with corrupted file as empty one, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Dec 15 2017, 7:43 PM · Origin-Gitorious, Format-Git

Nov 7 2017

ardumont closed T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?), a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git
ardumont closed T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git
ardumont closed T814: Gitorious import: unexisting object retrieval makes the loading fail, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Nov 7 2017, 6:22 PM · Origin-Gitorious, Format-Git

Oct 27 2017

ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T822: Gitorious import: ObjectFormatException raised when badly formatted object (around date?).
Oct 27 2017, 11:00 AM · Origin-Gitorious, Format-Git

Oct 26 2017

ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T819: Gitorious import: ObjectFormatException raised when badly formatted tag object exists in the repository.
Oct 26 2017, 4:07 PM · Origin-Gitorious, Format-Git
ardumont closed T815: Gitorious import: Release time conversion issue when no release date is provided, a subtask of T674: Gitorious import: Examine ingestion logs for errors and list them if any, as Resolved.
Oct 26 2017, 1:11 PM · Origin-Gitorious, Format-Git
ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T816: Gitorious import: loose object parsing error with corrupted file as empty one.
Oct 26 2017, 11:20 AM · Origin-Gitorious, Format-Git
ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T814: Gitorious import: unexisting object retrieval makes the loading fail.
Oct 26 2017, 11:11 AM · Origin-Gitorious, Format-Git
ardumont added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T815: Gitorious import: Release time conversion issue when no release date is provided.
Oct 26 2017, 11:11 AM · Origin-Gitorious, Format-Git

Oct 3 2017

ardumont renamed T312: Gitorious import: ingest repositories from ingest Gitorious repositories to Gitorious import: ingest repositories.
Oct 3 2017, 10:14 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont added a subtask for T312: Gitorious import: ingest repositories: T674: Gitorious import: Examine ingestion logs for errors and list them if any.
Oct 3 2017, 10:14 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont added a parent task for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T312: Gitorious import: ingest repositories.
Oct 3 2017, 10:14 AM · Origin-Gitorious, Format-Git
ardumont removed a parent task for T312: Gitorious import: ingest repositories: T674: Gitorious import: Examine ingestion logs for errors and list them if any.
Oct 3 2017, 10:13 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont removed a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T312: Gitorious import: ingest repositories.
Oct 3 2017, 10:13 AM · Origin-Gitorious, Format-Git

Jul 28 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

For information, the last injection has been done. The remaining errors:

Jul 28 2017, 11:11 AM · Origin-Gitorious, Format-Git
ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

(but we should have a list of those repos, for posterity).

Jul 28 2017, 10:29 AM · Origin-Gitorious, Format-Git

Jul 27 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

These should be rescheduled and driven to successful completion.

Jul 27 2017, 3:03 PM · Origin-Gitorious, Format-Git
zack added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.
  • we send something that was not a git repository.
  • integrity error (which is expected for now)
Jul 27 2017, 12:02 PM · Origin-Gitorious, Format-Git

Jul 26 2017

ardumont added a comment to T674: Gitorious import: Examine ingestion logs for errors and list them if any.

After much learning on how to read and extract logs from our kibana instance, here is the error repartition.

Jul 26 2017, 12:51 PM · Origin-Gitorious, Format-Git

Jun 6 2017

ardumont added a comment to T312: Gitorious import: ingest repositories.

As of now, ingestion, after multiple (re)schedulings, has been done.

Jun 6 2017, 1:34 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Apr 26 2017

ardumont added a comment to T312: Gitorious import: ingest repositories.

Update on this.

Apr 26 2017, 10:41 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Apr 7 2017

zack added a project to T312: Gitorious import: ingest repositories: Archive content.
Apr 7 2017, 11:00 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Feb 15 2017

ardumont added a comment to T312: Gitorious import: ingest repositories.

Visit dates have been fixed for the origins already injected.

Feb 15 2017, 7:34 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack added a parent task for T312: Gitorious import: ingest repositories: T674: Gitorious import: Examine ingestion logs for errors and list them if any.
Feb 15 2017, 4:13 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack added a subtask for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T312: Gitorious import: ingest repositories.
Feb 15 2017, 4:13 PM · Origin-Gitorious, Format-Git
zack removed a subtask for T312: Gitorious import: ingest repositories: T674: Gitorious import: Examine ingestion logs for errors and list them if any.
Feb 15 2017, 4:13 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack removed a parent task for T674: Gitorious import: Examine ingestion logs for errors and list them if any: T312: Gitorious import: ingest repositories.
Feb 15 2017, 4:13 PM · Origin-Gitorious, Format-Git

Feb 12 2017

zack renamed T312: Gitorious import: ingest repositories from ingest gitorious repositories to ingest Gitorious repositories.
Feb 12 2017, 6:37 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack moved T312: Gitorious import: ingest repositories from Restricted Project Column to Restricted Project Column on the Restricted Project board.
Feb 12 2017, 6:37 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack added a project to T312: Gitorious import: ingest repositories: Restricted Project.
Feb 12 2017, 6:13 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Feb 11 2017

ardumont added a comment to T312: Gitorious import: ingest repositories.

Command to trigger the messages (from worker01):

cat /srv/storage/space/mirrors/gitorious.org/full_mapping.txt | SWH_WORKER_INSTANCE=swh_loader_git_disk ./load_gitorious.py --root-repositories /srv/storage/space/mirrors/gitorious.org/mnt/repositories

(The script defaults to use the right queue 'swh_loader_git_express' and the right origin-date 'Wed, 30 Mar 2016 09:40:04 +0200')

Feb 11 2017, 2:11 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Feb 10 2017

ardumont changed the status of T312: Gitorious import: ingest repositories from Open to Work in Progress.
Feb 10 2017, 8:15 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont added a comment to T312: Gitorious import: ingest repositories.

start-date: Fri Feb 10 16:40:00 UTC 2017

Feb 10 2017, 5:48 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
ardumont renamed T674: Gitorious import: Examine ingestion logs for errors and list them if any from Gitorious import: Reference errors after ingestion to Gitorious import: Examine ingestion logs for errors and list them if any.
Feb 10 2017, 3:18 PM · Origin-Gitorious, Format-Git
olasd renamed T674: Gitorious import: Examine ingestion logs for errors and list them if any from Reference errors after ingestion to Gitorious import: Reference errors after ingestion.
Feb 10 2017, 2:39 PM · Origin-Gitorious, Format-Git
ardumont created T674: Gitorious import: Examine ingestion logs for errors and list them if any.
Feb 10 2017, 12:42 PM · Origin-Gitorious, Format-Git
olasd added a comment to T312: Gitorious import: ingest repositories.

The full mapping of gitorious repositories URLs to on-disk location is at uffizi:/srv/storage/space/mirrors/gitorious.org/full_mapping.txt

Feb 10 2017, 12:37 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

May 25 2016

rdicosmo added a parent task for T312: Gitorious import: ingest repositories: Unknown Object (Maniphest Task).
May 25 2016, 4:05 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

May 13 2016

olasd changed the visibility for Format-Git.
May 13 2016, 5:22 PM
olasd changed the visibility for T360: create gid 5000 and add swhworker to it (to ingest gitorious repos).
May 13 2016, 5:09 PM · System administration, Origin-Gitorious, Format-Git
olasd changed the visibility for T343: retrieve gitorious repositories from the gitorious valhalla.
May 13 2016, 5:09 PM · Origin-Gitorious, Format-Git
olasd changed the visibility for T312: Gitorious import: ingest repositories.
May 13 2016, 5:08 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

May 12 2016

olasd added a comment to T312: Gitorious import: ingest repositories.

I'm now running a git fsck on all the repositories. Output and results in worker01:/tmp/fsck.

May 12 2016, 6:42 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
olasd added a comment to T312: Gitorious import: ingest repositories.

I've collapsed the two mappings into a single file: /srv/softwareheritage/mirrors/gitorious.org/full_mapping.txt

May 12 2016, 5:16 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack added a comment to T312: Gitorious import: ingest repositories.

Here are all the information I have about the on-disk gitorious layout (credit: astrid):

May 12 2016, 4:00 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
olasd closed T360: create gid 5000 and add swhworker to it (to ingest gitorious repos) as Resolved.

Deployed the uid+gid changes and added the filesystem to uffizi:/etc/exports

May 12 2016, 2:44 PM · System administration, Origin-Gitorious, Format-Git
olasd closed T360: create gid 5000 and add swhworker to it (to ingest gitorious repos), a subtask of T312: Gitorious import: ingest repositories, as Resolved.
May 12 2016, 2:44 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
olasd added a comment to T360: create gid 5000 and add swhworker to it (to ingest gitorious repos).

To prepare for this, I moved temp-drydock to uid=10000 on worker01. If CI is broken it's my fault.

May 12 2016, 2:28 PM · System administration, Origin-Gitorious, Format-Git

Apr 1 2016

zack added a comment to T360: create gid 5000 and add swhworker to it (to ingest gitorious repos).

for reference, see the content of the gitorious disk image uffizi:/srv/softwareheritage/mirrors/gitorious.org/gitorious.img

Apr 1 2016, 12:03 PM · System administration, Origin-Gitorious, Format-Git
zack created T360: create gid 5000 and add swhworker to it (to ingest gitorious repos).
Apr 1 2016, 12:02 PM · System administration, Origin-Gitorious, Format-Git

Mar 29 2016

zack closed T343: retrieve gitorious repositories from the gitorious valhalla as Resolved.

This is now done. I'm running an fsck on the retrieved file system image just in case.

Mar 29 2016, 12:05 PM · Origin-Gitorious, Format-Git
zack closed T343: retrieve gitorious repositories from the gitorious valhalla, a subtask of T312: Gitorious import: ingest repositories, as Resolved.
Mar 29 2016, 12:05 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Mar 10 2016

zack removed a project from T343: retrieve gitorious repositories from the gitorious valhalla: Developers.
Mar 10 2016, 5:54 PM · Origin-Gitorious, Format-Git
zack removed projects from T312: Gitorious import: ingest repositories: Developers, Staff.
Mar 10 2016, 5:51 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack removed a project from T343: retrieve gitorious repositories from the gitorious valhalla: Staff.
Mar 10 2016, 5:49 PM · Origin-Gitorious, Format-Git

Mar 9 2016

zack changed the status of T343: retrieve gitorious repositories from the gitorious valhalla, a subtask of T312: Gitorious import: ingest repositories, from Open to Work in Progress.
Mar 9 2016, 10:27 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack changed the status of T343: retrieve gitorious repositories from the gitorious valhalla from Open to Work in Progress.

The transfer is now in progress on uffizi:/srv/softwareheritage/mirros/gitorious.org/, within a screen session of my user with title "gitorious-transfer".

Mar 9 2016, 10:27 AM · Origin-Gitorious, Format-Git
zack claimed T343: retrieve gitorious repositories from the gitorious valhalla.
Mar 9 2016, 9:53 AM · Origin-Gitorious, Format-Git

Mar 5 2016

zack lowered the priority of T312: Gitorious import: ingest repositories from High to Normal.
Mar 5 2016, 10:41 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack created T343: retrieve gitorious repositories from the gitorious valhalla.
Mar 5 2016, 10:41 AM · Origin-Gitorious, Format-Git
zack raised the priority of T312: Gitorious import: ingest repositories from Normal to High.
Mar 5 2016, 10:38 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack added a comment to T312: Gitorious import: ingest repositories.

We are now all set to start (after having automated it properly…) the transfer of Gitorious stuff to SWH.

Mar 5 2016, 10:36 AM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Feb 27 2016

zack added a comment to T312: Gitorious import: ingest repositories.

Here is the complete list of URL that can be used to "git clone" (via HTTPS) all the repositories available from the Gitorious valhalla:

.

Feb 27 2016, 3:30 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Feb 22 2016

olasd set the image for Format-Git to Unknown Object (File).
Feb 22 2016, 8:19 PM
zack renamed T312: Gitorious import: ingest repositories from ingest archived gitorious repositories to ingest gitorious repositories.
Feb 22 2016, 12:37 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git
zack created T312: Gitorious import: ingest repositories.
Feb 22 2016, 12:28 PM · Archive coverage, Restricted Project, Origin-Gitorious, Format-Git

Sep 10 2015

olasd removed a member for Format-Git: olasd.
Sep 10 2015, 11:00 AM
olasd renamed Format-Git from Git to Format-Git.
Sep 10 2015, 10:57 AM
olasd created Format-Git.
Sep 10 2015, 10:53 AM