ingest Google Code Git repositories
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	ardumont
	Feb 10 2017, 12:20 PM

Description

We have retrieved the git repositories and not yet ingested them.
This task is about the actual ingestion using our loader-git.

Note:

Like every other mirror/backup, it's stored at /srv/storage/space/mirrors/, under a dedicated root directory 'code.google.com' (in uffizi).
/srv/storage/space/mirrors/code.google.com/sources/INDEX.filesystem to list all googlecode's repositories on disk.

Requirements:

filtering the git repositories (we only have the INDEX.filesystem which lists of all googlecode repositories for now, be it of types git, svn or hg). There is a project.json in the same folder as the archive which contains the mention 'repoType' with possible value as either 'git', 'svn', or 'hg'.

As we did for the googlecode's svn repositories, we need to reconstruct their url: https://<project-name>.googlecode.com/

all git repositories are archive files (mostly zip). So, we either need to uncompress every archive first or as with the googlecode svn loader, let the worker uncompress first the archive in a temporary directory and then load the git repository.

at last, generate a INDEX-git-archive (same structure as the one we used for gitorious) with format: <origin_url> <path-to-git-repository-tree-or-archive>

Related Objects
Search...

Status	Assigned	Task
		Unknown Object (Maniphest Task)
Migrated	gitlab-migration	T367 ingest Google Code repositories
Migrated	gitlab-migration	T675 Google Code Git import: Examine ingestion logs for errors and list them if any
Migrated	gitlab-migration	T673 ingest Google Code Git repositories

Event Timeline

ardumont created this task.Feb 10 2017, 12:20 PM

ardumont claimed this task.Feb 10 2017, 12:26 PM

ardumont updated the task description. (Show Details)

zack added a parent task: T367: ingest Google Code repositories.Feb 10 2017, 12:28 PM

ardumont created subtask T675: Google Code Git import: Examine ingestion logs for errors and list them if any.Feb 10 2017, 12:43 PM

ardumont mentioned this in rSPPROFe458f7b405c4: swh::deploy::worker::swh_loader_git_disk: Add disk injection worker.Feb 10 2017, 3:52 PM

ardumont mentioned this in rSPSITE1c4e5337c34e: data/defaults: Add swh-loader-git-disk instance.

ardumont mentioned this in rSPSITE1c7d84449f1f: data/hostname/swh-workers: Deploy swh-loader-git-disk worker.Feb 10 2017, 4:07 PM

ardumont mentioned this in rSPPROFd8d3fc32b656: worker::swh_loader_git_disk: Don't consume from swh_loader_git queue.Feb 10 2017, 4:12 PM

zack renamed this task from inject googlecode's git repositories into swh to ingest Google Code Git repositories.Feb 12 2017, 6:14 PM

zack added a project: Restricted Project.

zack moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Feb 12 2017, 6:37 PM

ardumont updated the task description. (Show Details)Feb 13 2017, 4:36 PM

ardumont mentioned this in rDSNIPd2d5613cc5e4: googlecode: Filter googlecode repositories on their type.Feb 13 2017, 9:09 PM

ardumont mentioned this in rDLDG6063c187497a: loader: Add GitLoaderFromArchive.

ardumont mentioned this in rDLDG2c90051c0e28: base: Open an optional cleanup step in base loader.

ardumont mentioned this in rDLDG673efd0d2cd6: GitLoader task: Update the queue to consume messages from.Feb 14 2017, 6:59 PM

ardumont mentioned this in rDLDG3761ca6e28a4: ArchiveGitLoader task: Update the queue name to consume messages.

ardumont mentioned this in rDLDGf6cb2845ae86: GitLoaderFromArchive: Add docstrings information.

ardumont mentioned this in rSPPROFe3ee8c2a6a2f: swh::deploy::worker::swh_loader_git_archive: Add git-archive worker.

ardumont mentioned this in rSPSITE1eeee68549fe: data/hostname/swh-workers: Add swh-loader-git-archive worker configuration.

ardumont mentioned this in rSPSITEfd6f9122d1aa: data/hostname/worker01: Deploy swh-loader-git-archive.

at last, generate a full_mapping.txt (mirroring the one from gitorious) mentioning <origin_url> <path-to-git-repository-tree-or-archive>.

INDEX-git-archives is the file listing only zip archive file with git repositories.
It also maps the the origin-url to use for the injection.

$ head /srv/storage/space/mirrors/code.googlecode.com/sources/INDEX-git-archives
https://00000books.googlecode.com /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/0/00000books/00000books-source-archive.zip
https://0000som143-osmand.googlecode.com /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/0/0000som143-osmand/0000som143-osmand-source-archive.zip
https://001coldblade-authenticator.googlecode.com /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/0/001coldblade-authenticator/001coldblade-authenticator-source-archive.zip
https://0043113-aa.googlecode.com /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/0/0043113-aa/0043113-aa-source-archive.zip
https://005-iphone-project.googlecode.com /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/0/005-iphone-project/005-iphone-project-source-archive.zip
https://007xsq-sadsad.googlecode.com /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/0/007xsq-sadsad/007xsq-sadsad-source-archive.zip
https://00mohit-mohit.googlecode.com /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/0/00mohit-mohit/00mohit-mohit-source-archive.zip
https://010pepe010-moog.googlecode.com /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/0/010pepe010-moog/010pepe010-moog-source-archive.zip
https://010smithzhang-ddd.googlecode.com /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/0/010smithzhang-ddd/010smithzhang-ddd-source-archive.zip
https://014miharu-hirayama.googlecode.com /srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/0/014miharu-hirayama/014miharu-hirayama-source-archive.zip
...

As in T617, the origin date to use for injection is 'Tue, 3 May 2016 17:16:32 +0200'. We retrieved all googlecode repositories together (git, svn, hg).

ardumont mentioned this in T682: Ingest Google Code Mercurial repositories.Feb 15 2017, 1:58 PM

ardumont added a subtask: T682: Ingest Google Code Mercurial repositories.

ardumont updated the task description. (Show Details)Feb 15 2017, 2:01 PM

zack removed a subtask: T682: Ingest Google Code Mercurial repositories.Feb 15 2017, 4:03 PM

ardumont mentioned this in rSPSITE1654e5613641: data/hostname/swh-workers: Deploy swh-loader-git-archive on swh-workers.Feb 15 2017, 4:04 PM

zack removed a subtask: T675: Google Code Git import: Examine ingestion logs for errors and list them if any.Feb 15 2017, 4:13 PM

zack added a parent task: T675: Google Code Git import: Examine ingestion logs for errors and list them if any.

starting-date: 2017-02-15 14:42:27,724

The command to trigger the messages is (from worker01 but should be limited to it):

$ cat /srv/storage/space/mirrors/code.google.com/sources/INDEX-git-archives | SWH_WORKER_INSTANCE=swh_loader_git_archive ./load_googlecode.py --visit-date 'Tue, 3 May 2016 17:16:32 +0200'
{origin_url: 'https://zvaigzdinas-hih.googlecode.com', archive_path: '/srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/z/zvaigzdinas-hih/zvaigzdinas-hih-source-archive.zip', date: 'Tue, 3 May 2016 17:16:32 +0200'}
{origin_url: 'https://zvalenti-generations.googlecode.com', archive_path: '/srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/z/zvalenti-generations/zvalenti-generations-source-archive.zip', date: 'Tue, 3 May 2016 17:16:
32 +0200'}
{origin_url: 'https://zvarioz.googlecode.com', archive_path: '/srv/storage/space/mirrors/code.google.com/sources/v2/code.google.com/z/zvarioz/zvarioz-source-archive.zip', date: 'Tue, 3 May 2016 17:16:32 +0200'}

where: