Page MenuHomeSoftware Heritage
Feed Advanced Search

Dec 19 2017

ardumont created T908: mercurial loader: Define scheduler task(s).
Dec 19 2017, 2:05 PM · Mercurial loader
ardumont created T907: mercurial loader: Align mercurial loader with other loaders.
Dec 19 2017, 2:04 PM · Mercurial loader
ardumont created T906: mercurial loader: Debian package.
Dec 19 2017, 1:59 PM · System administration, Mercurial loader
ardumont claimed T329: hg / mercurial loader.
Dec 19 2017, 1:58 PM · Mercurial loader

Dec 15 2017

olasd closed T567: adapt SQL storage for repository snapshot objects, a subtask of T565: embrace repository snapshot object in the data model (meta task), as Resolved.
Dec 15 2017, 3:37 PM · General

Dec 13 2017

olasd closed T566: specify serialization format for repository snapshot objects, a subtask of T565: embrace repository snapshot object in the data model (meta task), as Resolved.
Dec 13 2017, 11:30 AM · General

Dec 12 2017

seirl added a parent task for T565: embrace repository snapshot object in the data model (meta task): T887: Vault: "snapshot" cooker.
Dec 12 2017, 3:45 PM · General
seirl closed T530: Software Heritage Vault, a subtask of T508: prototype: git archive from SWH, as Resolved.
Dec 12 2017, 3:44 PM · Vault, General
seirl closed T508: prototype: git archive from SWH, a subtask of T67: prototype: git clone from SWH, as Resolved.
Dec 12 2017, 3:43 PM · Vault, General
seirl closed T508: prototype: git archive from SWH as Resolved.

This is already well beyond a prototype, so I'm closing this task.

Dec 12 2017, 3:43 PM · Vault, General
seirl closed T67: prototype: git clone from SWH as Resolved.

As the "prototype" part of the vault is definitely over, I'm closing this task.

Dec 12 2017, 3:42 PM · Vault, General

Nov 10 2017

olasd added a comment to T335: specify the URI scheme swh:... to point to software heritage objects.

We can always allow people to truncate the identifier to some arbitrary (shorter) length. The canonical URI would be the full identifier, but our URI resolver can recognize shortened identifiers and point to a disambiguation page with all the objects whose identifier starts with the given string.

Nov 10 2017, 7:49 PM · General

Nov 6 2017

rdicosmo added a comment to T335: specify the URI scheme swh:... to point to software heritage objects.

I agree with all the suggestions: the full id should definitely contain all
this information.
Nevertheless, the sheer length of the result *may* turn out to be a blocker
for adoption as a reference to software in the academic publishing
framework. We can propose this, and see if we need to also provide a
shorter backup if really there is a strong negative feedback.

Nov 6 2017, 3:19 PM · General
zack added a comment to T335: specify the URI scheme swh:... to point to software heritage objects.

I'm not opposed to having explicit hash scheme names in the IDs—it is a good idea, only to be weighed against the cost in terms of length.
But we should also have schema version numbers, in case more radical changes will be needed in the future, e.g., renaming the object types in the graph.
If we retain both suggestions, that would give:

  • swh:1:revision:sha1_git:<git sha1 of a revision>
  • swh:1:content:blake2s256:<blake2s256 of a content>
Nov 6 2017, 3:04 PM · General
olasd added a comment to T335: specify the URI scheme swh:... to point to software heritage objects.

I've been thinking about this in relation to T836.

Nov 6 2017, 2:58 PM · General

Nov 5 2017

olasd added a subtask for T565: embrace repository snapshot object in the data model (meta task): T830: Remove tables occurrence and occurrence_history.
Nov 5 2017, 9:29 PM · General

Oct 17 2017

zack added a subtask for T807: dogfooding: ingest the Software Heritage forge into the archive (via the canonical URLs): T329: hg / mercurial loader.
Oct 17 2017, 3:44 PM · General
zack added a parent task for T329: hg / mercurial loader: T807: dogfooding: ingest the Software Heritage forge into the archive (via the canonical URLs).
Oct 17 2017, 3:44 PM · Mercurial loader
zack added a subtask for T807: dogfooding: ingest the Software Heritage forge into the archive (via the canonical URLs): T328: svn / subversion loader.
Oct 17 2017, 3:44 PM · General
zack created T807: dogfooding: ingest the Software Heritage forge into the archive (via the canonical URLs).
Oct 17 2017, 3:43 PM · General

Oct 16 2017

fiendish added a revision to T329: hg / mercurial loader: D256: mercurial bundle20 parser/loader.
Oct 16 2017, 10:06 PM · Mercurial loader

Oct 6 2017

zack closed T5: pg_hash: postgres datatype for checksums as Wontfix.
Oct 6 2017, 2:55 PM · General

Sep 15 2017

zack added a project to T508: prototype: git archive from SWH: Vault.
Sep 15 2017, 10:12 AM · Vault, General
zack added a project to T67: prototype: git clone from SWH: Vault.
Sep 15 2017, 10:03 AM · Vault, General
zack assigned T647: support software deposit via SWORD protocol (meta task) to ardumont.
Sep 15 2017, 9:58 AM · SWORD deposit, General
zack closed T598: Store content -> revision cache in azure table storage, a subtask of T547: Azure prototype: Content provenance information API, as Wontfix.
Sep 15 2017, 9:58 AM · General
zack closed T547: Azure prototype: Content provenance information API as Wontfix.

we're taking a different route for this now, based on @grouss WIP

Sep 15 2017, 9:57 AM · General

Sep 5 2017

zack added a project to T647: support software deposit via SWORD protocol (meta task): SWORD deposit.
Sep 5 2017, 10:28 AM · SWORD deposit, General
zack removed a project from T647: support software deposit via SWORD protocol (meta task): Deposit.
Sep 5 2017, 10:28 AM · SWORD deposit, General

Sep 4 2017

ardumont added a project to T647: support software deposit via SWORD protocol (meta task): Deposit.
Sep 4 2017, 1:16 PM · SWORD deposit, General
ardumont added a subtask for T647: support software deposit via SWORD protocol (meta task): T718: SWORD deposit: backend server.
Sep 4 2017, 11:47 AM · SWORD deposit, General

Aug 4 2017

seirl added a parent task for T565: embrace repository snapshot object in the data model (meta task): T531: Vault cookers.
Aug 4 2017, 4:27 PM · General

Jun 12 2017

olasd closed T536: Enrich task return status to register partial completions, a subtask of T533: Allow loaders to register partial state (meta task), as Resolved.
Jun 12 2017, 3:32 PM · General

May 9 2017

fiendish added a comment to T329: hg / mercurial loader.

I don't know if I should finish cleaning up slow_loader for code review, since the hglib interface is so slow as to be next to useless.

May 9 2017, 5:57 PM · Mercurial loader

May 7 2017

zack closed T57: git commit IRC notifications on #swh-devel as Resolved.

done (by @olasd last week)

May 7 2017, 3:50 PM · General

Apr 24 2017

olasd changed the status of T691: complete object storage mirror on Azure (meta task) from Open to Work in Progress.

We now have a "full" content mirror on azure (the data for each 16th bucket is up to date as of the time the snapshot was taken).

Apr 24 2017, 3:06 PM · General

Apr 7 2017

zack added a project to T367: ingest Google Code repositories: Archive content.
Apr 7 2017, 10:59 AM · Archive coverage, Restricted Project

Mar 14 2017

olasd closed T189: centralized syslog as Resolved.

systemd-journal-remote has been configured to receive logs on pergamon; systemd-journal-upload has been configured on all machines to push the journal messages to pergamon.

Mar 14 2017, 3:57 PM · General

Mar 6 2017

fiendish closed T591: lister for bitbucket repositories, a subtask of T561: ingest bitbucket (meta task), as Resolved.
Mar 6 2017, 5:45 PM · Archive coverage, Origin-Bitbucket

Mar 1 2017

zack created T691: complete object storage mirror on Azure (meta task).
Mar 1 2017, 9:32 AM · General

Feb 21 2017

ardumont placed T367: ingest Google Code repositories up for grabs.
Feb 21 2017, 11:46 AM · Archive coverage, Restricted Project

Feb 15 2017

ardumont changed the status of T673: ingest Google Code Git repositories, a subtask of T367: ingest Google Code repositories, from Open to Work in Progress.
Feb 15 2017, 7:53 PM · Archive coverage, Restricted Project
zack added a parent task for T329: hg / mercurial loader: T682: Ingest Google Code Mercurial repositories.
Feb 15 2017, 4:15 PM · Mercurial loader
zack added a subtask for T367: ingest Google Code repositories: T617: ingest Google Code Subversion repositories.
Feb 15 2017, 4:08 PM · Archive coverage, Restricted Project
zack removed a subtask for T367: ingest Google Code repositories: T328: svn / subversion loader.
Feb 15 2017, 4:06 PM · Archive coverage, Restricted Project
zack added a subtask for T367: ingest Google Code repositories: T682: Ingest Google Code Mercurial repositories.
Feb 15 2017, 4:05 PM · Archive coverage, Restricted Project
zack removed a subtask for T329: hg / mercurial loader: T682: Ingest Google Code Mercurial repositories.
Feb 15 2017, 4:03 PM · Mercurial loader
ardumont added a subtask for T329: hg / mercurial loader: T682: Ingest Google Code Mercurial repositories.
Feb 15 2017, 1:59 PM · Mercurial loader

Feb 13 2017

ardumont claimed T367: ingest Google Code repositories.
Feb 13 2017, 3:32 PM · Archive coverage, Restricted Project

Feb 12 2017

zack moved T367: ingest Google Code repositories from Restricted Project Column to Restricted Project Column on the Restricted Project board.
Feb 12 2017, 6:37 PM · Archive coverage, Restricted Project
zack renamed T547: Azure prototype: Content provenance information API from Prototype: Content provenance information API to Azure prototype: Content provenance information API.
Feb 12 2017, 6:17 PM · General
zack renamed T367: ingest Google Code repositories from ingest code.google.com repositories to ingest Google Code repositories.
Feb 12 2017, 6:15 PM · Archive coverage, Restricted Project

Feb 10 2017

zack added a subtask for T367: ingest Google Code repositories: T673: ingest Google Code Git repositories.
Feb 10 2017, 12:28 PM · Archive coverage, Restricted Project
zack added a project to T533: Allow loaders to register partial state (meta task): General.
Feb 10 2017, 8:44 AM · General

Feb 9 2017

zack closed T573: Lister engineering as Resolved.

this has been done by @olasd a while ago, as part of the work described at https://www.softwareheritage.org/2016/11/09/listing-47-million-repositories-refactoring-our-github-lister/

Feb 9 2017, 7:46 PM · General
zack added a subtask for T508: prototype: git archive from SWH: T530: Software Heritage Vault.
Feb 9 2017, 7:43 PM · Vault, General
zack closed T240: content archiver, a subtask of T239: preserve at least 2 copies of each content object, as Resolved.
Feb 9 2017, 7:35 PM · General
zack closed T383: stay up to date w.r.t. new Debian uploads as Invalid.

this is a duplicate of T120

Feb 9 2017, 1:09 PM · Restricted Project, General, Origin-Debian

Feb 8 2017

zack assigned T561: ingest bitbucket (meta task) to fiendish.
Feb 8 2017, 11:58 AM · Archive coverage, Origin-Bitbucket
zack assigned T329: hg / mercurial loader to fiendish.

We now have a (empty) Diffusion repo for this: https://forge.softwareheritage.org/source/swh-loader-mercurial/

Feb 8 2017, 11:57 AM · Mercurial loader

Feb 2 2017

zack closed T614: Open up the Software Heritage API to the world as Resolved.

*drum roll*
*tsssss*

Feb 2 2017, 9:56 PM · Restricted Project, General, System administration, Web app
olasd closed T616: rate limiting for anonymous access to the API, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Feb 2 2017, 10:43 AM · Restricted Project, General, System administration, Web app

Feb 1 2017

zack closed T655: tweak HTTP auth for first API public release, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Feb 1 2017, 5:47 PM · Restricted Project, General, System administration, Web app
olasd changed the status of T655: tweak HTTP auth for first API public release, a subtask of T614: Open up the Software Heritage API to the world, from Open to Work in Progress.
Feb 1 2017, 4:26 PM · Restricted Project, General, System administration, Web app
zack closed T621: content review for the API doc page, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Feb 1 2017, 12:29 PM · Restricted Project, General, System administration, Web app

Jan 31 2017

olasd closed T653: Split the scheduler (rabbitmq) out of the moma virtual machine, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Jan 31 2017, 1:43 PM · Restricted Project, General, System administration, Web app
olasd closed T615: set up pg_logical-based master/slave replication for the postgres db, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Jan 31 2017, 1:23 PM · Restricted Project, General, System administration, Web app

Jan 30 2017

zack changed the status of T621: content review for the API doc page, a subtask of T614: Open up the Software Heritage API to the world, from Open to Work in Progress.
Jan 30 2017, 4:54 PM · Restricted Project, General, System administration, Web app

Jan 26 2017

zack created T647: support software deposit via SWORD protocol (meta task).
Jan 26 2017, 4:41 PM · SWORD deposit, General

Jan 25 2017

ardumont closed T620: face lift for the API doc page, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Jan 25 2017, 1:41 PM · Restricted Project, General, System administration, Web app

Jan 24 2017

ardumont closed Unknown Object (Maniphest Task), a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Jan 24 2017, 11:20 AM · Restricted Project, General, System administration, Web app

Jan 20 2017

olasd changed the status of T616: rate limiting for anonymous access to the API, a subtask of T614: Open up the Software Heritage API to the world, from Open to Work in Progress.
Jan 20 2017, 7:36 PM · Restricted Project, General, System administration, Web app
olasd closed T618: debian packaging of flask-limiter, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Jan 20 2017, 12:08 PM · Restricted Project, General, System administration, Web app

Jan 11 2017

olasd changed the status of T618: debian packaging of flask-limiter, a subtask of T614: Open up the Software Heritage API to the world, from Open to Work in Progress.
Jan 11 2017, 5:53 PM · Restricted Project, General, System administration, Web app
zack moved T614: Open up the Software Heritage API to the world from Restricted Project Column to Restricted Project Column on the Restricted Project board.
Jan 11 2017, 4:39 PM · Restricted Project, General, System administration, Web app
zack added a project to T614: Open up the Software Heritage API to the world: Restricted Project.
Jan 11 2017, 4:38 PM · Restricted Project, General, System administration, Web app

Jan 9 2017

olasd changed the status of T615: set up pg_logical-based master/slave replication for the postgres db, a subtask of T614: Open up the Software Heritage API to the world, from Open to Work in Progress.
Jan 9 2017, 3:05 PM · Restricted Project, General, System administration, Web app

Jan 4 2017

zack updated the task description for T614: Open up the Software Heritage API to the world.
Jan 4 2017, 8:30 PM · Restricted Project, General, System administration, Web app
zack updated the task description for T614: Open up the Software Heritage API to the world.
Jan 4 2017, 8:25 PM · Restricted Project, General, System administration, Web app
zack added a project to T614: Open up the Software Heritage API to the world: General.
Jan 4 2017, 8:14 PM · Restricted Project, General, System administration, Web app

Dec 6 2016

ardumont closed T610: Update indexers' information about the name and version of the tools used for their computations as Resolved.
Dec 6 2016, 12:27 PM · Indexer, General
ardumont closed T610: Update indexers' information about the name and version of the tools used for their computations, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Dec 6 2016, 12:27 PM · Indexer, General
ardumont added a comment to T610: Update indexers' information about the name and version of the tools used for their computations.

Remains to update the azure workers with the latest indexer.
I'm on it.

Dec 6 2016, 11:43 AM · Indexer, General

Dec 2 2016

ardumont created T610: Update indexers' information about the name and version of the tools used for their computations.
Dec 2 2016, 1:29 PM · Indexer, General

Nov 23 2016

olasd closed T585: Re-engineer github lister, a subtask of T573: Lister engineering, as Resolved.
Nov 23 2016, 4:09 PM · General
olasd closed T382: stay up to date w.r.t. new GitHub repositories as Resolved.

Done as of T585.

Nov 23 2016, 4:06 PM · Restricted Project, General, Origin-GitHub

Nov 22 2016

ardumont closed T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task) as Resolved.
Nov 22 2016, 10:38 AM · Indexer, General

Nov 18 2016

ardumont closed T596: Add license indexer as Resolved.
Nov 18 2016, 3:53 PM · Indexer, General
ardumont closed T596: Add license indexer, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Nov 18 2016, 3:53 PM · Indexer, General

Nov 15 2016

ardumont added a comment to T596: Add license indexer.

Sum up and status:

  • Use of fossology's nomossa (nomos standalone version)
  • DB schema updated (swh-storage)
  • Api endpoints opened to add/read nomos's output (swh-storage)
  • New Indexer added (swh-indexer)
  • Puppet manifest for that indexer (swh-profile, swh-site)
  • Deploy DB upgrade (uffizi)
  • Deploy swh-storage upgrade (uffizi)
  • Deploy indexer on azure nodes
  • Feed existing contents stored in azure to keep up with other indexers
  • Update missing licenses in storage
  • Feed failed contents (the ones that were tagged, license unknown) back in
Nov 15 2016, 4:37 PM · Indexer, General
ardumont changed the status of T596: Add license indexer, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), from Open to Work in Progress.
Nov 15 2016, 4:31 PM · Indexer, General
ardumont changed the status of T596: Add license indexer from Open to Work in Progress.
Nov 15 2016, 4:31 PM · Indexer, General

Nov 5 2016

ardumont closed T597: Check reader git did its job, a subtask of T595: Read sha1s from git repositories and send them for archive in azure, as Resolved.
Nov 5 2016, 6:46 PM · Indexer, General
ardumont closed T597: Check reader git did its job as Resolved.
Nov 5 2016, 6:46 PM · Indexer, General
ardumont added a comment to T597: Check reader git did its job.

Updated previous comment. It's ok.

Nov 5 2016, 6:46 PM · Indexer, General

Nov 4 2016

ardumont updated the task description for T597: Check reader git did its job.
Nov 4 2016, 3:33 PM · Indexer, General
ardumont renamed T597: Check reader git did its job from Check that reader git did its job to Check reader git did its job.
Nov 4 2016, 3:23 PM · Indexer, General
ardumont added a comment to T597: Check reader git did its job.

Current analysis' details in P120

Nov 4 2016, 3:14 PM · Indexer, General
ardumont changed the status of T597: Check reader git did its job from Open to Work in Progress.
Nov 4 2016, 3:14 PM · Indexer, General
ardumont changed the status of T597: Check reader git did its job, a subtask of T595: Read sha1s from git repositories and send them for archive in azure, from Open to Work in Progress.
Nov 4 2016, 3:14 PM · Indexer, General