I don't know if I should finish cleaning up slow_loader for code review, since the hglib interface is so slow as to be next to useless.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Sep 5 2017
Sep 5 2017
zack added a project to T647: support software deposit via SWORD protocol (meta task): SWORD deposit.
Sep 4 2017
Sep 4 2017
Aug 4 2017
Aug 4 2017
seirl added a parent task for T565: embrace repository snapshot object in the data model (meta task): T531: Vault cookers.
Jun 12 2017
Jun 12 2017
olasd closed T536: Enrich task return status to register partial completions, a subtask of T533: Allow loaders to register partial state (meta task), as Resolved.
May 9 2017
May 9 2017
May 7 2017
May 7 2017
done (by @olasd last week)
Apr 24 2017
Apr 24 2017
olasd changed the status of T691: complete object storage mirror on Azure (meta task) from Open to Work in Progress.
We now have a "full" content mirror on azure (the data for each 16th bucket is up to date as of the time the snapshot was taken).
Apr 7 2017
Apr 7 2017
Mar 14 2017
Mar 14 2017
systemd-journal-remote has been configured to receive logs on pergamon; systemd-journal-upload has been configured on all machines to push the journal messages to pergamon.
Mar 6 2017
Mar 6 2017
fiendish closed T591: lister for bitbucket repositories, a subtask of T561: ingest bitbucket (meta task), as Resolved.
Mar 1 2017
Mar 1 2017
Feb 21 2017
Feb 21 2017
Feb 15 2017
Feb 15 2017
ardumont changed the status of T673: ingest Google Code Git repositories, a subtask of T367: ingest Google Code repositories, from Open to Work in Progress.
zack added a parent task for T329: hg / mercurial loader: T682: Ingest Google Code Mercurial repositories.
zack added a subtask for T367: ingest Google Code repositories: T617: ingest Google Code Subversion repositories.
zack added a subtask for T367: ingest Google Code repositories: T682: Ingest Google Code Mercurial repositories.
zack removed a subtask for T329: hg / mercurial loader: T682: Ingest Google Code Mercurial repositories.
ardumont added a subtask for T329: hg / mercurial loader: T682: Ingest Google Code Mercurial repositories.
Feb 13 2017
Feb 13 2017
Feb 12 2017
Feb 12 2017
zack moved T367: ingest Google Code repositories from Restricted Project Column to Restricted Project Column on the Restricted Project board.
zack renamed T547: Azure prototype: Content provenance information API from Prototype: Content provenance information API to Azure prototype: Content provenance information API.
zack renamed T367: ingest Google Code repositories from ingest code.google.com repositories to ingest Google Code repositories.
Feb 10 2017
Feb 10 2017
zack added a subtask for T367: ingest Google Code repositories: T673: ingest Google Code Git repositories.
Feb 9 2017
Feb 9 2017
this has been done by @olasd a while ago, as part of the work described at https://www.softwareheritage.org/2016/11/09/listing-47-million-repositories-refactoring-our-github-lister/
zack closed T240: content archiver, a subtask of T239: preserve at least 2 copies of each content object, as Resolved.
this is a duplicate of T120
Feb 8 2017
Feb 8 2017
We now have a (empty) Diffusion repo for this: https://forge.softwareheritage.org/source/swh-loader-mercurial/
Feb 2 2017
Feb 2 2017
*drum roll*
*tsssss*
olasd closed T616: rate limiting for anonymous access to the API, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Feb 1 2017
Feb 1 2017
zack closed T655: tweak HTTP auth for first API public release, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
olasd changed the status of T655: tweak HTTP auth for first API public release, a subtask of T614: Open up the Software Heritage API to the world, from Open to Work in Progress.
zack closed T621: content review for the API doc page, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Jan 31 2017
Jan 31 2017
olasd closed T653: Split the scheduler (rabbitmq) out of the moma virtual machine, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
olasd closed T615: set up pg_logical-based master/slave replication for the postgres db, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Jan 30 2017
Jan 30 2017
zack changed the status of T621: content review for the API doc page, a subtask of T614: Open up the Software Heritage API to the world, from Open to Work in Progress.
Jan 26 2017
Jan 26 2017
Jan 25 2017
Jan 25 2017
ardumont closed T620: face lift for the API doc page, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Jan 24 2017
Jan 24 2017
ardumont closed Unknown Object (Maniphest Task), a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Jan 20 2017
Jan 20 2017
olasd changed the status of T616: rate limiting for anonymous access to the API, a subtask of T614: Open up the Software Heritage API to the world, from Open to Work in Progress.
olasd closed T618: debian packaging of flask-limiter, a subtask of T614: Open up the Software Heritage API to the world, as Resolved.
Jan 11 2017
Jan 11 2017
olasd changed the status of T618: debian packaging of flask-limiter, a subtask of T614: Open up the Software Heritage API to the world, from Open to Work in Progress.
zack moved T614: Open up the Software Heritage API to the world from Restricted Project Column to Restricted Project Column on the Restricted Project board.
Jan 9 2017
Jan 9 2017
olasd changed the status of T615: set up pg_logical-based master/slave replication for the postgres db, a subtask of T614: Open up the Software Heritage API to the world, from Open to Work in Progress.
Jan 4 2017
Jan 4 2017
Dec 6 2016
Dec 6 2016
ardumont added a comment to T610: Update indexers' information about the name and version of the tools used for their computations.
Remains to update the azure workers with the latest indexer.
I'm on it.
Dec 2 2016
Dec 2 2016
Nov 23 2016
Nov 23 2016
Done as of T585.
Nov 22 2016
Nov 22 2016
Nov 18 2016
Nov 18 2016
ardumont closed T596: Add license indexer, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Nov 15 2016
Nov 15 2016
Sum up and status:
- Use of fossology's nomossa (nomos standalone version)
- DB schema updated (swh-storage)
- Api endpoints opened to add/read nomos's output (swh-storage)
- New Indexer added (swh-indexer)
- Puppet manifest for that indexer (swh-profile, swh-site)
- Deploy DB upgrade (uffizi)
- Deploy swh-storage upgrade (uffizi)
- Deploy indexer on azure nodes
- Feed existing contents stored in azure to keep up with other indexers
- Update missing licenses in storage
- Feed failed contents (the ones that were tagged, license unknown) back in
ardumont changed the status of T596: Add license indexer, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), from Open to Work in Progress.
Nov 5 2016
Nov 5 2016
ardumont closed T597: Check reader git did its job, a subtask of T595: Read sha1s from git repositories and send them for archive in azure, as Resolved.
Updated previous comment. It's ok.
Nov 4 2016
Nov 4 2016
ardumont renamed T597: Check reader git did its job from Check that reader git did its job to Check reader git did its job.
Current analysis' details in P120
ardumont changed the status of T597: Check reader git did its job, a subtask of T595: Read sha1s from git repositories and send them for archive in azure, from Open to Work in Progress.
ardumont closed T595: Read sha1s from git repositories and send them for archive in azure as Resolved.
ardumont closed T594: Properly package universal-ctags, a subtask of T589: Add ctags content indexer, as Resolved.
Not packaged in debian yet but for us.
That's a start :D
Oct 27 2016
Oct 27 2016
zack renamed T589: Add ctags content indexer from Add semantic indexer to Add ctags content indexer.
ardumont closed T589: Add ctags content indexer, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
ardumont changed the status of T595: Read sha1s from git repositories and send them for archive in azure from Open to Work in Progress.
ardumont added a comment to T595: Read sha1s from git repositories and send them for archive in azure.
reader git is currently running from worker01.
Oct 25 2016
Oct 25 2016
ardumont edited parent tasks for T595: Read sha1s from git repositories and send them for archive in azure, added: Unknown Object (Maniphest Task); removed: T594: Properly package universal-ctags.
ardumont edited parent tasks for T595: Read sha1s from git repositories and send them for archive in azure, added: Unknown Object (Maniphest Task); removed: T594: Properly package universal-ctags.
ardumont raised the priority of T595: Read sha1s from git repositories and send them for archive in azure from Low to Normal.
ardumont added a comment to T595: Read sha1s from git repositories and send them for archive in azure.
Related:
- 86928fa8ff8c491ce830387080ab47e668204f4f
- 50754523e84b4bff4e1a9c6765302cc3796d70b0
Oct 22 2016
Oct 22 2016
zack changed the status of T585: Re-engineer github lister, a subtask of T573: Lister engineering, from Open to Work in Progress.
zack added a parent task for T329: hg / mercurial loader: T593: ingest bitbucket hg/mercurial repositories.
zack renamed T561: ingest bitbucket (meta task) from ingest bitbucket.com repositories (meta task) to ingest bitbucket repositories (meta task).
zack renamed T561: ingest bitbucket (meta task) from ingest bitbucket.com repositories to ingest bitbucket.com repositories (meta task).
Thanks for reporting this, we should definitely add bitbucket.com to the coverage of our archive.
I'll add subtasks to this meta-task to structure more precisely what we need to do to achieve this goal.
Oct 17 2016
Oct 17 2016
| Tools | Pros | Cons |
| exuberant-ctags - http://ctags.sourceforge.net/ | - debian-packaged | - development stalled (last commit on 2014) |
| - extensible | - 41 languages supported | |
| - difficulties to deal with files' content alone | ||
| - no recent release (last official 5.8 dates back from 2009, the actual debian's is 5.9~svn20110310-11) | ||
| universal-ctags - http://docs.ctags.io/en/latest/ | - 84 supported languages | - manual install (not debian-packaged) |
| - extensible | - not released yet (planned for dec 2016, https://github.com/universal-ctags/ctags/milestone/3) | |
| - active development (last commit 12/10/2016) | - same limit as exuberant-ctags to recognize data from file's content alone | |
| - fork from exuberant-ctags | ||