Page MenuHomeSoftware Heritage
Feed Advanced Search

Advanced Search
Use Results
Hide Query

Nov 15 2016

ardumont added a comment to T596: Add license indexer.

Sum up and status:

  • Use of fossology's nomossa (nomos standalone version)
  • DB schema updated (swh-storage)
  • Api endpoints opened to add/read nomos's output (swh-storage)
  • New Indexer added (swh-indexer)
  • Puppet manifest for that indexer (swh-profile, swh-site)
  • Deploy DB upgrade (uffizi)
  • Deploy swh-storage upgrade (uffizi)
  • Deploy indexer on azure nodes
  • Feed existing contents stored in azure to keep up with other indexers
  • Update missing licenses in storage
  • Feed failed contents (the ones that were tagged, license unknown) back in
Nov 15 2016, 4:37 PM · Indexer, General
ardumont changed the status of T596: Add license indexer, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), from Open to Work in Progress.
Nov 15 2016, 4:31 PM · Indexer, General
ardumont changed the status of T596: Add license indexer from Open to Work in Progress.
Nov 15 2016, 4:31 PM · Indexer, General

Nov 5 2016

ardumont closed T597: Check reader git did its job, a subtask of T595: Read sha1s from git repositories and send them for archive in azure, as Resolved.
Nov 5 2016, 6:46 PM · Indexer, General
ardumont closed T597: Check reader git did its job as Resolved.
Nov 5 2016, 6:46 PM · Indexer, General
ardumont added a comment to T597: Check reader git did its job.

Updated previous comment. It's ok.

Nov 5 2016, 6:46 PM · Indexer, General
ardumont edited P120 check reader git did its job.
Nov 5 2016, 6:44 PM · Indexer

Nov 4 2016

ardumont edited P120 check reader git did its job.
Nov 4 2016, 3:39 PM · Indexer
ardumont updated the task description for T597: Check reader git did its job.
Nov 4 2016, 3:33 PM · Indexer, General
ardumont edited P120 check reader git did its job.
Nov 4 2016, 3:27 PM · Indexer
ardumont edited P120 check reader git did its job.
Nov 4 2016, 3:27 PM · Indexer
ardumont renamed T597: Check reader git did its job from Check that reader git did its job to Check reader git did its job.
Nov 4 2016, 3:23 PM · Indexer, General
ardumont added a comment to T597: Check reader git did its job.

Current analysis' details in P120

Nov 4 2016, 3:14 PM · Indexer, General
ardumont created P120 check reader git did its job.
Nov 4 2016, 3:14 PM · Indexer
ardumont changed the status of T597: Check reader git did its job from Open to Work in Progress.
Nov 4 2016, 3:14 PM · Indexer, General
ardumont changed the status of T597: Check reader git did its job, a subtask of T595: Read sha1s from git repositories and send them for archive in azure, from Open to Work in Progress.
Nov 4 2016, 3:14 PM · Indexer, General
ardumont created T597: Check reader git did its job.
Nov 4 2016, 3:13 PM · Indexer, General
ardumont closed T595: Read sha1s from git repositories and send them for archive in azure as Resolved.
Nov 4 2016, 3:10 PM · Indexer, General
ardumont closed T594: Properly package universal-ctags as Resolved.
Nov 4 2016, 2:22 PM · Indexer, General
ardumont closed T594: Properly package universal-ctags, a subtask of T589: Add ctags content indexer, as Resolved.
Nov 4 2016, 2:22 PM · Indexer, General
ardumont added a comment to T594: Properly package universal-ctags.

Not packaged in debian yet but for us.
That's a start :D

Nov 4 2016, 2:22 PM · Indexer, General
ardumont created T596: Add license indexer.
Nov 4 2016, 2:19 PM · Indexer, General
ardumont added a comment to T594: Properly package universal-ctags.

0406a25ea86ea04fd8ceaac34a81f1dca3e00fa8

Nov 4 2016, 2:17 PM · Indexer, General

Oct 27 2016

zack renamed T589: Add ctags content indexer from Add semantic indexer to Add ctags content indexer.
Oct 27 2016, 5:38 PM · Indexer, General
ardumont closed T589: Add ctags content indexer as Resolved.
Oct 27 2016, 4:02 PM · Indexer, General
ardumont closed T589: Add ctags content indexer, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Oct 27 2016, 4:02 PM · Indexer, General
ardumont changed the status of T595: Read sha1s from git repositories and send them for archive in azure from Open to Work in Progress.
Oct 27 2016, 3:48 PM · Indexer, General
ardumont added a comment to T595: Read sha1s from git repositories and send them for archive in azure.

reader git is currently running from worker01.

Oct 27 2016, 3:48 PM · Indexer, General

Oct 26 2016

ardumont edited P119 Problem when running reader-git in uffizi.
Oct 26 2016, 4:23 PM · Git loader
ardumont edited P119 Problem when running reader-git in uffizi.
Oct 26 2016, 4:18 PM · Git loader
ardumont created P119 Problem when running reader-git in uffizi.
Oct 26 2016, 4:18 PM · Git loader

Oct 25 2016

ardumont edited parent tasks for T595: Read sha1s from git repositories and send them for archive in azure, added: Unknown Object (Maniphest Task); removed: T594: Properly package universal-ctags.
Oct 25 2016, 6:55 PM · Indexer, General
ardumont edited parent tasks for T595: Read sha1s from git repositories and send them for archive in azure, added: Unknown Object (Maniphest Task); removed: T594: Properly package universal-ctags.
Oct 25 2016, 6:55 PM · Indexer, General
ardumont raised the priority of T595: Read sha1s from git repositories and send them for archive in azure from Low to Normal.
Oct 25 2016, 6:52 PM · Indexer, General
ardumont added a comment to T595: Read sha1s from git repositories and send them for archive in azure.

Related:

  • 86928fa8ff8c491ce830387080ab47e668204f4f
  • 50754523e84b4bff4e1a9c6765302cc3796d70b0
Oct 25 2016, 6:52 PM · Indexer, General
ardumont created T595: Read sha1s from git repositories and send them for archive in azure.
Oct 25 2016, 6:51 PM · Indexer, General
ardumont updated the task description for T594: Properly package universal-ctags.
Oct 25 2016, 6:48 PM · Indexer, General
ardumont created T594: Properly package universal-ctags.
Oct 25 2016, 6:47 PM · Indexer, General

Oct 24 2016

ardumont created P118 Language unknown repartition.
Oct 24 2016, 1:55 PM · Indexer
ardumont edited P116 swh language recognized repartition.
Oct 24 2016, 11:42 AM · Indexer
ardumont updated the title for P117 swh mimetype recognized repartition - updated from swh mimetype recognized repartition to swh mimetype recognized repartition - updated.
Oct 24 2016, 11:36 AM · Indexer

Oct 21 2016

ardumont updated the language for P117 swh mimetype recognized repartition - updated from autodetect to sql.
Oct 21 2016, 11:38 AM · Indexer
ardumont edited P116 swh language recognized repartition.
Oct 21 2016, 11:37 AM · Indexer
ardumont created P117 swh mimetype recognized repartition - updated.
Oct 21 2016, 11:36 AM · Indexer
ardumont created P116 swh language recognized repartition.
Oct 21 2016, 11:29 AM · Indexer

Oct 17 2016

ardumont added a comment to T589: Add ctags content indexer.
ToolsProsCons
exuberant-ctags - http://ctags.sourceforge.net/- debian-packaged- development stalled (last commit on 2014)
- extensible- 41 languages supported
- difficulties to deal with files' content alone
- no recent release (last official 5.8 dates back from 2009, the actual debian's is 5.9~svn20110310-11)
universal-ctags - http://docs.ctags.io/en/latest/- 84 supported languages- manual install (not debian-packaged)
- extensible- not released yet (planned for dec 2016, https://github.com/universal-ctags/ctags/milestone/3)
- active development (last commit 12/10/2016)- same limit as exuberant-ctags to recognize data from file's content alone
- fork from exuberant-ctags
Oct 17 2016, 12:08 PM · Indexer, General
ardumont updated the task description for T589: Add ctags content indexer.
Oct 17 2016, 11:43 AM · Indexer, General
ardumont created T589: Add ctags content indexer.
Oct 17 2016, 11:37 AM · Indexer, General
ardumont closed T584: Deploy and run latest indexer code as Resolved.
Oct 17 2016, 10:32 AM · Indexer, General
ardumont closed T584: Deploy and run latest indexer code, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Oct 17 2016, 10:32 AM · Indexer, General

Oct 13 2016

ardumont added a comment to T584: Deploy and run latest indexer code.

currently running

Oct 13 2016, 8:31 PM · Indexer, General
ardumont closed T583: Update puppet manifests according to latest developments as Resolved.
Oct 13 2016, 8:31 PM · Indexer, General
ardumont closed T583: Update puppet manifests according to latest developments, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Oct 13 2016, 8:31 PM · Indexer, General
ardumont changed the status of T584: Deploy and run latest indexer code from Open to Work in Progress.
Oct 13 2016, 6:56 PM · Indexer, General
ardumont changed the status of T584: Deploy and run latest indexer code, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), from Open to Work in Progress.
Oct 13 2016, 6:56 PM · Indexer, General
ardumont updated the task description for T584: Deploy and run latest indexer code.
Oct 13 2016, 4:58 PM · Indexer, General
ardumont updated the task description for T584: Deploy and run latest indexer code.
Oct 13 2016, 4:07 PM · Indexer, General
ardumont created T584: Deploy and run latest indexer code.
Oct 13 2016, 4:03 PM · Indexer, General
ardumont closed T581: Text content orchestrator, a subtask of T577: Mimetype/Encoding indexer, as Resolved.
Oct 13 2016, 3:53 PM · Indexer, General
ardumont closed T581: Text content orchestrator as Resolved.
Oct 13 2016, 3:53 PM · Indexer, General
ardumont closed T582: Orchestrator - Optional filtering on existing contents, a subtask of T575: Pipe contents copied to azure to the indexer orchestrator, as Resolved.
Oct 13 2016, 3:53 PM · Indexer, General
ardumont closed T582: Orchestrator - Optional filtering on existing contents as Resolved.
Oct 13 2016, 3:53 PM · Indexer, General
ardumont closed T575: Pipe contents copied to azure to the indexer orchestrator as Resolved.
Oct 13 2016, 3:53 PM · Indexer, General
ardumont closed T575: Pipe contents copied to azure to the indexer orchestrator, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Oct 13 2016, 3:53 PM · Indexer, General
ardumont created T583: Update puppet manifests according to latest developments.
Oct 13 2016, 3:40 PM · Indexer, General
ardumont closed T580: Start indexing computations (mimetype, language) as Resolved.
Oct 13 2016, 3:34 PM · Indexer, General
ardumont closed T580: Start indexing computations (mimetype, language), a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Oct 13 2016, 3:34 PM · Indexer, General
ardumont renamed T582: Orchestrator - Optional filtering on existing contents from Orchestrator - Optional filtering on existing content to Orchestrator - Optional filtering on existing contents.
Oct 13 2016, 2:15 PM · Indexer, General
ardumont created T582: Orchestrator - Optional filtering on existing contents.
Oct 13 2016, 2:15 PM · Indexer, General
ardumont updated the task description for T581: Text content orchestrator.
Oct 13 2016, 2:10 PM · Indexer, General

Oct 11 2016

ardumont changed the status of T580: Start indexing computations (mimetype, language) from Open to Work in Progress.
Oct 11 2016, 8:30 PM · Indexer, General
ardumont changed the status of T580: Start indexing computations (mimetype, language), a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), from Open to Work in Progress.
Oct 11 2016, 8:30 PM · Indexer, General
ardumont added a comment to T575: Pipe contents copied to azure to the indexer orchestrator.

As of today, this is the archiver (archiver-uffizi) which is in charge of copying data from our objstorages (uffizi, banco) to azure.
(currently this is deployed on uffizi).

Oct 11 2016, 6:07 PM · Indexer, General
ardumont created T581: Text content orchestrator.
Oct 11 2016, 6:02 PM · Indexer, General
ardumont added a subtask for T580: Start indexing computations (mimetype, language): T579: Puppetize subset of indexers to start indexing computations.
Oct 11 2016, 2:40 PM · Indexer, General
ardumont added a parent task for T579: Puppetize subset of indexers to start indexing computations: T580: Start indexing computations (mimetype, language).
Oct 11 2016, 2:40 PM · Puppet recipes, Indexer, General

Oct 10 2016

ardumont created P112 [invalid] connection error when producing message for orchestrator - ardumont not in swhworker group!.
Oct 10 2016, 10:48 PM · Indexer
ardumont created T580: Start indexing computations (mimetype, language).
Oct 10 2016, 10:48 PM · Indexer, General
ardumont updated the title for P112 [invalid] connection error when producing message for orchestrator - ardumont not in swhworker group! from connection error when producing message for orchestrator to [invalid] connection error when producing message for orchestrator - ardumont not in swhworker group!.
Oct 10 2016, 10:48 PM · Indexer
ardumont added a comment to T580: Start indexing computations (mimetype, language).

Pre-requisite steps:

  • migrate db version to 85
  • stop workers (currently ongoing...)
  • Upgrade uffizi's python3-swh.storage
  • send sha1s contents to orchestrator
Oct 10 2016, 10:48 PM · Indexer, General
ardumont created T579: Puppetize subset of indexers to start indexing computations.
Oct 10 2016, 10:48 PM · Puppet recipes, Indexer, General
ardumont closed T579: Puppetize subset of indexers to start indexing computations, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Oct 10 2016, 10:48 PM · Indexer, General
ardumont closed T579: Puppetize subset of indexers to start indexing computations as Resolved.
Oct 10 2016, 10:48 PM · Puppet recipes, Indexer, General
ardumont closed T578: Language indexer, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Oct 10 2016, 10:48 PM · Indexer, General
ardumont added a comment to T578: Language indexer.

R78:f5056c028f6fc02cd127c5a9a72c3d27868238a0
?

Oct 10 2016, 10:48 PM · Indexer, General
ardumont closed T578: Language indexer as Resolved.
Oct 10 2016, 10:48 PM · Indexer, General
ardumont closed T576: Indexer orchestrator as Resolved.
Oct 10 2016, 10:48 PM · Indexer, General
ardumont closed T576: Indexer orchestrator, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Oct 10 2016, 10:48 PM · Indexer, General
ardumont closed T577: Mimetype/Encoding indexer as Resolved.
Oct 10 2016, 10:47 PM · Indexer, General
ardumont closed T577: Mimetype/Encoding indexer, a subtask of T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task), as Resolved.
Oct 10 2016, 10:47 PM · Indexer, General

Oct 7 2016

ardumont added a comment to T578: Language indexer.

Check debsources' enum for language

Oct 7 2016, 10:33 AM · Indexer, General

Oct 5 2016

ardumont updated the task description for T578: Language indexer.
Oct 5 2016, 11:27 AM · Indexer, General
ardumont updated the task description for T577: Mimetype/Encoding indexer.
Oct 5 2016, 11:26 AM · Indexer, General
ardumont updated the task description for T576: Indexer orchestrator.
Oct 5 2016, 11:24 AM · Indexer, General
ardumont updated the task description for T576: Indexer orchestrator.
Oct 5 2016, 11:21 AM · Indexer, General
ardumont created T578: Language indexer.
Oct 5 2016, 11:19 AM · Indexer, General
ardumont created T577: Mimetype/Encoding indexer.
Oct 5 2016, 11:17 AM · Indexer, General
ardumont shifted T575: Pipe contents copied to azure to the indexer orchestrator from the Restricted Space space to the S1 Public space.
Oct 5 2016, 11:08 AM · Indexer, General
ardumont shifted T574: Pipeline copy content to azure and then compute multiple indexes independently (meta task) from the Restricted Space space to the S1 Public space.
Oct 5 2016, 11:08 AM · Indexer, General
ardumont shifted T576: Indexer orchestrator from the Restricted Space space to the S1 Public space.
Oct 5 2016, 11:07 AM · Indexer, General
First
Prev
Next
About · Developer information · Legal