Projects Core Loader

Core LoaderFolder
ActivePublic
Watch Project

Members
View All

This project does not have any members.

Watchers
View All

This project does not have any watchers.

Details

Description

loader plumbings used by other loaders

Recent Activity
View All

Tue, Feb 19

olasd added a parent task for T1461: Add loader-related metrics to swh-loader-core: T1535: Deploy prometheus-statsd-exporter to gather per-worker metrics.

Tue, Feb 19, 2:25 PM · Core Loader, Metrics/monitoring

Jan 8 2019

olasd triaged T1461: Add loader-related metrics to swh-loader-core as High priority.

Jan 8 2019, 11:45 AM · Core Loader, Metrics/monitoring

Dec 13 2018

vlorentz closed T1300: Release swh-loader-core v0.0.36 as Resolved.

Dec 13 2018, 1:32 PM · Core Loader

Dec 5 2018

vlorentz closed T1393: swh/loader/core/loader.py has no proper test coverage as Resolved.

Resolved by D770.

Dec 5 2018, 3:23 PM · Core Loader

Dec 4 2018

vlorentz added a parent task for T1393: swh/loader/core/loader.py has no proper test coverage: T1411: at least 80% SLOC coverage in all components.

Dec 4 2018, 11:32 AM · Core Loader

Nov 27 2018

vlorentz triaged T1393: swh/loader/core/loader.py has no proper test coverage as High priority.

Nov 27 2018, 6:24 PM · Core Loader

Oct 31 2018

vlorentz renamed T1300: Release swh-loader-core v0.0.36 from Release swh-loader-core v0.0.35 to Release swh-loader-core v0.0.36.

Oct 31 2018, 12:18 PM · Core Loader

vlorentz triaged T1300: Release swh-loader-core v0.0.36 as Normal priority.

Oct 31 2018, 10:42 AM · Core Loader

Oct 10 2018

ardumont closed T1238: Loader: refactor loader tests to avoid code duplication as Resolved.

Oct 10 2018, 2:41 PM · Core Loader

Oct 4 2018

zack closed T336: "save code now", a subtask of T1022: SWORD deposit requesting to save content existing on an external code hosting platform, as Resolved.

Oct 4 2018, 11:47 AM · Core Loader, SWORD deposit

zack renamed T1238: Loader: refactor loader tests to avoid code duplication from Loader: Refactor the loader tests to reuse to Loader: refactor loader tests to avoid code duplication.

Oct 4 2018, 9:06 AM · Core Loader

Oct 3 2018

ardumont triaged T1238: Loader: refactor loader tests to avoid code duplication as Normal priority.

Oct 3 2018, 4:13 PM · Core Loader

Jul 23 2018

moranegg closed T960: draft specs for deposit with incomplete tarball , a subtask of T1021: SWORD deposit of metadata about an existing SWH object, as Resolved.

Jul 23 2018, 4:36 PM · Core Loader, SWORD deposit

Jun 27 2018

zack triaged T1021: SWORD deposit of metadata about an existing SWH object as Normal priority.

Jun 27 2018, 8:26 AM · Core Loader, SWORD deposit

zack triaged T1022: SWORD deposit requesting to save content existing on an external code hosting platform as Normal priority.

Jun 27 2018, 8:09 AM · Core Loader, SWORD deposit

Apr 11 2018

rdicosmo updated the task description for T1022: SWORD deposit requesting to save content existing on an external code hosting platform.

Apr 11 2018, 7:07 PM · Core Loader, SWORD deposit

rdicosmo added a subtask for T1022: SWORD deposit requesting to save content existing on an external code hosting platform: T336: "save code now".

Apr 11 2018, 7:06 PM · Core Loader, SWORD deposit

rdicosmo created T1022: SWORD deposit requesting to save content existing on an external code hosting platform.

Apr 11 2018, 7:06 PM · Core Loader, SWORD deposit

rdicosmo added a subtask for T1021: SWORD deposit of metadata about an existing SWH object: T960: draft specs for deposit with incomplete tarball .

Apr 11 2018, 7:01 PM · Core Loader, SWORD deposit

rdicosmo created T1021: SWORD deposit of metadata about an existing SWH object.

Apr 11 2018, 7:00 PM · Core Loader, SWORD deposit

Mar 9 2018

ardumont closed T982: failing worker consumes remaining tasks without processing them as Resolved.

Well, after much digging documentations and some tryouts. Finally gave in to solution 3.

Mar 9 2018, 11:17 AM · Core Loader

Mar 7 2018

ardumont added a comment to T982: failing worker consumes remaining tasks without processing them.

|---------------------------------+---------------------------------------+---------------------------------+------------------------------------|
| Possible solution               | Pros                                  | Cons                            | Question                           |
|---------------------------------+---------------------------------------+---------------------------------+------------------------------------|
| 8. Celery signal on postrun     | - separation of concern               | Not working, in OOM kill        | Documentation [3]                  |
|                                 |                                       | scenario, task is killed        |                                    |
|                                 |                                       | (tested)                        |                                    |
|---------------------------------+---------------------------------------+---------------------------------+------------------------------------|

Mar 7 2018, 3:48 PM · Core Loader

Mar 5 2018

ardumont added a comment to T982: failing worker consumes remaining tasks without processing them.

Still asserting the possibilities (and reading documentation):

Mar 5 2018, 4:36 PM · Core Loader

ardumont added a comment to T982: failing worker consumes remaining tasks without processing them.

So far, the solutions i foresee:

Mar 5 2018, 12:41 PM · Core Loader

ardumont triaged T982: failing worker consumes remaining tasks without processing them as Normal priority.

Mar 5 2018, 11:41 AM · Core Loader

Jan 31 2018

ardumont created P219 latest svn errors - investigate missing input.

Jan 31 2018, 9:28 AM · Core Loader

Dec 7 2017

ardumont closed T871: Migrate swh-storage api functions relative to indexers to swh-indexer as Resolved.

Dec 7 2017, 5:28 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer

ardumont closed T872: Deploy and restart indexers as Resolved.

Dec 7 2017, 5:28 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer

ardumont closed T872: Deploy and restart indexers, a subtask of T871: Migrate swh-storage api functions relative to indexers to swh-indexer, as Resolved.

Dec 7 2017, 5:28 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer

ardumont updated the task description for T872: Deploy and restart indexers.

Dec 7 2017, 5:24 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer

ardumont updated the task description for T872: Deploy and restart indexers.

Dec 7 2017, 5:10 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer

ardumont updated the task description for T872: Deploy and restart indexers.

Dec 7 2017, 5:10 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer

ardumont changed the status of T872: Deploy and restart indexers from Open to Work in Progress.

Dec 7 2017, 4:41 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer

ardumont changed the status of T872: Deploy and restart indexers, a subtask of T871: Migrate swh-storage api functions relative to indexers to swh-indexer, from Open to Work in Progress.

Dec 7 2017, 4:41 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer

ardumont edited parent tasks for T872: Deploy and restart indexers, added: T871: Migrate swh-storage api functions relative to indexers to swh-indexer; removed: T867: Separate indexers' database model to its own database - meta task.

Dec 7 2017, 4:40 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer

ardumont added a subtask for T871: Migrate swh-storage api functions relative to indexers to swh-indexer: T872: Deploy and restart indexers.

Dec 7 2017, 4:40 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer

ardumont updated the task description for T872: Deploy and restart indexers.

Dec 7 2017, 4:40 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer

ardumont updated the task description for T871: Migrate swh-storage api functions relative to indexers to swh-indexer.

Dec 7 2017, 4:39 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer

ardumont added projects to T872: Deploy and restart indexers: Core Loader, SWORD deposit.

Dec 7 2017, 3:39 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer

ardumont updated the task description for T871: Migrate swh-storage api functions relative to indexers to swh-indexer.

Dec 7 2017, 12:28 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer

ardumont updated the task description for T871: Migrate swh-storage api functions relative to indexers to swh-indexer.

Dec 7 2017, 11:26 AM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer

ardumont updated the task description for T871: Migrate swh-storage api functions relative to indexers to swh-indexer.

Dec 7 2017, 10:38 AM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer

ardumont updated the task description for T871: Migrate swh-storage api functions relative to indexers to swh-indexer.

Dec 7 2017, 10:38 AM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer

ardumont added projects to T871: Migrate swh-storage api functions relative to indexers to swh-indexer: Core Loader, SWORD deposit.

Dec 7 2017, 10:35 AM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer

Oct 30 2017

ardumont closed T804: swh-loader-core: doesn't properly batch object insertions (for big requests) as Resolved by committing rDLDBASE329142d5f161: core.loader: Properly batch object insertions for big requests.

Oct 30 2017, 4:49 PM · Core Loader

ardumont added a comment to T804: swh-loader-core: doesn't properly batch object insertions (for big requests).

The size limit should superimpose itself to the number of items limit, but the behavior should be the same.

Oct 30 2017, 2:21 PM · Core Loader

olasd added a comment to T804: swh-loader-core: doesn't properly batch object insertions (for big requests).

I'm going to try to clarify my point:

Oct 30 2017, 1:51 PM · Core Loader

Oct 24 2017

ardumont added a comment to T804: swh-loader-core: doesn't properly batch object insertions (for big requests).

The core loader uses a queue system to bundle loading of objects.

Oct 24 2017, 7:11 PM · Core Loader

Oct 13 2017

zack added a project to T804: swh-loader-core: doesn't properly batch object insertions (for big requests): Core Loader.

Oct 13 2017, 2:23 PM · Core Loader

zack renamed Core Loader from VCS Loader to Core Loader.

Oct 13 2017, 2:23 PM