Page MenuHomeSoftware Heritage

Core LoaderFolder
ActivePublic

Members

  • This project does not have any members.

Watchers

  • This project does not have any watchers.

Details

Description

loader plumbings used by other loaders

Recent Activity

Tue, Feb 19

olasd added a parent task for T1461: Add loader-related metrics to swh-loader-core: T1535: Deploy prometheus-statsd-exporter to gather per-worker metrics.
Tue, Feb 19, 2:25 PM · Core Loader, Metrics/monitoring

Jan 8 2019

olasd triaged T1461: Add loader-related metrics to swh-loader-core as High priority.
Jan 8 2019, 11:45 AM · Core Loader, Metrics/monitoring

Dec 13 2018

vlorentz closed T1300: Release swh-loader-core v0.0.36 as Resolved.
Dec 13 2018, 1:32 PM · Core Loader

Dec 5 2018

vlorentz closed T1393: swh/loader/core/loader.py has no proper test coverage as Resolved.

Resolved by D770.

Dec 5 2018, 3:23 PM · Core Loader

Dec 4 2018

vlorentz added a parent task for T1393: swh/loader/core/loader.py has no proper test coverage: T1411: at least 80% SLOC coverage in all components.
Dec 4 2018, 11:32 AM · Core Loader

Nov 27 2018

vlorentz triaged T1393: swh/loader/core/loader.py has no proper test coverage as High priority.
Nov 27 2018, 6:24 PM · Core Loader

Oct 31 2018

vlorentz renamed T1300: Release swh-loader-core v0.0.36 from Release swh-loader-core v0.0.35 to Release swh-loader-core v0.0.36.
Oct 31 2018, 12:18 PM · Core Loader
vlorentz triaged T1300: Release swh-loader-core v0.0.36 as Normal priority.
Oct 31 2018, 10:42 AM · Core Loader

Oct 10 2018

ardumont closed T1238: Loader: refactor loader tests to avoid code duplication as Resolved.
Oct 10 2018, 2:41 PM · Core Loader

Oct 4 2018

zack closed T336: "save code now", a subtask of T1022: SWORD deposit requesting to save content existing on an external code hosting platform, as Resolved.
Oct 4 2018, 11:47 AM · Core Loader, SWORD deposit
zack renamed T1238: Loader: refactor loader tests to avoid code duplication from Loader: Refactor the loader tests to reuse to Loader: refactor loader tests to avoid code duplication.
Oct 4 2018, 9:06 AM · Core Loader

Oct 3 2018

ardumont triaged T1238: Loader: refactor loader tests to avoid code duplication as Normal priority.
Oct 3 2018, 4:13 PM · Core Loader

Jul 23 2018

moranegg closed T960: draft specs for deposit with incomplete tarball , a subtask of T1021: SWORD deposit of metadata about an existing SWH object, as Resolved.
Jul 23 2018, 4:36 PM · Core Loader, SWORD deposit

Jun 27 2018

zack triaged T1021: SWORD deposit of metadata about an existing SWH object as Normal priority.
Jun 27 2018, 8:26 AM · Core Loader, SWORD deposit
zack triaged T1022: SWORD deposit requesting to save content existing on an external code hosting platform as Normal priority.
Jun 27 2018, 8:09 AM · Core Loader, SWORD deposit

Apr 11 2018

rdicosmo updated the task description for T1022: SWORD deposit requesting to save content existing on an external code hosting platform.
Apr 11 2018, 7:07 PM · Core Loader, SWORD deposit
rdicosmo added a subtask for T1022: SWORD deposit requesting to save content existing on an external code hosting platform: T336: "save code now".
Apr 11 2018, 7:06 PM · Core Loader, SWORD deposit
rdicosmo created T1022: SWORD deposit requesting to save content existing on an external code hosting platform.
Apr 11 2018, 7:06 PM · Core Loader, SWORD deposit
rdicosmo added a subtask for T1021: SWORD deposit of metadata about an existing SWH object: T960: draft specs for deposit with incomplete tarball .
Apr 11 2018, 7:01 PM · Core Loader, SWORD deposit
rdicosmo created T1021: SWORD deposit of metadata about an existing SWH object.
Apr 11 2018, 7:00 PM · Core Loader, SWORD deposit

Mar 9 2018

ardumont closed T982: failing worker consumes remaining tasks without processing them as Resolved.

Well, after much digging documentations and some tryouts. Finally gave in to solution 3.

Mar 9 2018, 11:17 AM · Core Loader

Mar 7 2018

ardumont added a comment to T982: failing worker consumes remaining tasks without processing them.
|---------------------------------+---------------------------------------+---------------------------------+------------------------------------|
| Possible solution               | Pros                                  | Cons                            | Question                           |
|---------------------------------+---------------------------------------+---------------------------------+------------------------------------|
| 8. Celery signal on postrun     | - separation of concern               | Not working, in OOM kill        | Documentation [3]                  |
|                                 |                                       | scenario, task is killed        |                                    |
|                                 |                                       | (tested)                        |                                    |
|---------------------------------+---------------------------------------+---------------------------------+------------------------------------|
Mar 7 2018, 3:48 PM · Core Loader

Mar 5 2018

ardumont added a comment to T982: failing worker consumes remaining tasks without processing them.

Still asserting the possibilities (and reading documentation):

Mar 5 2018, 4:36 PM · Core Loader
ardumont added a comment to T982: failing worker consumes remaining tasks without processing them.

So far, the solutions i foresee:

Mar 5 2018, 12:41 PM · Core Loader
ardumont triaged T982: failing worker consumes remaining tasks without processing them as Normal priority.
Mar 5 2018, 11:41 AM · Core Loader

Jan 31 2018

ardumont created P219 latest svn errors - investigate missing input.
Jan 31 2018, 9:28 AM · Core Loader

Dec 7 2017

ardumont closed T871: Migrate swh-storage api functions relative to indexers to swh-indexer as Resolved.
Dec 7 2017, 5:28 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer
ardumont closed T872: Deploy and restart indexers as Resolved.
Dec 7 2017, 5:28 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer
ardumont closed T872: Deploy and restart indexers, a subtask of T871: Migrate swh-storage api functions relative to indexers to swh-indexer, as Resolved.
Dec 7 2017, 5:28 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer
ardumont updated the task description for T872: Deploy and restart indexers.
Dec 7 2017, 5:24 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer
ardumont updated the task description for T872: Deploy and restart indexers.
Dec 7 2017, 5:10 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer
ardumont updated the task description for T872: Deploy and restart indexers.
Dec 7 2017, 5:10 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer
ardumont changed the status of T872: Deploy and restart indexers from Open to Work in Progress.
Dec 7 2017, 4:41 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer
ardumont changed the status of T872: Deploy and restart indexers, a subtask of T871: Migrate swh-storage api functions relative to indexers to swh-indexer, from Open to Work in Progress.
Dec 7 2017, 4:41 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer
ardumont edited parent tasks for T872: Deploy and restart indexers, added: T871: Migrate swh-storage api functions relative to indexers to swh-indexer; removed: T867: Separate indexers' database model to its own database - meta task.
Dec 7 2017, 4:40 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer
ardumont added a subtask for T871: Migrate swh-storage api functions relative to indexers to swh-indexer: T872: Deploy and restart indexers.
Dec 7 2017, 4:40 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer
ardumont updated the task description for T872: Deploy and restart indexers.
Dec 7 2017, 4:40 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer
ardumont updated the task description for T871: Migrate swh-storage api functions relative to indexers to swh-indexer.
Dec 7 2017, 4:39 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer
ardumont added projects to T872: Deploy and restart indexers: Core Loader, SWORD deposit.
Dec 7 2017, 3:39 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer
ardumont updated the task description for T871: Migrate swh-storage api functions relative to indexers to swh-indexer.
Dec 7 2017, 12:28 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer
ardumont updated the task description for T871: Migrate swh-storage api functions relative to indexers to swh-indexer.
Dec 7 2017, 11:26 AM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer
ardumont updated the task description for T871: Migrate swh-storage api functions relative to indexers to swh-indexer.
Dec 7 2017, 10:38 AM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer
ardumont updated the task description for T871: Migrate swh-storage api functions relative to indexers to swh-indexer.
Dec 7 2017, 10:38 AM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer
ardumont added projects to T871: Migrate swh-storage api functions relative to indexers to swh-indexer: Core Loader, SWORD deposit.
Dec 7 2017, 10:35 AM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer

Oct 30 2017

ardumont closed T804: swh-loader-core: doesn't properly batch object insertions (for big requests) as Resolved by committing rDLDBASE329142d5f161: core.loader: Properly batch object insertions for big requests.
Oct 30 2017, 4:49 PM · Core Loader
ardumont added a comment to T804: swh-loader-core: doesn't properly batch object insertions (for big requests).

The size limit should superimpose itself to the number of items limit, but the behavior should be the same.

Oct 30 2017, 2:21 PM · Core Loader
olasd added a comment to T804: swh-loader-core: doesn't properly batch object insertions (for big requests).

I'm going to try to clarify my point:

Oct 30 2017, 1:51 PM · Core Loader

Oct 24 2017

ardumont added a comment to T804: swh-loader-core: doesn't properly batch object insertions (for big requests).

The core loader uses a queue system to bundle loading of objects.

Oct 24 2017, 7:11 PM · Core Loader

Oct 13 2017

zack added a project to T804: swh-loader-core: doesn't properly batch object insertions (for big requests): Core Loader.
Oct 13 2017, 2:23 PM · Core Loader
zack renamed Core Loader from VCS Loader to Core Loader.
Oct 13 2017, 2:23 PM