In D2135#49709, @ardumont wrote:
Build has FAILED

I'm not sure the ci can build a diff when not on master...
[tox] $ git reset --hard origin/master
HEAD is now at c0af471 tox.ini: Fix py3 environment to only tests the swh/loader tests
[tox] $ git clean -fd -f
[tox] $ git submodule update --init --recursive
[tox] $ arc patch --diff 7167 --nocommit --nobranch --force --conduit-uri=https://forge.softwareheritage.org/ ********
This diff is against origin/package-loader...

Oct 13 2019, 5:43 PM · Debian loader, Core Loader

swh-public-ci added a comment to D2135: package.debian: Re-implement debian loader within the package loader mechanism.

Build has FAILED

Oct 13 2019, 5:38 PM · Debian loader, Core Loader

Oct 12 2019

ardumont added inline comments to D2135: package.debian: Re-implement debian loader within the package loader mechanism.

Oct 12 2019, 8:05 PM · Debian loader, Core Loader

ardumont added a comment to D2135: package.debian: Re-implement debian loader within the package loader mechanism.

Build has FAILED

Oct 12 2019, 8:02 PM · Debian loader, Core Loader

ardumont updated the summary of D2135: package.debian: Re-implement debian loader within the package loader mechanism.

Oct 12 2019, 8:00 PM · Debian loader, Core Loader

Oct 1 2019

ardumont closed T2017: package loader: Discuss revision metadata normalization as Resolved.

Oct 1 2019, 6:46 PM · Metadata workflow, Restricted Project, Core Loader

ardumont added a parent task for T2017: package loader: Discuss revision metadata normalization: T1389: Implement a base "package" loader for package managers.

Oct 1 2019, 6:46 PM · Metadata workflow, Restricted Project, Core Loader

ardumont renamed T2017: package loader: Discuss revision metadata normalization from Discuss revision metadata normalization to package loader: Discuss revision metadata normalization.

Oct 1 2019, 6:45 PM · Metadata workflow, Restricted Project, Core Loader

ardumont added a comment to T2017: package loader: Discuss revision metadata normalization.

For the following, we will focus on package loader metadata.

Oct 1 2019, 2:31 PM · Metadata workflow, Restricted Project, Core Loader

Sep 30 2019

moranegg added projects to T2017: package loader: Discuss revision metadata normalization: Restricted Project, Metadata workflow.

Sep 30 2019, 11:22 AM · Metadata workflow, Restricted Project, Core Loader

Sep 28 2019

ardumont updated the task description for T2017: package loader: Discuss revision metadata normalization.

Sep 28 2019, 1:32 PM · Metadata workflow, Restricted Project, Core Loader

Sep 27 2019

ardumont triaged T2017: package loader: Discuss revision metadata normalization as Normal priority.

Sep 27 2019, 12:18 PM · Metadata workflow, Restricted Project, Core Loader

Sep 24 2019

ardumont added a comment to D2028: [wip] package-loader: Implement a common package loader mechanism.

roh, i did not want to close it...
I just wanted to update the diff... oh, well!

Sep 24 2019, 2:39 PM · Npm loader, PyPI loader, Origin-GNU, Core Loader

ardumont closed D2028: [wip] package-loader: Implement a common package loader mechanism.

Sep 24 2019, 2:37 PM · Npm loader, PyPI loader, Origin-GNU, Core Loader

ardumont updated the summary of D2028: [wip] package-loader: Implement a common package loader mechanism.

Sep 24 2019, 10:08 AM · Npm loader, PyPI loader, Origin-GNU, Core Loader

Apr 8 2019

ardumont closed D1364: swh.loader.core.utils: Add coverage on utils module.

Apr 8 2019, 11:37 AM · Core Loader

swh-public-ci added a comment to D1364: swh.loader.core.utils: Add coverage on utils module.

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/113/ for more details.

Apr 8 2019, 11:30 AM · Core Loader

ardumont updated the diff for D1364: swh.loader.core.utils: Add coverage on utils module.

test_utils: Use @patch directly to declare expected behavior

Apr 8 2019, 11:28 AM · Core Loader

vlorentz accepted D1364: swh.loader.core.utils: Add coverage on utils module.

Apr 8 2019, 10:00 AM · Core Loader

vlorentz added a comment to D1364: swh.loader.core.utils: Add coverage on utils module.

@patch('foo')
def test_bar(foo):
    foo.return_value = 42
    pass # ...

Apr 8 2019, 9:59 AM · Core Loader

Apr 6 2019

swh-public-ci added a comment to D1364: swh.loader.core.utils: Add coverage on utils module.

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/112/ for more details.

Apr 6 2019, 11:46 AM · Core Loader

ardumont added a project to D1364: swh.loader.core.utils: Add coverage on utils module: Core Loader.

Apr 6 2019, 11:45 AM · Core Loader

ardumont closed D1337: core.loader: Adapt according to latest storage api change.

Apr 6 2019, 10:07 AM · Core Loader

ardumont added a comment to D1337: core.loader: Adapt according to latest storage api change.

In a further step, it'd be nicer to make the loader-core counters compatible with the parsing done in swh.storage, so we can get a breakdown of operations per loader without jumping through too many hoops

Apr 6 2019, 9:30 AM · Core Loader

swh-public-ci added a comment to D1337: core.loader: Adapt according to latest storage api change.

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/104/ for more details.

Apr 6 2019, 9:29 AM · Core Loader

ardumont updated the diff for D1337: core.loader: Adapt according to latest storage api change.

Remove badly named counter

Apr 6 2019, 9:27 AM · Core Loader

Apr 5 2019

ardumont triaged T1626: loader-core: Unify counter with the storage's results as Normal priority.

Apr 5 2019, 9:44 AM · Core Loader

ardumont added inline comments to D1337: core.loader: Adapt according to latest storage api change.

Apr 5 2019, 9:42 AM · Core Loader

Apr 4 2019

olasd accepted D1337: core.loader: Adapt according to latest storage api change.

It feels a little weird to have different counters in the dict and in the logs, but this is after filtering out the existing objects so the difference should be minimal anyway.

Apr 4 2019, 5:23 PM · Core Loader

swh-public-ci added a comment to D1337: core.loader: Adapt according to latest storage api change.

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/103/ for more details.

Apr 4 2019, 2:43 PM · Core Loader

swh-public-ci added a comment to D1337: core.loader: Adapt according to latest storage api change.

Build has FAILED

Apr 4 2019, 11:47 AM · Core Loader

Apr 3 2019

Harbormaster failed remote builds in B5056: Diff 4289 for D1337: core.loader: Adapt according to latest storage api change!

Apr 3 2019, 6:16 PM · Core Loader

swh-public-ci added a comment to D1337: core.loader: Adapt according to latest storage api change.

Build has FAILED

Apr 3 2019, 6:16 PM · Core Loader

ardumont updated the diff for D1337: core.loader: Adapt according to latest storage api change.

Fix api call to origin_visit_update api

Apr 3 2019, 6:15 PM · Core Loader

olasd added inline comments to D1337: core.loader: Adapt according to latest storage api change.

Apr 3 2019, 3:43 PM · Core Loader

olasd added inline comments to D1337: core.loader: Adapt according to latest storage api change.

Apr 3 2019, 3:42 PM · Core Loader

ardumont updated the summary of D1337: core.loader: Adapt according to latest storage api change.

Apr 3 2019, 3:25 PM · Core Loader

swh-public-ci added a comment to D1337: core.loader: Adapt according to latest storage api change.

Build has FAILED

Apr 3 2019, 3:15 PM · Core Loader

Harbormaster failed remote builds in B5047: Diff 4280 for D1337: core.loader: Adapt according to latest storage api change!

Apr 3 2019, 3:15 PM · Core Loader

ardumont updated the diff for D1337: core.loader: Adapt according to latest storage api change.

snapshot_add now takes a list of snapshot as input

Apr 3 2019, 3:14 PM · Core Loader

ardumont added a comment to D1337: core.loader: Adapt according to latest storage api change.

In D1337#28947, @olasd wrote:

I wonder if it wouldn't make sense to just turn the counter attribute into a dict compatible with the output of the storage functions...

Apr 3 2019, 3:10 PM · Core Loader

ardumont added inline comments to D1337: core.loader: Adapt according to latest storage api change.

Apr 3 2019, 3:10 PM · Core Loader

vlorentz added inline comments to D1337: core.loader: Adapt according to latest storage api change.

Apr 3 2019, 2:57 PM · Core Loader

olasd added a comment to D1337: core.loader: Adapt according to latest storage api change.

I wonder if it wouldn't make sense to just turn the counter attribute into a dict compatible with the output of the storage functions...

Apr 3 2019, 2:51 PM · Core Loader

Harbormaster failed remote builds in B5037: Diff 4270 for D1337: core.loader: Adapt according to latest storage api change!

Apr 3 2019, 2:20 PM · Core Loader

swh-public-ci added a comment to D1337: core.loader: Adapt according to latest storage api change.

Build has FAILED

Apr 3 2019, 2:20 PM · Core Loader

ardumont updated the summary of D1337: core.loader: Adapt according to latest storage api change.

Apr 3 2019, 2:20 PM · Core Loader

ardumont added a project to D1337: core.loader: Adapt according to latest storage api change: Core Loader.

Apr 3 2019, 2:19 PM · Core Loader

Feb 19 2019

olasd added a parent task for T1461: Add loader-related metrics to swh-loader-core: T1535: Deploy prometheus-statsd-exporter to gather per-worker metrics.

Feb 19 2019, 2:25 PM · Core Loader, Metrics/monitoring

Jan 8 2019

olasd triaged T1461: Add loader-related metrics to swh-loader-core as High priority.

Jan 8 2019, 11:45 AM · Core Loader, Metrics/monitoring

Dec 13 2018

vlorentz closed T1300: Release swh-loader-core v0.0.36 as Resolved.

Dec 13 2018, 1:32 PM · Core Loader

Dec 5 2018

vlorentz closed T1393: swh/loader/core/loader.py has no proper test coverage as Resolved.

Resolved by D770.

Dec 5 2018, 3:23 PM · Core Loader

Dec 4 2018

vlorentz added a parent task for T1393: swh/loader/core/loader.py has no proper test coverage: T1411: reach a minimum of 80% SLOC coverage across all components.

Dec 4 2018, 11:32 AM · Core Loader

Nov 27 2018

vlorentz triaged T1393: swh/loader/core/loader.py has no proper test coverage as High priority.

Nov 27 2018, 6:24 PM · Core Loader

Oct 31 2018

vlorentz renamed T1300: Release swh-loader-core v0.0.36 from Release swh-loader-core v0.0.35 to Release swh-loader-core v0.0.36.

Oct 31 2018, 12:18 PM · Core Loader

vlorentz triaged T1300: Release swh-loader-core v0.0.36 as Normal priority.

Oct 31 2018, 10:42 AM · Core Loader

Oct 10 2018

ardumont closed T1238: Loader: refactor loader tests to avoid code duplication as Resolved.

Oct 10 2018, 2:41 PM · Core Loader

Oct 4 2018

zack closed T336: "save code now", a subtask of T1022: SWORD deposit requesting to save content existing on an external code hosting platform, as Resolved.

Oct 4 2018, 11:47 AM · Core Loader, SWORD deposit

zack renamed T1238: Loader: refactor loader tests to avoid code duplication from Loader: Refactor the loader tests to reuse to Loader: refactor loader tests to avoid code duplication.

Oct 4 2018, 9:06 AM · Core Loader

Oct 3 2018

ardumont triaged T1238: Loader: refactor loader tests to avoid code duplication as Normal priority.

Oct 3 2018, 4:13 PM · Core Loader

Jul 23 2018

moranegg closed T960: draft specs for deposit with incomplete tarball , a subtask of T1021: SWORD deposit of metadata about an existing SWH object, as Resolved.

Jul 23 2018, 4:36 PM · Core Loader, SWORD deposit

Jun 27 2018

zack triaged T1021: SWORD deposit of metadata about an existing SWH object as Normal priority.

Jun 27 2018, 8:26 AM · Core Loader, SWORD deposit

zack triaged T1022: SWORD deposit requesting to save content existing on an external code hosting platform as Normal priority.

Jun 27 2018, 8:09 AM · Core Loader, SWORD deposit

Apr 11 2018

rdicosmo updated the task description for T1022: SWORD deposit requesting to save content existing on an external code hosting platform.

Apr 11 2018, 7:07 PM · Core Loader, SWORD deposit

rdicosmo added a subtask for T1022: SWORD deposit requesting to save content existing on an external code hosting platform: T336: "save code now".

Apr 11 2018, 7:06 PM · Core Loader, SWORD deposit

rdicosmo created T1022: SWORD deposit requesting to save content existing on an external code hosting platform.

Apr 11 2018, 7:06 PM · Core Loader, SWORD deposit

rdicosmo added a subtask for T1021: SWORD deposit of metadata about an existing SWH object: T960: draft specs for deposit with incomplete tarball .

Apr 11 2018, 7:01 PM · Core Loader, SWORD deposit

rdicosmo created T1021: SWORD deposit of metadata about an existing SWH object.

Apr 11 2018, 7:00 PM · Core Loader, SWORD deposit

Mar 9 2018

ardumont closed T982: failing worker consumes remaining tasks without processing them as Resolved.

Well, after much digging documentations and some tryouts. Finally gave in to solution 3.

Mar 9 2018, 11:17 AM · Core Loader

Mar 7 2018

ardumont added a comment to T982: failing worker consumes remaining tasks without processing them.

|---------------------------------+---------------------------------------+---------------------------------+------------------------------------|
| Possible solution               | Pros                                  | Cons                            | Question                           |
|---------------------------------+---------------------------------------+---------------------------------+------------------------------------|
| 8. Celery signal on postrun     | - separation of concern               | Not working, in OOM kill        | Documentation [3]                  |
|                                 |                                       | scenario, task is killed        |                                    |
|                                 |                                       | (tested)                        |                                    |
|---------------------------------+---------------------------------------+---------------------------------+------------------------------------|

Mar 7 2018, 3:48 PM · Core Loader