Hi @douardda @vlorentz @zack please have a look :)

Build is green

Add mapping for definitions and harvests
Add functions map_row, map_definition, map_harvest to check whether swh archive is able to map clearlydefined object or not

launchpad listing in progress and it seems to display the same behavior as the cgit lister (T2998#57500) [1]

Build is green

Add mapping of definitions and harvests

lister-cran status: run ko [1]

Build is green

Add mapping of definitions and harvests

gitea lister instance (https://try.gitea.io/api/v1), status ok:

The fix is deployed on webapp1 and solved the problem.

Build is green

In D4962#125005, @ardumont wrote:

yep, looks good to me.

Thanks.

Rebase

yep, looks good to me.

That installs a similar pattern as the other production node (moma regarding webapp/storage deployment)

lgtm so far (did not read the test)

Build is green

Add mapping of definitions and harvests

Build is green

Remove debian folder committed by mistake

Build is green

Add *Lister.from_configfile test for stateless listers (cgit, cran, pypi)

Build is green

• add a test to check lister instantiation with configuration file

The storage version v0.21.1 is deployed in staging, the problem looks fixed :

❯ curl -s  https://webapp.staging.swh.network/api/1/origin/https://gitlab.com/miwc/miwc.github.io.git/visit/latest/\?require_snapshot\=true | jq ''
{
  "origin": "https://gitlab.com/miwc/miwc.github.io.git",
  "date": "2020-12-07T18:21:58.967952+00:00",
  "type": "git",
  "visit": 1,
  "status": "full",
  "snapshot": "759b36e0e3e81e8cbf601181829571daa645b5d2",
  "metadata": {},
  "origin_url": "https://webapp.staging.swh.network/api/1/origin/https://gitlab.com/miwc/miwc.github.io.git/get/",
  "snapshot_url": "https://webapp.staging.swh.network/api/1/snapshot/759b36e0e3e81e8cbf601181829571daa645b5d2/"
}

Build is green

Build has FAILED

Fix typos

Update cgit/cran/pypi lister to allow credentials to be passed along during instantiation

In D4961#124944, @ardumont wrote:

Well, we can't really do that, we currently totally inhibit the lister.from_configfile call.

Ah right, I guess we should add the test in test_lister.py then. I will update D4962 then.

yes, indeed and that's what i'm currently doing ;)

(great news about launchpad ;)

Well, we can't really do that, we currently totally inhibit the lister.from_configfile call.

Ah right, I guess we should add the test in test_lister.py then. I will update D4962 then.

In D4961#124940, @ardumont wrote:

load_from_envvar = mocker.patch("swh.lister.pattern.load_from_envvar")
load_from_envvar.return_value = {"credentials": {}}

Well, we can't really do that, we currently totally inhibit the lister.from_configfile call.

Well, we can't really do that, we currently totally inhibit the lister.from_configfile call.

cgit lister should flush origins after each page, which instance has been listed here ?

Note that this lister seems to need some writing improvments though.
It seemed to have flushed the writing only at the end of the listing.
If that's the real behavior (i'll need to check), that won't bode well for relatively high dimensioned instance like the cgit eclispe instance for example.

I'll check if i missed others as well.

Other ported listers might have the same issue so I guess they should be fixed too.

In D4961#124922, @ardumont wrote:

Could you also update the celery tasks tests by mocking credentials configuration coming from configuration file ?

good idea, i'll check that.

---

Independently from this diff but still i think possibly related to stateless lister in general, you might be interested by [1]

[1] https://forge.softwareheritage.org/T2998#57500

Could you also update the celery tasks tests by mocking credentials configuration coming from configuration file ?

one cgit lister scheduled, status, it finished ok but [1]

LGTM. Other ported listers might have the same issue so I guess they should be fixed too.

Build is green

Add mapping of definitions and harvests

lgtm

Build is green

Add mapping of definitions and harvests

Build has FAILED

Add mapping of definitions and harvest

Build has FAILED

Add mapping of definitions and harvest

Lister phabricator deployed with one instance (swh), status ok: