Page MenuHomeSoftware Heritage
Feed All Stories

Today

TG1999 updated subscribers of D4931: Add mapping of definitions and harvests.

Hi @douardda @vlorentz @zack please have a look :)

Thu, Jan 28, 4:15 PM
swh-public-ci added a comment to D4931: Add mapping of definitions and harvests.

Build is green

Thu, Jan 28, 4:14 PM
TG1999 updated the diff for D4931: Add mapping of definitions and harvests.

Add mapping for definitions and harvests
Add functions map_row, map_definition, map_harvest to check whether swh archive is able to map clearlydefined object or not

Thu, Jan 28, 4:12 PM
ardumont added a comment to T2998: Deploy lister next-gen in staging.

launchpad listing in progress and it seems to display the same behavior as the cgit lister (T2998#57500).

Thu, Jan 28, 4:06 PM · System administration, Lister
ardumont triaged T3003: next gen lister: Make lister flush their visit state regularly as Normal priority.
Thu, Jan 28, 4:05 PM
swh-public-ci added a comment to D4931: Add mapping of definitions and harvests.

Build is green

Thu, Jan 28, 3:58 PM
TG1999 updated the diff for D4931: Add mapping of definitions and harvests.

Add mapping of definitions and harvests

Thu, Jan 28, 3:57 PM
ardumont updated the task description for T2998: Deploy lister next-gen in staging.
Thu, Jan 28, 3:56 PM · System administration, Lister
ardumont triaged T3002: Current next-gen lister cran failing to list as Normal priority.
Thu, Jan 28, 3:55 PM · Origin-CRAN, Lister
ardumont created T3002: Current next-gen lister cran failing to list.
Thu, Jan 28, 3:55 PM · Origin-CRAN, Lister
ardumont added a comment to T2998: Deploy lister next-gen in staging.

lister-cran status: run ko [1]

Thu, Jan 28, 3:54 PM · System administration, Lister
ardumont updated the task description for T2998: Deploy lister next-gen in staging.
Thu, Jan 28, 3:46 PM · System administration, Lister
vsellier changed the status of T2975: Disk replacement on esnode1 from Open to Work in Progress.
Thu, Jan 28, 3:44 PM · System administration
ardumont updated the task description for T2998: Deploy lister next-gen in staging.
Thu, Jan 28, 3:42 PM · System administration, Lister
ardumont updated the task description for T2998: Deploy lister next-gen in staging.
Thu, Jan 28, 3:39 PM · System administration, Lister
swh-public-ci added a comment to D4931: Add mapping of definitions and harvests.

Build is green

Thu, Jan 28, 3:38 PM
ardumont updated the task description for T2998: Deploy lister next-gen in staging.
Thu, Jan 28, 3:37 PM · System administration, Lister
TG1999 updated the diff for D4931: Add mapping of definitions and harvests.

Add mapping of definitions and harvests

Thu, Jan 28, 3:37 PM
ardumont updated the task description for T2998: Deploy lister next-gen in staging.
Thu, Jan 28, 3:36 PM · System administration, Lister
ardumont added a comment to T2998: Deploy lister next-gen in staging.

gitea lister instance (https://try.gitea.io/api/v1), status ok:

Thu, Jan 28, 3:36 PM · System administration, Lister
vsellier closed T3001: Webapp is not displaying the origin type on the search results as Resolved.

The fix is deployed on webapp1 and solved the problem.

Thu, Jan 28, 3:33 PM · Storage manager, Web app
anlambert closed T2992: Port launchpad lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Thu, Jan 28, 3:28 PM · Sprint 2021 01, Scheduling utilities
anlambert closed D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 3:28 PM
anlambert closed T2992: Port launchpad lister to the new Lister API as Resolved by committing rDLSf86200470025: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 3:28 PM · Lister
anlambert committed rDLSf86200470025: launchpad: Reimplement lister using new Lister API (authored by anlambert).
launchpad: Reimplement lister using new Lister API
Thu, Jan 28, 3:28 PM
swh-public-ci added a comment to D4962: launchpad: Reimplement lister using new Lister API.

Build is green

Thu, Jan 28, 3:26 PM
ardumont updated the task description for T2998: Deploy lister next-gen in staging.
Thu, Jan 28, 3:26 PM · System administration, Lister
anlambert added a comment to D4962: launchpad: Reimplement lister using new Lister API.

yep, looks good to me.

Thanks.

Thu, Jan 28, 3:24 PM
anlambert updated the diff for D4962: launchpad: Reimplement lister using new Lister API.

Rebase

Thu, Jan 28, 3:23 PM
ardumont accepted D4962: launchpad: Reimplement lister using new Lister API.

yep, looks good to me.

Thu, Jan 28, 3:21 PM
vsellier closed D4963: webapp1: use the same deployment pattern than moma.
Thu, Jan 28, 3:19 PM
vsellier committed rSPSITEb82b0d93c2ec: webapp1: use the same deployment pattern than moma (authored by vsellier).
webapp1: use the same deployment pattern than moma
Thu, Jan 28, 3:18 PM
ardumont added inline comments to D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 3:17 PM
anlambert added inline comments to D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 3:15 PM
ardumont accepted D4963: webapp1: use the same deployment pattern than moma.

That installs a similar pattern as the other production node (moma regarding webapp/storage deployment)

Thu, Jan 28, 3:13 PM
ardumont added a comment to D4962: launchpad: Reimplement lister using new Lister API.

lgtm so far (did not read the test)

Thu, Jan 28, 3:11 PM
vsellier requested review of D4963: webapp1: use the same deployment pattern than moma.
Thu, Jan 28, 3:10 PM
swh-public-ci added a comment to D4931: Add mapping of definitions and harvests.

Build is green

Thu, Jan 28, 3:02 PM
TG1999 updated the diff for D4931: Add mapping of definitions and harvests.

Add mapping of definitions and harvests

Thu, Jan 28, 3:01 PM
swh-public-ci added a comment to D4962: launchpad: Reimplement lister using new Lister API.

Build is green

Thu, Jan 28, 2:58 PM
anlambert added inline comments to D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 2:56 PM
anlambert updated the diff for D4962: launchpad: Reimplement lister using new Lister API.

Remove debian folder committed by mistake

Thu, Jan 28, 2:54 PM
anlambert added inline comments to D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 2:54 PM
ardumont closed D4961: Make stateless lister constructors compatible with credentials.
Thu, Jan 28, 2:50 PM
ardumont committed rDLSae17b6b9a04a: Make stateless lister constructors compatible with credentials (authored by ardumont).
Make stateless lister constructors compatible with credentials
Thu, Jan 28, 2:50 PM
ardumont added inline comments to D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 2:48 PM
swh-public-ci added a comment to D4961: Make stateless lister constructors compatible with credentials.

Build is green

Thu, Jan 28, 2:47 PM
ardumont updated the summary of D4961: Make stateless lister constructors compatible with credentials.
Thu, Jan 28, 2:44 PM
ardumont retitled D4961: Make stateless lister constructors compatible with credentials from cgit: Make lister constructor compatible to Make stateless lister constructors compatible with credentials.
Thu, Jan 28, 2:44 PM
ardumont updated the diff for D4961: Make stateless lister constructors compatible with credentials.

Add *Lister.from_configfile test for stateless listers (cgit, cran, pypi)

Thu, Jan 28, 2:43 PM
swh-public-ci added a comment to D4962: launchpad: Reimplement lister using new Lister API.

Build is green

Thu, Jan 28, 2:41 PM
anlambert updated the diff for D4962: launchpad: Reimplement lister using new Lister API.
  • add a test to check lister instantiation with configuration file
Thu, Jan 28, 2:37 PM
vsellier added a comment to T3001: Webapp is not displaying the origin type on the search results.

The storage version v0.21.1 is deployed in staging, the problem looks fixed :

❯ curl -s  https://webapp.staging.swh.network/api/1/origin/https://gitlab.com/miwc/miwc.github.io.git/visit/latest/\?require_snapshot\=true | jq ''
{
  "origin": "https://gitlab.com/miwc/miwc.github.io.git",
  "date": "2020-12-07T18:21:58.967952+00:00",
  "type": "git",
  "visit": 1,
  "status": "full",
  "snapshot": "759b36e0e3e81e8cbf601181829571daa645b5d2",
  "metadata": {},
  "origin_url": "https://webapp.staging.swh.network/api/1/origin/https://gitlab.com/miwc/miwc.github.io.git/get/",
  "snapshot_url": "https://webapp.staging.swh.network/api/1/snapshot/759b36e0e3e81e8cbf601181829571daa645b5d2/"
}
Thu, Jan 28, 2:36 PM · Storage manager, Web app
swh-public-ci added a comment to D4961: Make stateless lister constructors compatible with credentials.

Build is green

Thu, Jan 28, 2:36 PM
Harbormaster failed remote builds in B18846: Diff 17705 for D4961: Make stateless lister constructors compatible with credentials!
Thu, Jan 28, 2:32 PM
swh-public-ci added a comment to D4961: Make stateless lister constructors compatible with credentials.

Build has FAILED

Thu, Jan 28, 2:32 PM
ardumont updated the diff for D4961: Make stateless lister constructors compatible with credentials.

Fix typos

Thu, Jan 28, 2:31 PM
ardumont updated the diff for D4961: Make stateless lister constructors compatible with credentials.

Update cgit/cran/pypi lister to allow credentials to be passed along during instantiation

Thu, Jan 28, 2:29 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTO68331b5e1684: Updated backport on buster-swh from debian/0.21.1-1_swh1 (unstable-swh) (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated backport on buster-swh from debian/0.21.1-1_swh1 (unstable-swh)
Thu, Jan 28, 2:24 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTO3f1dc5ce0043: Merge tag 'debian/0.21.1-1_swh1' into debian/buster-swh (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Merge tag 'debian/0.21.1-1_swh1' into debian/buster-swh
Thu, Jan 28, 2:24 PM
anlambert added a comment to D4961: Make stateless lister constructors compatible with credentials.

Well, we can't really do that, we currently totally inhibit the lister.from_configfile call.

Ah right, I guess we should add the test in test_lister.py then. I will update D4962 then.

yes, indeed and that's what i'm currently doing ;)

(great news about launchpad ;)

Thu, Jan 28, 2:21 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTOd2b062bcdc0c: pristine-tar data for swh-storage_0.21.1.orig.tar.gz (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
pristine-tar data for swh-storage_0.21.1.orig.tar.gz
Thu, Jan 28, 2:19 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTO3d71ec67c3d2: Updated debian changelog for version 0.21.1 (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Updated debian changelog for version 0.21.1
Thu, Jan 28, 2:19 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTO9580e4564e33: Update upstream source from tag 'debian/upstream/0.21.1' (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
Update upstream source from tag 'debian/upstream/0.21.1'
Thu, Jan 28, 2:19 PM
Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org> committed rDSTO57d30664b74c: New upstream version 0.21.1 (authored by Jenkins for Software Heritage <jenkins@jenkins-debian1.internal.softwareheritage.org>).
New upstream version 0.21.1
Thu, Jan 28, 2:19 PM
ardumont added a comment to D4961: Make stateless lister constructors compatible with credentials.

Well, we can't really do that, we currently totally inhibit the lister.from_configfile call.

Ah right, I guess we should add the test in test_lister.py then. I will update D4962 then.

Thu, Jan 28, 2:18 PM
anlambert added a comment to D4961: Make stateless lister constructors compatible with credentials.

load_from_envvar = mocker.patch("swh.lister.pattern.load_from_envvar")
load_from_envvar.return_value = {"credentials": {}}

Well, we can't really do that, we currently totally inhibit the lister.from_configfile call.

Thu, Jan 28, 2:13 PM
vsellier closed T2988: Improve cgit lister to add last modification date of the repos as Resolved.
Thu, Jan 28, 2:10 PM · CGit lister, Lister
ardumont added a comment to D4961: Make stateless lister constructors compatible with credentials.

Well, we can't really do that, we currently totally inhibit the lister.from_configfile call.

Thu, Jan 28, 2:08 PM
anlambert requested review of D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 2:06 PM
anlambert added a revision to T2992: Port launchpad lister to the new Lister API: D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 2:03 PM · Lister
ardumont added a comment to T2998: Deploy lister next-gen in staging.

cgit lister should flush origins after each page, which instance has been listed here ?

Thu, Jan 28, 2:02 PM · System administration, Lister
vsellier closed D4960: Correctly return origin_visit_status.type value everywhere.
Thu, Jan 28, 2:01 PM
vsellier committed rDSTO76de53cb261f: Correctly return origin_visit_status.type value everywhere (authored by vsellier).
Correctly return origin_visit_status.type value everywhere
Thu, Jan 28, 2:01 PM
anlambert added a comment to T2998: Deploy lister next-gen in staging.

Note that this lister seems to need some writing improvments though.
It seemed to have flushed the writing only at the end of the listing.
If that's the real behavior (i'll need to check), that won't bode well for relatively high dimensioned instance like the cgit eclispe instance for example.

Thu, Jan 28, 1:55 PM · System administration, Lister
ardumont added a comment to D4961: Make stateless lister constructors compatible with credentials.

I'll check if i missed others as well.

Thu, Jan 28, 1:54 PM
ardumont added a comment to D4961: Make stateless lister constructors compatible with credentials.

Other ported listers might have the same issue so I guess they should be fixed too.

Thu, Jan 28, 1:50 PM
anlambert added a comment to D4961: Make stateless lister constructors compatible with credentials.

Could you also update the celery tasks tests by mocking credentials configuration coming from configuration file ?

good idea, i'll check that.


Independently from this diff but still i think possibly related to stateless lister in general, you might be interested by [1]

[1] https://forge.softwareheritage.org/T2998#57500

Thu, Jan 28, 1:50 PM
ardumont updated the test plan for D4961: Make stateless lister constructors compatible with credentials.
Thu, Jan 28, 1:49 PM
ardumont added a comment to D4961: Make stateless lister constructors compatible with credentials.

Could you also update the celery tasks tests by mocking credentials configuration coming from configuration file ?

Thu, Jan 28, 1:48 PM
ardumont added a comment to T2998: Deploy lister next-gen in staging.

one cgit lister scheduled, status, it finished ok but [1]

Thu, Jan 28, 1:47 PM · System administration, Lister
anlambert accepted D4961: Make stateless lister constructors compatible with credentials.

LGTM. Other ported listers might have the same issue so I guess they should be fixed too.

Thu, Jan 28, 1:45 PM
ardumont requested review of D4961: Make stateless lister constructors compatible with credentials.
Thu, Jan 28, 1:38 PM
ardumont added a revision to T2998: Deploy lister next-gen in staging: D4961: Make stateless lister constructors compatible with credentials.
Thu, Jan 28, 1:34 PM · System administration, Lister
swh-public-ci added a comment to D4931: Add mapping of definitions and harvests.

Build is green

Thu, Jan 28, 1:30 PM
TG1999 updated the diff for D4931: Add mapping of definitions and harvests.

Add mapping of definitions and harvests

Thu, Jan 28, 1:29 PM
ardumont accepted D4960: Correctly return origin_visit_status.type value everywhere.

lgtm

Thu, Jan 28, 1:29 PM
TG1999 retitled D4931: Add mapping of definitions and harvests from Add Orchestrator: Add mapping of definitions and harvests to Add mapping of definitions and harvests.
Thu, Jan 28, 12:39 PM
swh-public-ci added a comment to D4931: Add mapping of definitions and harvests.

Build is green

Thu, Jan 28, 12:35 PM
TG1999 updated the diff for D4931: Add mapping of definitions and harvests.

Add mapping of definitions and harvests

Thu, Jan 28, 12:34 PM
vsellier requested review of D4960: Correctly return origin_visit_status.type value everywhere.
Thu, Jan 28, 12:23 PM
Harbormaster failed remote builds in B18840: Diff 17699 for D4931: Add mapping of definitions and harvests!
Thu, Jan 28, 12:18 PM
swh-public-ci added a comment to D4931: Add mapping of definitions and harvests.

Build has FAILED

Thu, Jan 28, 12:18 PM
TG1999 updated the diff for D4931: Add mapping of definitions and harvests.

Add mapping of definitions and harvest

Thu, Jan 28, 12:17 PM
Harbormaster failed remote builds in B18838: Diff 17697 for D4931: Add mapping of definitions and harvests!
Thu, Jan 28, 12:17 PM
swh-public-ci added a comment to D4931: Add mapping of definitions and harvests.

Build has FAILED

Thu, Jan 28, 12:17 PM
TG1999 updated the diff for D4931: Add mapping of definitions and harvests.

Add mapping of definitions and harvest

Thu, Jan 28, 12:16 PM
ardumont updated the task description for T2998: Deploy lister next-gen in staging.
Thu, Jan 28, 12:14 PM · System administration, Lister
ardumont updated the task description for T2998: Deploy lister next-gen in staging.
Thu, Jan 28, 12:13 PM · System administration, Lister
ardumont added a comment to T2998: Deploy lister next-gen in staging.

Lister phabricator deployed with one instance (swh), status ok:

Thu, Jan 28, 12:13 PM · System administration, Lister