Page MenuHomeSoftware Heritage

anlambert (Antoine Lambert)
User

Projects (7)

User Details

User Since
Jul 3 2017, 10:14 AM (186 w, 3 d)

Recent Activity

Today

anlambert added inline comments to D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 2:56 PM
anlambert updated the diff for D4962: launchpad: Reimplement lister using new Lister API.

Remove debian folder committed by mistake

Thu, Jan 28, 2:54 PM
anlambert added inline comments to D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 2:54 PM
anlambert updated the diff for D4962: launchpad: Reimplement lister using new Lister API.
  • add a test to check lister instantiation with configuration file
Thu, Jan 28, 2:37 PM
anlambert added a comment to D4961: Make stateless lister constructors compatible with credentials.

Well, we can't really do that, we currently totally inhibit the lister.from_configfile call.

Ah right, I guess we should add the test in test_lister.py then. I will update D4962 then.

yes, indeed and that's what i'm currently doing ;)

(great news about launchpad ;)

Thu, Jan 28, 2:21 PM
anlambert added a comment to D4961: Make stateless lister constructors compatible with credentials.

load_from_envvar = mocker.patch("swh.lister.pattern.load_from_envvar")
load_from_envvar.return_value = {"credentials": {}}

Well, we can't really do that, we currently totally inhibit the lister.from_configfile call.

Thu, Jan 28, 2:13 PM
anlambert requested review of D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 2:06 PM
anlambert added a revision to T2992: Port launchpad lister to the new Lister API: D4962: launchpad: Reimplement lister using new Lister API.
Thu, Jan 28, 2:03 PM · Lister
anlambert added a comment to T2998: Deploy lister next-gen in staging.

Note that this lister seems to need some writing improvments though.
It seemed to have flushed the writing only at the end of the listing.
If that's the real behavior (i'll need to check), that won't bode well for relatively high dimensioned instance like the cgit eclispe instance for example.

Thu, Jan 28, 1:55 PM · System administration, Lister
anlambert added a comment to D4961: Make stateless lister constructors compatible with credentials.

Could you also update the celery tasks tests by mocking credentials configuration coming from configuration file ?

good idea, i'll check that.


Independently from this diff but still i think possibly related to stateless lister in general, you might be interested by [1]

[1] https://forge.softwareheritage.org/T2998#57500

Thu, Jan 28, 1:50 PM
anlambert accepted D4961: Make stateless lister constructors compatible with credentials.

LGTM. Other ported listers might have the same issue so I guess they should be fixed too.

Thu, Jan 28, 1:45 PM

Yesterday

anlambert added inline comments to D4957: gitlab: Deal with missing or trailing / in url input.
Wed, Jan 27, 7:13 PM
anlambert accepted D4957: gitlab: Deal with missing or trailing / in url input.
Wed, Jan 27, 7:10 PM
anlambert added inline comments to D4957: gitlab: Deal with missing or trailing / in url input.
Wed, Jan 27, 7:10 PM
anlambert added a comment to D4958: cli: Make cli work with new lister....

Legacy option should be removed soon so no tests needed here ;-)

Wed, Jan 27, 6:46 PM
anlambert accepted D4958: cli: Make cli work with new lister....
Wed, Jan 27, 6:46 PM
anlambert added inline comments to D4957: gitlab: Deal with missing or trailing / in url input.
Wed, Jan 27, 6:42 PM
anlambert requested changes to D4957: gitlab: Deal with missing or trailing / in url input.

This would be cleaner to process the url in the lister constructor instead (same as in debian one).

Wed, Jan 27, 6:11 PM
anlambert changed the status of T2992: Port launchpad lister to the new Lister API from Open to Work in Progress.
Wed, Jan 27, 4:05 PM · Lister
anlambert changed the status of T2992: Port launchpad lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, from Open to Work in Progress.
Wed, Jan 27, 4:05 PM · Sprint 2021 01, Scheduling utilities
anlambert added a comment to D4955: Drop no longer swh.lister.core.{indexing,page_by_page}_lister.

@anlambert, thanks, jsyk, i'm working on fixing failing test in the chroot debian in the end goal to deploy the listers in staging.

Wed, Jan 27, 4:03 PM
anlambert accepted D4955: Drop no longer swh.lister.core.{indexing,page_by_page}_lister.
Wed, Jan 27, 3:54 PM
anlambert accepted D4954: cgit: Don't stop the listing when a repository page is not available.

Looks good !

Wed, Jan 27, 3:03 PM
anlambert added a comment to D4954: cgit: Don't stop the listing when a repository page is not available.

Less code to modify and easier to read and understand the intended behavior from my point of view.

maybe so.

One pro I see for your implementation is if at some point the main loop (get_pages)
fails to fetch something (which we don't currently expect to see the assert added
here), we'll have more information available in the log instead of an AssertionError.
We'll indeed have an actual HttpError with the status code and everything in the
stacktrace.

All in all, i think you are right ;)

Wed, Jan 27, 2:54 PM
anlambert added a comment to D4954: cgit: Don't stop the listing when a repository page is not available.

You should rather catch the requests.exceptions.HTTPError originally raised by _get_and_parse method.

picture me curious, why?

(it's missing the eventual logging.warning instruction but its use is debatable as well ;)

Wed, Jan 27, 2:38 PM
anlambert requested changes to D4954: cgit: Don't stop the listing when a repository page is not available.

You should rather catch the requests.exceptions.HTTPError originally raised by _get_and_parse method.

Wed, Jan 27, 2:24 PM
anlambert triaged T2999: Optimize the number of HTTP requests sent by the cgit lister as Normal priority.
Wed, Jan 27, 1:52 PM · CGit lister
anlambert accepted D4953: cgit: Add support for last_update information during listing.

Looks good to me !

Wed, Jan 27, 1:35 PM
anlambert requested changes to D4953: cgit: Add support for last_update information during listing.

Thanks for tackling this !

Wed, Jan 27, 12:09 PM
anlambert claimed T2992: Port launchpad lister to the new Lister API.
Wed, Jan 27, 10:35 AM · Lister

Tue, Jan 26

anlambert closed D4952: tests/resources: Fix mypy 0.800 errors.
Tue, Jan 26, 5:37 PM
anlambert committed rDWAPPSab5401d73db1: tests/resources: Fix mypy 0.800 errors (authored by anlambert).
tests/resources: Fix mypy 0.800 errors
Tue, Jan 26, 5:37 PM
anlambert closed D4925: debian: Reimplement lister using new Lister API.
Tue, Jan 26, 5:30 PM
anlambert closed T2979: Port debian lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Tue, Jan 26, 5:29 PM · Sprint 2021 01, Scheduling utilities
anlambert closed T2979: Port debian lister to the new Lister API as Resolved by committing rDLSbb0184c004d2: debian: Reimplement lister using new Lister API.
Tue, Jan 26, 5:29 PM · Lister, Sprint 2021 01
anlambert committed rDLSbb0184c004d2: debian: Reimplement lister using new Lister API (authored by anlambert).
debian: Reimplement lister using new Lister API
Tue, Jan 26, 5:29 PM
anlambert updated the diff for D4925: debian: Reimplement lister using new Lister API.

Rebase

Tue, Jan 26, 5:23 PM
anlambert requested review of D4952: tests/resources: Fix mypy 0.800 errors.
Tue, Jan 26, 5:23 PM
anlambert updated the diff for D4925: debian: Reimplement lister using new Lister API.

Rebase and remove conftest.py file

Tue, Jan 26, 5:08 PM
anlambert accepted D4949: tests: remove all unused conftests.

Thanks !

Tue, Jan 26, 5:01 PM
anlambert requested changes to D4949: tests: remove all unused conftests.

Looks good but the commit message could be more detailed and rephrased to something like:

Tue, Jan 26, 4:51 PM
anlambert accepted D4948: gitlab: Implement keyset-based pagination listing.

Looks good to me ! I added a comment regarding further improvements that could be added to the lister but not related to that diff.

Tue, Jan 26, 4:19 PM
anlambert updated the diff for D4925: debian: Reimplement lister using new Lister API.

Rebase

Tue, Jan 26, 3:25 PM
anlambert closed T2989: Port CRAN lister to the new Lister API as Resolved by committing rDLS22eeb0956ec1: cran: Retrieve last update date for each listed package.
Tue, Jan 26, 3:24 PM · Lister
anlambert closed T2989: Port CRAN lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, as Resolved.
Tue, Jan 26, 3:24 PM · Sprint 2021 01, Scheduling utilities
anlambert committed rDLS22eeb0956ec1: cran: Retrieve last update date for each listed package (authored by anlambert).
cran: Retrieve last update date for each listed package
Tue, Jan 26, 3:24 PM
anlambert closed D4945: cran: Reimplement lister using new Lister API.
Tue, Jan 26, 3:24 PM
anlambert committed rDLS6f40ab4c5730: cran: Reimplement lister using new Lister API (authored by anlambert).
cran: Reimplement lister using new Lister API
Tue, Jan 26, 3:24 PM
anlambert closed T2911: Redirect https://www.softwareheritage.org/schema/2018/deposit to https://docs.softwareheritage.org/devel/swh-deposit/specs/protocol-reference.html, a subtask of T2625: create and publish xmlns schema for the specific swh-deposit metadata, as Resolved.
Tue, Jan 26, 3:22 PM · Restricted Project, SWORD deposit, Scientific Community Building
anlambert closed T2911: Redirect https://www.softwareheritage.org/schema/2018/deposit to https://docs.softwareheritage.org/devel/swh-deposit/specs/protocol-reference.html as Resolved.

Following URLs are now redirected to the desired page:

Tue, Jan 26, 3:22 PM · Website
anlambert updated the diff for D4945: cran: Reimplement lister using new Lister API.

Rebase

Tue, Jan 26, 3:15 PM
anlambert added inline comments to D4945: cran: Reimplement lister using new Lister API.
Tue, Jan 26, 2:42 PM
anlambert updated the diff for D4945: cran: Reimplement lister using new Lister API.

Update: Add test for parse_packaged_date and fix implementation.

Tue, Jan 26, 2:41 PM
anlambert updated the diff for D4925: debian: Reimplement lister using new Lister API.

Update: Address some @tenma comments

Tue, Jan 26, 2:23 PM
anlambert accepted D4940: gitlab: Support authentication.

Looks good !

Tue, Jan 26, 2:14 PM
anlambert added inline comments to D4925: debian: Reimplement lister using new Lister API.
Tue, Jan 26, 12:04 PM
anlambert added inline comments to D4925: debian: Reimplement lister using new Lister API.
Tue, Jan 26, 12:00 PM
anlambert added a comment to T2994: Use keyset pagination in Gitlab lister.

On a related note, I found this list of Community-Hosted GitLab Instances. Most of them have public access and could be added to the set of Gitlab instances listed by Software Heritage.

Tue, Jan 26, 11:19 AM · Origin-GitLab, Lister
anlambert triaged T2994: Use keyset pagination in Gitlab lister as Normal priority.
Tue, Jan 26, 11:08 AM · Origin-GitLab, Lister

Mon, Jan 25

anlambert requested review of D4945: cran: Reimplement lister using new Lister API.
Mon, Jan 25, 7:51 PM
anlambert added a revision to T2989: Port CRAN lister to the new Lister API: D4945: cran: Reimplement lister using new Lister API.
Mon, Jan 25, 7:48 PM · Lister
anlambert requested changes to D4940: gitlab: Support authentication.

Sorry I accepted the diff before seeing you used Basic Auth which I think is not working.

Mon, Jan 25, 7:40 PM
anlambert accepted D4940: gitlab: Support authentication.

I do not see any section regarding basic HTTP authentication for API requests in Gitlab API doc. Are you sure it is working ?

Mon, Jan 25, 7:38 PM
anlambert accepted D4944: gitlab: Add support for last_update information during listing.

You should add a test to check the last_update field value in scheduler database is not None.

Mon, Jan 25, 7:28 PM
anlambert requested review of D4925: debian: Reimplement lister using new Lister API.
Mon, Jan 25, 7:16 PM
anlambert closed D4942: tests: Fix errors after swh-scheduler API update.
Mon, Jan 25, 6:27 PM
anlambert committed rDLSea8ecee54185: tests: Fix errors after swh-scheduler API update (authored by anlambert).
tests: Fix errors after swh-scheduler API update
Mon, Jan 25, 6:27 PM
anlambert added a comment to D4940: gitlab: Support authentication.

Build has FAILED

Patch application report for D4940 (id=17591)

Rebasing onto b6a69b2ed9...

Current branch diff-target is up to date.
Changes applied before test
commit b66ffc6c441dba8d01325bf8c648ba667f14798c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Jan 25 18:15:21 2021 +0100

    gitlab: make url mandatory and add type

commit cc9cccdcc284738a3e20ea1a7a71f8c814c2ace6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Jan 25 15:32:49 2021 +0100

    gitlab: Support authentication
    
    Related to T2987

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/162/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/162/console

Mon, Jan 25, 6:22 PM
anlambert added inline comments to D4940: gitlab: Support authentication.
Mon, Jan 25, 6:21 PM
anlambert renamed T2989: Port CRAN lister to the new Lister API from Port cran lister to the new Lister API to Port CRAN lister to the new Lister API.
Mon, Jan 25, 5:52 PM · Lister
anlambert updated the diff for D4942: tests: Fix errors after swh-scheduler API update.

Remove debian/conftest.py modified by black from the commit.

Mon, Jan 25, 5:16 PM
anlambert requested review of D4942: tests: Fix errors after swh-scheduler API update.
Mon, Jan 25, 5:13 PM
anlambert added inline comments to D4940: gitlab: Support authentication.
Mon, Jan 25, 4:16 PM
anlambert added inline comments to D4940: gitlab: Support authentication.
Mon, Jan 25, 4:06 PM
anlambert accepted D4907: Reimplement Gitea lister using new Lister API.

Looks good to me !

Mon, Jan 25, 3:41 PM · Sprint 2021 01, Lister
anlambert accepted D4926: Port cgit lister to the new lister api.

Looks good to me !

Mon, Jan 25, 3:06 PM · Sprint 2021 01, Lister
anlambert claimed T2989: Port CRAN lister to the new Lister API.
Mon, Jan 25, 2:57 PM · Lister
anlambert changed the status of T2989: Port CRAN lister to the new Lister API, a subtask of T2442: Provide a unified API for listers to interact with the scheduler, from Open to Work in Progress.
Mon, Jan 25, 2:57 PM · Sprint 2021 01, Scheduling utilities
anlambert changed the status of T2989: Port CRAN lister to the new Lister API from Open to Work in Progress.
Mon, Jan 25, 2:57 PM · Lister
anlambert closed D4938: bitbucket: Pick random credentials in configuration and improve logging.
Mon, Jan 25, 2:44 PM
anlambert committed rDLS59c9abb9162b: bitbucket: Pick random credentials in configuration and improve logging (authored by anlambert).
bitbucket: Pick random credentials in configuration and improve logging
Mon, Jan 25, 2:44 PM
anlambert updated the diff for D4938: bitbucket: Pick random credentials in configuration and improve logging.

Rebase

Mon, Jan 25, 2:35 PM
anlambert accepted D4937: gitlab: Let the lister compute the internal project listing page.

Great !

Mon, Jan 25, 2:30 PM
anlambert closed D4924: model/ListedOrigin: Set extra_loader_arguments type to Dict[str, Any].
Mon, Jan 25, 2:19 PM
anlambert committed rDSCH2906b4e8a085: model/ListedOrigin: Set extra_loader_arguments type to Dict[str, Any] (authored by anlambert).
model/ListedOrigin: Set extra_loader_arguments type to Dict[str, Any]
Mon, Jan 25, 2:19 PM
anlambert added a subtask for T2442: Provide a unified API for listers to interact with the scheduler: T2987: Port gitlab lister to the new `swh.lister.pattern.Lister` API.
Mon, Jan 25, 2:18 PM · Sprint 2021 01, Scheduling utilities
anlambert added a parent task for T2987: Port gitlab lister to the new `swh.lister.pattern.Lister` API: T2442: Provide a unified API for listers to interact with the scheduler.
Mon, Jan 25, 2:18 PM · Lister
anlambert triaged T2992: Port launchpad lister to the new Lister API as Normal priority.
Mon, Jan 25, 2:17 PM · Lister
anlambert triaged T2991: Port packagist lister to the new Lister API as Normal priority.
Mon, Jan 25, 2:16 PM · Lister
anlambert lowered the priority of T2990: Port GNU lister to the new Lister API from High to Normal.
Mon, Jan 25, 2:14 PM · Origin-GNU, Lister
anlambert triaged T2990: Port GNU lister to the new Lister API as High priority.
Mon, Jan 25, 2:14 PM · Origin-GNU, Lister
anlambert requested review of D4938: bitbucket: Pick random credentials in configuration and improve logging.
Mon, Jan 25, 2:13 PM
anlambert triaged T2989: Port CRAN lister to the new Lister API as Normal priority.
Mon, Jan 25, 2:13 PM · Lister
anlambert updated the diff for D4924: model/ListedOrigin: Set extra_loader_arguments type to Dict[str, Any].

Rebase

Mon, Jan 25, 2:11 PM
anlambert added inline comments to D4907: Reimplement Gitea lister using new Lister API.
Mon, Jan 25, 2:08 PM · Sprint 2021 01, Lister
anlambert requested changes to D4907: Reimplement Gitea lister using new Lister API.
Mon, Jan 25, 2:00 PM · Sprint 2021 01, Lister
anlambert accepted D4932: gitlab: Add test on incremental implementation.

Looks good to me !

Mon, Jan 25, 12:35 PM
anlambert accepted D4933: gitlab: Add test on rate-limit support.

Looks good to me !

Mon, Jan 25, 12:33 PM
anlambert accepted D4930: gitlab: Add tests on pagination.

Looks good to me !

Mon, Jan 25, 12:28 PM
anlambert accepted D4929: gitlab: Port to the new lister api.
Mon, Jan 25, 12:17 PM