I have made a few comments inline. All in all, as this lister makes a single request, I think we can drop the whole ratelimit handling logic which will make it much, much simpler.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Jan 15 2021
Jan 15 2021
tenma moved T2956: Port PyPI lister to the new Lister API from todo to code review on the Sprint 2021 01 board.
tenma retitled D4867: Reimplement PyPI lister using new Lister API from [WIP] Reimplement PyPI lister using new Lister API to Reimplement PyPI lister using new Lister API.
Build is green
Use requests.Response.raise_for_status to handle errors generically
Jan 14 2021
Jan 14 2021
Build is green
Rebase
Thanks for the remarks, will go through them.
Sorry, I clicked "submit" too early.
Not covered: authentication, max retries, small corner cases (see below)
tenma moved T2956: Port PyPI lister to the new Lister API from Backlog to todo on the Sprint 2021 01 board.
tenma moved T2955: Port Bitbucket lister to the new Lister API from in-progress to code review on the Sprint 2021 01 board.
tenma retitled D4843: Reimplement Bitbucket lister using new Lister API from [WIP] Reimplement Bitbucket lister using new Lister API to Reimplement Bitbucket lister using new Lister API.
Jan 13 2021
Jan 13 2021
Build is green
Fix copyrights and rebase
anlambert changed the status of T2955: Port Bitbucket lister to the new Lister API from Open to Work in Progress.
Build is green
anlambert retitled D4843: Reimplement Bitbucket lister using new Lister API from WIP new Bitbucket lister to [WIP] Reimplement Bitbucket lister using new Lister API.
Update: Remove old range lister task and update tasks tests.
Build is green
Improved tests
lgtm (minus the build failure that is ;)
Harbormaster failed remote builds in B18325: Diff 17181 for D4843: Reimplement Bitbucket lister using new Lister API!
Build has FAILED
- add and improve tests
Build is green
refactor auth and backoff
Jan 12 2021
Jan 12 2021
Jan 11 2021
Jan 11 2021
i wonder if this can be closed... was this ever implemented? and it's true that our gitweb is really a cgit instance... i wonder if *anyone* still runs gitweb with any significantly large number of repos, considering how much faster cgit is.
Dec 10 2020
Dec 10 2020
olasd triaged T2875: GNU lister/loader: no recent visits for origins that have definitely received updates as Normal priority.
Oct 26 2020
Oct 26 2020
douardda added projects to T2585: Properly handle deleted/no longer available origins: Core Loader, Lister.
Oct 23 2020
Oct 23 2020
Oct 21 2020
Oct 21 2020
Sep 30 2020
Sep 30 2020
Listed (oneshot full + recurring incremental) and loaded (as far as I can tell).
Sep 29 2020
Sep 29 2020
I've sent an email to the fsfe.
Sep 28 2020
Sep 28 2020
The lister is deployed, this forge is not listed though (codeberg.org is).
Can this be closed now? What's missing? Adding a listing task?
Sep 22 2020
Sep 22 2020
moranegg moved T1946: Improve run_a_new_lister.rst file from Backlog to developers (docs/devel/) on the Documentation board.
moranegg moved T1979: Remarks on the tutorial "Run a new lister" from Backlog to developers (docs/devel/) on the Documentation board.
Sep 21 2020
Sep 21 2020
ardumont closed T2608: Deploy launchpad and gitea listers on production, a subtask of T1734: Create a Lister for launchpad.net, as Resolved.
ardumont closed T2608: Deploy launchpad and gitea listers on production, a subtask of T2313: Archive git.fsfe.org (Gitea), as Resolved.
Sep 17 2020
Sep 17 2020
vsellier closed T2577: Test gitea lister on staging environment, a subtask of T2313: Archive git.fsfe.org (Gitea), as Resolved.
everything seems to work well, the production deployment will be done in T2608
Sep 15 2020
Sep 15 2020
We'll check with @anlambert so it lands soon and be deployed.
ardumont closed T2070: lister: Failure during listing on a fresh new db (unicity constraint) as Resolved.
ardumont added a comment to T2070: lister: Failure during listing on a fresh new db (unicity constraint).
Possibly related to [1] fixed by [2]
Sep 14 2020
Sep 14 2020
That doesn't seem good/normal, does it?
In T2577#48486, @vsellier wrote:An email was sent on the swh-devel mailing list to ask for reviews.
The deployment in production will be performed in the middle of week 38 is no problems are raised.
An email was sent on the swh-devel mailing list to ask for reviews.
The deployment in production will be performed in the middle of week 38 is no problems are raised.
An email was sent on the swh-devel mailing list to ask for reviews.
The deployment in production will be performed in the middle of week 38 is no problems are raised.
Sep 10 2020
Sep 10 2020
vsellier reopened T2577: Test gitea lister on staging environment, a subtask of T2313: Archive git.fsfe.org (Gitea), as Work in Progress.
reopened to validate to complete process from the listing to the loading of some repository
vsellier closed T2577: Test gitea lister on staging environment, a subtask of T2313: Archive git.fsfe.org (Gitea), as Resolved.
The test of new version v0.1.4 including the fix on the the range split, the uid change and the incremental task fix is ok.
Build is green
Rework commit message
add --doctest-modules to the arguments of pytest in tox.ini
Tested in the docker-environment, the problem is not reproduced anymore with 5 concurrent listers.
add --doctest-modules to the arguments of pytest in tox.ini