Although I feel this is more of arrow library issue rather than swh issue.
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Aug 12 2019
Aug 12 2019
Thanks @nahimilega, changing locale worked out fine
Aug 8 2019
Aug 8 2019
I solved this issue by changing my system locale to 'en_US'.
You can refer to these tutorials to change your system
https://www.tecmint.com/set-system-locales-in-linux/
https://askubuntu.com/questions/89976/how-do-i-change-the-default-locale-in-ubuntu-server
Jul 19 2019
Jul 19 2019
ardumont closed T1923: Implement Packagist Lister, a subtask of T1776: packagist (PHP) Lister, as Resolved.
Jul 18 2019
Jul 18 2019
nahimilega closed T1890: pypi lister: Add tests as Resolved by committing rDLS08ade29e6de0: swh.lister.pypi: Add tests.
Jul 14 2019
Jul 14 2019
nahimilega added a revision to T1890: pypi lister: Add tests: D1733: swh.lister.core: Add tests for simple lister.
Jul 10 2019
Jul 10 2019
ardumont added a comment to T1906: list-gitlab-full on gitlab.com crashes with: TypeError: 'NoneType' object cannot be interpreted as an integer.
I think it's related to T1863.
ardumont updated the task description for T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing.
vlorentz updated the task description for T1906: list-gitlab-full on gitlab.com crashes with: TypeError: 'NoneType' object cannot be interpreted as an integer.
Jul 8 2019
Jul 8 2019
Jul 4 2019
Jul 4 2019
ardumont updated the title for P459 ugly bitbucket lister workaround from irk: unstuck bitbucket lister to ugly unstucking bitbucket lister.
ardumont added projects to P459 ugly bitbucket lister workaround: Lister, Bitbucket lister, Origin-Bitbucket.
I have cc-ed swh-devel@inria.fr, no idea if that will work or not though
ardumont changed the status of T1859: bitbucket lister: Deal with connection errors from Open to Work in Progress.
I have reported the issue to the atlassian team [1] with mostly the following analysis:
So possibly a blacklist per range of ips.
Jul 3 2019
Jul 3 2019
Jul 1 2019
Jul 1 2019
ardumont renamed T1835: List/Ingest major cgit instances from List major cgit instances to List/Ingest major cgit instances.
Getting gnu-savannah instance out of the task (T1451 exists for it already).
ardumont renamed T1865: gitlab lister: make full listing on large instance more robust to concurrency writings from gitlab lister: full listing on large instance does not handle correctly concurrency writings to gitlab lister: make full listing on large instance more robust to concurrency writings.
ardumont updated the task description for T1866: gitlab-lister: Investigate rate limit implementation further.
Jun 30 2019
Jun 30 2019
I did not find anything specific to rate limit policy for salsa.
I found an api usage best practices chapter though [1].
ardumont triaged T1866: gitlab-lister: Investigate rate limit implementation further as Normal priority.
ardumont renamed T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing from gitlab lister: Full lister fails to retrieve the range information it needs to start listing to gitlab.com: Full lister fails to retrieve the range information it needs to start listing.
ardumont added a comment to T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing.
I'll just concentrate on fixing the urls and respawn those tasks for now.
ardumont changed the status of T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing from Open to Work in Progress.
Heads up.
Jun 29 2019
Jun 29 2019
ardumont updated the task description for T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.
ardumont updated the task description for T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing.
ardumont triaged T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing as High priority.
ardumont updated the task description for T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.
ardumont added a comment to T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.
I'd be inclined to use a composition of solution:
- having the listing policy determined at cgit instance lister initialization (3.)
- Since 1. has already been done in the past, use that instead as a fallback (eclipse and freedesktop might be popular enough to sustain the load for a tad more requests than the current bare listing we do).
ardumont renamed T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern from cgit lister adaptations to deal with cgit.freedesktop.org to cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.
Jun 28 2019
Jun 28 2019
Related T1659?
It's not really related, because gitweb and cgit are two different things.
ardumont updated the task description for T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.
Getting cgit.freedesktop.org out of the scope of this task.
See T1861 for it.
ardumont removed a subtask for T1835: List/Ingest major cgit instances: T1451: ingest GNU Savannah Git repositories.
ardumont updated the task description for T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.
ardumont triaged T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern as Normal priority.
The instances (all except freedesktop) are done being listed.
nahimilega changed the status of T1835: List/Ingest major cgit instances from Work in Progress to Open.
ardumont changed the status of T1835: List/Ingest major cgit instances from Open to Work in Progress.
ardumont changed the status of T1451: ingest GNU Savannah Git repositories, a subtask of T1835: List/Ingest major cgit instances, from Open to Work in Progress.
Here we go, it's starting...
swh-lister=> select instance, count(*) from cgit_repo group by instance; instance | count --------------+------- gnu-kernel | 1002 gnu-savannah | 1018 tor | 492 (1 row)
ardumont added a subtask for T1835: List/Ingest major cgit instances: T1451: ingest GNU Savannah Git repositories.
swh-public-ci added a comment to D1660: Split models into smaller chunks to avoid oversized db transactions.
Build has FAILED
Harbormaster failed remote builds in B6557: Diff 5556 for D1660: Split models into smaller chunks to avoid oversized db transactions!
ardumont added a project to D1660: Split models into smaller chunks to avoid oversized db transactions: Lister.
Trying to determine if it's a blacklist per ip, i rescheduled this with another worker (worker02).
It got stuck with the same error.
ardumont added projects to T1859: bitbucket lister: Deal with connection errors: Bitbucket lister, Lister.
http://git.upsilon.cc/ (@zack's git repositories) might be relevant.
Jun 26 2019
Jun 26 2019
It's currently running now.
9k listed so far.
Maybe not, i see task-type references in the scheduler db...
It's not been deployed through the scheduler though (it most probably anterior to the scheduler).