Page MenuHomeSoftware Heritage
Feed Advanced Search

Mon, Aug 12

vlorentz added a comment to T1947: Unsupported locale 'en_IN' in arrow lib.

Although I feel this is more of arrow library issue rather than swh issue.

Mon, Aug 12, 4:08 PM · Scheduling utilities
GoodwillHunter added a comment to T1947: Unsupported locale 'en_IN' in arrow lib.

Thanks @nahimilega, changing locale worked out fine

Mon, Aug 12, 3:30 PM · Scheduling utilities

Thu, Aug 8

nahimilega updated subscribers of T1947: Unsupported locale 'en_IN' in arrow lib.

I solved this issue by changing my system locale to 'en_US'.
You can refer to these tutorials to change your system
https://www.tecmint.com/set-system-locales-in-linux/
https://askubuntu.com/questions/89976/how-do-i-change-the-default-locale-in-ubuntu-server

Thu, Aug 8, 10:28 PM · Scheduling utilities
GoodwillHunter triaged T1947: Unsupported locale 'en_IN' in arrow lib as Normal priority.
Thu, Aug 8, 10:10 PM · Scheduling utilities
nahimilega added a project to T1946: Improve run_a_new_lister.rst file: Lister.
Thu, Aug 8, 10:00 PM · Lister

Jul 19 2019

ardumont closed T1923: Implement Packagist Lister, a subtask of T1776: packagist (PHP) Lister, as Resolved.
Jul 19 2019, 4:48 PM · Lister, Archive coverage
ardumont closed T1923: Implement Packagist Lister as Resolved.
Jul 19 2019, 4:48 PM · Lister, Archive coverage
ardumont added a revision to T1923: Implement Packagist Lister: D1584: swh.lister.packagist.
Jul 19 2019, 4:36 PM · Lister, Archive coverage
ardumont triaged T1924: Deploy packagist Lister as Normal priority.
Jul 19 2019, 4:36 PM · Lister, Archive coverage
ardumont triaged T1923: Implement Packagist Lister as Normal priority.
Jul 19 2019, 4:35 PM · Lister, Archive coverage

Jul 18 2019

nahimilega closed T1890: pypi lister: Add tests as Resolved by committing rDLS08ade29e6de0: swh.lister.pypi: Add tests.
Jul 18 2019, 6:50 PM · Origin-Pypi, Lister

Jul 14 2019

nahimilega added a revision to T1890: pypi lister: Add tests: D1733: swh.lister.core: Add tests for simple lister.
Jul 14 2019, 8:28 PM · Origin-Pypi, Lister

Jul 10 2019

ardumont added a comment to T1906: list-gitlab-full on gitlab.com crashes with: TypeError: 'NoneType' object cannot be interpreted as an integer.

I think it's related to T1863.

Jul 10 2019, 10:19 AM · Origin-GitLab, Lister
ardumont updated the task description for T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing.
Jul 10 2019, 10:17 AM · Lister
vlorentz triaged T1906: list-gitlab-full on gitlab.com crashes with: TypeError: 'NoneType' object cannot be interpreted as an integer as Normal priority.
Jul 10 2019, 10:16 AM · Origin-GitLab, Lister
vlorentz updated the task description for T1906: list-gitlab-full on gitlab.com crashes with: TypeError: 'NoneType' object cannot be interpreted as an integer.
Jul 10 2019, 10:13 AM · Origin-GitLab, Lister
vlorentz created T1906: list-gitlab-full on gitlab.com crashes with: TypeError: 'NoneType' object cannot be interpreted as an integer.
Jul 10 2019, 10:12 AM · Origin-GitLab, Lister

Jul 8 2019

ardumont triaged T1890: pypi lister: Add tests as Normal priority.
Jul 8 2019, 10:29 AM · Origin-Pypi, Lister

Jul 4 2019

ardumont updated the title for P459 ugly unstucking bitbucket lister from irk: unstuck bitbucket lister to ugly unstucking bitbucket lister.
Jul 4 2019, 4:58 PM · Origin-Bitbucket, Bitbucket lister, Lister
ardumont added projects to P459 ugly unstucking bitbucket lister: Lister, Bitbucket lister, Origin-Bitbucket.
Jul 4 2019, 4:57 PM · Origin-Bitbucket, Bitbucket lister, Lister
ardumont updated the task description for T1859: bitbucket lister: Deal with connection errors.
Jul 4 2019, 4:36 PM · Lister, Bitbucket lister
ardumont added a comment to T1859: bitbucket lister: Deal with connection errors.

I have cc-ed swh-devel@inria.fr, no idea if that will work or not though

Jul 4 2019, 4:31 PM · Lister, Bitbucket lister
ardumont changed the status of T1859: bitbucket lister: Deal with connection errors from Open to Work in Progress.
Jul 4 2019, 11:30 AM · Lister, Bitbucket lister
ardumont added a comment to T1859: bitbucket lister: Deal with connection errors.

I have reported the issue to the atlassian team [1] with mostly the following analysis:

Jul 4 2019, 11:30 AM · Lister, Bitbucket lister
ardumont added a comment to T1859: bitbucket lister: Deal with connection errors.

So possibly a blacklist per range of ips.

Jul 4 2019, 10:49 AM · Lister, Bitbucket lister
ardumont updated the task description for T1859: bitbucket lister: Deal with connection errors.
Jul 4 2019, 10:01 AM · Lister, Bitbucket lister
ardumont updated the task description for T1859: bitbucket lister: Deal with connection errors.
Jul 4 2019, 10:01 AM · Lister, Bitbucket lister

Jul 3 2019

ardumont placed T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing up for grabs.
Jul 3 2019, 3:26 PM · Lister

Jul 1 2019

ardumont renamed T1835: List/Ingest major cgit instances from List major cgit instances to List/Ingest major cgit instances.
Jul 1 2019, 4:52 PM · Lister
ardumont added a comment to T1835: List/Ingest major cgit instances.

Getting gnu-savannah instance out of the task (T1451 exists for it already).

Jul 1 2019, 4:51 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jul 1 2019, 4:04 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jul 1 2019, 4:04 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jul 1 2019, 4:04 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jul 1 2019, 4:03 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jul 1 2019, 4:01 PM · Lister
ardumont renamed T1865: gitlab lister: make full listing on large instance more robust to concurrency writings from gitlab lister: full listing on large instance does not handle correctly concurrency writings to gitlab lister: make full listing on large instance more robust to concurrency writings.
Jul 1 2019, 10:16 AM · Lister
ardumont updated the task description for T1866: gitlab-lister: Investigate rate limit implementation further.
Jul 1 2019, 10:04 AM · Lister

Jun 30 2019

ardumont added a comment to T1866: gitlab-lister: Investigate rate limit implementation further.

I did not find anything specific to rate limit policy for salsa.
I found an api usage best practices chapter though [1].

Jun 30 2019, 6:50 PM · Lister
ardumont triaged T1866: gitlab-lister: Investigate rate limit implementation further as Normal priority.
Jun 30 2019, 11:55 AM · Lister
ardumont added a project to T1865: gitlab lister: make full listing on large instance more robust to concurrency writings: Lister.
Jun 30 2019, 11:08 AM · Lister
ardumont renamed T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing from gitlab lister: Full lister fails to retrieve the range information it needs to start listing to gitlab.com: Full lister fails to retrieve the range information it needs to start listing.
Jun 30 2019, 11:00 AM · Lister
ardumont added a comment to T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing.

I'll just concentrate on fixing the urls and respawn those tasks for now.

Jun 30 2019, 10:56 AM · Lister
ardumont changed the status of T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing from Open to Work in Progress.

Heads up.

Jun 30 2019, 10:47 AM · Lister

Jun 29 2019

ardumont updated the task description for T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.
Jun 29 2019, 3:57 PM · CGit lister, Lister
ardumont updated the task description for T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing.
Jun 29 2019, 9:54 AM · Lister
ardumont triaged T1863: gitlab.com: Full lister fails to retrieve the range information it needs to start listing as High priority.
Jun 29 2019, 9:53 AM · Lister
ardumont updated the task description for T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.
Jun 29 2019, 9:15 AM · CGit lister, Lister
ardumont added a comment to T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.

I'd be inclined to use a composition of solution:

  • having the listing policy determined at cgit instance lister initialization (3.)
  • Since 1. has already been done in the past, use that instead as a fallback (eclipse and freedesktop might be popular enough to sustain the load for a tad more requests than the current bare listing we do).
Jun 29 2019, 9:13 AM · CGit lister, Lister
ardumont renamed T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern from cgit lister adaptations to deal with cgit.freedesktop.org to cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.
Jun 29 2019, 9:10 AM · CGit lister, Lister

Jun 28 2019

ardumont added a comment to T1800: gitweb lister.

Related T1659?

It's not really related, because gitweb and cgit are two different things.

Jun 28 2019, 8:48 PM · Lister
ardumont updated the task description for T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.
Jun 28 2019, 8:36 PM · CGit lister, Lister
ardumont added a comment to T1835: List/Ingest major cgit instances.

Getting cgit.freedesktop.org out of the scope of this task.
See T1861 for it.

Jun 28 2019, 8:14 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 8:14 PM · Lister
ardumont removed a subtask for T1835: List/Ingest major cgit instances: T1451: ingest GNU Savannah Git repositories.
Jun 28 2019, 8:12 PM · Lister
ardumont closed T1835: List/Ingest major cgit instances as Resolved.
Jun 28 2019, 8:12 PM · Lister
ardumont updated the task description for T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern.
Jun 28 2019, 8:11 PM · CGit lister, Lister
ardumont triaged T1861: cgit lister: Adapt lister to deal with inconsistent `git clone uri` pattern as Normal priority.
Jun 28 2019, 8:11 PM · CGit lister, Lister
ardumont added a comment to T1835: List/Ingest major cgit instances.

The instances (all except freedesktop) are done being listed.

Jun 28 2019, 8:09 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 8:08 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 8:07 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 8:06 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 7:54 PM · Lister
ardumont claimed T1835: List/Ingest major cgit instances.
Jun 28 2019, 7:37 PM · Lister
nahimilega changed the status of T1835: List/Ingest major cgit instances from Work in Progress to Open.
Jun 28 2019, 7:36 PM · Lister
ardumont claimed T1835: List/Ingest major cgit instances.
Jun 28 2019, 7:34 PM · Lister
ardumont changed the status of T1835: List/Ingest major cgit instances from Open to Work in Progress.
Jun 28 2019, 7:33 PM · Lister
ardumont changed the status of T1451: ingest GNU Savannah Git repositories, a subtask of T1835: List/Ingest major cgit instances, from Open to Work in Progress.
Jun 28 2019, 7:33 PM · Lister
ardumont added a comment to T1835: List/Ingest major cgit instances.

Here we go, it's starting...

swh-lister=> select instance, count(*) from cgit_repo group by instance;
   instance   | count
--------------+-------
 gnu-kernel   |  1002
 gnu-savannah |  1018
 tor          |   492
(1 row)
Jun 28 2019, 7:24 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 7:23 PM · Lister
nahimilega updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 7:22 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 7:02 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 6:49 PM · Lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 6:48 PM · Lister
ardumont updated subscribers of T1835: List/Ingest major cgit instances.
Jun 28 2019, 5:39 PM · Lister
ardumont added a subtask for T1835: List/Ingest major cgit instances: T1451: ingest GNU Savannah Git repositories.
Jun 28 2019, 5:26 PM · Lister
swh-public-ci added a comment to D1660: Split models into smaller chunks to avoid oversized db transactions.

Build has FAILED

Jun 28 2019, 3:54 PM · Lister
Harbormaster failed remote builds in B6557: Diff 5556 for D1660: Split models into smaller chunks to avoid oversized db transactions!
Jun 28 2019, 3:54 PM · Lister
anlambert closed D1660: Split models into smaller chunks to avoid oversized db transactions.
Jun 28 2019, 3:51 PM · Lister
ardumont accepted D1660: Split models into smaller chunks to avoid oversized db transactions.
Jun 28 2019, 3:50 PM · Lister
ardumont added a project to D1660: Split models into smaller chunks to avoid oversized db transactions: Lister.
Jun 28 2019, 3:50 PM · Lister
ardumont added a comment to T1859: bitbucket lister: Deal with connection errors.

Trying to determine if it's a blacklist per ip, i rescheduled this with another worker (worker02).
It got stuck with the same error.

Jun 28 2019, 3:06 PM · Lister, Bitbucket lister
ardumont added projects to T1859: bitbucket lister: Deal with connection errors: Bitbucket lister, Lister.
Jun 28 2019, 3:05 PM · Lister, Bitbucket lister
ardumont updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 12:02 PM · Lister
anlambert updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 12:01 PM · Lister
ardumont removed a project from T1835: List/Ingest major cgit instances: CGit lister.
Jun 28 2019, 11:59 AM · Lister
anlambert updated the task description for T1835: List/Ingest major cgit instances.
Jun 28 2019, 11:44 AM · Lister
ardumont added a comment to T1835: List/Ingest major cgit instances.

http://git.upsilon.cc/ (@zack's git repositories) might be relevant.

Jun 28 2019, 11:38 AM · Lister

Jun 26 2019

ardumont closed T1849: Deploy bitbucket lister as Resolved.

It's currently running now.
9k listed so far.

Jun 26 2019, 3:08 PM · Bitbucket lister, Lister
ardumont closed T1826: bitbucket lister does not work as Resolved.
Jun 26 2019, 3:08 PM · Bitbucket lister, Lister
ardumont updated the task description for T1849: Deploy bitbucket lister.
Jun 26 2019, 1:32 PM · Bitbucket lister, Lister
ardumont added a comment to T1849: Deploy bitbucket lister.

Maybe not, i see task-type references in the scheduler db...

Jun 26 2019, 1:32 PM · Bitbucket lister, Lister
ardumont updated the task description for T1849: Deploy bitbucket lister.
Jun 26 2019, 1:31 PM · Bitbucket lister, Lister
ardumont added a comment to T1849: Deploy bitbucket lister.

It's not been deployed through the scheduler though (it most probably anterior to the scheduler).

Jun 26 2019, 1:29 PM · Bitbucket lister, Lister
ardumont updated the task description for T1849: Deploy bitbucket lister.
Jun 26 2019, 1:24 PM · Bitbucket lister, Lister
ardumont updated the task description for T1849: Deploy bitbucket lister.
Jun 26 2019, 1:24 PM · Bitbucket lister, Lister
ardumont changed the status of T1849: Deploy bitbucket lister from Open to Work in Progress.
Jun 26 2019, 1:21 PM · Bitbucket lister, Lister
ardumont closed D1634: lister: Type correctly the 'indexable' column.
Jun 26 2019, 11:19 AM · Bitbucket lister, Lister
ardumont closed D1632: bitbucket: Allow incremental lister to start properly the first time.
Jun 26 2019, 11:19 AM · Bitbucket lister, Lister
ardumont closed D1635: indexing-lister: Allow to define flush packet size.
Jun 26 2019, 11:19 AM · Bitbucket lister, Lister
ardumont closed D1633: bitbucket: Unify logging instructions.
Jun 26 2019, 11:19 AM · Bitbucket lister, Lister