This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

Sentry issue: SWH-LISTER-5K

Raising priority as it's been four months since the lister is stuck in production.

Our initial contact (Erik von Zijst) is no longer there, but you can try pinging other who were part of the discussion :

• TJ Kells <tkells@atlassian.com>
• Daniel Tao <dtao@atlassian.com>
• Ming Gong <mgong@atlassian.com>
• Tom Kane <tkane@atlassian.com>

In T4239#84980, @olasd wrote:

We had a mysterious error 500 issue with one buggy repository in the bitbucket API in the past, which we had reported to their Jira to no avail, but someone at Octobus (I think it was @marmoute or @Alphare ?) managed to reach out to one of the devs at atlassian to actually debug and fix the issue.

Maybe this contact could be notified again?

good idea ^

We had a mysterious error 500 issue with one buggy repository in the bitbucket API in the past, which we had reported to their Jira to no avail, but someone at Octobus (I think it was @marmoute or @Alphare ?) managed to reach out to one of the devs at atlassian to actually debug and fix the issue.

A possible workaround for this without coding anything just yet would be to upgrade the scheduler's listers record for bitbucket to the unstucking date you mention first [1]

Looks good to me, time to land this !

Build is green

Revert changes to the credentials setting method, and improve corresponding test

Build is green

Remove tenacity wait parameter

Build is green

Remove the persistent isort conftest modification from diff

Build is green

Have test coverage for credentials setting

Build is green

• apply @vlorentz suggestions
• use tenacity retrying decorator through lister.utils thanks to D4869
• remove setters no more useful after D4872
• adapt tests accordingly

Build is green

• fix assert
• fix typing
• add a warning on credentials

Thanks for the remarks, will go through them.

Sorry, I clicked "submit" too early.

Not covered: authentication, max retries, small corner cases (see below)

Build is green

Fix copyrights and rebase

Build is green

Update: Remove old range lister task and update tasks tests.

Build is green

Improved tests

lgtm (minus the build failure that is ;)

Build has FAILED

• add and improve tests

Build is green

refactor auth and backoff

They are still working on it...
(last update from 04/09/2019 mentioning it's still wip on their dev team side).

(they apparently allows the user to re-open but when actually trying to, it's not working...).

Ticket [1] is closed because i did not reply fast enough.

I have cc-ed swh-devel@inria.fr, no idea if that will work or not though

I have reported the issue to the atlassian team [1] with mostly the following analysis:

So possibly a blacklist per range of ips.

Trying to determine if it's a blacklist per ip, i rescheduled this with another worker (worker02).
It got stuck with the same error.

It's currently running now.
9k listed so far.

Maybe not, i see task-type references in the scheduler db...

It's not been deployed through the scheduler though (it most probably anterior to the scheduler).

Build is green
See https://jenkins.softwareheritage.org/job/DLS/job/tox/270/ for more details.

Update according to review

indexable_type does not seem to be needed as a class attribute; and I'm not a fan of the name.