It would be nice if the tests for this journal client used an actual storage with a journal writer, rather than fully mocked topics. Doing this would have caught the original breakage.

It would be nice if the tests for this journal client used an actual storage with a journal writer, rather than fully mocked topics. Doing this would have caught the original breakage.

and indexer 0.6.1 is now packaged. We have everything we need to unstuck it now.

\o/

example: https://archive.softwareheritage.org/swh:1:cnt:a6463d2ce390990c31c2f2fa8019606721f0ca13;origin=http://git.savannah.gnu.org/git/gnugo.git/

I think it qualifies, yes

@vlorentz I'm not sure it is an easy hack, could you review the task and decide?

Yes, but you should just open a diff instead.

And does Jenkins also run code in non-master branches through the pipeline?

and apologies for the delay.

Thanks for the detailed reply 🙂, and apologies for the delay. Things are clearer now 👍

hey, I would like to contribute.

The pytest tests are succeeding in the swh-indexer module, but failing in some other modules. Since this issue pertains to only the swh-indexer module, it shouldn't cause problems, right?

Hi, I'd like to take up this issue as my first issue here :) . But before I take it up, I just had a few queries:

sample with our shiny sentry: https://sentry.softwareheritage.org/share/issue/f4a40625783b4a5588980005ddc5a5e6/

Fixed by D896.

Does it still happen? The journal client changed a lot since this task was open, including switching backend library.

Fixed by @olasd

That's the postgresql statement_timeout variable that we set for some methods on the storage backends.

In T1788#37373, @ardumont wrote:

We can investigate 2 things:

• check postgresql options to kill queries that takes too long (solely indexer-db right now) -> and find some way to report those

We can investigate 2 things:

• check postgresql options to kill queries that takes too long (solely indexer-db right now) -> and find some way to report those
• push the proxy client storage idea (started within the T1389 for the storage, currently wip) up to the indexer-storage.

Note: ... comments stack pop ... (-> been there a while apparently)

In the end, it was missing initialization data steps.

current status:

I went through all the tests in test_storage.py. It appears that only content_fossology_license_get needs to be refactored. All other storage methods return a dictionary or a list of dictionaries, where each dictionary has multiple keys.

I am familiar with the web APIs and I went through the discussion in T782. When you say output a single dictionary, I believe you mean something like this

{
  sha1: [
    {tool: TOOL, licenses: [licences]},
    {tool: TOOL, licenses: [licences]}
  ],

• swh-indexer-journal-client: Update cli to latest version
• swh-indexer-journal-client: Fix configuration change