Deployed the following:

It would be nice if the tests for this journal client used an actual storage with a journal writer, rather than fully mocked topics. Doing this would have caught the original breakage.

The swh-indexer stack is deployed on staging and the initial loading is done.
The volumes are quite low :

It would be nice if the tests for this journal client used an actual storage with a journal writer, rather than fully mocked topics. Doing this would have caught the original breakage.

I propose meeting in the middle and having the following policies:

and indexer 0.6.1 is now packaged. We have everything we need to unstuck it now.

In T2780#53415, @olasd wrote:

Is this supposed to be persistent (and keep the full history of all messages), or transient (and used for "real-time" clients)? IOW, what are the storage requirements for this?

Where is the list of topics that need to be created?

It's unclear what the prefix should be. swh.storage uses swh.journal.objects, we can either use that one too, or a new one, eg. swh.journal.indexed

I think we should definitely use a different prefix as swh.storage, as the ACLs for third parties should be separate.

this a description of the pipeline to clarify the interaction between the components (source: P883) :

Is this supposed to be persistent (and keep the full history of all messages), or transient (and used for "real-time" clients)? IOW, what are the storage requirements for this?

T2814 needs to be released before

Closing in favor of T2686

I like the Final version attribute idea. But to be clear, this means we will need to add successive versions as extra classes in swh.model.model when we change the schema (which is not necessarily a bad thing), and remove them when we are sure they are not around anymore

Since this "migration problem" also concerns cassandra, maybe an simple approach would be to add a Final version attribute to all model entities (a simple monotonic integer).

Thanks to @seirl using the full journal to do a graph export (and therefore having the time to check whether all objects were there), we've found a bunch of bugs in the journal backfiller / configuration preventing large objects to be added.

After resetting a local consumer to these offsets, I was completely unable to reproduce this issue.

(the backfill had, in fact, completed within a month)

At this point, I don't think we'll make it much better with postgres as source.

I suspect this hasn't happened recently

Build is green

• Rework commit message
• Reuse same string format to reduce diff stat change

Build is green

Use pprint_key function as suggested in irc ;)

Build is green

Fix according to review \m/

checking for dict instance seems completely arbitrary, and doesn't catch other non-hashable types.

I suggest this instead:

try:
    key_str = hash_to_hex(key)
except TypeError:
    key_str = repr(key)

checking for dict instance seems completely arbitrary, and doesn't catch other non-hashable types.

Build is green

In D3151#76517, @ardumont wrote:

Remove global var and put that definition local to its use

Remove global var and put that definition local to its use

Build is green

• Drop no longer required setUp method
• Specify the deinitialize/initialize in comments

Build is green

Fix tests (in the end, reset issue)

Build has FAILED

Keep initial assertions order

Build has FAILED