In D3981#98760, @douardda wrote:

Also this description of flavors should be documented somewhere (I mean somewhere else than the ci message)

In D3981#98777, @vlorentz wrote:

@douardda deduplication means it's not read_replica, so either mirror or default.

We want deduplication for default (it's the one written to by loaders), and also for mirror so they filter out duplicates from the journal.

In D3981#98519, @vlorentz wrote:

Is this a WIP? If yes, what are the -- ? comments?

And as discussed on IRC, we might want to use unique indices in read replicas too, so we have less differences between flavors.

Yeah, this definitely needs to happen before the new swh.loader.core is deployed.

Btw, sqitch uses the native scripting for the database engine (in case of postgres, psql scripts), so if we ever end up using it, we can keep this approach.

So, I've first attempted something along the lines of https://www.depesz.com/2008/06/18/conditional-ddl/, which uses a function to execute DDL commands stored in a string.

remove spurious flag on an \else block

Begrudgingly giving this an Accept.

+1 to the idea (and to the specific config as I have already bikeshed it)

In D3972#98084, @anlambert wrote:

In D3972#98074, @olasd wrote:

I'm okay with this change. But I think other team members should have a stab at the color of the bikeshed too.

Ack, I will wait before landing this then.

I really hate this, but save from making *_add able to take generators as arguments again, this is probably the most decent way out.

It took me a bit to understand what the sorteds were achieving (dedup the list while keeping it ordered), so I wouldn't mind if you turned them into an explicit loop, or at least if you added a comment.

I'm okay with this change. But I think other team members should have a stab at the color of the bikeshed too.

In T2373#48877, @ardumont wrote:

So content_missing call explodes mid-air client side (`"POST /content/missing
HTTP/1.1" 200 9475383` so client received the data).

It so happens that the content_missing api is taking an unlimited amount of
bytes ids as input [1] And then "tries" to stream to the client the results
(rpc layer in the middle makes that moot).

In D3967#97981, @ardumont wrote:

I think this should be moved to the sysadm-provisioning repository [1] instead.

[1] https://forge.softwareheritage.org/source/swh-sysadmin-provisioning/

Definitely yes.

Fix flake8 issue that snuck past pre-commit thanks to rebase shenanigans

Thanks! This indeed looks much less fragile than the concat version.

I don't like using concat and ERB templates to generate yaml files at all:

Bye bye .ini.

In D3963#97665, @olasd wrote:

I agree with the switch to yaml, but I don't think we should have plain text yaml files in the hiera variables.

We should switch the swh::global_conf hiera variable to be a yaml dict, and the way the file is generated by the profile::swh to inline this yaml (with a "warning" header).

Or we could drop the file altogether, as it's not being used at all now.

I agree with the switch to yaml, but I don't think we should have plain text yaml files in the hiera variables.

This has been pushed to production now.

Rebase

Rebase

Rebase

Rebase

Rebase

• Improve networks profile module documentation

After doing the basic kafka config, I've done the following to create the topics: