To create the tables, you could use

Oh, and now that we've moved workers to have a large swap space, the issue of downloading the full packfile in ram before rejecting it should be less disruptive than it's been in the past (where the whole worker would get killed because it ran out of its memory allocation).

For now, I've disabled our hardcoding of the TCP transport for GitHub origins.

Why are we pinning tree-sitter at all?

Thanks!

Should we submit a change to dulwich to get access to, at least, the raw author/committer lines (or even better raw unparsed headers)?

Thanks!

Around half the revision rows have now been migrated to bytes offsets. The migration is still ongoing.

Thanks for working on this!

In D6874#179031, @anlambert wrote:

Does reducing content_bytes (e.g. making it 50MB) help? How about going through nginx instead of hitting gunicorn directly?

I can try but aligning docker conf with production one seems the right way to proceed.

Great, thanks!

To avoid having to care about the order in which the instances are defined, I suggest making the script called by the timer unit itself hardcode the commands for the default instance, then use run-parts on the snippets generated for the other instances.

Does reducing content_bytes (e.g. making it 50MB) help? Any logs in the storage container? How about going through nginx instead of hitting gunicorn directly?

Awesome!

The shape of POLICY_ADDITIONAL_PARAMETERS is a bit clunky, with the dict key being POLICY_ADDITIONAL_PARAMETERS[visit_type][kwarg][policy]; having the nested dict "key" be POLICY_ADDITIONAL_PARAMETERS[visit_type][policy] = kwargs would be simpler?

statsd.timing/statsd.timed do full histograms. Do we really want to keep bucketed counts for all of these values, or just a running total?

What kinds of storage timeouts? I'm not against this on principle, but I'm a bit worried that this could be masking a real bug.

On jenkins-debian1:

Adding a bunch of indexes will also make the updates by the listers slower, which may have cascading effects, so we should really avoid that if possible.

The index won't help as half of the rows of the table have this property, and you'll still have to filter and sort all the results afterwards (so the index will, at best, be used to fetch the rows less efficiently than the seq scan did). I doubt PostgreSQL will even consider the index.

After thinking about it, we could set up a blackhole /dev/null alias on pergamon, and have gandi relay mail there (pergamon is already set up to receive mail, e.g. for the forge).

Who do you want behind it? (I don't think gandi supports a "blackhole" redirection)

SGTM. Thanks!

Looks like the bogus payment requests have stopped now. Disk usage on the PAAS instance is under control.

After restarting the instance, MySQL binary logs are disabled, which should avoid us getting some confusing hidden disk usage again.

While tracking down the root issue in WordPress, I noticed (from the php-fpm error logs) that WPML's languages weren't properly initialized, so I did that in the WordPress admin interface (enabled the same languages as the rest of the website).

I used the Stripe API to cancel the fraudulous payment intent (https://stripe.com/docs/api/payment_intents/cancel). (request req_J31DYN4PJsxdkK)

Root cause analysis points to a high amount of stripe webhooks being received on/around the time of the outage.

After running the reset master command on the mysql server, the disk quota usage is back down to ~2 GB, which matches usage from files + the actual MySQL data.

Response from Gandi support

Message sent to gandi simple hosting support:

After enabling the emergency console on the gandi interface, I couldn't find the disk usage culprit with du. It soon became obvious that the disk usage came from the database (which is the only bit of the disk that's not readble via the account used to connect to the emergency console).

Well, that wasn't going to work either, so I've now re-launched the migration using id ranges, with 16 parallel queries...

Why the duplication of profile::grafana into profile::grafana::backend?