Build is green

Improve docstring

Build is green

Build is green

Actually rework commit message (without forgetting to commit the message)

Build is green

Rework commit message

Build is green

• Rebase
• Adapt according to review (avoid constant)
• Update docstring to explicit the disabling of origins

Adapt according to review:

• start increment to 1 then adapt tests
• rename variable to a more meaningful name

In D5978#156375, @ardumont wrote:

Informally, if there is a single visit with a given status, then the number of
successive visits with that status is 1; but with the semantics defined here, it's 0.
This seems confusing to me; could you make counters start at 1 (unless there are no
visits at all, in which case 0 is correct)

yes, I sense it is confusing. Although starting the counter at 1 then resets it at 0
after when we switch to a new status is what's confusing to me...

Shouldn't we make that counter always start at 1 and then be reset to 1 instead?

cool

Informally, if there is a single visit with a given status, then the number of
successive visits with that status is 1; but with the semantics defined here, it's 0.
This seems confusing to me; could you make counters start at 1 (unless there are no
visits at all, in which case 0 is correct)

I'm not a huge fan of disabling origins forever; are you planning to relax this somehow? (eg. visit again a couple of years later)

I'm not a huge fan of disabling origins forever; are you planning to relax this somehow? (eg. visit again a couple of years later)

Informally, if there is a single visit with a given status, then the number of successive visits with that status is 1; but with the semantics defined here, it's 0. This seems confusing to me; could you make counters start at 1 (unless there are no visits at all, in which case 0 is correct)

Fix commit message

The probe is deployed on the monitoring: https://icinga.softwareheritage.org/dashboard#!/monitoring/service/show?host=belvedere.internal.softwareheritage.org&service=Postgresql%20replication%20lag%20%28belvedere%20-%3E%20somerset%29

Build is green

Build is green

Rebase

Add missing assertions on successive_visits

Build is green

Build is green

Build is green

Use correct commit range...

Build is green

Use correct commit range

Disable failing visits when reaching 2 (since the counter starts at 0)

Start successive_visits to 0 like in the model declaration

Mongo engine
db.serverStatus().storageEngine

"name" : "wiredTiger",
"supportsCommittedReads" : true,
"oldestRequiredTimestampForCrashRecovery" : Timestamp(0, 0),
"supportsPendingDrops" : true,
"dropPendingIdents" : NumberLong(0),
"supportsSnapshotReadConcern" : true,
"readOnly" : false,
"persistent" : true,
"backupCursorOpen" : false,
"supportsResumableIndexBuilds" : true

Still progressing. It's not fast though since it's deployed rather simply. It's doing
one origin at a time which can end up seemingly stuck behind a big repository (currently
[1]):

Deactivate failing visits (delegating to listers the act of activating back those
origins which gets live again). I have diffs which deal with this that needs some
rebase and work according to latest change (I need to get back to it) [1].

One remark inline.

Build is green

Build is green

Rebase on latest work

Rebase on latest work

Build is green

rebase

Improve the readability of the graphs

This should be resolved now (actually, on the 31st at 16).

Rehearse the run and make minor updates to make sure it runs right away this friday.

my understanding of the logs is that three OSDs (osd.3, osd.11, and osd.14 IIRC) were suddenly all stuck "waiting for subops". Unfortunately, while this is easy to debug while the issue is happening (dump_ops_in_flight), I didn't think of it at the time, given the hurry to fix the issue...

FTR I've tried to investigate a bit to find clues of what the origin of the outage was, but I did not find any obvious culprit.