heh that's funny, olasd and I also discussed this yesterday (though we were interested in changing objects rather than deleting):

09:06:17 <+vlorentz> so anyway, the reason I was checking this out, is that we could temporarily set max.compaction.lag.ms to a low value on swh.journal.objects.directory to trigger a compaction, so we don't need to change the replayer code for https://forge.softwareheritage.org/T4644#97858
09:06:17 -- Notice(swhbot): T4644 (submitter: swh-sentry-integration, owner: vlorentz, priority High, status: Open): replayer crashes on invalid directory entry name (b'gitter/gitter.xml') <https://forge.softwareheritage.org/T4644#97858>
09:06:42 <+vlorentz> if I understand all this correctly
09:07:20 <+olasd> yeah, that's plausible
09:08:00 <+olasd> we should probably set max.compaction.lag.ms to a low value anyway, there's multiple areas where we actually want frequent compaction
09:08:07 <+olasd> (e.g. takedowns)

Should a delete_object(object_type: str, object_key: KeyType) method be added to swh.journal.writer.interface.JournalWriterInterface?

yes, I think that would make sense.

https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4658#note_12824 is the process I've followed for a (lower stakes) removal of buggy raw_extrinsic_metadata messages from swh.journal (in staging).

In that specific situation the schema for object keys in kafka had changed (and we hadn't cleaned kafka up after the change), so it made sense to use a journal client to figure out the keys of the objects that needed removal. It probably makes sense to support both options (either removal through known message keys, where you only need to instantiate a producer, or removal using a kafka client with a message matcher).

After discussing D8833 and discussing the general API with @olasd, it feels like adding a delete method to swh.journal.writer.interface.JournalWriterInterface is the wrong approach. To try to capture our discussion

The journal operation is closely aligned with Kafka’s, so deleting data from the journal actually means laying out a tombstone in the journal. We should not rely on consumers of the journal actually seeing these tombstones to handle deletions. If they lag too much, the tombstone will eventually be removed (together with the actual data) from the journal.

JournalWriterInterface is currently implemented for memory, stream and kafka:

• memory is being used to perform tests, and except from ensuring that clients will not crash when seeing tombstones, deletion should not be a concern otherwise.
• stream is being used to serialize data from the archive later used as fixtures in our tests. It does not seem to be of any interest to have tombstones in these streams.
• kafka is where we actually store the journal data.

Therefore, we only should implement delete in KafkaJournalWriter. Takedown requests and other archive mangling procedure will use this explicitly, keeping how Kafka data model works in mind.

It might be worth double-checking and eventually adding a test about how swh.journal.JournalClient handles tombstone, though.

@douardda: would you agree?

In T4657#98361, @lunar wrote:

[...]
@douardda: would you agree?

Yes.