I think this should be hardened a bit. We could:

use chardet to try to detect the actual encoding of the content
or pass the encoding information from the caller (I guess the mime type detection also does some sort of encoding detection)
or add a errors='replace' to avoid exploding on invalid utf-8

This is not really explicit in the diff but the input content_data is guaranteed to be UTF-8 encoded if the content is textual.

When fetching content from the archive, the [[ https://forge.softwareheritage.org/source/swh-web/browse/master/swh/web/browse/utils.py$148-239 | request_content ]] function is used and will encode any textual content to UTF-8 before passing it to the prepare_content_for_display function.

Nevertheless, that code is quite a mess and should be improved / simplified but this is out of scope for that diff.

The tests should also be expanded to have more "adversarial" contents (legacy encodings, mojibake, ...).

We already have those kind of tests at an upper level in tests/browse/views/test_content.py. Refactoring the content preprocessing pipeline will allow us to add better tests though.

Finally, if that's possible, it'd be great to allow users to override the encoding detected in the UI, like we do for highlighting (but that's clearly out of scope for this diff).

Agreed, it exists cases where the detected encoding will not be the right one.

or add a errors='replace' to avoid exploding on invalid utf-8

I will add that just in case.

+1 on errors='replace'.

The tests should also be expanded to have more "adversarial" contents (legacy encodings, mojibake, ...).

We already have those kind of tests at an upper level in tests/browse/views/test_content.py

no we don't. There's only one test for non-utf8 text, and it's on a well-formed utf-16le file.