I created a paste P163 instead of directly commenting the results here.

date of computation: Tue May 29 2017

mean:

softwareheritage=> select avg(length) from content_language cl inner join content c on cl.id=c.sha1;
        avg
--------------------
 26862.385193867011
(1 row)

variance:

softwareheritage=> select variance(length) from content_language cl inner join content c on cl.id=c.sha1;
       variance
-----------------------
 125835685708.88915180
(1 row)

standard deviation:

softwareheritage=> select stddev(length) from content_language cl inner join content c on cl.id=c.sha1;
     stddev
-----------------
 355008.08759433
(1 row)

Those contents were extracted and stored in uffizi:/srv/storage/space/lists/content-language-id-size.txt.gz (format per line: <sha1>\t<length>).

Using https://forge.softwareheritage.org/rDSNIP49ffa63356d7bcee7ea259381d078a3a87359bed, graph https://forge.softwareheritage.org/F2250694 was drawn.

• take only the first 10k of the raw contents (as a possible configuration option).

This has been implemented, tested and deployed.
This pulled a fix on a limitation about concurrent tooling (name/version with a different configuration).

take only a percentage portion of the content (also a possible configuration option).

This has not been tested (so not implemented).

use the detected encoding from the mimetype indexer and pass along that optional information.

This has not been implemented.

The output of the mimetype indexer (based on the 'file' cli) for the encoding does not match python's encoding/decoding feature.
This would mean maintaining a translation dict somewhere in the code to convert appropriately.
That was the main reason that was not used in the initial implementation.

I did not find a simple output explaining all possible encoding from 'file' so i did not look further into that direction.

A new task about this has been added (T728).

Note:
I have extracted an 'actual' state of encodings detected though. It's an extract from content_mimetype table (as of 31/05/2017).
It's stored in uffizi:/srv/storage/space/lists/unique-encodings-found.txt (only 8 so far: binary, ebcdic, iso-8859-1, unknown-8bit, us-ascii, utf-16be, utf-16le, utf-8).