I add some issues with utf-8 decoding errors so now i use file early enough to help determine the encoding of the files and enrich the result.

So now, i exploit the encoding field to:

• reduce decoding errors
• filter out 'binary' encoding from language programming detection and ctags computations (it made no sense to me to permit it but i could be wrong).

Note that it's still possible to have decoding failures (since some encoding from the tool file make possibly no sense for python). Those contents are marked as such with a 'decoding_failure' key entry.

I also add lots of issues with many recognition tools which expects as default to have some default inputs:

• ctags
• github-linguist
• ohcount (blackduck's)

In some form or another, they expect either the filename (for the extension surely) or the language to be provided. If they don't, they simply don't do anything.

Anyway, that's why now, i chose randomly one filename from the cache.

The library python3-pygments works by analyzing the code (well it has some heuristics as well, the shebang for one) but still it works well.

I also tried to exploit the language name but it cannot work.
ctags knows, by default only 40 languages or so (42 i think), but python3-pygments (which i use here) now 404...
So that cannot go well even by a long shot when mapping...

The need evolved - cf. T574