swh/indexer/storage/db.py
135	`cm.mimetype like 'text/%'` `binary` only means that libmagic has no idea what the mimetype is; so `cm.mimetype != 'binary'` would not exclude binary content that libmagic could recognize.
147	^^

This revision is now accepted and ready to land.Nov 19 2018, 4:40 PM

Remove spurrious print statement

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/56/ for more details.

Harbormaster completed remote builds in B2455: Diff 2133.Nov 19 2018, 5:42 PM

ardumont added inline comments.Nov 19 2018, 5:48 PM

swh/indexer/storage/db.py
135	binary only means that libmagic has no idea what the mimetype is; so cm.mimetype != 'binary' would not exclude binary content that libmagic could recognize. Right! I was just re-implementing as it was before [1] Not that this cannot change, of course ;) [1]D630#inline-3605 cm.mimetype like 'text/%' Seems better and this seems to be using index as well so \m/.
147	barf!

vlorentz added inline comments.Nov 19 2018, 5:49 PM

swh/indexer/storage/db.py
135	It used to be on the encoding, not on the mimetype.

vlorentz added inline comments.Nov 19 2018, 5:51 PM

swh/indexer/storage/db.py
135	Seems better and this seems to be using index as well so \m/. Oh, I did not consider the performance issues of doing that. It's nice that it works out of the box :)

Improve filtering instructions as per @vlorentz's suggestion ;)
Fix hypothesis warning about twice @given wrapping (I did not see earlier)

ardumont marked an inline comment as done.Nov 19 2018, 5:56 PM

ardumont added inline comments.

swh/indexer/tests/storage/test_storage.py
1753	Double @given -> warnings.

Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/57/ for more details.

Harbormaster completed remote builds in B2457: Diff 2135.Nov 19 2018, 5:59 PM

Closed by commit rDCIDX469103d42691: tasks: Open new fossology license range indexer task (authored by ardumont). · Explain WhyNov 19 2018, 6:18 PM

This revision was automatically updated to reflect the committed changes.

huh, right!
Bug!?

In D677#14063, @ardumont wrote:

huh, right!
Bug!?

No, it's ok. I see that now that i slept ;)
We used to check for the encoding not being binary and now we check for the 'mimetype' in the form 'text/%'.
I thought that we detected an old bug. It was late ;)