I'll do this tomorrow when i'm fresh.

I'll do this tomorrow when i'm fresh.

Thinking more about this.

Developed, Packaged, Deployed, indexers unstuck \m/

In the mean time, as my main concern is not the indexer, i'll work around this to avoid stopping entirely the indexers (as some batch can then be stuck in the rescheduling loop).

Sounds like a bug against python3-magic.

Example Sha1 with that error is '099c7254742e2be54a86d03a3a1826a7b8e757d0':

The old tool is id 7, the new one is 9:

Depends on T761

Bogus mimetype values are identified by the following queries:

softwareheritage=> select count(*) from content_mimetype where mimetype LIKE '[%' or mimetype like '' and indexer_configuration_id=7;
 count
-------
 50733
(1 row)

Status:

• Final listing of bogus values: /srv/storage/space/lists/indexer/mimetype/sha1-with-bogus-values.txt.gz (50733)
• Queue reached the sane point.
• workers stopped.

I am waiting for the queue to drop at 10000 as that will avoid rescheduling the already done 10000 (well except for the new bogus values :)

There might be other bogus values in the stats that I haven't noticed.

I don't see how i can easily check this though since we don't have the sha1 provenance yet.

From the top of my head, i would say that i forgot to clean up those bogus values after the initial runs around december 2016.
I don't see how i can easily check this though since we don't have the sha1 provenance yet.