indexer: Remove redundant conditional check
indexer.storage: Filter out textual data when needed
- mimetype: no filtering is done.
- fossology_license: Filtering to only compute on textual contents.
Related T1312
Details
Details
Diff Detail
Diff Detail
- Repository
- rDCIDX Metadata indexer
- Lint
Automatic diff as part of commit; lint not applicable. - Unit
Automatic diff as part of commit; unit tests not applicable.
Event Timeline
Comment Actions
Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/55/ for more details.
Comment Actions
Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/56/ for more details.
| swh/indexer/storage/db.py | ||
|---|---|---|
| 136 ↗ | (On Diff #2131) |
Right! I was just re-implementing as it was before [1]
Seems better and this seems to be using index as well so \m/. |
| 148 ↗ | (On Diff #2131) | barf! |
| swh/indexer/storage/db.py | ||
|---|---|---|
| 136 ↗ | (On Diff #2131) | It used to be on the encoding, not on the mimetype. |
| swh/indexer/storage/db.py | ||
|---|---|---|
| 136 ↗ | (On Diff #2131) |
Oh, I did not consider the performance issues of doing that. It's nice that it works out of the box :) |
Comment Actions
- Improve filtering instructions as per @vlorentz's suggestion ;)
- Fix hypothesis warning about twice @given wrapping (I did not see earlier)
| swh/indexer/tests/storage/test_storage.py | ||
|---|---|---|
| 1754 ↗ | (On Diff #2133) | Double @given -> warnings. |
Comment Actions
Build is green
See https://jenkins.softwareheritage.org/job/DCIDX/job/tox/57/ for more details.
Comment Actions
No, it's ok. I see that now that i slept ;)
We used to check for the encoding not being binary and now we check for the 'mimetype' in the form 'text/%'.
I thought that we detected an old bug. It was late ;)