Differential D874

ContentIndexer: convert hash to bytes.
ClosedPublic
Actions

Authored by vlorentz on Dec 20 2018, 6:35 PM.

Details

Reviewers

olasd

Group Reviewers

Reviewers

Commits

rDCIDXc5684e667fc7: ContentIndexer: convert hash to bytes.

Summary

When received via the scheduler (which manipulates JSON), it's a string.

The RevisionIndexer uses the exact same conversion code.

Test Plan

docker-compose up
python3 -m swh.loader.git.loader --origin-url https://github.com/SoftwareHeritage/swh-storage.git
echo 'indexer_mimetype;oneshot;[["6dfe5dd2ab86d1ad3677285155027332fb35e9e5"]];{"policy_update": "update-dups"}' | python3 -m swh.scheduler.cli --cls remote --url http://localhost:5008/ task schedule /dev/stdin -c type -c policy -c args -c kwargs -d ';'

Diff Detail

Repository

rDCIDX Metadata indexer

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Dec 20 2018, 6:35 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptDec 20 2018, 6:35 PM

vlorentz added a child revision: D875: docker-dev: Make the mimetype indexer work..Dec 20 2018, 6:37 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/163/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tox/163/console

Harbormaster failed remote builds in B3229: Diff 2777!Dec 20 2018, 6:40 PM

vlorentz added a parent revision: D876: Make tests actually run..Dec 20 2018, 7:56 PM

I'm not too fond of silently accepting both data types (I'd rather have limitations of the scheduler handled on the task side rather than here) but I guess that's fine.

This also probably needs test coverage so we don't break it in the future.

swh/indexer/indexer.py
339	needs to be fixed

This revision is now accepted and ready to land.Dec 21 2018, 3:37 PM

In D874#18797, @olasd wrote:

I'm not too fond of silently accepting both data types (I'd rather have limitations of the scheduler handled on the task side rather than here) but I guess that's fine.

I agree, but we already do it all over the indexers :/