Page MenuHomeSoftware Heritage

Allow application/x-msgpack deserialization again
AbandonedPublic

Authored by ardumont on Tue, Oct 5, 12:20 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Maniphest Tasks
T3632: Investigate the ContentDisallowed exception
Summary

There is stacktrace all over the place about refusing to msgpack deserialization [1] for
some reason. Our default configuration is supposed to already allow it.

The doc says it supports both the name and the mimetype [2].
So as a tryout/short in the dark, here we go.

It's not tested though.

[1]

ContentDisallowed('Refusing to deserialize untrusted content of type
msgpack (application/x-msgpack)')

[2] https://docs.celeryproject.org/en/stable/userguide/configuration.html#std-setting-accept_content

Related to T3632

Test Plan

tox still happy

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 24224
Build 37806: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 37805: arc lint + arc unit

Event Timeline

ardumont retitled this revision from Allow application/x-msgpack deserialization to Allow application/x-msgpack deserialization again.Tue, Oct 5, 12:20 PM

Build is green

Patch application report for D6410 (id=23311)

Could not rebase; Attempt merge onto ecc14007aa...

Updating ecc1400..eaafbf1
Fast-forward
 swh/scheduler/celery_backend/config.py | 2 +-
 swh/scheduler/task.py                  | 9 ++++++---
 2 files changed, 7 insertions(+), 4 deletions(-)
Changes applied before test
commit eaafbf1cca64a0da83cf4c42d001e810f6ccba39
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Oct 5 12:07:55 2021 +0200

    Allow application/x-msgpack deserialization
    
    There is stacktrace all over the place about refusing to msgpack deserialization [1]
    
    This is a tryout
    
    [1]
ContentDisallowed('Refusing to deserialize untrusted content of type
msgpack (application/x-msgpack)')
```

Related to T3632

commit edb5b860959911468ca35f52887ea58fe32ad16e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date: Sat Oct 2 18:46:51 2021 +0200

Respect task configuration to allow ignoring task result event

This is needed to actually deactivate sending their result for most of the recurring
tasks to the listener.

Related to T3458
See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/453/ for more details.

as @vlorentz pointed out [1], this change should be irrelevant though...

[1] https://github.com/celery/kombu/blob/master/kombu/serialization.py#L369-L372

does not seem to be the proper fix.

FTR, using the celery cli tool directly from a development venv to interact with the celery server running in the docker compose test setup (as described there ) used to work ok, but not any more.

One have to specify the app, like:

celery --app=swh.scheduler.celery_backend.config.app status

[edit] I use celery 5.1.2 in my venv.

as @vlorentz pointed out [1], this change should be irrelevant though...

[1] https://github.com/celery/kombu/blob/master/kombu/serialization.py#L369-L372

does not seem to be the proper fix.

FTR, using the celery cli tool directly from a development venv to interact with the celery server running in the docker compose test setup (as described there ) used to work ok, but not any more.

One have to specify the app, like:

celery --app=swh.scheduler.celery_backend.config.app status

[edit] I use celery 5.1.2 in my venv.

See D6444