Page MenuHomeSoftware Heritage

Force json module use to decode requests text response
ClosedPublic

Authored by anlambert on Oct 24 2019, 4:32 PM.

Details

Summary

When debugging tests execution of swh-graph, I stumbled across that error:

graph_client = <RemoteGraphClient url=http://127.0.0.1:54177/graph/>

    def test_stats(graph_client):
>       stats = graph_client.stats()

swh/graph/tests/test_api_client.py:2: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
swh/graph/client.py:36: in stats
    return self.get('stats')
../swh-core/swh/core/api/__init__.py:220: in get
    return self._decode_response(response)
../swh-core/swh/core/api/__init__.py:246: in _decode_response
    return decode_response(response)
../swh-core/swh/core/api/serializers.py:31: in decode_response
    r = response.json(cls=SWHJSONDecoder)
/usr/lib/python3/dist-packages/requests/models.py:889: in json
    self.content.decode(encoding), **kwargs
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

s = '{"counts":{"nodes":21,"edges":23},"ratios":{"compression":1.578,"bits_per_node":9.524,"bits_per_edge":8.696,"avg_loca...":3.696},"indegree":{"min":0,"max":3,"avg":1.0952380952380953},"outdegree":{"min":0,"max":3,"avg":1.0952380952380953}}', encoding = None
cls = <class 'swh.core.api.serializers.SWHJSONDecoder'>, object_hook = None, parse_float = None, parse_int = None, parse_constant = None, object_pairs_hook = None, use_decimal = False, kw = {}

    def loads(s, encoding=None, cls=None, object_hook=None, parse_float=None,
            parse_int=None, parse_constant=None, object_pairs_hook=None,
            use_decimal=False, **kw):
        """Deserialize ``s`` (a ``str`` or ``unicode`` instance containing a JSON
        document) to a Python object.
    
        *encoding* determines the encoding used to interpret any
        :class:`str` objects decoded by this instance (``'utf-8'`` by
        default).  It has no effect when decoding :class:`unicode` objects.
    
        Note that currently only encodings that are a superset of ASCII work,
        strings of other encodings should be passed in as :class:`unicode`.
    
        *object_hook*, if specified, will be called with the result of every
        JSON object decoded and its return value will be used in place of the
        given :class:`dict`.  This can be used to provide custom
        deserializations (e.g. to support JSON-RPC class hinting).
    
        *object_pairs_hook* is an optional function that will be called with
        the result of any object literal decode with an ordered list of pairs.
        The return value of *object_pairs_hook* will be used instead of the
        :class:`dict`.  This feature can be used to implement custom decoders
        that rely on the order that the key and value pairs are decoded (for
        example, :func:`collections.OrderedDict` will remember the order of
        insertion). If *object_hook* is also defined, the *object_pairs_hook*
        takes priority.
    
        *parse_float*, if specified, will be called with the string of every
        JSON float to be decoded.  By default, this is equivalent to
        ``float(num_str)``. This can be used to use another datatype or parser
        for JSON floats (e.g. :class:`decimal.Decimal`).
    
        *parse_int*, if specified, will be called with the string of every
        JSON int to be decoded.  By default, this is equivalent to
        ``int(num_str)``.  This can be used to use another datatype or parser
        for JSON integers (e.g. :class:`float`).
    
        *parse_constant*, if specified, will be called with one of the
        following strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``.  This
        can be used to raise an exception if invalid JSON numbers are
        encountered.
    
        If *use_decimal* is true (default: ``False``) then it implies
        parse_float=decimal.Decimal for parity with ``dump``.
    
        To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
        kwarg. NOTE: You should use *object_hook* or *object_pairs_hook* instead
        of subclassing whenever possible.
    
        """
        if (cls is None and encoding is None and object_hook is None and
                parse_int is None and parse_float is None and
                parse_constant is None and object_pairs_hook is None
                and not use_decimal and not kw):
            return _default_decoder.decode(s)
        if cls is None:
            cls = JSONDecoder
        if object_hook is not None:
            kw['object_hook'] = object_hook
        if object_pairs_hook is not None:
            kw['object_pairs_hook'] = object_pairs_hook
        if parse_float is not None:
            kw['parse_float'] = parse_float
        if parse_int is not None:
            kw['parse_int'] = parse_int
        if parse_constant is not None:
            kw['parse_constant'] = parse_constant
        if use_decimal:
            if parse_float is not None:
                raise TypeError("use_decimal=True implies parse_float=Decimal")
            kw['parse_float'] = Decimal
>       return cls(encoding=encoding, **kw).decode(s)
E       TypeError: __init__() got an unexpected keyword argument 'encoding'

/usr/lib/python3/dist-packages/simplejson/__init__.py:535: TypeError

Indeed, when the simplejson module is present in the Phython environment (this is a pgadmin4 dependency for instance),
requests will use it to decode json but a call to response.json(cls=...) will raise an exception as the arguments
accepted by JSONDecoder are inconsistent between the standard library json and simplejson.

See https://github.com/psf/requests/issues/4842 for more details.

So ensure to use standard json module for decoding response texts and thus avoid possible errors when the
simplejson module is present in the Python environment.

Diff Detail

Repository
rDCORE Foundations and core functionalities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 8637
Build 12560: tox-on-jenkinsJenkins
Build 12559: arc lint + arc unit

Event Timeline

vlorentz added a subscriber: vlorentz.

This should be tested. (Sorry)

This revision now requires changes to proceed.Oct 25 2019, 1:01 PM

Update: Add test for JSON decoding in decode_response

I'm going to be annoying again, but let's pick another domain (eg. example.org), swh.org is an existing website unrelated to us.

This revision is now accepted and ready to land.Oct 25 2019, 3:06 PM