Urllib.request.urlopen Python3

I need some help with this.
I use Python 3.7.3 and 3.5.2
I noticed this when trying to get some code working on my Raspberry Pi (3.5 only)
I kept getting errors and set about fixing it…
I moved to my PC where I have both.
example code

import json
import urllib.request

def get_sensor_list():
    url = "https://aq.connectedhumber.org/api.php?action=list-devices&only-with-location=yes"
    return get_url_response(url)

def get_url_response(url):
    response = urllib.request.urlopen( url )
    json_data = json.loads( response.read() )
    return json_data

json_dict = get_sensor_list()
print(json_dict)

3.7.3 works nicely.
returns the list as expected, (same as in webbrowser)
(it’s just the code method used for the graphs.connectedhumber.org and my nearest device PM level)

3.5.2 (win10 an Pi) however errors with type error

Traceback (most recent call last):
  File "C:\Users\Ben\Desktop\url_get_test.py", line 13, in <module>
    json_dict = get_sensor_list()
  File "C:\Users\Ben\Desktop\url_get_test.py", line 6, in get_sensor_list
    return get_url_response(url)
  File "C:\Users\Ben\Desktop\url_get_test.py", line 10, in get_url_response
    json_data = json.loads( response.read() )
  File "C:\Users\Ben\AppData\Local\Programs\Python\Python35-32\lib\json\__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

Does anyone know what changed ?
All i can see is a deprecated usage of some arguments, but that shouldn’t matter.
Can anyone see or know of anything changed in 3.6 or 3.7 ?
https://docs.python.org/3/library/urllib.request.html

I can give the fix a
response.read().decode(‘utf-8’)

which is similar to what I do with serial reads.

But it seems PYthon3.7 can work out (luckily) what the stream is, so doesn’t default to binary.

I noticed, both return binary in the read if you print it out directly

def get_url_response(url):
    response = urllib.request.urlopen( url )
    print(response.read())
    json_data = json.loads( response.read() )
    return json_data

interestingly Python3.7 then fails on the next line with the json_data text read because of binary too at that point.
So just use
json_data = json.loads( response.read().decode('utf-8') )

just off to fix that in my nearest sensor post too. :slight_smile:

1 Like

Nice solution! I wonder if it’s actually because if a difference in the version of urllib that you’re using, since Python usually comes bundled with things like that pre-installed.