Understanding Python Errors

Reading errors

By now, you have likely encountered Python errors very often. Here is an example of an error:

In [1]:
s = 'hello'
s[0] = 'a'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-07a9845b0745> in <module>()
      1 s = 'hello'
----> 2 s[0] = 'a'

TypeError: 'str' object does not support item assignment

This is called an exception traceback. The exception is the error itself, and the traceback is the information that shows where it occured. The above traceback is quite simple (because the code producing the error is quite simple). The most important thing you should look at in an error is the last line, in this case:

'str' object does not support item assignment

Now let's look at a slightly more complex error:

In [2]:
import numpy as np

def subtract_smooth(x, y):
    y_new = y - median_filter(x, y, 1.)
    return y_new

def median_filter(x, y, width):
    y_new = np.zeros(y.shape)
    for i in range(len(x)):
        y_new[i] = np.median(y[np.abs(x - x[i]) < width * 0.5])
    return y_new
In [3]:
subtract_smooth(np.array([1,2,3,4,5]),np.array([4,5,6,8]))
/Users/tom/miniconda3/envs/dev/lib/python3.6/site-packages/ipykernel_launcher.py:10: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 4 but corresponding boolean dimension is 5
  # Remove the CWD from sys.path while we load stuff.
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-3-7bd1930fb255> in <module>()
----> 1 subtract_smooth(np.array([1,2,3,4,5]),np.array([4,5,6,8]))

<ipython-input-2-202311ce8ba8> in subtract_smooth(x, y)
      2 
      3 def subtract_smooth(x, y):
----> 4     y_new = y - median_filter(x, y, 1.)
      5     return y_new
      6 

<ipython-input-2-202311ce8ba8> in median_filter(x, y, width)
      8     y_new = np.zeros(y.shape)
      9     for i in range(len(x)):
---> 10         y_new[i] = np.median(y[np.abs(x - x[i]) < width * 0.5])
     11     return y_new

IndexError: index 4 is out of bounds for axis 1 with size 4

The error is now more complex. The first line shows what top-level code was executed when the error occured - in this case the call to subtract_smooth:

<ipython-input-17-7bd1930fb255> in <module>()
----> 1 subtract_smooth(np.array([1,2,3,4,5]),np.array([4,5,6,8]))

The next chunk shows where the error occured inside subtract_smooth:

<ipython-input-18-4b1062b3783d> in subtract_smooth(x, y)
      1 def subtract_smooth(x, y):
----> 2     y_new = y - median_filter(x, y, 1.)
      3     return y_new
      4 
      5 def median_filter(x, y, width):

you can see it happened when calling median_filter. Finally, we can see where the error occured inside median_filter:

<ipython-input-18-4b1062b3783d> in median_filter(x, y, width)
      6     y_new = np.zeros(y.shape)
      7     for i in range(len(x)):
----> 8         y_new[i] = np.median(y[np.abs(x - x[i]) < width * 0.5])
      9     return y_new

So tracebacks show you the full history of the error!

Now in the above case, the final error is:

ValueError: too many boolean indices

Why is this occuring? The only place that boolean indices are used here is when doing:

np.abs(x - x[i]) < width * 0.5

The issue is that if we look back at the original function call, there are more values for x than for y!

Using the debugger

In the above example, the code was still simple enough that we could guess the solution, but sometimes things are not so simple. One way to diagnose the issue would have been to print out the content of the variables in median_filter and run it again to see what was going on.

However, Python includes a debugger, which allows you to jump right in to where the error happened, and look at the variables. In the IPython notebook or in IPython, once an error has happened, you can run %debug, and you will see a ipdb> prompt (IPython debugger). You can then print out variables to see what they are set to. Let's try the above example again:

In [4]:
subtract_smooth(np.array([1,2,3,4,5]),np.array([4,5,6,8]))
/Users/tom/miniconda3/envs/dev/lib/python3.6/site-packages/ipykernel_launcher.py:10: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 4 but corresponding boolean dimension is 5
  # Remove the CWD from sys.path while we load stuff.
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-4-7bd1930fb255> in <module>()
----> 1 subtract_smooth(np.array([1,2,3,4,5]),np.array([4,5,6,8]))

<ipython-input-2-202311ce8ba8> in subtract_smooth(x, y)
      2 
      3 def subtract_smooth(x, y):
----> 4     y_new = y - median_filter(x, y, 1.)
      5     return y_new
      6 

<ipython-input-2-202311ce8ba8> in median_filter(x, y, width)
      8     y_new = np.zeros(y.shape)
      9     for i in range(len(x)):
---> 10         y_new[i] = np.median(y[np.abs(x - x[i]) < width * 0.5])
     11     return y_new

IndexError: index 4 is out of bounds for axis 1 with size 4
In [5]:
# Type %debug here

We can see that the boolean array that is being used as indices to y is too big. Much simpler! Type exit to exit the debugger.

Catching exceptions

In some cases, we know that errors might happen, and we don't want them to crash the code. For example, if we have to read in 1000 files, a few might be corrupt, and we just want to skip over them. Or we want to try something, and if it doesn't work, do something else. To do this, we can catch exceptions:

In [6]:
s = 'hello'
In [7]:
s[1] = 'a'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-c4127d7f42a9> in <module>()
----> 1 s[1] = 'a'

TypeError: 'str' object does not support item assignment
In [8]:
try:
    s[1] = 'a'
except:
    print("Can't set s[1]")
Can't set s[1]

The try...except contruct above catches all exceptions, but in some cases we want to be a bit more specific. The error that occurs above is a TypeError, which is just one kind of error (others include ValueError, SystemError, etc.). To catch just TypeError, you can do:

In [9]:
try:
    s[1] = 'a'
except TypeError:
    print("Can't set s[1]")
Can't set s[1]

If you catch other errors, TypeError will pass

In [10]:
try:
    s[1] = 'a'
except ValueError:    
    print("Can't set s[1]")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-10-08508459de36> in <module>()
      1 try:
----> 2     s[1] = 'a'
      3 except ValueError:
      4     print("Can't set s[1]")

TypeError: 'str' object does not support item assignment