We have already talked about built-in Python types, but there are more types that we did not speak about. One of these is the file() object which can be used to read or write files.

Let's try and get the contents of the file into IPython. We start off by creating a file object:

In [1]:
f = open('data/data.txt', 'r')


The open function is taking the data/data.txt file, opening it, and returning an object (which we call f) that can then be used to access the data.

Note that f is not the data in the file, it is what is called a file handle, which points to the file:

In [2]:
type(f)

Out[2]:
_io.TextIOWrapper

Now, simply type:

In [3]:
f.read()

Out[3]:
'RAJ        DEJ                          Jmag   e_Jmag\n2000 (deg) 2000 (deg) 2MASS             (mag)  (mag) \n---------- ---------- ----------------- ------ ------\n010.684737 +41.269035 00424433+4116085   9.453  0.052\n010.683469 +41.268585 00424403+4116069   9.321  0.022\n010.685657 +41.269550 00424455+4116103  10.773  0.069\n010.686026 +41.269226 00424464+4116092   9.299  0.063\n010.683465 +41.269676 00424403+4116108  11.507  0.056\n010.686015 +41.269630 00424464+4116106   9.399  0.045\n010.685270 +41.267124 00424446+4116016  12.070  0.035\n'

The read() function basically just read the whole file and put the contents inside a string.

Let's try this again:

In [4]:
f.read()

Out[4]:
''

What's happened? We read the file, and the file 'pointer' is now sitting at the end of the file, and there is nothing left to read.

Let's now try and do something more useful, and capture the contents of the file in a string:

In [5]:
f = open('data/data.txt', 'r')


Now data should contain a string with the contents of the file:

In [6]:
data

Out[6]:
'RAJ        DEJ                          Jmag   e_Jmag\n2000 (deg) 2000 (deg) 2MASS             (mag)  (mag) \n---------- ---------- ----------------- ------ ------\n010.684737 +41.269035 00424433+4116085   9.453  0.052\n010.683469 +41.268585 00424403+4116069   9.321  0.022\n010.685657 +41.269550 00424455+4116103  10.773  0.069\n010.686026 +41.269226 00424464+4116092   9.299  0.063\n010.683465 +41.269676 00424403+4116108  11.507  0.056\n010.686015 +41.269630 00424464+4116106   9.399  0.045\n010.685270 +41.267124 00424446+4116016  12.070  0.035\n'

But what we'd really like to do is read the file line by line. There are several ways to do this, the simplest of which is to use a for loop in the following way:

In [7]:
f = open('data/data.txt', 'r')
for line in f:
print(repr(line))

'RAJ        DEJ                          Jmag   e_Jmag\n'
'2000 (deg) 2000 (deg) 2MASS             (mag)  (mag) \n'
'---------- ---------- ----------------- ------ ------\n'
'010.684737 +41.269035 00424433+4116085   9.453  0.052\n'
'010.683469 +41.268585 00424403+4116069   9.321  0.022\n'
'010.685657 +41.269550 00424455+4116103  10.773  0.069\n'
'010.686026 +41.269226 00424464+4116092   9.299  0.063\n'
'010.683465 +41.269676 00424403+4116108  11.507  0.056\n'
'010.686015 +41.269630 00424464+4116106   9.399  0.045\n'
'010.685270 +41.267124 00424446+4116016  12.070  0.035\n'


Note that we are using repr() to show any invisible characters (this will be useful in a minute). Also note that we are now looping over a file rather than a list, and this automatically reads in the next line at each iteration. Each line is being returned as a string. Notice the \n at the end of each line - this is a line return character, which indicates the end of a line.

Now we're reading in a file line by line, what would be nice would be to get some values out of it. Let's examine the last line in detail. If we just type line we should see the last line that was printed in the loop:

In [8]:
line

Out[8]:
'010.685270 +41.267124 00424446+4116016  12.070  0.035\n'

We can first get rid of the \n character with:

In [9]:
line = line.strip()

In [10]:
line

Out[10]:
'010.685270 +41.267124 00424446+4116016  12.070  0.035'

Next, we can use what we learned about strings and lists to do:

In [11]:
columns = line.split()

In [12]:
columns

Out[12]:
['010.685270', '+41.267124', '00424446+4116016', '12.070', '0.035']

Finally, let's say we care about the object name (the 2MASS column), and the J band magnitude (the Jmag) column:

In [13]:
name = columns[2]
jmag = columns[3]

In [14]:
name

Out[14]:
'00424446+4116016'
In [15]:
jmag

Out[15]:
'12.070'

Note that jmag is a string, but if we want a floating point number, we can instead do:

In [16]:
jmag = float(columns[3])

In [17]:
jmag

Out[17]:
12.07

One last piece of information we need about files is how we can read a single line. This is done using:

line = f.readline()

We can put all this together to write a little script to read the data from the file and display the columns we care about to the screen! Here is is:

In [18]:
# Open file
f = open('data/data.txt', 'r')

# Loop over lines and extract variables of interest
for line in f:
line = line.strip()
columns = line.split()
name = columns[2]
jmag = float(columns[3])
print(name, jmag)

00424433+4116085 9.453
00424403+4116069 9.321
00424455+4116103 10.773
00424464+4116092 9.299
00424403+4116108 11.507
00424464+4116106 9.399
00424446+4116016 12.07


## Exercise 1¶

Here is a copy of the above code to read in the file. Modify this code so as to create a dictionary which gives Jmag for a given 2MASS name, i.e.

>>> jmag['00424455+4116103']
10.773



Then loop over the items in the dictionary and print out for each the source name and the Jmag value.

In [19]:
# EDIT THE CODE BELOW

# Open file
f = open('data/data.txt', 'r')

# Loop over lines and extract variables of interest
for line in f:
line = line.strip()
columns = line.split()
name = columns[2]
jmag = float(columns[3])


Bonus: can you figure out a way to make sure that you loop over the source names in alphabetical order?

## Writing files¶

To open a file for writing, use:

In [20]:
f = open('data_new.txt', 'w')


Then simply use f.write() to write any content to the file, for example:

In [21]:
f.write("Hello, World!\n")

Out[21]:
14

If you want to write multiple lines, you can either give a list of strings to the writelines() method:

In [22]:
f.writelines(['spam\n', 'egg\n', 'spam\n'])


or you can write them as a single string:

In [23]:
f.write('spam\negg\nspam\n')

Out[23]:
14

Once you have finished writing data to a file, you need to close it:

In [24]:
f.close()


(this also applies to reading files)

## The with-statement¶

As we have seen above, files must not just be opened but should be properly closed afterwards to make sure they are actually written before using them somewhere else. Sometimes writes to files get cached by Python to minimize actual writing to disk, which is comparably slow. Closing a file ensures that these changes are actually written.

To avoid forgetting to close a file there is the with-statement.

In [25]:
with open('data/data_new.txt', 'w') as f:
f.write('spam\n')


This opens the specified file and holds the file-object within f, as well as closing the file when the with-codeblock ends. Afterwards, the file is properly closed and not available anymore.

## Exercise 2¶

Continuing from the example in the 'Reading files' section, can you figure out a way to write out a new file containing two columns - the name of the source and the Jmag value?

In [26]:
# EDIT THE CODE BELOW

# Open file
f = open('data/data.txt', 'r')


The above shows you how you can read and write any data file. Of course, there are also functions that exist to help you read in data in certain formats (for example numpy contains a function numpy.loadtxt to read in arrays from files) but the key is that with the above, you can ready any file, so any other function is then just making your life easier.