«
»

How do I read a huge file line by line in Python?

By chris

This is taken from here and was written by rupe.

In Python, the most common way to read lines from a file is to do the following:

for line in open('myfile','r').readlines():
do_something(line)

When this is done, however, the readlines() function loads the entire file into memory as it runs. A better approach for large files is to use the fileinput module, as follows:

import fileinput
for line in fileinput.input(['myfile']):
do_something(line)

the fileinput.input() call reads lines sequentially, but doesn’t keep them in memory after they’ve been read.

This entry was posted on February 4, 2009 at 5:17 pm and is filed under Python. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

2 Responses to “How do I read a huge file line by line in Python?”

  1. Ettore Says:
    February 4, 2009 at 9:39 pm

    Thanks Chris, this could help me doing something I need at the moment ;) Very timely!

  2. della Says:
    February 5, 2009 at 10:05 am

    for line in open(‘myfile’): blah() does exactly the same thing: file objects can be iterated from python 2.3.

    Even better, starting from py 2.5, you can ensure that the file object is closed as soon as you exit from the block using it with the ‘with’ statement.

    with open(‘myfile’) as f:
    for line in f:
    blah()

    Of course there would be no problem in calling f.close(), but this ensures that the object gets closed however you exit from the block (exception, ‘break’ statement, etc.

Leave a Reply